Hand Gesture Recognition using Open CV - Sell Project

Sign-Language Note Simple-OpenCV-Calculator and this project are merged to one. Simple-OpenCV-Calculator will no longer be maintained. What I did here The first thing I did was, I created 44 gesture samples using OpenCV. For each gesture I captured 1200 images which were 50x50 pixels. All theses images were in grayscale which is stored in the gestures/ folder. The pictures were flipped using flipimages.py. This script flips every image along the vertical axis. Hence each gesture has 2400 images. Learned what a CNN is and how it works. Best resources were Tensorflow's official website and machinelearningmastery.com. Created a CNN which look a lot similar to this MNIST classifying model using both Tensorflow and Keras. If you want to add more gestures you might need to add your own layers and also tweak some parameters, that you have to do on your own. Then used the model which was trained using Keras on a video stream. As of today, I have stored the 44 gestures for which are 26 alphabets and 10 numbers of American Sign language and some other gestures. And trained the model on these images. There are a lot of details that I left. But these are the basic and main steps. Outcome Watch it here. Requirements Python 3.x Tensorflow 1.5 Keras OpenCV 3.4 h5py pyttsx3 A good grasp over the above 5 topics along with neural networks. Refer to the internet if you have problems with those. I myself am just a begineer in those. A good CPU (preferably with a GPU). Patience.... A lot of it. Installing the requirements Start your terminal of cmd depending on your os. If you have a NVidia GPU then make sure you have the prerequisites for Tensorflow GPU installation (Refer to official site). Then use this commmand pip install -r requirementsgpu.txt In case you do not have a GPU then use this command pip install -r requirementscpu.txt How to use this repo Before using this repo, let me warn about something. You will have no interactive interface that will tell you what to do. So you will have to figure out most of the stuff by yourself and also make some changes to the scripts if the needs arise. But here is a basic gist. Creating a gesture Watch the video guide for setting the hand histogram here. First set your hand histogram. You do not need to do it again if you have already done it. But you do need to do it if the lighting conditions change. To do so type the command given below and follow the instructions below. python sethandhist.py A windows "Set hand histogram" will appear. "Set hand histogram" will have 50 squares (5x10). Put your hand in those squares. Make sure your hand covers all the squares. Press 'c'. 1 other window will appear "Thresh". On pressing 'c' only white patches corresponding to the parts of the image which has your skin color should appear on the "Thresh" window. Make sure all the squares are covered by your hand. In case you are not successful then move your hand a little bit and press 'c' again. Repeat this until you get a good histogram. After you get a good histogram press 's' to save the histogram. All the windows close. I already have added 44 (0-43) gestures. It is on you if you want to add even more gestures or replace my gestures. Hence this step is OPTIONAL. To create your own gestures or replace my gestures do the following. It is done by the command given below. On starting executing this program, you will have to enter the gesture number and gesture name/text. Then an OpenCV window called "Capturing gestures" which will appear. In the webcam feed you will see a green window (inside which you will have to do your gesture) and a counter that counts the number of pictures stored. python creategestures.py Press 'c' when you are ready with your gesture. Capturing gesture will begin after a few seconds. Move your hand a little bit here and there. You can pause capturing by pressing 'c' and resume it by pressing 'c'. Capturing resumes after a few secondAfter the counter reaches 1200 the window will close automatically. After capturing all the gestures you can flip the images using python flipimages.py When you are done adding new gestures run the loadimages.py file once. You do not need to run this file again until and unless you add a new gesture. python loadimages.py Displaying all gestures To see all the gestures that are stored in 'gestures/' folder run this command python displayallgestures.py Training a model So training can be done with either Tensorflow or Keras. If you want to train using Tensorflow then run the cnntf.py file. If you want to train using Keras then use the cnnkeras.py file. python cnntf.py python cnnkeras.py If you use Tensorflow you will have the checkpoints and the metagraph file in the tmp/cnnmodel3 folder. If you use Keras you will have the model in the root directory by the name cnnmodelkeras2.h5. You do not need to retrain your model every time. In case you added or removed a gesture then you need to retrain it. Get model reports To get the classification reports a...