My work began with researching the electronic hardware compatibility between cameras and microcontrollers that would fit our stand-alone design. After understanding that an Arduino has to use a computer to run the software, I made the decision to go with a

Raspberry Pi which can run all of the software. I researched the possibility of using only one camera to function as both the video that the device shows on the video call and the video that the face tracking software uses. With regular web cameras, it did not seem that there would be a simple solution to tackling both functions and thus we went with two cameras. In the end, the general flow of how the webcam would work was that one camera's video feed is taken in which the face tracking algorithm looks for a face. If there is a face that is outside of the innermost bounding box (set by us) of the camera's frame then the 2-axis gimbal servo system will pan and tilt to make the face go into the bounding box. If there is no face detected at all then the gimbal will do a roaming function to move until a face is detected and then the facial detection function will pan and tilt until the face is inside the bounding box.

I worked with one of my friends on the software that integrates the face-tracking algorithm, vector calculation, and servo control into the Raspberry Pi to be completely stand-alone. First, I had to figure out which trained facial detection would work in our case. I started off by trying to get the OpenCV haar cascade face tracker to work on images and then a video feed. By testing multiple lighting conditions, movements (jumping, running, walking, and going out of frame), and distances from the camera, we noticed that the haar cascade had detection issues and took too much processing power for the Raspberry Pi to allow for real-time tracking. We needed to find a tracker optimized for multiple lighting conditions and less processing power. After testing 4 new trackers, the quickest and farthest in distance was the MTCNN tracker. Once the face tracking algorithm was established, we figured out the vector calculations that would translate the face to bounding box distance into servo movement. In order to get the fully integrated prototype running, I wrote the code that would put together all of the functions. 

Once the code was written and worked with testing on a regular desktop computer processor, it was time to make the Raspberry Pi execute everything on its own. I looked into the Raspberry Pi pin layout and wired all the servo motors and camera input. With all of the hardware assembled together, I was able to test the code to make sure that there were no failure cases. One issue we came across was that the servo motors were not stepping quickly enough for movements that were quick because the vector calculation had it so that the gimbal system would move to where it detected the face. In order to fix the code, I made the bounding box a little larger and the vector calculations would happen less frequently to both lower the processing power required and make more efficient movements.