In part 1 of this series of posts, we built a giant set of robotic googly eyes. In part 2, we brought the googly eyes to life using an Arduino. In this post, we’ll use OpenCV to make the googly eyes detect and follow people as they move around the room. More specifically, we’ll use OpenCV to detect faces on a webcam and move the googly eyes to look roughly in the direction of the largest face in view.

Project Overview

To detect faces, we’ll use a Python script and the OpenCV open source computer vision library. The Python script and OpenCV will run on a small computer running Ubuntu Linux. The Python script will grab an image from a webcam, feed the image into one of the OpenCV face detection algorithms, and, if any faces are detected, feed a scaled version of the x coordinate of the largest detected face to the Arduino over a serial port. Finally, the Arduino will run code to move the eyeballs back and forth based on the value received from the Python script.

Required Hardware

The hardware used for this project.

OpenCV and face detection require some raw compute power. I’ve successfully used both a March 2011 Core i7 MacBook Pro 13″ running MacOS X and a May 2015 Intel Core i7 NUC running Ubuntu 14.10 to run OpenCV and the face detection algorithms for this project. I tried using a Raspberry Pi 2 with the Raspberry Pi camera as well but the latency and frame rate were just too slow to create a convincing tracking / following effect.

I’d recommend the following hardware:

• A computer with an Intel Core i5 or Core i7 laptop or desktop chip made within the last three or four years. I used an Intel Core i7 NUC with 16GB DDR and a 120GB M.2 SATA SSD.
• An HD webcam with a resolution of 1280×720 that is compatible with Linux such as the Logitech C270 that I used.
• You’ll need one free USB port for the webcam and one free USB port for the Arduino.
• The googly eyes, Arduino, and stepper motor drivers described in two earlier posts.

Required Software

An earlier version of the OpenCV face detection script running on a MacBook Pro.

The software stack required to run this project is complex and somewhat difficult to install. But once it is installed, the basic Python script that connects OpenCV to the googly eyes is only a few hundred lines of code—and most of that is from the sample code included with OpenCV. All the software required for this project is open source and available for download for free.

Here’s the required software:

• Ubuntu 14.10
• NumPy and SciPy
• OpenCV 2.4.9
• PySerial
• screen (Linux utility used as a terminal emulator)

Installing Linux

Installing Linux never goes as easily as I think it should go—especially if trying to install an older, stable version of the OS on new hardware. I ran into issues with the Intel graphics drivers for my hardware not being a part of the Ubuntu 14.10 distribution. Here’s the general procedure to install the OS:

• Download the 64-bit PC (AMD64) desktop version of Ubuntu 14.10 from http://releases.ubuntu.com/14.10/.
• Follow the instructions here to create a bootable USB thumbdrive from the downloaded .iso file. These instructions are for creating the USB thumbdrive from a Mac. Other sites have similar instruction for working from a PC or another Linux box.
• Boot from the USB thumbdrive and follow the prompts to install the OS.

Unfortunately, my graphics driver issue prevented me from installing the OS on the first try. To successfully install the OS, I had to boot the installer with the nomodeset option added to the boot options. This would allow the installer GUI to run. Once the OS was installed, I had to add the nomodeset option to the OS boot options in order to run the Intel graphics driver installer GUI. Once the drivers were installed, I had to remove the nomodeset from the OS boot options to get the full performance of the Intel graphics on the NUC.

nomodeset added to the boot options during Linux installation.

To boot the installer with the nomodeset option, I followed these instructions. Bascially, boot the installer from the USB stick, and edit the boot options before starting the installer to have the nomodeset option in addition to the quiet and splash options.

Adding nomodeset to /etc/default/grub.

Once the OS was installed and booted, I edited /etc/default/grub to have the nomodeset option as well then ran sudo update-grub to propagate the changes into /boot/grub/grub.cfg. More details on this are available here. The solution BELOW the checked answer (not THE checked answer) is fairly detailed.

At this point, I updated the kernel to version 3.18.3 using these instructions and installed the Intel graphics driver. This required invoking dpkg to install the installer then running the installer itself.

Finally, I edited /etc/default/grub to remove the nomodeset option and rebuilt /boot/grub/grub.cfg with sudo update-grab. At this point my Linux box was running Ubuntu 14.10 with the 3.18.3 kernel and the 1.08 Intel graphics drivers. All was working well.

Now it was time to update the OS. Luckily that’s easy to do. Run these two commands:

• sudo apt-get update
• sudo apt-get upgrade

Installing the Webcam

Testing the webcam using the cheese utility.

I used a Logitech C270 webcam. It was cheap. It’s HD 720p. It works with Linux. I used the cheese application to test the webcam. It can be downloaded and installed with this command.

• sudo apt-get install cheese

Once installed, type cheese at a command prompt to launch and a small window with an image from the webcam should appear on the desktop.

Installing NumPy and SciPy

The OpenCV face detection sample program requires NumPy. SciPy is useful too. Might as well install both. Instructions to install both are on this page. Follow the instructions for Ubuntu. It’s one big long command:

• sudo apt-get install python-numpy python-scipy python-matplotlib ipython ipython-notebook python-pandas python-sympy python-nose

Installing OpenCV

The next step is to download, compile, and install OpenCV version 2.4.9. The process is relatively straightforward except that FFMPEG is no longer part of the Ubuntu 14.10 distribution. Rather than try to download FFMPEG from a different source, I just compiled OpenCV without FFMPEG support.

I followed these instructions to download, compile, and install OpenCV except that I used libtiff5 instead of libtiff4 and I ran cmake with the additional option -D WITH_FFMPEG=OFF to disable FFMPEG support. This option should come before the final ‘..’ in the cmake command line.

To pick up libtiff5 instead of libtiff4, I used this command to install the opencv dependencies:

• sudo apt-get install build-essential libgtk2.0-dev libjpeg-dev libtiff5-dev libjasper-dev libopenexr-dev cmake python-dev python-numpy python-tk libtbb-dev libeigen3-dev yasm libfaac-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev libx264-dev libqt4-dev libqt4-opengl-dev sphinx-common texlive-latex-extra libv4l-dev libdc1394-22-dev libavcodec-dev libavformat-dev libswscale-dev default-jdk ant libvtk5-qt4-dev

My arguments to cmake with FFMPEG support turned off were:

• cmake -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=ON -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D WITH_VTK=ON -D WITH_FFMPEG=OFF ..

Accessing the Serial Port

PySerial is used to access the serial port from Python. PySerial was installed either with the rest of the OS or one of the other packages. In any event, it was already present on my system after following all the above steps.

I use the Linux utility screen as a terminal emulator. It’s useful for communicating with the Arduino outside of the Arduino development environment too. Screen is not installed by default. The following command will install it:

• sudo apt-get install screen

Finally, any user that wishes to access the serial port needs to be placed into the dialout group. This can be done using the command below where $USER is the username. Once the command is executed, the user needs to log off then log back into the system for it to take effect.

• sudo usermod -a -G dialout $USER

Woohoo! Software Installed!

That concludes the installation of the OS and all the software packages required to run the Python face detection script and communicate with the Arduino. The next steps are to install a sketch on the Arduino to move the motors in response to commands on the serial port and to run the Python face detection script.

The Arduino Sketch

The Arduino sketch listens on the Arduino’s serial port for a position for each of the eyes and then moves the motors to that position. While the motors are running, it listens for the next position and will change the destination on the fly if the previous move has not completed.

Download the Arduino sketch from my github repository here. Install the sketch on the Arduino using the Arduino IDE. If needed, post two of this series contains links to download the IDE and some tutorials.

Connect the Arduino to the Linux box and execute the following command:

• screen /dev/ttyUSB0 19200

where /dev/ttyUSB0 is the Arduino’s serial port on the Linux box. Adafruit has a good tutorial on finding the name of the serial port in case it is not /dev/ttyUSB0 on your machine. 19200 is the baud rate. It should match the baud rate set in the Arduino sketch.

If screen starts then immediately exits, your user account needs to be added the dialout group using the usermod command a few paragraphs above.

Using screen to test connectivity to the Arduino and googly eyes.

Hit return a few times and you should get a prompt that says “EYES>.” Connect the motor power supply and execute the following commands:

• mt -800 -800
• mt 800 800
• mt 0 0

The motors should move a 1/4 turn CW, a half turn CCW, and finally a 1/4 turn CW and return to their initial position. Type CTRL-A then k to exit screen.

The Python Face Detection Script

In opencv-2.4.9/samples/python2, there’s a Python face detection script called facedetect.py. My script is based off this sample code. To run the sample as is, cd to opencv-2.4.9/samples/python2 then run:

• python facedetect.py

A window should launch showing live video from the webcam. Green rectangles should appear around any faces in the video. Blue rectangles should appear around nested eyeballs. Hit escape to exit.

Output of the default facedetect.py script. Detected faces are highlighted in green. Detected nested eyeballs are highlighted in blue.

I made the following changes to the script to make it work with the Arduino and the googly eyes:

1. Adjust video size — Capture the input video at the full resolution of the webcam.
2. Delete nested object detection — We don’t need to detect eyeballs.
3. Mirror the video — It’s a bit easier to walk around and watch what is happening with a mirror image vs a normal image so I used fliplr to flip the webcam image before displaying it.
4. Find the biggest face — What happens when multiple people are in the frame? I modified the code to calculate the areas of all the faces found and use the largest face. All the faces are highlighted in green and the face used for the position of the eyes is highlighted in red.
5. Send motor positions out the serial port — The final modification is to take the x value of the center of the red rectangle, scale it to the range -800 to +800, and send the scaled position out the serial port.

Create a new directory, cd to it, and download my script to it from my github repository here. The script is dependent on several files that are in the python2 samples directory. Copy the following files so they are in the same directory as the file fd2.py:

• opencv-2.4.9/samples/python2/common.py
• opencv-2.4.9/samples/python2/video.py
• opencv-2.4.9/data/haarcascades/haarcascade_frontalface_alt.xml

Connect the Arduino, adjust the pupils so they are looking straight down, and power up the motor drivers. Now launch the Python script using the following command. Be sure to replace /dev/ttyUSB0 with the actual name of the serial port connected to the Arduino.

• python fd2.py –serial /dev/ttyUSB0

Output of the modified face detection python script. The largest face is used to position the eyeballs and is highlighted in red with any smaller faces highlighted in green. The time to process the image and the position value sent to the Arduino are displayed in the upper left hand corner.

With any luck, the webcam video is displayed on the screen, the largest face is highlighted with a red rectangle, any extra faces are highlighted with green rectangles, and the googly eyes move back and forth tracking the face highlighted in the red rectangle.

The Results

In case you missed the YouTube video at the top of the post, here it is again:

The video shows me walking back and forth across the room while the eyes follow me. The monitor in the video shows a mirrored image from the webcam with my face highlighted.

In this post, we’ll add motion to the googly eyes using a microcontroller and some stepper motor drivers.

In part 1 of this series of posts, we built a giant set of robotic googly eyes. Now it’s time to animate them! To keep the project simple, we’ll use an Arduino Uno and a pair of Big Easy Driver stepper motor drivers. Let’s get started.

Required Parts

To animate the eyes, you’ll need the following parts:

• 1 Arduino Uno, $20
• 2 Big Easy Drivers, $20 each
• 10 3.5mm Pitch, 2-Pin Screw Terminals, $0.95 each
• 1 DC Power Supply for the Motors, 12-24V, 1A, $20

Total cost for these parts is about $90 excluding shipping and handling.

Prepare the Motor Drivers

The first step is to prepare the stepper motor drivers.

Big easy driver power connections, +3.3/5V selection pads (blue), Vref test point (green), and current setting pot (yellow).

Refer to the photo above as you complete the following steps:

1. Connect the M+ screw terminals on both drivers to the (+) lead of the 24V motor power supply and the GND screw terminals immediately next to the M+ terminals to the (-) lead of the 24V power supply as shown in the above photo. Do NOT connect the 24V motor power supply to the Arduino! Doing so will likely damage the Arduino.
2. Verify the +3.3/5V selection pads are not shorted or soldered together. These are highlighted above in light blue. If the pads are bridged together, use a soldering iron and some solder wick to clear the solder between them.
3. Calculate the reference voltage for the motor drivers. The reference voltage sets the current through the motor windings. Our motors have a rating of 0.5A. The reference voltage to set the current to 0.5A is computed as follows:
   

   Vref = I*(8*Rref) where I = 0.5A and Rref = 0.11Ω.
   Vref = 0.5*8.0*0.11 = 0.44V

   If you used different motors than I specified in part 1, be sure to use the current rating of your motors rather than 0.5A.

4. Next use a voltmeter and a small screwdriver to set the calculated reference voltage.
   • Connect the ground lead of the voltmeter to a convenient ground point in the circuit.
   • Place the positive lead of the voltmeter on the test point highlighted above in green.
   • Plug in and turn on the motor power supply.
   • Slowly adjust the current setting knob highlighted above in yellow until the voltage at the test point is just under the calculated reference voltage (0.44 volts).
   • Repeat this procedure for the 2nd stepper motor driver.
5. Turn off and unplug the motor power supply before moving to the next section.

My motors ran pretty hot (140°F) using the specified maximum current and calculated reference voltage. To make the motors run cooler, I adjusted the reference voltage down to half the calculated value and used 0.22 volts instead of 0.44 volts. The motors still had plenty of torque and ran significantly cooler (100°F).

Connect the Stepper Drivers to the Arduino and Motors

The next step is to connect the Arduino to the stepper drivers and the stepper drivers to the stepper motors. Below is a diagram of the connections between the stepper drivers, the Arduino, and the stepper motors. Below that are two checklists listing the signal names and their connections. Use the diagram and the checklists to connect the Arduino to the stepper drivers and then the stepper drivers to the stepper motors.

Arduino and Stepper Motor Connections

Connections Between the Arduino and the Stepper Drivers

Connections Between the Stepper Drivers and the Stepper Motors

The Completed Electronics

The photo below shows all the electronics completely wired.

All connections completed!

Preparing the Development Environment

If you’ve never used an Arduino before, download and install the Arduino Development Environment. The Arduino website has several good tutorials on how to use the development environment to write, compile, and download code.

We’re going to use the AccelStepper library to control the stepper motors. It can be downloaded here. It is installed by unzipping the distribution zip file into the libraries subfolder of your sketchbook.

The Arduino Sketch

Create a new sketch and paste the following code into it:

// include stepper library header file
#include <AccelStepper.h>

// motor speed in steps per second
#define motorSpeed 2400

// motor acceleration in steps per second per second
#define motorAccel 32000

// left eyeball step and direction pins
#define xstep 2
#define xdir  3

// right eyeball step and direction pins
#define ystep 4
#define ydir  5

// left eyeball accelstepper instance
AccelStepper stepper1 (1, xstep, xdir);

// right eyeball accelstepper instance
AccelStepper stepper2 (1, ystep, ydir);


void setup()
{
    // set step and direction pins to outputs
    pinMode (xstep, OUTPUT);
    pinMode (xdir,  OUTPUT);
    pinMode (ystep, OUTPUT);
    pinMode (ydir,  OUTPUT);

    // left eyeball stepper setup
    stepper1.setMaxSpeed(motorSpeed);
    stepper1.setSpeed(motorSpeed);
    stepper1.setAcceleration(motorAccel);

    // right eyeball stepper setup
    stepper2.setMaxSpeed(motorSpeed);
    stepper2.setSpeed(motorSpeed);
    stepper2.setAcceleration(motorAccel);
}


void loop()
{
    // main demo loop
    LookLeft ();
    delay (1000);
    LookRight ();
    delay (1000);
    LookDown ();
    delay (1000);
    RollEyes ();
    delay (1000);
    LookLeft ();
    delay (1000);
    LookDown ();
    delay (1000);
    CrossEyed ();
    delay (1000);
    LookUp ();
    delay (1000);
    LookDown ();
    delay (1000);
}


void LookLeft (void)
{
    stepper1.moveTo(+800);
    stepper2.moveTo(+800);
    
    while ((stepper1.distanceToGo() != 0) || (stepper2.distanceToGo() != 0)) {
        stepper1.run();
        stepper2.run();
    }
}


void LookRight (void)
{
    stepper1.moveTo(-800);
    stepper2.moveTo(-800);
    
    while ((stepper1.distanceToGo() != 0) || (stepper2.distanceToGo() != 0)) {
        stepper1.run();
        stepper2.run();
    }
}


void LookDown (void)
{
    stepper1.moveTo(0);
    stepper2.moveTo(0);
    
    while ((stepper1.distanceToGo() != 0) || (stepper2.distanceToGo() != 0)) {
        stepper1.run();
        stepper2.run();
    }
}


void LookUp (void)
{
    stepper1.moveTo(+1600);
    stepper2.moveTo(-1600);
    
    while ((stepper1.distanceToGo() != 0) || (stepper2.distanceToGo() != 0)) {
        stepper1.run();
        stepper2.run();
    }
}


void RollEyes (void)
{
    stepper1.moveTo(+3200);
    stepper2.moveTo(+3200);
    
    while ((stepper1.distanceToGo() != 0) || (stepper2.distanceToGo() != 0)) {
        stepper1.run();
        stepper2.run();
    }
    
    stepper1.setCurrentPosition (0);
    stepper1.moveTo(0);
    stepper2.setCurrentPosition (0);
    stepper2.moveTo(0);
}


void CrossEyed (void)
{
    stepper1.moveTo(-800);
    stepper2.moveTo(+800);
    
    while ((stepper1.distanceToGo() != 0) || (stepper2.distanceToGo() != 0)) {
        stepper1.run();
        stepper2.run();
    }
}

Running The Sketch

To compile, download, and run the sketch, follow these steps:

• Plug the Arduino into the computer’s USB port.
• Select the Arduino’s serial port from the Tools : Serial Port menu.
• Compile, download, and run the code by clicking the right arrow in the code editing window.

We want the eyeballs to be in a known position (looking straight down) when starting the Arduino. Follow these steps to move the eyeballs to the starting position and to start the Arduino with the eyeballs in the starting position:

• Unplug the Arduino from the computer’s USB port.
• Rotate both pupils until they’re looking straight down.
• Plug in the DC motor supply.
• Plug the Arduino into the computer’s USB port.

At this point, the eyeballs should start moving following the sequence outlined in the loop section of the sketch. This sequence is left : right : down : rolleyes : left : down : crosseyed : up : down.

Next Steps

In part three of this series of posts, we’re going to use OpenCV to detect faces and make the googly eyes track a person as they move around a room. This will first require the installation of Python, NumPy, OpenCV, and PySerial. Then we’ll modify the example OpenCV Python face detection code to send the location of a face on a webcam to the Arduino. Lastly, we’ll modify the Arduino code to listen for the location of a face and move the steppers to point the eyeballs towards that location.

Completed robotic googly eyes. The scleras (whites of the eyes) are about 8″ (200mm) in diameter with 5″ (125mm) pupils.

Build your very own pair of giant robot googly eyes! These eyes are built from laser cut acrylic panels and motorized using a pair of NEMA-14 stepper motors. This post is the first in a series of three posts. In this post, we’ll build the googly eyes. In the 2nd post, we’ll look at hardware and software to animate the googly eyes. In the 3rd and final post, we’ll connect the eyes to OpenCV to make them track motion and faces in a room.

3D Model

Before buying any parts or doing any building, I designed the eye balls in Autodesk Fusion 360. A 3D model of the eyeballs can be viewed online using their free 3D viewer software here. The 3D model was useful for making sure all the parts would fit together, building the bill of materials for the project, and seeing where all the pieces go when it was time to assemble the googly eyes. A Fusion 360 archive file for the googly eyes can be downloaded here.

Required Parts

The parts required to build a set of robotic googly eyes.

The parts for this project are listed below. The total cost was about $85 excluding the shipping and handling charges.

• 2 Pololu NEMA-14 35x28mm Stepper Motors, Item #1208, $12.95 ea
• 1 Pololu 2-Pack of Aluminum Mounting Hubs for 5mm Shafts, M3 Holes, Item #1998, $7.49 for 2
• 8 Keystone #6 Aluminum Round Spacers, 1/2″, Item #3466, $0.37 ea
• 6 Keystone 6-32 Aluminum Hex Standoffs, 1-3/4″, Item #1819, $1.00 ea
• 4 M3-0.50 x 8mm Black Oxide Button Head Socket Cap Screws, McMaster-Carr #91239A113, $6.43 for 100
• 12 6-32 x 0.375″ Button Head Socket Cap Head Screws, McMaster-Carr #92949A146, $3.93 for 100
• 8 M3-0.50 x 18mm Button Head Socket Cap Screws, McMaster-Carr #92095A472, $6.00 for 100
• 2 Laser Cut Acrylic Eye Whites (scleras)*
• 2 Laser Cut Acrylic Eye Blacks (pupils)*
• 1 Laser Cut Acrylic Wall Mounting Plate*

*DXF files for use at Ponoko are provided in the next section. The materials and cutting cost was $38.17.

Cutting the Acrylic Pieces

I had Ponoko cut the acrylic pieces for the googly eyes. Total cost excluding shipping was $38.17 at the time I built my pair of googly eyes. You’ll need the two design files below.

p2_nema_14_white_sclera_0.118.dxf
p2_nema_14_black_pupils_0.118.dxf

(Right click and choose save as if you have trouble downloading the files.)

Both of these files are sized to be cut from Ponoko’s P2-sized material. The file p2_nema_14_white_sclera_0.118.dxf contains two scleras (the white parts of the eyes). I cut these out of glossy white 3mm acrylic. The file p2_nema_14_black_pupils_0.118.dxf contains four pupils and two wall mounting brackets. I cut these out of glossy black 3mm acrylic.

Assembly

Start assembly by attaching the mounting hubs to the shafts of the stepper motors. The top of the hub should sit flush with the end of the shaft. Tighten the set screws to their final torque. The photo in the parts list above shows the hubs mounted to the stepper motors.

Postion of motor and spacers on the rear of the eye white. Notice the wires exit the stepper motor away from the 1.75″ spacers.

Next up is to connect the motors to the rear of the eye whites. Insert one of the M3-0.5 x 18mm screws through one of the four smaller holes on the front of the acrylic eye white. Place a spacer on the screw on the back side of the eye white then insert the screw into the corresponding mounting hole on one of the stepper motors. Point the side of the stepper motor with the wires down and away from the three larger 6-32 holes so that the wires will clear the spacers that are installed in the next step. Repeat this procedure for the remaining three M3 screws then repeat for the second eyeball.

The motor and spacers mounted to the rear of the eye whites.

Now connect the eye whites to the spacers. The spacers will connect the eyeballs to the wall mounting plate. Insert a 6-32 x 0.375″ screw through one of the three larger holes on the front side of the acrylic eye white. Tighten a spacer onto the screw on the rear of the eye white. Repeat this procedure for the two remaining 6-32 screw holes then repeat for the second eyeball.

Side view of the googly eyes mounted on the wall hanging bracket.

Acrylic scratches easily so carefully flip the assembly over and place it on a soft surface such as a towel. Position the wall mounting bracket over one of the eyeballs. Insert a 6-32 x 0.375 screw through the wall mounting bracket and into the corresponding spacer on the eyeball. I mounted my wall mounting bracket to the eyeballs such that the motor wires exited the bottom when the wall mounting bracket is hung on the wall. Holes are provided for either orientation. Finish attaching the first eyeball to the wall mounting bracket then repeat for the second eyeball.

Flip the eyeballs over and use the M3-0.5 x 8mm black oxide screws to mount the pupils to the mounting hubs on the ends of the stepper motors.

Congratulations! You’ve built your very own pair of giant robotic googly eyes!

Completed robotic googly eyes

Next Steps

In the next post in this series, we’ll look at making the eyeballs move using an Arduino and some simple stepper motor drivers. In the final post, we’ll use OpenCV to make the googly eyes track faces in the room.