Extending Python with C – Using OpenCV on the web

Update #2: I’ve set up a live demo for the sample app here.

Update: I got some feedback from this article asking things like “Wouldn’t it be easier to achieve this with ctype, cython, etc” and here is my answer: Yes it would! However most of these approaches would include a performance penalty and the purpose of this post was to demostrate how to create a Python extension and I already had the face detection C code from my previous OpenCV article. I’m planning on trying face recognition using the other approaches suggested to me and creating a benchmark to show whether I’m right on believing there would be this overhead with the other approaches.

This article is kind of a two-in-one, I’ll demonstrate how to create an extension to Python written in C and how to take the power of OpenCV to the web. In order to follow this article you must have OpenCV installed on your system, if you don’t have OpenCV you can read my previous article to learn how to get and compile OpenCV (there’s a small adjustment to be made on OpenCV’s source code if you’re planning to compile it on a Mac machine).

While OpenCV does indeed have a Python API, it can be quite difficult to get it running and as it’s so easy to extend Python with C or C++ and we can leverage the performance of these languages to process images it makes sense to write the processing algorithm in these languages and then wrap it with a Python extension.

I’ll be using the same facial detection technique used on my previous OpenCV article though this time we’ll apply it to still pictures and not video. We want something that can be used in Python like this:

import face
face_list = face.detect('/Users/raphaelcruzeiro/Documents/people.jpg')

Now that we know how we want our Python code to look like we can then proceed to create the extension in C. Let’s create a file named facerecognition.c and a setup.py.  On our setup.py we’ll specify the C source, the include path and the libraries we’re going to use:

from distutils.core import setup, Extension

module1 = Extension('face',
                    include_dirs = ['/usr/local/include'],
                    libraries = ['opencv_core', 'opencv_objdetect'],
                    library_dirs = ['/usr/local/lib'],
                    sources = ['facerecognition.c']
                    )

setup (name = 'Face CV Web',
       version = '1.0',
       description = 'A computer vision module.',
       ext_modules = [module1])

This script will then be used to compile our extension. And now to the C code on facerecognition.c:

#include <Python.h>
#include <opencv/cv.h>

#include <opencv2/highgui/highgui_c.h>

/* We'll store any errors encountered during the execution */
static PyObject *FaceError;

CvHaarClassifierCascade *cascade;
CvMemStorage            *storage;

PyObject *detectFaces(IplImage *img)
{
    int i;

    /* Detect all faces on the frame */
    CvSeq *faces = cvHaarDetectObjects(
                                       img,
                                       cascade,
                                       storage,
                                       1.1,
                                       3, 0,
                                       cvSize(50, 50),
                                       cvSize(50, 50)
                                      );

    /* This is how we create a Python list from C */
    PyObject *result = PyList_New(0);

    /* Now, let's mark each face */
    for(i = 0 ; i < (faces ? faces->total : 0) ; i++) {
        CvRect *r = (CvRect*)cvGetSeqElem(faces, i);

        /*
            I'm creating dictionaries to represent the face's
            position as dictionaries are considerably easier
            to create than custom objects and can be easily serialized
            to JSON on out Python code.
            To create a Python dictionary all you have to do is to call
            the Py_BuildValue function with a a formatting mask with
            brackets, the 's' represents a string, the 'i'represents a
            number and the 'O' represents a Python object, so if I want
            to create a Python dictionary like " { x : 5, y : 1 } " all
            I have to do is use the mask "{sisi}".
        */
        PyObject *origin = Py_BuildValue(
                                         "{sisi}",
                                         "x",
                                         r->x,
                                         "y",
                                         r->y
                                        );
        PyObject *size = Py_BuildValue(
                                       "{sisi}",
                                       "width",
                                       r->width,
                                       "height",
                                       r->height
                                      );
        PyObject *rect = Py_BuildValue(
                                       "{sOsO}",
                                       "origin",
                                       origin,
                                       "size",
                                       size
                                      );

        PyList_Append(result, rect);
    }

    return result;
}

static PyObject *
face_detect(PyObject *self, PyObject *args)
{
    const char *path;
    int sts;

    if (!PyArg_ParseTuple(args, "s", &path))
        return NULL;

    IplImage* frame;

    /* OpenCV stores the Haar Cascades for default on this location */
    char *file = "/usr/local/share/opencv/"
        "/haarcascades/haarcascade_frontalface_alt.xml";

    cascade = (CvHaarClassifierCascade*)cvLoad(file, 0, 0, 0);
    storage = cvCreateMemStorage(0);

    IplImage* image = cvLoadImage(path, CV_LOAD_IMAGE_COLOR);

    if(image == NULL) {
        PyErr_SetString(FaceError, "Image not found!");
        cvReleaseHaarClassifierCascade(&cascade);
        cvReleaseMemStorage(&storage);
        return NULL;
    }

     PyObject *result = detectFaces(image);

     cvReleaseHaarClassifierCascade(&cascade);
     cvReleaseMemStorage(&storage);
     cvReleaseImage(&image);

     return result;
}

static PyMethodDef FaceMethods[] = {
    {"detect",  face_detect, METH_VARARGS,
     "Detect the faces on an image."},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

PyMODINIT_FUNC
initface(void)
{
    PyObject *m;

    m = Py_InitModule("face", FaceMethods);
    if (m == NULL)
        return;

    FaceError = PyErr_NewException("face.error", NULL, NULL);
    Py_INCREF(FaceError);
    PyModule_AddObject(m, "error", FaceError);
}

That may seem like a lot to take in at once but it is in reality quite simple. A PyObject is a data structure that represents a Python object, this data structure contains the object’s reference count and a pointer to the objects’type. Unless we’re defining a new kind of Python object we can create our object using Py_BuildObject, this is done to create dictionaries on the above code.
To create a Python dictionary all you have to do is to call the Py_BuildValue function with a formatting mask with brackets, the s represents a string, the i represents a number and the O represents a Python object, so if you want to create a Python dictionary like ” { x : 5, y : 1 } ” all you have to do is use the mask “{sisi}”.

To make your functions visible to Python you must create a method definition table (PyMethodDef) and add it to the module initialization. Also on the module initialization you should initialize the FaceError variable as a Python exception to be thrown if there is an error opening the file, not doing so will cause Python to receive a SIGABRT from the OS.

To compile our module just type the command:

python setup.py build

And copy the resulting module, face.so, to Python’s site-packages directory.

You should now be able to call the module from Python like this:

import face
face_list = face.detect('/Users/raphaelcruzeiro/Documents/people.jpg')

Using the extension on a real application

I’ve created a sample Django application that leverages on this face detection capabilities exposed by our extension. While I won’t discuss every aspect of the Django application as it is out of the scope of this post, I’ll just show here the key aspects of the application (a link to the source repository can be found at the bottom of the article).

The web app allows the user to upload an image and then displays it on the browser. When the image is displayed an AJAX call is made to the server to evaluate the image and supply the client-side with the position of the faces.

from face import detect as _detect

def detect(request):
    print request.POST

    id = request.POST['id']
    image = ModelImage.objects.get(pk=id)

    return HttpResponse(json.dumps(_detect(image.img.path)))

This is how the server-side handles the AJAX request to process the image and this is how the AJAX call is made and the response processed on the client-side:

$(document).ready(function(){
    $.post('{% url detect_url %}', 'id={{ id }}', function(data){
        $.each(data, function(i, f){
            var face = $(' <div style="background: blue; opacity: 0.5; position: absolute;"></div>').css({
                'top' : f.origin.y + 'px',
                'left' : f.origin.x + 'px',
                'height' : f.size.height + 'px',
                'width' : f.size.width + 'px'
            });

            $('#mask').append(face);
        })
        }, 'json');
});

The image is contained within a relative positioned  div  with the id mask and our script basically adds an absolute positioned div on top of each face returned on the JSON response.

The full source code of this article can be found on https://github.com/raphaelcruzeiro/Face-Detector

See the live demo.