Writing a daemon for Unix and unix-like systems

A daemon is a specific kind of process that has the following characteristics:

• It’s long lived. A daemon usually only stops when the system shuts down.
• It runs on background. and has no controlling terminal whatsoever.

Services and servers are typically daemons, we have for example: sshd (the secure shell daemon), ftpd (ftp daemon), cron (a daemon that executes scheduled tasks). As you can see it’s an informal convention that the daemon process’s name ends with the letter d.

On a Unix environment, a daemon is usually created as a normal process forking a child process and immediately terminating itself , thus making the  init process adopt the daemon as it’s child process. The daemon then proceeds to disassociate itself from any controlling terminal and change it’s working directory to the root, or any other directory that is guaranteed not to be on a filesystem that may eventually need to be unmounted, and clear the process umask to ensure that any file created by the daemon has the correct set os permissions.

Forking is one of the beautiful things in a Unix environment. When the fork system call is issued, the operating system creates another process with its own address space loaded with an exact copy of all the memory segments of the parent process and both parent and child process are on the same state: the return of the fork call. On the parent, this call returns the PID of the child process or a negative number if an error has occurred, on the child, the call returns 0, so basically, it’s possible to distinguish the parent and child processes by the return of the  fork call.

And now back to daemons. When I need to turn a process into a daemon I usually like to use a standard “daemonizer” that I reuse across my projects. This “daemonizer” consists basically of a header and a source file:

daemon.h:

#ifndef DAEMON_H

#define DEAMON_H

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>

void startDaemon();

#endif

daemon.c:

#include "daemon.h"

void startDaemon()
{
    printf("Starting daemon...\n");

    pid_t pid, sid;

    pid = fork();

    if (pid < 0) { /* Something went wrong. */
        exit(EXIT_FAILURE);
    } else if (pid > 0) {
        printf("Daemon successfully started!\n");
        exit(EXIT_SUCCESS); /* The parent process happily
                             * terminates itself */
    }

    /* By setting the umask to 0 we will have full access
     * to any files generated by the daemon */
    umask(0);

    /* This creates a new session and makes the daemon its
     * leader and leader of a new process group and prevents
     * it from having any terminal */
    sid = setsid();
    if (sid < 0) {
        exit(EXIT_FAILURE);
    }

    /* Change the directory to root (although any directory
     * that is guaranteed not to be unmounted will do). */
    if (chdir("/") < 0) {
        exit(EXIT_FAILURE);
    }

    /* The main application loop */
    while (1) {
        sleep(10);

        /* Here you can make the daemon do any background
         * task you like */

    }
}

Writing a daemon for unix system is terribly easy, now just hit your keyboard and create something awesome from this!

Creating website previews using webkit in Qt

Recently I had to create a web crawler written in Python that generated page previews when indexing the web pages. The first idea that crossed my mind was to create Python C extension leveraging on the awesome browser engine webkit to render the page, however, I later decided to create a completely separated tool for rendering, since it would not be easy AFAIK to run Qt’s MOC during the compilation of the extension in a Python setup script. A standalone tool would not be so elegant comparing to my initial idea of creating an extension but ,as the wise man once said, ‘simple is better than complicated’.
The first thing I did was to create a Qt Console Application project and add the necessary modules to my .pro file:

QT       += core webkit network

And to ensure that the gui module was not excluded from the project by removing the following line:

QT       -= gui

After doing that I had everything ready to begin the coding.

The page loading and rendering was encapsulated on a class Snapshot:

#ifndef SNAPSHOT_H
#define SNAPSHOT_H
#include <QtCore>
#include <QtWebKit>
#include <QImage>
#include <QPainter>

class Snapshot : QObject
{
    Q_OBJECT

    QWebPage *page;
    QString *outputFilename;

public:
    Snapshot(QObject *parent = 0);
    void shot(QUrl url, QSize &size, QString *outputFilename = new QString("output.png"));

private slots:
    void doneLoading(bool);
};

#endif // SNAPSHOT_H

This header defines Snapshot as a simple class that inherits QObject, hence the Q_OBJECT macro, and a single private slot to be called when the page has finished loading. Now, let’s move to the implementation:

#include <QApplication>
#include "snapshot.h"

Snapshot::Snapshot(QObject *parent) : QObject(parent), page(new QWebPage)
{
}

void Snapshot::shot(QUrl url, QSize &size, QString *outputFilename)
{
    this->outputFilename = outputFilename;
    connect(page, SIGNAL(loadFinished(bool)), SLOT(doneLoading(bool)));
    page->mainFrame()->load(url);
    page->setViewportSize(size);
    page->mainFrame()->setScrollBarPolicy(Qt::Vertical, Qt::ScrollBarAlwaysOff);
}

void Snapshot::doneLoading(bool)
{
    QImage image(page->viewportSize(), QImage::Format_ARGB32);
    QPainter painter(&image);

    page->mainFrame()->render(&painter);

    painter.end();

    image.save(*outputFilename);

    delete outputFilename;
    QApplication::quit();
}

All the magic is basically done on the doneLoading slot where I created an image, used it as the graphic output for a painter and then passed the painter to the webpage component for rendering.

Pretty much everything for taking a website snapshot is in the above class, all that is missing is creating the object and calling the method shot from within a Qt application:

#include <QApplication>
#include "snapshot.h"

int main(int argc, char *argv[])
{
    if(argc > 0) {
        QApplication a(argc, argv);

        QSize size;

        if(argc > 1) {
            size = QSize(
                        QString(argv[2]).split(QChar('x'))[0].toInt(),
                        QString(argv[2]).split(QChar('x'))[1].toInt()
            );
        } else {
            size = QSize(1024, 768);
        }

        Snapshot shot;
        if (argc > 2) shot.shot(QUrl(QString(argv[1])), size, new QString(argv[3])); else shot.shot(QUrl(QString(argv[1])), size);

        return a.exec();
    }
}

With this main function as a point of entry, the tool can called from the terminal like this (you can also call it from a Python application like I did):

./webimage http://news.ycombinator.com 800x600 hackernews.png

And it will generate an image like this:

The source code can be found on github.

UPDATE: I forgot to mention, if you plan on using this on a Unix webserver be sure to have Xvfb installed for running a lightweight X server session during the program execution by running the tool like this:

xvfb-run --server-args="-screen 0, 1024x768x24 ./webimage http://news.ycombinator.com 800x600 hackernews.png

Navigating the filesystem and removing specific files with Python

So I decided to organize some external discs where I keep things like: TV series episodes, movies, etc; and I noticed that there were tons of useless rar files just sitting there making my directories look like a complete mess. To quickly clean my disks I decided to create a simple script and better yet: a reusable script so that whenever I download anything that comes as a lot of rar files, I can just clean them with one swift blow [add evil laugh here].

The script is really simple (as I tried to follow The Pythonic Way):

import sys
import os
import re

def main(args):
    print 'Searching path: %s' % args[1]
    rar_count = 0

    for dirname, dirnames, filenames in os.walk(args[1]):
         for filename in filenames:
             name, extension = os.path.splitext(filename)
             if extension == '.rar' or re.search(r'\.r\d+$', extension):
                 filepath = os.path.join(dirname, filename)
                 print 'Deleting: %s' % filepath
                 os.remove(filepath)
                 rar_count += 1
    print 'Removed %d rar files' % rar_count

if __name__ == "__main__":
    sys.exit(main(sys.argv))

This script can be used like this:

python rarkiller.py /Volumes/Backupdisk

Basically it will recursively navigate through the file system starting at the directory specified as argument and try to match the file extension to a regex that matched some rar file extensions, if it’s a match the file will be deleted. It’s really that simple.

Extending Python with C – Using OpenCV on the web

Update #2: I’ve set up a live demo for the sample app here.

Update: I got some feedback from this article asking things like “Wouldn’t it be easier to achieve this with ctype, cython, etc” and here is my answer: Yes it would! However most of these approaches would include a performance penalty and the purpose of this post was to demostrate how to create a Python extension and I already had the face detection C code from my previous OpenCV article. I’m planning on trying face recognition using the other approaches suggested to me and creating a benchmark to show whether I’m right on believing there would be this overhead with the other approaches.

This article is kind of a two-in-one, I’ll demonstrate how to create an extension to Python written in C and how to take the power of OpenCV to the web. In order to follow this article you must have OpenCV installed on your system, if you don’t have OpenCV you can read my previous article to learn how to get and compile OpenCV (there’s a small adjustment to be made on OpenCV’s source code if you’re planning to compile it on a Mac machine).

While OpenCV does indeed have a Python API, it can be quite difficult to get it running and as it’s so easy to extend Python with C or C++ and we can leverage the performance of these languages to process images it makes sense to write the processing algorithm in these languages and then wrap it with a Python extension.

I’ll be using the same facial detection technique used on my previous OpenCV article though this time we’ll apply it to still pictures and not video. We want something that can be used in Python like this:

import face
face_list = face.detect('/Users/raphaelcruzeiro/Documents/people.jpg')

Now that we know how we want our Python code to look like we can then proceed to create the extension in C. Let’s create a file named facerecognition.c and a setup.py.  On our setup.py we’ll specify the C source, the include path and the libraries we’re going to use:

from distutils.core import setup, Extension

module1 = Extension('face',
                    include_dirs = ['/usr/local/include'],
                    libraries = ['opencv_core', 'opencv_objdetect'],
                    library_dirs = ['/usr/local/lib'],
                    sources = ['facerecognition.c']
                    )

setup (name = 'Face CV Web',
       version = '1.0',
       description = 'A computer vision module.',
       ext_modules = [module1])

This script will then be used to compile our extension. And now to the C code on facerecognition.c:

#include <Python.h>
#include <opencv/cv.h>

#include <opencv2/highgui/highgui_c.h>

/* We'll store any errors encountered during the execution */
static PyObject *FaceError;

CvHaarClassifierCascade *cascade;
CvMemStorage            *storage;

PyObject *detectFaces(IplImage *img)
{
    int i;

    /* Detect all faces on the frame */
    CvSeq *faces = cvHaarDetectObjects(
                                       img,
                                       cascade,
                                       storage,
                                       1.1,
                                       3, 0,
                                       cvSize(50, 50),
                                       cvSize(50, 50)
                                      );

    /* This is how we create a Python list from C */
    PyObject *result = PyList_New(0);

    /* Now, let's mark each face */
    for(i = 0 ; i < (faces ? faces->total : 0) ; i++) {
        CvRect *r = (CvRect*)cvGetSeqElem(faces, i);

        /*
            I'm creating dictionaries to represent the face's
            position as dictionaries are considerably easier
            to create than custom objects and can be easily serialized
            to JSON on out Python code.
            To create a Python dictionary all you have to do is to call
            the Py_BuildValue function with a a formatting mask with
            brackets, the 's' represents a string, the 'i'represents a
            number and the 'O' represents a Python object, so if I want
            to create a Python dictionary like " { x : 5, y : 1 } " all
            I have to do is use the mask "{sisi}".
        */
        PyObject *origin = Py_BuildValue(
                                         "{sisi}",
                                         "x",
                                         r->x,
                                         "y",
                                         r->y
                                        );
        PyObject *size = Py_BuildValue(
                                       "{sisi}",
                                       "width",
                                       r->width,
                                       "height",
                                       r->height
                                      );
        PyObject *rect = Py_BuildValue(
                                       "{sOsO}",
                                       "origin",
                                       origin,
                                       "size",
                                       size
                                      );

        PyList_Append(result, rect);
    }

    return result;
}

static PyObject *
face_detect(PyObject *self, PyObject *args)
{
    const char *path;
    int sts;

    if (!PyArg_ParseTuple(args, "s", &path))
        return NULL;

    IplImage* frame;

    /* OpenCV stores the Haar Cascades for default on this location */
    char *file = "/usr/local/share/opencv/"
        "/haarcascades/haarcascade_frontalface_alt.xml";

    cascade = (CvHaarClassifierCascade*)cvLoad(file, 0, 0, 0);
    storage = cvCreateMemStorage(0);

    IplImage* image = cvLoadImage(path, CV_LOAD_IMAGE_COLOR);

    if(image == NULL) {
        PyErr_SetString(FaceError, "Image not found!");
        cvReleaseHaarClassifierCascade(&cascade);
        cvReleaseMemStorage(&storage);
        return NULL;
    }

     PyObject *result = detectFaces(image);

     cvReleaseHaarClassifierCascade(&cascade);
     cvReleaseMemStorage(&storage);
     cvReleaseImage(&image);

     return result;
}

static PyMethodDef FaceMethods[] = {
    {"detect",  face_detect, METH_VARARGS,
     "Detect the faces on an image."},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

PyMODINIT_FUNC
initface(void)
{
    PyObject *m;

    m = Py_InitModule("face", FaceMethods);
    if (m == NULL)
        return;

    FaceError = PyErr_NewException("face.error", NULL, NULL);
    Py_INCREF(FaceError);
    PyModule_AddObject(m, "error", FaceError);
}

That may seem like a lot to take in at once but it is in reality quite simple. A PyObject is a data structure that represents a Python object, this data structure contains the object’s reference count and a pointer to the objects’type. Unless we’re defining a new kind of Python object we can create our object using Py_BuildObject, this is done to create dictionaries on the above code.
To create a Python dictionary all you have to do is to call the Py_BuildValue function with a formatting mask with brackets, the s represents a string, the i represents a number and the O represents a Python object, so if you want to create a Python dictionary like ” { x : 5, y : 1 } ” all you have to do is use the mask “{sisi}”.

To make your functions visible to Python you must create a method definition table (PyMethodDef) and add it to the module initialization. Also on the module initialization you should initialize the FaceError variable as a Python exception to be thrown if there is an error opening the file, not doing so will cause Python to receive a SIGABRT from the OS.

To compile our module just type the command:

python setup.py build

And copy the resulting module, face.so, to Python’s site-packages directory.

You should now be able to call the module from Python like this:

import face
face_list = face.detect('/Users/raphaelcruzeiro/Documents/people.jpg')

Using the extension on a real application

I’ve created a sample Django application that leverages on this face detection capabilities exposed by our extension. While I won’t discuss every aspect of the Django application as it is out of the scope of this post, I’ll just show here the key aspects of the application (a link to the source repository can be found at the bottom of the article).

The web app allows the user to upload an image and then displays it on the browser. When the image is displayed an AJAX call is made to the server to evaluate the image and supply the client-side with the position of the faces.

from face import detect as _detect

def detect(request):
    print request.POST

    id = request.POST['id']
    image = ModelImage.objects.get(pk=id)

    return HttpResponse(json.dumps(_detect(image.img.path)))

This is how the server-side handles the AJAX request to process the image and this is how the AJAX call is made and the response processed on the client-side:

$(document).ready(function(){
    $.post('{% url detect_url %}', 'id={{ id }}', function(data){
        $.each(data, function(i, f){
            var face = $(' <div style="background: blue; opacity: 0.5; position: absolute;"></div>').css({
                'top' : f.origin.y + 'px',
                'left' : f.origin.x + 'px',
                'height' : f.size.height + 'px',
                'width' : f.size.width + 'px'
            });

            $('#mask').append(face);
        })
        }, 'json');
});

The image is contained within a relative positioned  div  with the id mask and our script basically adds an absolute positioned div on top of each face returned on the JSON response.

The full source code of this article can be found on https://github.com/raphaelcruzeiro/Face-Detector

See the live demo.

An early look at my PDF annotator iPad app

So, I’ve finally taken some time to finish the core of my iPad app (which I am still calling PdfAnnotator for the lack of a better name). One of the challenges of this app was creating a workaround for the CGPDFDocumentRef‘s aggressive cache that made my app’s memory consumption climb up to the highs. (I’m going to write an entire post about this issue and the solution I came up with)

Anyway, here is a first look at the core functionality of my app:

I decided to open-source the core of the app as I believe it is a pretty decent Pdf application with zooming capabilities and a paging control that displays thumbnails, etc. The code is licensed under the MIT license and can be found at: https://github.com/raphaelcruzeiro/PdfAnnotator-for-iPad

The final app will use the open-source core but will bring some features that I haven’t yet seen in any of the competitors. As soon as I have something new to show I’ll post it here.

Computer Vision with OpenCV

What is OpenCV

OpenCV is a cross-platform, open-source library for real time computer vision. Originally created by Intel and now supported by  Willow Garage (a robotics research lab), OpenCV is an elegant and effective library for computer vision and image processing. The library was originally written in C and as of 2009 it also includes a C++ interface.

Getting OpenCV

Getting OpenCV is extremely easy. First of all, be sure to have CMake installed (if you’re on a Mac you can have it installed with MacPorts: sudo port install cmake). With CMake installed you are ready to build the code, just checkout the latest tested version of OpenCV:

svn co https://code.ros.org/svn/opencv/branches/2.3

Run CMake on the OpenCV directory: (this will most likely take a while)

sudo cmake -G "Unix Makefiles" .

And then the usual make and make install:

sudo make -j8
sudo make install

I had some compiling errors about some missing declaration when running make on my Mac, I worked around this issue by including the zlib header on OpenCV-2.3.0/modules/highgui/src/grfmt_png.cpp and then the make went just fine.

Playing with OpenCV

Here is a small sample I wrote using OpenCV for facial detection :(this sample was loosely based on the sample found on the book Learning OpenCV by Gary Bradsky and Adrian Kaehler)

#include <stdio.h>
#include <opencv/cv.h>
#include <opencv2/highgui/highgui_c.h>

CvHaarClassifierCascade *cascade;
CvMemStorage            *storage;

void detectFaces(IplImage *img);

int main (int argc, const char * argv[])
{
    cvNamedWindow( "Source", 1 );

    /* Turn on and capture video from the first available camera
       Found in the system */
    CvCapture *capture = cvCreateCameraCapture(0);

    IplImage* frame;

    int key;

    char *file = "/usr/local/share/opencv/"
        "/haarcascades/haarcascade_frontalface_alt.xml";

    cascade = (CvHaarClassifierCascade*)cvLoad(file, 0, 0, 0);
    storage = cvCreateMemStorage(0);

    if(!capture) {
        printf("Error: no camera detected.");
        return -1;
    }

    while(1) {
        frame = cvQueryFrame(capture);

        if(!frame) break;

        detectFaces(frame);

        key = cvWaitKey(10);

       /* If the users presses the Esc key finish the program */
       if(key == 27) break;
    }

    /* Cleaning up */
    cvReleaseCapture(&capture);
    cvReleaseHaarClassifierCascade(&cascade);
    cvReleaseMemStorage(&storage);
    cvDestroyWindow("Source");
}

void detectFaces(IplImage *img)
{
    int i;

    /* Detect all faces on the frame */
    CvSeq *faces = cvHaarDetectObjects(
                                       img,
                                       cascade,
                                       storage,
                                       1.1,
                                       3, 0,
                                       cvSize(50, 50),
                                       cvSize(50, 50)
                                      );

    /* Now, let's mark each face */
    for(i = 0 ; i < (faces ? faces->total : 0) ; i++) {
        CvRect *r = (CvRect*)cvGetSeqElem(faces, i);
        cvRectangle(
                    img,
                    cvPoint(r->x, r->y),
                    cvPoint(r->x + r->width, r->y + r->height),
                    CV_RGB(0, 230, 0),
                    1,
                    8,
                    0
                   );

        CvFont a;
        cvInitFont(&a, CV_FONT_HERSHEY_TRIPLEX, 1.0f, 1.0f, 1.0f, 4, CV_AA);

        char face[30];
        sprintf(face, "Face %d", i + 1);

        CvSize textSize;
        int ymin = 0;
        cvGetTextSize(face, &a, &textSize, &ymin);

        cvPutText(
                  img,
                  face,
                  cvPoint(r->x + ((r->width / 2) - textSize.width / 2), r ->y + r->height + 25),
                  &a,
                  CV_RGB(0, 230, 0)
                 );
    }

    cvShowImage("Source", img);
}

Be sure to link against libopencv_core, libopencv_objdetect and libopencv_highgui. This code will compile on all major platforms.