Creating the Tech4Good Awards Glowstick Voting System with Python and Javascript

Overview

Every year the Tech4Good Awards are held to celebrate "innovation in technology that makes the world a better place". During the ceremony the audience are asked to vote for their favourite contenders and traditionally, in keeping with the award's remit, this has involved something a little innovative. Paul Hayes has been doing this pro-bono and asked if I could help out this year.

The auditorium through a fish-eye lens.

Sketching the problem

Our processing workflow started with ten shots per second of a glowstick wielding audience and ended with a continuously updated, big-screen projected representation of current voting intentions. The question was which particular route to take. Cue head-scratching.

First pass

The favoured solution was to keep everything in the land of Javascript (JS). You can now access web-cam images with a fair degree of reliability (see below) client-side, and JS's image-processing power is coming on leaps and bounds. The rather wonderful JSFeat library seemed like a good starting point, and Paul had the ear of its very helpful creator. But a few things like blob-detection and color-filtering (it seems heavily geared towards grayscales) would have to be hacked together, with the clock ticking down rapidly. Some progress was made (blob-detection is a doddle, right?) but failing WebRCT contexts (webcam images failing to make it through the rather new API to JS-land) and the lack of really mature utility functions made it all seem a little ambitious.

Python to the rescue

I have come to take it for granted that for pretty much any problem involving computer software, there will be a set of well-written, fairly intuitive, mature libraries to hand that can be easily stitched together into something resembling a solution. So much so that I sometimes like to make things hard for myself and hold the Python in reserve.
Anyway, given the time constraints it seemed sensible for me to focus on a Python solution while Paul grappled manfully with a full JS pipeline. I knew OpenCV, the mammoth computer-vision library, would be involved as there's an eminently usable Python wrapper. I have a fair amount of experience with it, having used it as the basis for a little startup venture a couple of years ago, and have been generally impressed. I also wanted to investigate SimpleCV, a computer vision framework written in Python and sitting atop OpenCV, among others. It looked a little more user friendly than py-opencv and that proved to be the case.

Mashing things together

With Paul refining the browser-end, my task was to harvest images from the webcam, process them with SimpleCV, producing a vote-count, and get that count to a browser-page, wherein Paul's web-shaders could render them in glorious technicolor.
The obvious solution was to use a web-socket, by means of which a Python server could create a two-way relay system with the browser-based graphics JS code. This used to be a little non-trivial, from memory, but these days there are very user friendly solutions kicking around. I'm a Flask Python kinda guy so settled on Miguel Grindberg's brilliant Flask-SocketIO. With a working computer-vision solution in modular Pythonic form, our funky-visuals web-page would be able to drive it through the web-socket as often as required (10 updates a second didn't break a sweat) and reflect the results returned.
The working prototype is sketched below:

Making it work

With a working prototype and a less than funky pulsating circle placeholder to register voting numbers, it was Paul's job to hook the backend to the front, fiddle about a bit (to give it it's technical term) and have the whole thing ready for an auspicious audience including Mariella Frostrup, in the morning. In what must have been one of the most frenetic coding sessions of his life he battled valiantly against weird Linux/MacOS OpenCV data-ordering differences, the lack of a suitable venue to practice in, glowsticks in the post and the like. While I SMSed the odd bit of feedback from an unbreakable engagement down the pub.
The first task was to use the fact that we had a stable image platform and create a visual mask to isolate the voting area and remove bright spots such as door-signs, monitors, stair-lights, anything that might pass our 'bright, roughly vertical rectangle' test.

Making an image-mask. We use the output from the fish-eye lens (left), to isolate the voting area of interest and remove light-pollutants (centre), producing a mask (right) we can apply to output from the webcam.

With the mask in place, refinements were made to the algorithm to accommodate fish-eye lens effects, real-world light intensities, relative pixel-blob sizes and the like. Paul found that sending calibration values back to Python from Javascript (gotta love those two-way web-sockets) was a huge time-saver - this he hacked on the day, time counting down. As is often the case, the time spent getting decent feedback to parametric changes pays off in spades.
The images below show how well the set-up performed, easily capturing the audience's consensus.

The webcam image is masked (left) before being processed. The centre image shows the blobs detected and, outlined in red, those assumed to be voting glowsticks. The accuracy was pretty much spot on, certainly good enough to measure the audience's preferences. The vote-count was then beautifully rendered using a webgl shaders, these produced with the rather fine glsl.js library, a subset of the WebGL library focusing on OpenGL shaders.

Conclusion

Given the time-constraints it was gratifying to get something up and working, that it worked so well was icing on the cake. But the real stars of the show are the languages and libraries that made it possible. One of the joys of coding these days is the ready availability of mature, well documented open-source libraries. If something isn't quite right you can change the source-code but often it's fine as it is, a labour of love even. So hats off to Python, Javascript, OpenCV, SimpleCV, Flask and Flask-SocketIO, socket.io, WebGL, glsl.js among others.