Every year the Tech4Good Awards are held to celebrate "innovation in technology that makes the world a better place". During the ceremony the audience are asked to vote for their favourite contenders and traditionally, in keeping with the award's remit, this has involved something a little innovative. Paul Hayes has been doing this pro-bono and asked if I could help out this year.
Sketching the problem
Our processing workflow started with ten shots per second of a glowstick wielding audience and ended with a continuously updated, big-screen projected representation of current voting intentions. The question was which particular route to take. Cue head-scratching.
Python to the rescue
I have come to take it for granted that for pretty much any problem involving computer software, there will be a set of well-written, fairly intuitive, mature libraries to hand that can be easily stitched together into something resembling a solution. So much so that I sometimes like to make things hard for myself and hold the Python in reserve.
Anyway, given the time constraints it seemed sensible for me to focus on a Python solution while Paul grappled manfully with a full JS pipeline. I knew OpenCV, the mammoth computer-vision library, would be involved as there's an eminently usable Python wrapper. I have a fair amount of experience with it, having used it as the basis for a little startup venture a couple of years ago, and have been generally impressed. I also wanted to investigate SimpleCV, a computer vision framework written in Python and sitting atop OpenCV, among others. It looked a little more user friendly than py-opencv and that proved to be the case.
Mashing things together
With Paul refining the browser-end, my task was to harvest images from the webcam, process them with SimpleCV, producing a vote-count, and get that count to a browser-page, wherein Paul's web-shaders could render them in glorious technicolor.
The obvious solution was to use a web-socket, by means of which a Python server could create a two-way relay system with the browser-based graphics JS code. This used to be a little non-trivial, from memory, but these days there are very user friendly solutions kicking around. I'm a Flask Python kinda guy so settled on Miguel Grindberg's brilliant Flask-SocketIO. With a working computer-vision solution in modular Pythonic form, our funky-visuals web-page would be able to drive it through the web-socket as often as required (10 updates a second didn't break a sweat) and reflect the results returned.
The working prototype is sketched below:
Making it work
With a working prototype and a less than funky pulsating circle placeholder to register voting numbers, it was Paul's job to hook the backend to the front, fiddle about a bit (to give it it's technical term) and have the whole thing ready for an auspicious audience including Mariella Frostrup, in the morning. In what must have been one of the most frenetic coding sessions of his life he battled valiantly against weird Linux/MacOS OpenCV data-ordering differences, the lack of a suitable venue to practice in, glowsticks in the post and the like. While I SMSed the odd bit of feedback from an unbreakable engagement down the pub.
The first task was to use the fact that we had a stable image platform and create a visual mask to isolate the voting area and remove bright spots such as door-signs, monitors, stair-lights, anything that might pass our 'bright, roughly vertical rectangle' test.
The images below show how well the set-up performed, easily capturing the audience's consensus.