Getting it Out There

Python, Javascript and Web-visualizations

PyData London 2014

Your host for this afternoon's entertainment - Kyran Dale / http://kyrandale.com

Talk Structure

  1. Very Brief Intro to JS from Pythonista perspective
  2. JSON - how to glue JS to Python
  3. Serving the data
  4. Some Use Cases
  5. Summing up

The Dream of the Pythonista

The Dream of the Pythonista

The Dream of the Pythonista

Javascript

A few personal observations

Javascript

Oops

  • Import Error: no module named foo
  • Global by default
  • Ecmascript's rough edges (looping, params etc.)
  • The Ugly:
    
    setGroundPlane: function(w, h, texture) {
        w = typeof w !== 'undefined'? w: 2048;
        h = typeof h !== 'undefined'? h: 2048;
    [ ... ]
        for(var d in drawables){
            this.drawables[d].update(drawables[d]);
                   
  • Crockford's 'Javscript: the good parts'

So Many Standards to Choose From

There are lots and lots of frameworks

jQuery, Dojo-toolkit, ExtJS, Twitter-Bootstrap, YUI, MooTools, Kendo, Prototype, SproutCore, Ember.js, Blossom, Google-closure, Cappuccino, Tower.js, DHTMLX, CoreChain, qooxdoo, Mojito, Meteor, Qatrix, Knockout.js, Backbone.js, AngularJS, Spine, Agility.js, Sammy.js, JavascriptMVC, Batman.js, Soma.js... ad. inf.

It's All Too Beautiful!

Getting Help

Typical thread

Question

“ Does anyone know how to set my Grunt config file to ignore temporary files in the watch directory? ”

Answer

“ Have you tried using Gulp? ”

The Goods

  • Pretty much 100% installation base
  • Arms war among the engines: orders of magnitude increases in performance
  • One fairly quickly normalises to syntax quirks - prototyping is actually quite cool
  • Exhilarating times: flipside of wild-west
  • Increasingly powerful Visualization tools: D3 and THREE.js (WebGL) to the fore

Cool Libraries We Have Known

D3, the JS Visualization Powerhouse (+Vega)

Cool Libraries We Have Known

D3 + Crossfilter.js: JS Can Crunch Numbers

Server-side Javascript?

  • Nothing to compare with Numpy, Scipy etc: the glory of Python is its libraries
  • Although strides made with modularity (require) still usual JS quirks
  • Aside from a few edge cases (high asynchronous web-traffic) no really compelling case for experienced Pythonista

Coffeescript

  • Python/Ruby ish syntactic sugar
  • Difficulties debugging
  • Really not advisable for JS newbies IMO

JSON is the glue between Python and JS

JSON for data delivery

Example


{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Product",
    "description": "A product from Acme's catalog",
    "type": "object",
    "properties": {
        "id": {
            "description": "The unique identifier for a product",
            "type": "integer"
        }
    },
    "required": ["id"]
}
          

Python's json module



#data.json:
#{"foo": [1, 2, 3], "baa": {"Hello": "World!"}
In [1]  import json
# load data from a file
In [2]  with open("data.json") as json_file:
            json_data = json.load(json_file)
        json_data
Out[3]: {u'baa': {u'Hello': u'World!'}, u'foo': [1, 2, 3]}
# or from a string
In [4]  jdata = json.load('{"foo": [1, 2, 3], "baa": {"Hello": "World!"}')
In [5]: jdata
Out[6]: {u'baa': {u'Hello': u'World!'}, u'foo': [1, 2, 3]}
# and equivalent json.dump and json.dumps
          

Server to Client

Lightweight Servers

A one line solution

python -m SimpleHTTPServer 

Gotchas: streaming video and web-sockets issues.

Lightweight Servers

Flask


from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()             
          

Lightweight Servers

Flask: towards a basic RESTful API


from flask import Flask, jsonify
app = Flask(__name__)

import db

@app.route("/api/v1/runs/< int:id>", methods=['GET'])
def get_runs(id):
    run = db.get_run(id=id)
    if(run):
        return jsonify({'run': run})
    abort(404) # resource not found

if __name__ == "__main__":
    app.run()
          

Databases

  • Mongo BSON based, which lends a certain convenience. Useful for Meteor Framework
  • But Python makes working with JSON easy. Native data is trivial, other stuff fairly easily serialized
  • Just write custom-serializer:
    
    with open('entry.json', 'w', encoding='utf-8') as f:
            json.dump(entry, f, default=customserializer.to_json)
    # need complimentary customserializer.from_json to load
                  

Web-sockets

Use-case: Delivering interactive data from a Python engine

We have a pre-existing Python simulator and wish to push its results to the web without having to rewrite it in Javascript. With web-sockets the simulator need only provide its state in JSON(izable) form and the server negotiate an ongoing stream with the client. This data provides the basis for a dynamic presentation in the browser.

Web-sockets

Python engines, JS visuals

Web-sockets

Native-GUI to Web-GUI

Visualizing the Weather

Old Version

Visualizing the Weather

Interactive data v 0.1

Visualizing the Weather

Interactive data v 0.2

Visualizing the Weather

Modules ho!

  • cleaning the data-files: pymongo, re, glob
  • Python tailor-made for such string manipulations
  • producing geo-JSON: scipy, geojson, pymongo
  • No matter how obscure, Python delivers:
    
    # convert British national grid coords to world
    lat, lng = OSGB36toWGS84(e, n)
                   

Visualizing the Weather

Producing an interpolated grid


from scipy.interpolate import Rbf, griddata, interp2d
[ ... ]
def get_month_year_interp(self, mm, yyyy, col='sun', grid_res_deg=0.3):
    data = self.get_month_year(mm, yyyy)
    x, y, z = self.get_xyz(data, col)
    # produce interpolated grid data 
    interp = interp2d(x, y, z)
    [ ... ]
                col_val= interp(lat, lng)
            

Conclusion

  • Javascript not just a necessary evil
  • Some first-class libraries, and enough performance to be dangerous
  • Perfect compliment to Python's data crunching fu
  • We can off-load most visualizations to the web and this is only going to get better and easier