Getting Your Python Data onto a Browser

PyData London 1st July 2014

A short, practical guide to making the leap from OS based visualizations to something you can share with the world at the click of a button. Starring Python, Javascript and D3.

Kyran Dale / http://kyrandale.com

Talk Structure

  • Brief introduction to main themes and a few tips and sanity checks
  • The form of data (JSON when possible)
  • A 'Hello World' for beginning Python+JS development
  • Better ways of getting the data from Python to Javascript
  • What to do with it when you get there...
  • ... a little D3 intro
  • Suggestion for further explorations
  • Summation and some useful resources

Intro

This is a compliment to previous PyData talk with a practical edge. Anyone made quesy by the sight of code should leave the room now.
The stress will be on developing some best-practices and getting a firm foundation in place for building Python + Javascript visualizations.
Some very cool initiatives at the moment like the D3-based mpld3 (see IPython Notebook), Vincent (Vega-based) and d3py. But this talk is about building from ground up - once you have that ability most intermediate solutions will feel restrictive. Novel, creative data-visualization is a key aim here.

Why now is a good time to be doing this

  • If you are into data you are into data-visualization
  • Relatively, browsers are steroid-enhanced, 500 pound gorillas compared with yesteryear
  • Javascript engines now powerful enough to hack the maths
  • Some wonderful, mature libraries out there, D3 being a standout
  • People's expectations re. web-visualization are increasing - pre-rendered pngs don't hack it anymore
  • It's fun... honestly

Cool, random discovery for the day

Courtesy of DLMF-NIST

A few sanity checks for Javascript

  • Avoid all frameworks early on
  • Avoid Coffescript et. al. by same token
  • Read Crockford's 'Javascript: the good parts'
  • Being solid with JS will help in long run

The best advice you'll ever receive

Good editor + linter = relative joy

Tips for web data-visualization

  • Do as much data pre-processing as possible in Python
  • With D3 et. al. start simply but try and understand profoundly
  • Know enough about the Document Object Model to be dangerous
  • Think hard about the form data will take (e.g. Micha's Rule)

Micha's Rule: Don't store data in keys of a JSON-blob

D3 friendly


data = [
    {'title': 'foo', 'message': 'bar', (...)},
    {'title': 'baz', 'message': 'qux', (...)}, ... 
]
          

Avoid this sort of thing


data = {
    'foo': {'message': 'bar', (...)},
    'baz':{'message': 'qux', (...)}, ...
}
          

JSON is the glue between Python and JS

JSON for data delivery

Example


{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Product",
    "description": "A product from Acme's catalog",
    "type": "object",
    "properties": {
        "id": {
            "description": "The unique identifier for a product",
            "type": "integer"
        }
    },
    "required": ["id"]
}
          

Python's json module


#data.json:
#{"foo": [1, 2, 3], "baa": {"Hello": "World!"}
In [1]  import json
# load data from a file
In [2]  with open("data.json") as json_file:
            json_data = json.load(json_file)
        json_data
Out[3]: {u'baa': {u'Hello': u'World!'}, u'foo': [1, 2, 3]}
# or from a string
In [4]  jdata = json.load('{"foo": [1, 2, 3], "baa": {"Hello": "World!"}')
In [5]: jdata
Out[6]: {u'baa': {u'Hello': u'World!'}, u'foo': [1, 2, 3]}
# and equivalent json.dump and json.dumps
          

PyData to Javascript: A Hello World


# server.py
import json


data = [{'title':'Hello World!', 'body':"Your big, blue, roundness impresses us all."}]
json.dump(data, open('demo_data.json', 'w'))
          


<!DOCTYPE html>



Running a development server

The easy way, from the project directory just type and enter


              $ python -m SimpleHTTPServer
            

Open the default address in your browser and hey presto!

Note: node's http-server is faster and can handle streaming video etc..


$ npm install http-server -g; http-server -p 8000             

A trivial data pipeline from Python -> JSON -> Javascript but most data visualization work will be increasingly sophisticated variations on this theme.

A little organization

Best-practice 101, getting those files organized early will save a lot of time in the long run.

Adjust the index.html entry-point




<link rel="stylesheet" href="static/css/style.css">


json data from the '/api/<id>' call

<script src="http://d3js.org/d3.v3.min.js"></script> <script src="static/js/simplechart.js"></script> <script src="static/js/script.js"></script>

A little Flask-based RESTful API


#!flask/bin/python
from flask import Flask, make_response
import pandas as pd

app = Flask(__name__)

dummy_data = {
    0: pd.DataFrame({'name':['A', 'B', 'C', 'D'], 'value':[4, 2, 8, 5]}),
    1: pd.DataFrame({'name':['A', 'B', 'C'], 'value':[13, 29, 9]}),
    2: pd.DataFrame({'name':['A', 'B', 'C','D', 'E', 'F'], 'value':[3, 12, 9, 21, 15, 7]})
    }

@app.route('/')
def index():
    return make_response(open('index.html').read())


@app.route('/api/<int:id>')
def api(id):
    return make_response(dummy_data[id].to_json(orient='records'))


if __name__ == '__main__':
    app.run(debug = True)
            

Charts Jim, but not as we know them

  • D3 was inspired by Wilkinson's classic text 'Grammar of Graphics', as was ggplot
  • Allows full control over all elements of visualization
  • The pay-off for all this expressive power is a steep (some would say vertiginous) learning curve
  • But basic elements fairly simple - important to understand them well...

A Simple Barchart

Building a chart with D3 (condensed)


var SimpleChart = function() {
    var margin = {top: 15, right: 30, bottom: 15, left: 40},
    width = 600 - margin.left - margin.right,
    height = 400 - margin.top - margin.bottom;
    var x = d3.scale.ordinal().rangeRoundBands([0, width], 0.1);
    var y = d3.scale.linear().range([height, 0]);
    var xAxis = d3.svg.axis().scale(x).orient("bottom");
    var yAxis = d3.svg.axis().scale(y).orient("left");
    var chart = function(selection) {
        selection.each(function(data) {
            chart.svg = d3.select(this).selectAll('g').data([data]);
            var gEnter = chart.svg.enter().append('g');
            gEnter.append("g").attr("class", "x axis");
            gEnter.append("g").attr("class", "y axis");
            gEnter.append("g").attr("class", "plot-area");
            x.domain(data.map(function(d) { return d.name; }));
            y.domain([0, d3.max(data, function(d) { return d.value; })]);
            chart.svg
                .attr("width", width + margin.left + margin.right)
                .attr("height", height + margin.top + margin.bottom);
            var g = chart.svg
                .attr("transform", "translate(" + margin.left + "," + margin.top + ")");
            g.select('.x.axis')
                .attr("transform", "translate(0," + height + ")")
                .transition().duration(1000).call(xAxis);
            g.select('.y.axis')
                .transition().duration(1000).call(yAxis);
            var cd = g.select('.plot-area').selectAll('.bar')
                .data(data);
            cd.enter().append("svg:rect")
                .attr("class", "bar");
            cd.exit()remove();
            cd.transition().duration(1000)
                .attr("x", function(d) { return x(d.name); })
                .attr("y", function(d) { return y(d.value); })
                .attr("height", function(d) { return height - y(d.value); })
                .attr("width", x.rangeBand());
        });
    };
    return chart;
};
            

D3 101

  • SELECT a DOM element: svg = select('svg')
  • BIND some DATA to it: bars = svg.selectAll('.bar').data()
  • ENTER and APPEND: bars.enter().append('rect').attr('class', 'bar')
  • EXIT and REMOVE: bars.exit().remove()
  • Think of stamping a DOM-element with a data-dependent cutter

Reactive charts and data-filters

Using Angular Directives and Bootstrap to create control-panel with html


 
  

Histograms and barcharts with menu-filter and chart-filter

Reactive charts and data-filters

Charts responding to data change

Summing up

  • Now is a great time to be doing web visualizations
  • WebGL, SVG, CSS3D all pretty solid and getting more so
  • Some hugely powerful, mature libraries and a fast language
  • Python natural compliment to Javascript - no replacement for the ease it brings to data-analysis
  • Hope this talk has provided at least a small springboard
  • Get out there, play around and have some fun

Resources