[metrics-team] JavaScript on metrics.torproject.org

Tue Jan 12 12:00:01 UTC 2016

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Thomas,

thanks for moving this discussion forward.  Reading about the various
tools and frameworks was sure an interesting read.

I wonder though if we should take one step back, ignore the discussion
of possible web frameworks for the moment, and just compare a future
Tor Metrics that requires some amount of client-side JavaScript to one
that doesn't.

I know that you favor a Metrics with some JavaScript, and there are
certainly good arguments for that.  But that is also the case for a
Metrics that works entirely without JavaScript.  I'd want us to stay
open with regard to this decision until we have a good idea of all the
pros and cons and until we have buy-in from the rest of Tor.  It seems
that we can still have a good discussion of web frameworks once this
general decision for or against JavaScript is made.

What do you think about building a small prototype of a visualization
that requires client-side JavaScript and another one that doesn't?
Most people are not web developers, so they can hardly follow the
discussion of web techniques here.  Giving them an actual example
might make it easier for them to understand the differences.  I'd be
willing to help with building these prototypes.

I have more thoughts on the web frameworks and on requirements like
avoiding packages not contained in Debian stable, but I'll hold them
back until we have that discussion.

All the best,
Karsten

On 08/01/16 15:28, tl wrote:
> Two additions:
> 
> - just found a list of popular isomprphic JS web frameworks:
> http://isomorphic.net/libraries but nothing about which one
> interacts the best with D3 - a decision about a simple templating
> language on top of express would have to be made too. Handlebars is
> popular, but I’m not an expert.
> 
> And, while thinking about it, I’m increasingly inclined to promote
> a solution where all pages that don’t require strong interactivity
> are realized with a simple templating engine (like Handlebars). For
> the few visualizations that benefit heavily from client side JS,
> the templating engine would have to be replaced by an isomorphic
> webframework like React. That way, not all pages would be build the
> same way but most of them would be quite simple to build and
> maintain.
> 
> 
> Still waiting for comments and ideas before moving this topic to
> tor-dev.
> 
> 
> ciao thms
> 
> 
>> On 07.01.2016, at 14:56, tl <tl at rat.io> wrote:
>> 
>> Okay, answering my own post ;-)
>> 
>> tl;dr: no Javascript for most of the site JavaScript as
>> enhancements to pages where sensible server side based on Node,
>> Express, React, MongoDB visualizations with D3
>> 
>> 
>> I favor a solution where we have a site working fine without
>> JavaScript and in some places even finer with JavaScript. With
>> "some places" I mean additional interface elements to control
>> (and play with) visualizations and maybe some additional site
>> navigation too. There’s no question that omitting JavaScript is
>> both praticable and a good thing to do, for reasons of security
>> as welll as performance. But it is also undeniable that
>> JavaScript can facilitate some awesome user interfaces especially
>> with data visualizations that are not possible otherwise.
>> JavaScript usage has been a contested topic right from the start
>> in the 1990ies. My stance is: banning JS from ones personal
>> browser is absolutely fine and reasonable and forcing JS on all
>> users is unacceptable. Abolishing JS  however is a shame since it
>> can be tremendously useful in building user interfaces, and
>> interfaces are the connection between the user and the data
>> (read: not negligible). Building interfaces that work fine
>> without JavaScript but offer easier and more fluid interaction
>> with JavaScript is harder than doing it only one way (either with
>> or without JS) but it is possible without nasty hacks or
>> excessive bloat. I think the extra effort is worth it and this is
>> the way to go. Since IMO this settles the question if we
>> should/could use JavaScript on the site it follows that we should
>> also port the current metrics site backend to JavaScript/Node and
>> base its visualizations on D3.js. That way the whole thing will
>> be based on only one technology - but of course will still
>> require mastering the building blocks. But now, enough with the
>> fundamentalist foreword and down into the drudgery...
>> 
>> 
>> It looks like we don’t need a full blown CMS since only two or
>> three rather technical people will maintain the website. Therefor
>> we can go with one of the established web frameworks and build
>> the little admin interfaces we might need ourselves. A popular JS
>> web framework stack is known by the name MEAN which mimics the
>> LAMP acronym and stands for MongoDB, Express, Angular.js and
>> Node: * Node.js is the JavaScript runtime in which all server
>> functionality is hosted/embedded and is available as Debian
>> stable package * MongoDB is a very popular database, based on
>> JSON, available as Debian stable package as well. I worked a lot
>> with it when aggregating data for the 'visionion' visualization
>> project and although I’m not partucularily fond of it it makes
>> working with slightly unregular document types rather easy. It
>> provides mapReduce functionality to eventually aggregate a little
>> data by the side and its JSON based data schema makes it easy to
>> access from a web app. But I wouldn’t mind to work with
>> PostgreSQL either. * Express is the de-facto standard for web
>> servers in the Node ecosystem and a little more than that. It can
>> be extended with some little templating engine to build sizeable
>> static websites which would be all we need if we go the non-JS
>> way on the frontend. * Angular.JS is a full blown web application
>> framework (developed and backed by Google) - providing the view
>> to Express instead of some small templating engine - that is
>> rather popular (also Arturo recommends it ) and provides all you
>> could ever need in one (big) box. The big downside is that it
>> needs JavaScript on the client side.
>> 
>> That’s the MEAN stack. Node and Express are de facto standards,
>> MongoDB may be just fine. The interesting question is what to use
>> for the view layer on top of Express: a simple templating engine
>> doesn’t provide the JavaScript interface we desire to implement
>> elaborate visualization interfaces while Angular.JS doesn’t
>> provide the non-JS baseline considered mandatory. There exist of
>> course a plethora of JS web app frameworks - too many to analyse
>> them here. A few of them provide the new hotness "isomorphic
>> rendering" which means that the site can be rendered on the
>> server as well as in the browser. The original aim is to speed up
>> initial pageload and optimize for SEO, but I hope it can be used
>> to implement a non-JS-and-JS-too-site. If the browser does not
>> support JavaScript it renders a page that was pregenerated on the
>> server. For most of the metrics site there will be no difference
>> at all since they don’t need any JavaScript. For visualizations
>> though it can make a big difference. Visualization controls will
>> be just (nicely styled) forms and interactions will require a
>> roundtrip to the server. Visualizations will only be able to show
>> limited amounts of data because they can’t fetch data in the
>> background. Some of the worst problems with this approach can be
>> mitigated by loading the visualizations themselves into iFrames,
>> reducing flicker and loading times. If the browser is willing to
>> execute JavaScript though additional data is fetched in the
>> background, zooms and sliders and tooltips appear, animations
>> make it easier to understand effects etc. It will be the same
>> information, but it will be accessible much easier and make more
>> sense. It remains to be seen how much code duplication that
>> involves. Some of the interface elements would have to build as
>> HTML form elements as well as D3 elements. There duplication is
>> unavoidable. Those interface improvements that rely on fetching
>> data in the background should be implementable rather
>> seamlessly.
>> 
>> All this is not exactly trivial. There will be obstacles but
>> after some googling I’m quite convinced that there are sensible
>> solutions too. React [0] (developed and backed by Facebook) is by
>> far the most prominent of the frameworks to support isomorphic
>> rendering [1]. React is a fine piece of technology on its own,
>> rather popular (not as much as AngularJS, but still) and well
>> supported too. In contrast to the all-encompassing AngularJS it
>> only aims to be a view layer and integrates nicely with Express
>> [2]. There is a problem though - React tries to control the DOM
>> just like D3, the visualization engine - but there exist
>> solutions to that [3]. Another, maybe less involved solution for
>> isomorphic rendering is called Rendr [4]. It’s less well known
>> and less supported though, so my first shot would be React. If
>> none of these work out as hoped there’s always the possibilty to
>> use a simple templating engine with Express and add
>> JavaScript-based interactivity through jQuery to some selected
>> visualizations that we think are worth the extra effort. That may
>> not be very elegant or scalable, and more 2005 than 2015, but as
>> a backup strategy it’s proven and works.
>> 
>> 
>> A question that I have is about security: the latest version of
>> Node in Debian stable is 0.10.29 from July 2014. That’s not bad.
>> I wonder though if it’s save to load just about any module
>> available to Node out there - as Node is the (hopefully) secure
>> sandbox it runs in - or if we have to get Debian-stable npm
>> packages for Express, React etc too? That might prove difficult…
>> 
>> 
>> Ciao thms
>> 
>> 
>> [0]
>> http://blog.yld.io/2015/06/10/getting-started-with-react-and-node-js/
>>
>> 
[1]
https://strongloop.com/strongblog/node-js-react-isomorphic-javascript-why-it-matters/
>> [2] https://github.com/reactjs/express-react-views [3]
>> http://oli.me.uk/2015/09/09/d3-within-react-the-right-way/ [4]
>> https://github.com/rendrjs/rendr
>> 
>> 
>> 
>> 
>>> On 06.01.2016, at 22:30, tl <tl at rat.io> wrote:
>>> 
>>> Hi,
>>> 
>>> Karsten, Letty and me recently had a conversation about the use
>>> of JavaScript on https://metrics.torproject.org. The problem
>>> space in brief is that the metrics site needs nice
>>> visualizations which need nice interfaces which can profit very
>>> much from some nice JavaScript whereas there are a lot of nice
>>> people who very reasonably insist on deactivating JavaScript in
>>> their nice Tor browsers. So: what to do? A side aspect of this
>>> discussion is that the metrics website needs a technical
>>> overhaul. It’s based on some hard to maintain servlet/JSP and
>>> while we’re on it we could as well port it to some
>>> JavaScript/Node based framework on the server side if we decide
>>> to use more JavaScript anywy. Following are some notes from the
>>> discussion.
>>> 
>>> +
>>> 
>>> Possible outcomes: 1. Don't require any JavaScript, works
>>> without issues in high security mode. 2. Only require
>>> JavaScript for some visualizations where it makes sense, users
>>> will have to switch to medium-high security to watch those. 2b.
>>> For visualizations that require JavaScript to be really
>>> useful, provide a simplified version that works without
>>> client-side JavaScript. 3. Require JavaScript for most
>>> visualizations by first looking at making them as usable and as
>>> useful as possible, rather than worrying about users having to
>>> change their Tor Browser security setting.
>>> 
>>> Benefits and possibilities of using JavaScript - Ability to
>>> load new data from the server, based on user input, rather than
>>> pre-loading all the data that the user might want to look at. -
>>> Handle user input more directly than via HTML forms which
>>> require a server round-trip. - Let the user decide whether they
>>> want to lower their security in order to look at a
>>> visualization, rather than making that decision for them. -
>>> It's sufficient to host static content, which can be mirrored
>>> much more easily than dynamic content.
>>> 
>>> Possible approaches for passing on JavaScript - Avoid full page
>>> reloads by using iframes. - Investigate more graph frameworks
>>> than just D3.js, including gnuplot, R/ggplot2, others.  A major
>>> reason for using D3.js is its tight integration with user input
>>> elements. - Make heavy use of CSS 3 features/tricks. - Add
>>> tooltips to canvas.
>>> 
>>> Next steps: - Find out why the bubble graph takes so long to
>>> load.  Try to avoid the parts that take long in future graphs. 
>>> - In addition to the bottom-up approach taken here, also take
>>> a top-down approach by thinking about contents and only worry
>>> about possible implementations later.
>>> 
>>> Next steps after that: - Think about possible web frameworks,
>>> like Angular.js if we decide to use JavaScript or Django if we
>>> decide against it. - Look into alternatives to Bootstrap, like
>>> Bourbon.io.
>>> 
>>> +
>>> 
>>> So far, so unclear. Input welcome!
>>> 
>>> 
>>> ciao, thms ---- 
>>> _______________________________________________ metrics-team
>>> mailing list metrics-team at lists.torproject.org 
>>> https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team
>>
>>
>>
>>
>>
>>
>>> 
< he not busy being born is busy dying >
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________ metrics-team
>> mailing list metrics-team at lists.torproject.org 
>> https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team
>
>> 
> 
> 
> 
> 
> < he not busy being born is busy dying >
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________ metrics-team
> mailing list metrics-team at lists.torproject.org 
> https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team
> 

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQEcBAEBAgAGBQJWlOrBAAoJEJD5dJfVqbCrt4MH/jAMVdDzMq6+XnXqP93eWp1r
Oivt9mk3x+k9LIP2WFvxVrUb6FTfeJ6adoBVDjZus9+QLm+EkO0FYIAwwH+M9VdC
IXKmu/SBiPrxz3pKuT7Sw1UZl28RE5iUTioNVT40VdlWBvXA4Kfv4NrquqFfYkfk
JGBIQMDFrIc8PhI+12qvlkKTG9Xrig2CHfHm7/xC9MQnPfly9+7OuDLEaoqftBp8
wMmKapiGA1LYKaUhA7qtK2zKjxioZDv7n44KYjsZsbkJLbnEbghot0lzkHI4DXID
BGQ2RqnfHVI6Yjit+HQgfos+FuC+OAODrTUIMnV16iEmjOjDRHC3NMe6NRw9XUM=
=5Ycf
-----END PGP SIGNATURE-----