[metrics-team] JavaScript on metrics.torproject.org

tl tl at rat.io
Fri Jan 8 14:28:48 UTC 2016

Two additions:

- just found a list of popular isomprphic JS web frameworks: http://isomorphic.net/libraries but nothing about which one interacts the best with D3
- a decision about a simple templating language on top of express would have to be made too. Handlebars is popular, but I’m not an expert.

And, while thinking about it, I’m increasingly inclined to promote a solution where all pages that don’t require strong interactivity are realized with a simple templating engine (like Handlebars). For the few visualizations that benefit heavily from client side JS, the templating engine would have to be replaced by an isomorphic webframework like React. That way, not all pages would be build the same way but most of them would be quite simple to build and maintain.

Still waiting for comments and ideas before moving this topic to tor-dev.


> On 07.01.2016, at 14:56, tl <tl at rat.io> wrote:
> Okay, answering my own post ;-)
> tl;dr:
> no Javascript for most of the site
> JavaScript as enhancements to pages where sensible
> server side based on Node, Express, React, MongoDB
> visualizations with D3
> I favor a solution where we have a site working fine without JavaScript and in some places even finer with JavaScript. With "some places" I mean additional interface elements to control (and play with) visualizations and maybe some additional site navigation too. There’s no question that omitting JavaScript is both praticable and a good thing to do, for reasons of security as welll as performance. But it is also undeniable that JavaScript can facilitate some awesome user interfaces especially with data visualizations that are not possible otherwise. JavaScript usage has been a contested topic right from the start in the 1990ies. My stance is: banning JS from ones personal browser is absolutely fine and reasonable and forcing JS on all users is unacceptable. Abolishing JS  however is a shame since it can be tremendously useful in building user interfaces, and interfaces are the connection between the user and the data (read: not negligible).
> Building interfaces that work fine without JavaScript but offer easier and more fluid interaction with JavaScript is harder than doing it only one way (either with or without JS) but it is possible without nasty hacks or excessive bloat. I think the extra effort is worth it and this is the way to go.
> Since IMO this settles the question if we should/could use JavaScript on the site it follows that we should also port the current metrics site backend to JavaScript/Node and base its visualizations on D3.js. That way the whole thing will be based on only one technology - but of course will still require mastering the building blocks.
> But now, enough with the fundamentalist foreword and down into the drudgery...
> It looks like we don’t need a full blown CMS since only two or three rather technical people will maintain the website. Therefor we can go with one of the established web frameworks and build the little admin interfaces we might need ourselves.
> A popular JS web framework stack is known by the name MEAN which mimics the LAMP acronym and stands for MongoDB, Express, Angular.js and Node:
> * Node.js is the JavaScript runtime in which all server functionality is hosted/embedded and is available as Debian stable package
> * MongoDB is a very popular database, based on JSON, available as Debian stable package as well. I worked a lot with it when aggregating data for the 'visionion' visualization project and although I’m not partucularily fond of it it makes working with slightly unregular document types rather easy. It provides mapReduce functionality to eventually aggregate a little data by the side and its JSON based data schema makes it easy to access from a web app. But I wouldn’t mind to work with PostgreSQL either.
> * Express is the de-facto standard for web servers in the Node ecosystem and a little more than that. It can be extended with some little templating engine to build sizeable static websites which would be all we need if we go the non-JS way on the frontend.
> * Angular.JS is a full blown web application framework (developed and backed by Google) - providing the view to Express instead of some small templating engine - that is rather popular (also Arturo recommends it ) and provides all you could ever need in one (big) box. The big downside is that it needs JavaScript on the client side.
> That’s the MEAN stack. Node and Express are de facto standards, MongoDB may be just fine. The interesting question is what to use for the view layer on top of Express: a simple templating engine doesn’t provide the JavaScript interface we desire to implement elaborate visualization interfaces while Angular.JS doesn’t provide the non-JS baseline considered mandatory.
> There exist of course a plethora of JS web app frameworks - too many to analyse them here. A few of them provide the new hotness "isomorphic rendering" which means that the site can be rendered on the server as well as in the browser. The original aim is to speed up initial pageload and optimize for SEO, but I hope it can be used to implement a non-JS-and-JS-too-site. If the browser does not support JavaScript it renders a page that was pregenerated on the server.
> For most of the metrics site there will be no difference at all since they don’t need any JavaScript. For visualizations though it can make a big difference. Visualization controls will be just (nicely styled) forms and interactions will require a roundtrip to the server. Visualizations will only be able to show limited amounts of data because they can’t fetch data in the background. Some of the worst problems with this approach can be mitigated by loading the visualizations themselves into iFrames, reducing flicker and loading times. If the browser is willing to execute JavaScript though additional data is fetched in the background, zooms and sliders and tooltips appear, animations make it easier to understand effects etc. It will be the same information, but it will be accessible much easier and make more sense. It remains to be seen how much code duplication that involves. Some of the interface elements would have to build as HTML form elements as well as D3 elements. There duplication is unavoidable. Those interface improvements that rely on fetching data in the background should be implementable rather seamlessly.
> All this is not exactly trivial. There will be obstacles but after some googling I’m quite convinced that there are sensible solutions too.
> React [0] (developed and backed by Facebook) is by far the most prominent of the frameworks to support isomorphic rendering [1]. React is a fine piece of technology on its own, rather popular (not as much as AngularJS, but still) and well supported too. In contrast to the all-encompassing AngularJS it only aims to be a view layer and integrates nicely with Express [2]. There is a problem though - React tries to control the DOM just like D3, the visualization engine - but there exist solutions to that [3].
> Another, maybe less involved solution for isomorphic rendering is called Rendr [4]. It’s less well known and less supported though, so my first shot would be React.
> If none of these work out as hoped there’s always the possibilty to use a simple templating engine with Express and add JavaScript-based interactivity through jQuery to some selected visualizations that we think are worth the extra effort. That may not be very elegant or scalable, and more 2005 than 2015, but as a backup strategy it’s proven and works.
> A question that I have is about security: the latest version of Node in Debian stable is 0.10.29 from July 2014. That’s not bad. I wonder though if it’s save to load just about any module available to Node out there - as Node is the (hopefully) secure sandbox it runs in - or if we have to get Debian-stable npm packages for Express, React etc too? That might prove difficult…
> Ciao
> thms
> [0] http://blog.yld.io/2015/06/10/getting-started-with-react-and-node-js/
> [1] https://strongloop.com/strongblog/node-js-react-isomorphic-javascript-why-it-matters/
> [2] https://github.com/reactjs/express-react-views
> [3] http://oli.me.uk/2015/09/09/d3-within-react-the-right-way/
> [4] https://github.com/rendrjs/rendr
>> On 06.01.2016, at 22:30, tl <tl at rat.io> wrote:
>> Hi,
>> Karsten, Letty and me recently had a conversation about the use of JavaScript on https://metrics.torproject.org. The problem space in brief is that the metrics site needs nice visualizations which need nice interfaces which can profit very much from some nice JavaScript whereas there are a lot of nice people who very reasonably insist on deactivating JavaScript in their nice Tor browsers. So: what to do?
>> A side aspect of this discussion is that the metrics website needs a technical overhaul. It’s based on some hard to maintain servlet/JSP and while we’re on it we could as well port it to some JavaScript/Node based framework on the server side if we decide to use more JavaScript anywy.
>> Following are some notes from the discussion.
>> +
>> Possible outcomes:
>> 1. Don't require any JavaScript, works without issues in high
>> security mode.
>> 2. Only require JavaScript for some visualizations where it makes
>> sense, users will have to switch to medium-high security to watch those.
>> 2b. For visualizations that require JavaScript to be really useful,
>> provide a simplified version that works without client-side JavaScript.
>> 3. Require JavaScript for most visualizations by first looking at
>> making them as usable and as useful as possible, rather than worrying
>> about users having to change their Tor Browser security setting.
>> Benefits and possibilities of using JavaScript
>> - Ability to load new data from the server, based on user input,
>> rather than pre-loading all the data that the user might want to look at.
>> - Handle user input more directly than via HTML forms which require a
>> server round-trip.
>> - Let the user decide whether they want to lower their security in
>> order to look at a visualization, rather than making that decision for
>> them.
>> - It's sufficient to host static content, which can be mirrored much
>> more easily than dynamic content.
>> Possible approaches for passing on JavaScript
>> - Avoid full page reloads by using iframes.
>> - Investigate more graph frameworks than just D3.js, including
>> gnuplot, R/ggplot2, others.  A major reason for using D3.js is its
>> tight integration with user input elements.
>> - Make heavy use of CSS 3 features/tricks.
>> - Add tooltips to canvas.
>> Next steps:
>> - Find out why the bubble graph takes so long to load.  Try to avoid
>> the parts that take long in future graphs.
>> - In addition to the bottom-up approach taken here, also take a
>> top-down approach by thinking about contents and only worry about
>> possible implementations later.
>> Next steps after that:
>> - Think about possible web frameworks, like Angular.js if we decide
>> to use JavaScript or Django if we decide against it.
>> - Look into alternatives to Bootstrap, like Bourbon.io.
>> +
>> So far, so unclear. Input welcome!
>> ciao,
>> thms
>> ----
>> _______________________________________________
>> metrics-team mailing list
>> metrics-team at lists.torproject.org
>> https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team
> < he not busy being born is busy dying >
> _______________________________________________
> metrics-team mailing list
> metrics-team at lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team

< he not busy being born is busy dying >

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20160108/3c89b026/attachment.sig>

More information about the metrics-team mailing list