[metrics-team] JavaScript on metrics.torproject.org

tl tl at rat.io
Wed Jan 13 00:00:08 UTC 2016

Hi Karsten,


I think we have a misunderstanding here. I’m speaking about two ways of using JavaScript: server-side and client side. Client side I’m totally in favor of a site that works without JavaScript as far as possible - and that is quite far - but adds functionality for those that have JavaScript enabled and that is not possible to convey without client side Javascript. Server side there are reasons why I think it could be beneficial to use a JavaScript framework, but I have no idea about the security implications


I think that almost all of what the Metrics site or other sites in the Tot network need can be done without client side JavaScript. These are static pages, serving text and a few bitmaps and there’s absolutely no reason why they should demand the use of JavaScript. Sites like Atlas and Globe could be done perfectly fine without client side JavaScript and if I understood correctly they where done with one of the heaviest JavaScript frameworks around just because it was the fastest way to do it - which often translates to "that was the tool that we knew already".

But client side JavaScript comes into play when you want interactive visualizations. There are 2 graph visualizations on the bottom of a random Globe page I visited [0]. These graphs can be generated on the server and delivered to the client as SVG vector graphic or as PNG bitmap graphic. But there is also a slider under these graphs with which you can control the period to be shown. This is client side JavaScript. It allows you to intuitively select a period and to update the graph accordingly without a whole page reload - thus allowing to play with the data, making the experience of drilling for details fluid and seemingly effortless blabla. Ahem, well, sort of, but you get the idea. Without client side JavaScript you have to resort to form-based interfaces. These don’t need to look bad either - CSS  made the 1990 form styles go away for good - and you can mitigate the page-reload problem by putting the graphics in an iframe. So it’s not all black or white but you can easily imagine that these tricks will only get you so far while some features are just not possible achieve without client side JavaScript. For example you won’t get animations without JavaScript (there are special cases where CSS animations can save your ass, but not in general and not with reasonable effort as far as I know). Animations and fluid interactions are not just gimmicky nice-to-haves. Animations can be very important to understand transitions and thereby correlations, good interaction is key to successfully investigating the data. Look for example at the dc.js standard exampel [1] and try a) not to be impressed and b) to do that without client side JavaScript. No way!

So: there is stuff that is as cool to have as impossible to do without client side JavaScript. We can always say "we’ll try" - and we should! - but we will ony get that far. And like Letty I want to able to do the cool stuff too. Most of the work is in aggregating the data and setting the whole thing up. That people with high security settings won’t see it shouldn’t mean that nobody else can profit from it.

It’s easy to do meaningful rudimentary non-scripted static versions of JavaScript heavy visualizations. It’s not so easy to do them well and as good as posssible. There certainly is the danger of moral hazard: putting much love into the shiny scripted "experience" and neglecting the tedious, somehow bolted on form-based version. I hope that the isomorphic JavaScript frameworks that I wrote about in my last post help here, becauese they allow to render server side what normally the client would render. But that really has to be tested to be sure and I fear that a quick Proof of Concept will not reveal all problems and possibilities.

Realisitically I think we should define which visualizations we want, like: what is the basic stuff that really needs to be there - and do that without client side JavaScript as good as possible. There will inevitably always be ways to make it even better with client side JavaScript, and we should try to implement some of those too, as examples and tool boxes for how to build client side JavaScript that goes the extra mile and stays accessible to non-Javascript enabled browsers. Hopefully the isomorphic web frameworks will enable us to do that without too much extra work. Apart from that we can’t do much more than appeal to people to take the need for non-scripted web pages serious and use the tools and examples that we are able to provide.

Now server side JavaScript:
given the above somehow passionate plea for _some_ client side JavaScript I sort of took it for granted that porting the backend to JavaScript too would be the sensible thing to do, especially if we find that an isomorphic web framework like React really delivers on the promise and makes it easier for us to serve scripted visualizations that are rendered on the server if the client has JavaScript disabled. There are more and more web developers that can code JavaScript and feel at home in a Node.js-based environment but for whom a Java- or Python-based web framework would be mostly intimidating (you can count me in). So: why not?! Maybe not because of security concerns, and as I said already: I have no idea what is secure in that realm and what isn’t. Node.js is in Debian stable, but what does that mean? Is therefor every package that runs inside Node.js safe too? Or is it not?

One important thing to consider is that client side JavaScript can’t harm anybody who hasn’t enabled JavaScript in the first place. Sure, if we provide cool interactive JS-driven visualizations we provide incentive to enable it

You might have problems with moral hazard, but might also have problems with no hazard at all because not many people want to work under a strict non-JS regime.

> On 12.01.2016, at 13:00, Karsten Loesing <karsten at torproject.org> wrote:


> What do you think about building a small prototype of a visualization
> that requires client-side JavaScript and another one that doesn't?
> Most people are not web developers, so they can hardly follow the
> discussion of web techniques here.  Giving them an actual example
> might make it easier for them to understand the differences.  I'd be
> willing to help with building these prototypes.

Take Lettys visualization and take away the animation and the time slider. Take away the time sliders from [0]. Never have something like [1]. I could pick some more examples from the D3 gallery [2]. Do we really need more proof?

I’m getting a bit edgy and probably I'm not the best embassador with respect to the usage of JavaScript: I had discussions pro/con JavaScript in leftist circles since 20 years, I had discussions pro/con fax machines before and in between I had discussions pro/con CSS, because some die hard anarchists considered it street to stick with their rotten Netscape Communicator 4.7.6. I find these discussions most often very arbitrary: mail is okay, but fax is not. Fax is okay, but computers are not. Computers are okay, but not databases. The web is okay, but not JavaScript, and so on. Why not just say: it depends. Can it harm somebody, can it profit somebody, can somebody profit without someone else being harmed?

I totally respect security concerns. I absolutely understand concerns about moral hazard like in the example above and I think we should have that in mind and try hard to mitigate it. I find it a good and satisfying challenge to achieve as much as possible without JavaScript.
But I also think that JavaScript is a cool technology, that being able to script visualizations provides really useful and otherwise unachievable benefits and that we should not give those away because some people can’t profit from them. That sort of standardization on the lowest level only causes frustration.

> I have more thoughts on the web frameworks and on requirements like
> avoiding packages not contained in Debian stable, but I'll hold them
> back until we have that discussion.

I’d love to hear about those thoughts right away!


[0] https://globe.torproject.org/#/relay/7C16F60CD2AEB2208ADF5B69187DDF8A61C85EAE
[1] https://dc-js.github.io/dc.js/
[2] https://github.com/mbostock/d3/wiki/Gallery
[3] http://arstechnica.com/security/2016/01/going-forward-the-tor-project-wants-to-be-less-reliant-on-us-govt-funding/

> All the best,
> Karsten
> On 08/01/16 15:28, tl wrote:
> > Two additions:
> >
> > - just found a list of popular isomprphic JS web frameworks:
> > http://isomorphic.net/libraries but nothing about which one
> > interacts the best with D3 - a decision about a simple templating
> > language on top of express would have to be made too. Handlebars is
> > popular, but I’m not an expert.
> >
> > And, while thinking about it, I’m increasingly inclined to promote
> > a solution where all pages that don’t require strong interactivity
> > are realized with a simple templating engine (like Handlebars). For
> > the few visualizations that benefit heavily from client side JS,
> > the templating engine would have to be replaced by an isomorphic
> > webframework like React. That way, not all pages would be build the
> > same way but most of them would be quite simple to build and
> > maintain.
> >
> >
> > Still waiting for comments and ideas before moving this topic to
> > tor-dev.
> >
> >
> > ciao thms
> >
> >
> >> On 07.01.2016, at 14:56, tl <tl at rat.io> wrote:
> >>
> >> Okay, answering my own post ;-)
> >>
> >> tl;dr: no Javascript for most of the site JavaScript as
> >> enhancements to pages where sensible server side based on Node,
> >> Express, React, MongoDB visualizations with D3
> >>
> >>
> >> I favor a solution where we have a site working fine without
> >> JavaScript and in some places even finer with JavaScript. With
> >> "some places" I mean additional interface elements to control
> >> (and play with) visualizations and maybe some additional site
> >> navigation too. There’s no question that omitting JavaScript is
> >> both praticable and a good thing to do, for reasons of security
> >> as welll as performance. But it is also undeniable that
> >> JavaScript can facilitate some awesome user interfaces especially
> >> with data visualizations that are not possible otherwise.
> >> JavaScript usage has been a contested topic right from the start
> >> in the 1990ies. My stance is: banning JS from ones personal
> >> browser is absolutely fine and reasonable and forcing JS on all
> >> users is unacceptable. Abolishing JS  however is a shame since it
> >> can be tremendously useful in building user interfaces, and
> >> interfaces are the connection between the user and the data
> >> (read: not negligible). Building interfaces that work fine
> >> without JavaScript but offer easier and more fluid interaction
> >> with JavaScript is harder than doing it only one way (either with
> >> or without JS) but it is possible without nasty hacks or
> >> excessive bloat. I think the extra effort is worth it and this is
> >> the way to go. Since IMO this settles the question if we
> >> should/could use JavaScript on the site it follows that we should
> >> also port the current metrics site backend to JavaScript/Node and
> >> base its visualizations on D3.js. That way the whole thing will
> >> be based on only one technology - but of course will still
> >> require mastering the building blocks. But now, enough with the
> >> fundamentalist foreword and down into the drudgery...
> >>
> >>
> >> It looks like we don’t need a full blown CMS since only two or
> >> three rather technical people will maintain the website. Therefor
> >> we can go with one of the established web frameworks and build
> >> the little admin interfaces we might need ourselves. A popular JS
> >> web framework stack is known by the name MEAN which mimics the
> >> LAMP acronym and stands for MongoDB, Express, Angular.js and
> >> Node: * Node.js is the JavaScript runtime in which all server
> >> functionality is hosted/embedded and is available as Debian
> >> stable package * MongoDB is a very popular database, based on
> >> JSON, available as Debian stable package as well. I worked a lot
> >> with it when aggregating data for the 'visionion' visualization
> >> project and although I’m not partucularily fond of it it makes
> >> working with slightly unregular document types rather easy. It
> >> provides mapReduce functionality to eventually aggregate a little
> >> data by the side and its JSON based data schema makes it easy to
> >> access from a web app. But I wouldn’t mind to work with
> >> PostgreSQL either. * Express is the de-facto standard for web
> >> servers in the Node ecosystem and a little more than that. It can
> >> be extended with some little templating engine to build sizeable
> >> static websites which would be all we need if we go the non-JS
> >> way on the frontend. * Angular.JS is a full blown web application
> >> framework (developed and backed by Google) - providing the view
> >> to Express instead of some small templating engine - that is
> >> rather popular (also Arturo recommends it ) and provides all you
> >> could ever need in one (big) box. The big downside is that it
> >> needs JavaScript on the client side.
> >>
> >> That’s the MEAN stack. Node and Express are de facto standards,
> >> MongoDB may be just fine. The interesting question is what to use
> >> for the view layer on top of Express: a simple templating engine
> >> doesn’t provide the JavaScript interface we desire to implement
> >> elaborate visualization interfaces while Angular.JS doesn’t
> >> provide the non-JS baseline considered mandatory. There exist of
> >> course a plethora of JS web app frameworks - too many to analyse
> >> them here. A few of them provide the new hotness "isomorphic
> >> rendering" which means that the site can be rendered on the
> >> server as well as in the browser. The original aim is to speed up
> >> initial pageload and optimize for SEO, but I hope it can be used
> >> to implement a non-JS-and-JS-too-site. If the browser does not
> >> support JavaScript it renders a page that was pregenerated on the
> >> server. For most of the metrics site there will be no difference
> >> at all since they don’t need any JavaScript. For visualizations
> >> though it can make a big difference. Visualization controls will
> >> be just (nicely styled) forms and interactions will require a
> >> roundtrip to the server. Visualizations will only be able to show
> >> limited amounts of data because they can’t fetch data in the
> >> background. Some of the worst problems with this approach can be
> >> mitigated by loading the visualizations themselves into iFrames,
> >> reducing flicker and loading times. If the browser is willing to
> >> execute JavaScript though additional data is fetched in the
> >> background, zooms and sliders and tooltips appear, animations
> >> make it easier to understand effects etc. It will be the same
> >> information, but it will be accessible much easier and make more
> >> sense. It remains to be seen how much code duplication that
> >> involves. Some of the interface elements would have to build as
> >> HTML form elements as well as D3 elements. There duplication is
> >> unavoidable. Those interface improvements that rely on fetching
> >> data in the background should be implementable rather
> >> seamlessly.
> >>
> >> All this is not exactly trivial. There will be obstacles but
> >> after some googling I’m quite convinced that there are sensible
> >> solutions too. React [0] (developed and backed by Facebook) is by
> >> far the most prominent of the frameworks to support isomorphic
> >> rendering [1]. React is a fine piece of technology on its own,
> >> rather popular (not as much as AngularJS, but still) and well
> >> supported too. In contrast to the all-encompassing AngularJS it
> >> only aims to be a view layer and integrates nicely with Express
> >> [2]. There is a problem though - React tries to control the DOM
> >> just like D3, the visualization engine - but there exist
> >> solutions to that [3]. Another, maybe less involved solution for
> >> isomorphic rendering is called Rendr [4]. It’s less well known
> >> and less supported though, so my first shot would be React. If
> >> none of these work out as hoped there’s always the possibilty to
> >> use a simple templating engine with Express and add
> >> JavaScript-based interactivity through jQuery to some selected
> >> visualizations that we think are worth the extra effort. That may
> >> not be very elegant or scalable, and more 2005 than 2015, but as
> >> a backup strategy it’s proven and works.
> >>
> >>
> >> A question that I have is about security: the latest version of
> >> Node in Debian stable is 0.10.29 from July 2014. That’s not bad.
> >> I wonder though if it’s save to load just about any module
> >> available to Node out there - as Node is the (hopefully) secure
> >> sandbox it runs in - or if we have to get Debian-stable npm
> >> packages for Express, React etc too? That might prove difficult…
> >>
> >>
> >> Ciao thms
> >>
> >>
> >> [0]
> >> http://blog.yld.io/2015/06/10/getting-started-with-react-and-node-js/
> >>
> >>
> [1]
> https://strongloop.com/strongblog/node-js-react-isomorphic-javascript-why-it-matters/
> >> [2] https://github.com/reactjs/express-react-views [3]
> >> http://oli.me.uk/2015/09/09/d3-within-react-the-right-way/ [4]
> >> https://github.com/rendrjs/rendr
> >>
> >>
> >>
> >>
> >>> On 06.01.2016, at 22:30, tl <tl at rat.io> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Karsten, Letty and me recently had a conversation about the use
> >>> of JavaScript on https://metrics.torproject.org. The problem
> >>> space in brief is that the metrics site needs nice
> >>> visualizations which need nice interfaces which can profit very
> >>> much from some nice JavaScript whereas there are a lot of nice
> >>> people who very reasonably insist on deactivating JavaScript in
> >>> their nice Tor browsers. So: what to do? A side aspect of this
> >>> discussion is that the metrics website needs a technical
> >>> overhaul. It’s based on some hard to maintain servlet/JSP and
> >>> while we’re on it we could as well port it to some
> >>> JavaScript/Node based framework on the server side if we decide
> >>> to use more JavaScript anywy. Following are some notes from the
> >>> discussion.
> >>>
> >>> +
> >>>
> >>> Possible outcomes: 1. Don't require any JavaScript, works
> >>> without issues in high security mode. 2. Only require
> >>> JavaScript for some visualizations where it makes sense, users
> >>> will have to switch to medium-high security to watch those. 2b.
> >>> For visualizations that require JavaScript to be really
> >>> useful, provide a simplified version that works without
> >>> client-side JavaScript. 3. Require JavaScript for most
> >>> visualizations by first looking at making them as usable and as
> >>> useful as possible, rather than worrying about users having to
> >>> change their Tor Browser security setting.
> >>>
> >>> Benefits and possibilities of using JavaScript - Ability to
> >>> load new data from the server, based on user input, rather than
> >>> pre-loading all the data that the user might want to look at. -
> >>> Handle user input more directly than via HTML forms which
> >>> require a server round-trip. - Let the user decide whether they
> >>> want to lower their security in order to look at a
> >>> visualization, rather than making that decision for them. -
> >>> It's sufficient to host static content, which can be mirrored
> >>> much more easily than dynamic content.
> >>>
> >>> Possible approaches for passing on JavaScript - Avoid full page
> >>> reloads by using iframes. - Investigate more graph frameworks
> >>> than just D3.js, including gnuplot, R/ggplot2, others.  A major
> >>> reason for using D3.js is its tight integration with user input
> >>> elements. - Make heavy use of CSS 3 features/tricks. - Add
> >>> tooltips to canvas.
> >>>
> >>> Next steps: - Find out why the bubble graph takes so long to
> >>> load.  Try to avoid the parts that take long in future graphs.
> >>> - In addition to the bottom-up approach taken here, also take
> >>> a top-down approach by thinking about contents and only worry
> >>> about possible implementations later.
> >>>
> >>> Next steps after that: - Think about possible web frameworks,
> >>> like Angular.js if we decide to use JavaScript or Django if we
> >>> decide against it. - Look into alternatives to Bootstrap, like
> >>> Bourbon.io.
> >>>
> >>> +
> >>>
> >>> So far, so unclear. Input welcome!
> >>>
> >>>
> >>> ciao, thms ----
> >>> _______________________________________________ metrics-team
> >>> mailing list metrics-team at lists.torproject.org
> >>> https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team
> >>
> >>
> >>
> >>
> >>
> >>
> >>>
> < he not busy being born is busy dying >
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________ metrics-team
> >> mailing list metrics-team at lists.torproject.org
> >> https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team
> >
> >>
> >
> >
> >
> >
> > < he not busy being born is busy dying >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________ metrics-team
> > mailing list metrics-team at lists.torproject.org
> > https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team
> >

< he not busy being born is busy dying >

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20160113/ebe9e179/attachment-0001.sig>

More information about the metrics-team mailing list