[metrics-team] JavaScript on metrics.torproject.org

Karsten Loesing karsten at torproject.org
Fri Jan 15 13:25:59 UTC 2016


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 13/01/16 01:00, tl wrote:
> Hi Karsten,

Hi thms,

> tl,;dr

Indeed, this was a really long posting, and I understand that it's due
to the fact that this is a very broad discussion.  But we should both
try to keep our postings as short as possible, both to prevent this
discussion from stalling and to make it easier for others to follow
along and chime in.

> I think we have a misunderstanding here. I’m speaking about two
> ways of using JavaScript: server-side and client side. Client side
> I’m totally in favor of a site that works without JavaScript as far
> as possible - and that is quite far - but adds functionality for
> those that have JavaScript enabled and that is not possible to
> convey without client side Javascript. Server side there are
> reasons why I think it could be beneficial to use a JavaScript
> framework, but I have no idea about the security implications
> 
> 
> Longer:
> 
> I think that almost all of what the Metrics site or other sites in
> the Tot network need can be done without client side JavaScript.
> These are static pages, serving text and a few bitmaps and there’s
> absolutely no reason why they should demand the use of JavaScript.
> Sites like Atlas and Globe could be done perfectly fine without
> client side JavaScript and if I understood correctly they where
> done with one of the heaviest JavaScript frameworks around just
> because it was the fastest way to do it - which often translates to
> "that was the tool that we knew already".

Agreed.  And yes, we didn't have a discussion like this for Atlas or
Globe.  Maybe they would have written differently if we had.

> But client side JavaScript comes into play when you want
> interactive visualizations. There are 2 graph visualizations on the
> bottom of a random Globe page I visited [0]. These graphs can be
> generated on the server and delivered to the client as SVG vector
> graphic or as PNG bitmap graphic. But there is also a slider under
> these graphs with which you can control the period to be shown.
> This is client side JavaScript. It allows you to intuitively select
> a period and to update the graph accordingly without a whole page
> reload - thus allowing to play with the data, making the experience
> of drilling for details fluid and seemingly effortless blabla.
> Ahem, well, sort of, but you get the idea. Without client side
> JavaScript you have to resort to form-based interfaces. These don’t
> need to look bad either - CSS  made the 1990 form styles go away
> for good - and you can mitigate the page-reload problem by putting
> the graphics in an iframe. So it’s not all black or white but you
> can easily imagine that these tricks will only get you so far while
> some features are just not possible achieve without client side
> JavaScript. For example you won’t get animations without JavaScript
> (there are special cases where CSS animations can save your ass,
> but not in general and not with reasonable effort as far as I
> know). Animations and fluid interactions are not just gimmicky
> nice-to-haves. Animations can be very important to understand
> transitions and thereby correlations, good interaction is key to
> successfully investigating the data. Look for example at the dc.js
> standard exampel [1] and try a) not to be impressed and b) to do
> that without client side JavaScript. No way!
> 
> So: there is stuff that is as cool to have as impossible to do
> without client side JavaScript. We can always say "we’ll try" - and
> we should! - but we will ony get that far. And like Letty I want to
> able to do the cool stuff too. Most of the work is in aggregating
> the data and setting the whole thing up. That people with high
> security settings won’t see it shouldn’t mean that nobody else can
> profit from it.
> 
> It’s easy to do meaningful rudimentary non-scripted static versions
> of JavaScript heavy visualizations. It’s not so easy to do them
> well and as good as posssible. There certainly is the danger of
> moral hazard: putting much love into the shiny scripted
> "experience" and neglecting the tedious, somehow bolted on
> form-based version. I hope that the isomorphic JavaScript
> frameworks that I wrote about in my last post help here, becauese
> they allow to render server side what normally the client would
> render. But that really has to be tested to be sure and I fear that
> a quick Proof of Concept will not reveal all problems and
> possibilities.
> 
> Realisitically I think we should define which visualizations we
> want, like: what is the basic stuff that really needs to be there -
> and do that without client side JavaScript as good as possible.
> There will inevitably always be ways to make it even better with
> client side JavaScript, and we should try to implement some of
> those too, as examples and tool boxes for how to build client side
> JavaScript that goes the extra mile and stays accessible to
> non-Javascript enabled browsers. Hopefully the isomorphic web
> frameworks will enable us to do that without too much extra work.
> Apart from that we can’t do much more than appeal to people to take
> the need for non-scripted web pages serious and use the tools and
> examples that we are able to provide.

I think the first question we need to answer is how interactive we
*need* our visualizations to be.  I agree that Globe's slider for the
time period shown is useful, and I can see how animations can help
understanding correlations better.  But we should be able to come up
with a few concrete examples where less interactive visualizations are
simply insufficient to educate the public about certain aspects the
Tor network.

That's why I made a "site map" of Metrics that shows all graphs that
are either currently implemented or that are planned to be added at
some point.

https://people.torproject.org/~karsten/volatile/metrics-sitemap.pdf

Can you maybe pick two or three of these visualizations, or suggest
related ones, where you think that interactivity based on client-side
JavaScript would make the visualization significantly better?

> Now server side JavaScript: given the above somehow passionate plea
> for _some_ client side JavaScript I sort of took it for granted
> that porting the backend to JavaScript too would be the sensible
> thing to do, especially if we find that an isomorphic web framework
> like React really delivers on the promise and makes it easier for
> us to serve scripted visualizations that are rendered on the server
> if the client has JavaScript disabled. There are more and more web
> developers that can code JavaScript and feel at home in a
> Node.js-based environment but for whom a Java- or Python-based web
> framework would be mostly intimidating (you can count me in). So:
> why not?! Maybe not because of security concerns, and as I said
> already: I have no idea what is secure in that realm and what
> isn’t. Node.js is in Debian stable, but what does that mean? Is
> therefor every package that runs inside Node.js safe too? Or is it
> not?
> 
> One important thing to consider is that client side JavaScript
> can’t harm anybody who hasn’t enabled JavaScript in the first
> place. Sure, if we provide cool interactive JS-driven
> visualizations we provide incentive to enable it
> 
> You might have problems with moral hazard, but might also have
> problems with no hazard at all because not many people want to work
> under a strict non-JS regime.

I think many of these concerns are addressed on this wiki page:

https://trac.torproject.org/projects/tor/wiki/org/operations/Guidelines

Regarding packages running inside of Node.js, specifically those that
are installed using npm, those would fall under "Packages installed
via third-party package managers (really bad)".  And as you can see,
Metrics is guilty of doing that by requiring some R packages, even
though I think we could as well switch to Debian packages for those.
But given that we might away from them, I'll leave everything as it is.

>> On 12.01.2016, at 13:00, Karsten Loesing <karsten at torproject.org>
>> wrote:
> 
> [...]
> 
>> What do you think about building a small prototype of a
>> visualization that requires client-side JavaScript and another
>> one that doesn't? Most people are not web developers, so they can
>> hardly follow the discussion of web techniques here.  Giving them
>> an actual example might make it easier for them to understand the
>> differences.  I'd be willing to help with building these
>> prototypes.
> 
> Take Lettys visualization and take away the animation and the time
> slider. Take away the time sliders from [0]. Never have something
> like [1]. I could pick some more examples from the D3 gallery [2].
> Do we really need more proof?

I could imagine taking away the animation and time slider from Letty's
visualization and still having a very useful visualization.

And I could imagine taking away the time slider from the Globe graph.

What I cannot imagine yet is never having something like that specific
D3.js sample visualization, but just because I don't know how much
we'd be missing in the specific case of visualizing Tor network data.
 Right now, I can live without it.  I can't say yet whether I will be
addicted by it in the future.  This is related to my request above for
specific examples that would benefit vastly from more interactivity.

> I’m getting a bit edgy and probably I'm not the best embassador
> with respect to the usage of JavaScript: I had discussions pro/con
> JavaScript in leftist circles since 20 years, I had discussions
> pro/con fax machines before and in between I had discussions
> pro/con CSS, because some die hard anarchists considered it street
> to stick with their rotten Netscape Communicator 4.7.6. I find
> these discussions most often very arbitrary: mail is okay, but fax
> is not. Fax is okay, but computers are not. Computers are okay, but
> not databases. The web is okay, but not JavaScript, and so on. Why
> not just say: it depends. Can it harm somebody, can it profit
> somebody, can somebody profit without someone else being harmed?

While I see your point, I want to note that this situation is a bit
different.  Tor is both shipping the Tor Browser and advocating that
users stay safe on the Net which may include turning on the
high-security mode.  At the same time Tor is providing content on Tor
Metrics that requires users to lower their security setting, at least
for the really cool stuff.  You see the contradiction there.  That's
why I don't want to make this whole decision lightly.

> I totally respect security concerns. I absolutely understand
> concerns about moral hazard like in the example above and I think
> we should have that in mind and try hard to mitigate it. I find it
> a good and satisfying challenge to achieve as much as possible
> without JavaScript. But I also think that JavaScript is a cool
> technology, that being able to script visualizations provides
> really useful and otherwise unachievable benefits and that we
> should not give those away because some people can’t profit from
> them. That sort of standardization on the lowest level only causes
> frustration.

I hear you with regard to developer happiness and wanting to use cool
technology.  That's certainly a factor here.

But please also hear me with regard to user happiness (which includes
Tor Browser users on high security) and system administrators (who
really hate it when we use non-standard package managers that put
their systems at risk).

>> I have more thoughts on the web frameworks and on requirements
>> like avoiding packages not contained in Debian stable, but I'll
>> hold them back until we have that discussion.
> 
> I’d love to hear about those thoughts right away!

I think I included all those arguments above.

Moving forward, I suggest we (you, I, others) write two prototypes to
try out two different directions:

 - MEAN (or just the EN part of it) with React and D3 to draw a graph
either on the server or on the client, depending on whether the client
has JavaScript or not.

 - Node with D3 to draw either the bubble graph or Letty's
visualization, and iframes and CSS magic, to see how far we could get
when abstaining from client-side JavaScript.

My sense is that we'll make more progress on this discussion by trying
out some concrete approaches than by reinforcing our arguments given
before.

It's fine if you don't want to hack on these prototypes.  Just let me
know, and I'll pick one and hack away.  But even if you don't want to
write code, now's still a good time to suggest different prototypes if
you hope that the result will help us better in deciding whether or
not to support client-side JavaScript on Tor Metrics.

> Ciao thms

All the best,
Karsten



> [0]
> https://globe.torproject.org/#/relay/7C16F60CD2AEB2208ADF5B69187DDF8A61C85EAE
>
> 
[1] https://dc-js.github.io/dc.js/
> [2] https://github.com/mbostock/d3/wiki/Gallery [3]
> http://arstechnica.com/security/2016/01/going-forward-the-tor-project-wants-to-be-less-reliant-on-us-govt-funding/
>
> 
> 
> 
> 
>> All the best, Karsten
>> 
>> 
>> On 08/01/16 15:28, tl wrote:
>>> Two additions:
>>> 
>>> - just found a list of popular isomprphic JS web frameworks: 
>>> http://isomorphic.net/libraries but nothing about which one 
>>> interacts the best with D3 - a decision about a simple
>>> templating language on top of express would have to be made
>>> too. Handlebars is popular, but I’m not an expert.
>>> 
>>> And, while thinking about it, I’m increasingly inclined to
>>> promote a solution where all pages that don’t require strong
>>> interactivity are realized with a simple templating engine
>>> (like Handlebars). For the few visualizations that benefit
>>> heavily from client side JS, the templating engine would have
>>> to be replaced by an isomorphic webframework like React. That
>>> way, not all pages would be build the same way but most of them
>>> would be quite simple to build and maintain.
>>> 
>>> 
>>> Still waiting for comments and ideas before moving this topic
>>> to tor-dev.
>>> 
>>> 
>>> ciao thms
>>> 
>>> 
>>>> On 07.01.2016, at 14:56, tl <tl at rat.io> wrote:
>>>> 
>>>> Okay, answering my own post ;-)
>>>> 
>>>> tl;dr: no Javascript for most of the site JavaScript as 
>>>> enhancements to pages where sensible server side based on
>>>> Node, Express, React, MongoDB visualizations with D3
>>>> 
>>>> 
>>>> I favor a solution where we have a site working fine without 
>>>> JavaScript and in some places even finer with JavaScript.
>>>> With "some places" I mean additional interface elements to
>>>> control (and play with) visualizations and maybe some
>>>> additional site navigation too. There’s no question that
>>>> omitting JavaScript is both praticable and a good thing to
>>>> do, for reasons of security as welll as performance. But it
>>>> is also undeniable that JavaScript can facilitate some
>>>> awesome user interfaces especially with data visualizations
>>>> that are not possible otherwise. JavaScript usage has been a
>>>> contested topic right from the start in the 1990ies. My
>>>> stance is: banning JS from ones personal browser is
>>>> absolutely fine and reasonable and forcing JS on all users is
>>>> unacceptable. Abolishing JS  however is a shame since it can
>>>> be tremendously useful in building user interfaces, and 
>>>> interfaces are the connection between the user and the data 
>>>> (read: not negligible). Building interfaces that work fine 
>>>> without JavaScript but offer easier and more fluid
>>>> interaction with JavaScript is harder than doing it only one
>>>> way (either with or without JS) but it is possible without
>>>> nasty hacks or excessive bloat. I think the extra effort is
>>>> worth it and this is the way to go. Since IMO this settles
>>>> the question if we should/could use JavaScript on the site it
>>>> follows that we should also port the current metrics site
>>>> backend to JavaScript/Node and base its visualizations on
>>>> D3.js. That way the whole thing will be based on only one
>>>> technology - but of course will still require mastering the
>>>> building blocks. But now, enough with the fundamentalist
>>>> foreword and down into the drudgery...
>>>> 
>>>> 
>>>> It looks like we don’t need a full blown CMS since only two
>>>> or three rather technical people will maintain the website.
>>>> Therefor we can go with one of the established web frameworks
>>>> and build the little admin interfaces we might need
>>>> ourselves. A popular JS web framework stack is known by the
>>>> name MEAN which mimics the LAMP acronym and stands for
>>>> MongoDB, Express, Angular.js and Node: * Node.js is the
>>>> JavaScript runtime in which all server functionality is
>>>> hosted/embedded and is available as Debian stable package *
>>>> MongoDB is a very popular database, based on JSON, available
>>>> as Debian stable package as well. I worked a lot with it when
>>>> aggregating data for the 'visionion' visualization project
>>>> and although I’m not partucularily fond of it it makes 
>>>> working with slightly unregular document types rather easy.
>>>> It provides mapReduce functionality to eventually aggregate a
>>>> little data by the side and its JSON based data schema makes
>>>> it easy to access from a web app. But I wouldn’t mind to work
>>>> with PostgreSQL either. * Express is the de-facto standard
>>>> for web servers in the Node ecosystem and a little more than
>>>> that. It can be extended with some little templating engine
>>>> to build sizeable static websites which would be all we need
>>>> if we go the non-JS way on the frontend. * Angular.JS is a
>>>> full blown web application framework (developed and backed by
>>>> Google) - providing the view to Express instead of some small
>>>> templating engine - that is rather popular (also Arturo
>>>> recommends it ) and provides all you could ever need in one
>>>> (big) box. The big downside is that it needs JavaScript on
>>>> the client side.
>>>> 
>>>> That’s the MEAN stack. Node and Express are de facto
>>>> standards, MongoDB may be just fine. The interesting question
>>>> is what to use for the view layer on top of Express: a simple
>>>> templating engine doesn’t provide the JavaScript interface we
>>>> desire to implement elaborate visualization interfaces while
>>>> Angular.JS doesn’t provide the non-JS baseline considered
>>>> mandatory. There exist of course a plethora of JS web app
>>>> frameworks - too many to analyse them here. A few of them
>>>> provide the new hotness "isomorphic rendering" which means
>>>> that the site can be rendered on the server as well as in the
>>>> browser. The original aim is to speed up initial pageload and
>>>> optimize for SEO, but I hope it can be used to implement a
>>>> non-JS-and-JS-too-site. If the browser does not support
>>>> JavaScript it renders a page that was pregenerated on the 
>>>> server. For most of the metrics site there will be no
>>>> difference at all since they don’t need any JavaScript. For
>>>> visualizations though it can make a big difference.
>>>> Visualization controls will be just (nicely styled) forms and
>>>> interactions will require a roundtrip to the server.
>>>> Visualizations will only be able to show limited amounts of
>>>> data because they can’t fetch data in the background. Some of
>>>> the worst problems with this approach can be mitigated by
>>>> loading the visualizations themselves into iFrames, reducing
>>>> flicker and loading times. If the browser is willing to 
>>>> execute JavaScript though additional data is fetched in the 
>>>> background, zooms and sliders and tooltips appear,
>>>> animations make it easier to understand effects etc. It will
>>>> be the same information, but it will be accessible much
>>>> easier and make more sense. It remains to be seen how much
>>>> code duplication that involves. Some of the interface
>>>> elements would have to build as HTML form elements as well as
>>>> D3 elements. There duplication is unavoidable. Those
>>>> interface improvements that rely on fetching data in the
>>>> background should be implementable rather seamlessly.
>>>> 
>>>> All this is not exactly trivial. There will be obstacles but 
>>>> after some googling I’m quite convinced that there are
>>>> sensible solutions too. React [0] (developed and backed by
>>>> Facebook) is by far the most prominent of the frameworks to
>>>> support isomorphic rendering [1]. React is a fine piece of
>>>> technology on its own, rather popular (not as much as
>>>> AngularJS, but still) and well supported too. In contrast to
>>>> the all-encompassing AngularJS it only aims to be a view
>>>> layer and integrates nicely with Express [2]. There is a
>>>> problem though - React tries to control the DOM just like D3,
>>>> the visualization engine - but there exist solutions to that
>>>> [3]. Another, maybe less involved solution for isomorphic
>>>> rendering is called Rendr [4]. It’s less well known and less
>>>> supported though, so my first shot would be React. If none of
>>>> these work out as hoped there’s always the possibilty to use
>>>> a simple templating engine with Express and add 
>>>> JavaScript-based interactivity through jQuery to some
>>>> selected visualizations that we think are worth the extra
>>>> effort. That may not be very elegant or scalable, and more
>>>> 2005 than 2015, but as a backup strategy it’s proven and
>>>> works.
>>>> 
>>>> 
>>>> A question that I have is about security: the latest version
>>>> of Node in Debian stable is 0.10.29 from July 2014. That’s
>>>> not bad. I wonder though if it’s save to load just about any
>>>> module available to Node out there - as Node is the
>>>> (hopefully) secure sandbox it runs in - or if we have to get
>>>> Debian-stable npm packages for Express, React etc too? That
>>>> might prove difficult…
>>>> 
>>>> 
>>>> Ciao thms
>>>> 
>>>> 
>>>> [0] 
>>>> http://blog.yld.io/2015/06/10/getting-started-with-react-and-node-js/
>>>>
>>>>
>>
>>>> 
[1]
>> https://strongloop.com/strongblog/node-js-react-isomorphic-javascript-why-it-matters/
>>>>
>> 
[2] https://github.com/reactjs/express-react-views [3]
>>>> http://oli.me.uk/2015/09/09/d3-within-react-the-right-way/
>>>> [4] https://github.com/rendrjs/rendr
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On 06.01.2016, at 22:30, tl <tl at rat.io> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> Karsten, Letty and me recently had a conversation about the
>>>>> use of JavaScript on https://metrics.torproject.org. The
>>>>> problem space in brief is that the metrics site needs nice 
>>>>> visualizations which need nice interfaces which can profit
>>>>> very much from some nice JavaScript whereas there are a lot
>>>>> of nice people who very reasonably insist on deactivating
>>>>> JavaScript in their nice Tor browsers. So: what to do? A
>>>>> side aspect of this discussion is that the metrics website
>>>>> needs a technical overhaul. It’s based on some hard to
>>>>> maintain servlet/JSP and while we’re on it we could as well
>>>>> port it to some JavaScript/Node based framework on the
>>>>> server side if we decide to use more JavaScript anywy.
>>>>> Following are some notes from the discussion.
>>>>> 
>>>>> +
>>>>> 
>>>>> Possible outcomes: 1. Don't require any JavaScript, works 
>>>>> without issues in high security mode. 2. Only require 
>>>>> JavaScript for some visualizations where it makes sense,
>>>>> users will have to switch to medium-high security to watch
>>>>> those. 2b. For visualizations that require JavaScript to be
>>>>> really useful, provide a simplified version that works
>>>>> without client-side JavaScript. 3. Require JavaScript for
>>>>> most visualizations by first looking at making them as
>>>>> usable and as useful as possible, rather than worrying
>>>>> about users having to change their Tor Browser security
>>>>> setting.
>>>>> 
>>>>> Benefits and possibilities of using JavaScript - Ability
>>>>> to load new data from the server, based on user input,
>>>>> rather than pre-loading all the data that the user might
>>>>> want to look at. - Handle user input more directly than via
>>>>> HTML forms which require a server round-trip. - Let the
>>>>> user decide whether they want to lower their security in
>>>>> order to look at a visualization, rather than making that
>>>>> decision for them. - It's sufficient to host static
>>>>> content, which can be mirrored much more easily than
>>>>> dynamic content.
>>>>> 
>>>>> Possible approaches for passing on JavaScript - Avoid full
>>>>> page reloads by using iframes. - Investigate more graph
>>>>> frameworks than just D3.js, including gnuplot, R/ggplot2,
>>>>> others.  A major reason for using D3.js is its tight
>>>>> integration with user input elements. - Make heavy use of
>>>>> CSS 3 features/tricks. - Add tooltips to canvas.
>>>>> 
>>>>> Next steps: - Find out why the bubble graph takes so long
>>>>> to load.  Try to avoid the parts that take long in future
>>>>> graphs. - In addition to the bottom-up approach taken here,
>>>>> also take a top-down approach by thinking about contents
>>>>> and only worry about possible implementations later.
>>>>> 
>>>>> Next steps after that: - Think about possible web
>>>>> frameworks, like Angular.js if we decide to use JavaScript
>>>>> or Django if we decide against it. - Look into alternatives
>>>>> to Bootstrap, like Bourbon.io.
>>>>> 
>>>>> +
>>>>> 
>>>>> So far, so unclear. Input welcome!
>>>>> 
>>>>> 
>>>>> ciao, thms ---- 
>>>>> _______________________________________________
>>>>> metrics-team mailing list
>>>>> metrics-team at lists.torproject.org 
>>>>> https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>
>>>>> 
< he not busy being born is busy dying >
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________ metrics-team 
>>>> mailing list metrics-team at lists.torproject.org 
>>>> https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team
>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>> 
< he not busy being born is busy dying >
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________ metrics-team 
>>> mailing list metrics-team at lists.torproject.org 
>>> https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team
>>>
>>
>
>>> 
> 
> 
> 
> 
> < he not busy being born is busy dying >
> 
> 
> 
> 
> 

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQEcBAEBAgAGBQJWmPNnAAoJEJD5dJfVqbCrlugH/27VeZaGJ3Xu+TWZocqlwIqr
GiBLEUuKPTTmOZOpC875yce4GU4ks8I8YcY+5djHMCis+kwy1lebf90F1batg9dk
c23cbLDxLb9sskqFNfeqeCxD+/WwHmMhNAuJ5Ubgad7MRJt7FY0s2m3YOosrGB+c
UWC3FaL6iQzyAPxR4PR9McFDKR+zwxysOvdhO1QtSL+dcwzSBID26s2AYeN7oZYy
UKN4SURHjdNW15UvhDFd1HWqkQ5XjNJWKT7gqII0iJaYtFSmQt8UCldzoAqWacZm
BKOZ0uLy+QFBD3Tu5kCcd3ktgUO2EKllx71eMOGYnIEgJOBzuYhXsyaxjZ1x72s=
=G8QK
-----END PGP SIGNATURE-----


More information about the metrics-team mailing list