[metrics-bugs] #28799 [Metrics/Website]: Use readr's read_csv() to speed up drawing graphs (was: Use R.cache to speed up drawing graphs)

[Tor Bug Tracker & Wiki]

#28799: Use readr's read_csv() to speed up drawing graphs
-----------------------------+--------------------------
 Reporter:  karsten          |          Owner:  karsten
     Type:  enhancement      |         Status:  accepted
 Priority:  Low              |      Milestone:
Component:  Metrics/Website  |        Version:
 Severity:  Normal           |     Resolution:
 Keywords:                   |  Actual Points:
Parent ID:                   |         Points:
 Reviewer:                   |        Sponsor:
-----------------------------+--------------------------
Changes (by karsten):

 * status:  merge_ready => accepted
 * priority:  Medium => Low


Old description:

> Let's use R.cache to speed up drawing graphs. I already prepared a patch
> that I'm going to post here as soon as I have a ticket number. From the
> commit message:
>
> Over two years ago, in commit 1f90b72 from October 2016, we made our user
> graphs faster by avoiding to read the large .csv file on demand.  Instead
> we read it once as part of the daily update, saved it to disk as .RData
> file using R's save() function, and loaded it back to memory using R's
> load() function when drawing a graph.
>
> This approach worked okay. It just had two disadvantages:
>
>  1. We had to write a small amount of R code for each graph type, which
> is why we only did it for graphs with large .csv files.
>  2. Running these small R script as part of the daily update made it
> harder to move away from Ant towards a Java-only execution model.
>
> The new approach implemented in this commit uses R.cache, which caches
> data for use by concurrent Rserve clients. The first time we read a .csv
> file we save it to the cache, and all subsequent times we just load it
> back from the cache. We're using the file name and last modified time as
> key in the cache to avoid using stale data. We're also clearing the cache
> on startup to avoid running out of disk space.
>
> One somewhat unwanted side effect is that drawing the first graph from a
> new .csv file may take a few more seconds as compared to drawing
> subsequent graphs. This seems acceptable, though.
>
> Requires installing the R.cache package from CRAN, which is available on
> Debian as r-cran-r.cache.

New description:

 Let's use R.cache to speed up drawing graphs. I already prepared a patch
 that I'm going to post here as soon as I have a ticket number. From the
 commit message:

 Over two years ago, in commit 1f90b72 from October 2016, we made our user
 graphs faster by avoiding to read the large .csv file on demand.  Instead
 we read it once as part of the daily update, saved it to disk as .RData
 file using R's save() function, and loaded it back to memory using R's
 load() function when drawing a graph.

 This approach worked okay. It just had two disadvantages:

  1. We had to write a small amount of R code for each graph type, which is
 why we only did it for graphs with large .csv files.
  2. Running these small R script as part of the daily update made it
 harder to move away from Ant towards a Java-only execution model.

 The new approach implemented in this commit uses R.cache, which caches
 data for use by concurrent Rserve clients. The first time we read a .csv
 file we save it to the cache, and all subsequent times we just load it
 back from the cache. We're using the file name and last modified time as
 key in the cache to avoid using stale data. We're also clearing the cache
 on startup to avoid running out of disk space.

 One somewhat unwanted side effect is that drawing the first graph from a
 new .csv file may take a few more seconds as compared to drawing
 subsequent graphs. This seems acceptable, though.

 Requires installing the R.cache package from CRAN, which is available on
 Debian as r-cran-r.cache.

 '''Edit: Turns out that we don't want R.cache but readr's read_csv()
 instead. See comments below.'''

--

Comment:

 Thanks for looking! Merged with a small
 [https://gitweb.torproject.org/user/karsten/metrics-
 web.git/commit/?h=task-28799-2&id=fecafc07b99798946308bbb3615c15bb0ce6a30f
 tweak], and deployed.

 Setting back to accepted for the remaining graphs after we gathered some
 more experience with this new approach. That could easily happen in 2019.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/28799#comment:8>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online