[metrics-team] plots

John Williams john.b.williams at gmail.com
Mon May 29 15:49:14 UTC 2017


On Mon, May 29, 2017 at 9:01 AM, Karsten Loesing <karsten at torproject.org>
wrote:

> Hi John,
>
> On 29.05.17 13:35, John Williams wrote:
> > Hi Karsten
> >
> > The modern R tool chain  has the tidyverse
> > <https://blog.rstudio.org/2016/09/15/tidyverse-1-0-0/> as it core.  The
> > tidyverse library is a convenience library for installing and loading the
> > core set of R packages that comprise the tidyverse - ggplot2, dplyr,
> tidyr,
> > readr, purrr & tibble. A dozen other packages are installed but not
> > explicitly loaded by tidyverse.
> >
> > A high rate of R package innovation currently exists within & beyond
> > the tidyverse (e.g., the shiny tool chain) and it would be difficult to
> > exclude CRAN <https://cran.r-project.org/>-compliant packages from use.
> >
> > I have high confidence in the people & motivations
> > <https://rviews.rstudio.com/2016/10/12/interview-with-j-j-allaire/>
> behind
> > the tidyverse & shiny tool chains.
> >
> > Speaking of shiny, a shiny server
> > <https://www.rstudio.com/products/shiny/shiny-server> installed in the
> Tor
> > metrics subnet will enable rapid innovation in Tor metrics. Instead of
> > creating plot code to called by javascript, we'll create interactive
> shiny
> > apps linked by iframe,  and/or standalone, data-centric web apps, just
> > using R.
>
> Can you elaborate on that?  How would we use Shiny Server without
> JavaScript?  (I'm not asking, because I think it's impossible.  I'm
> asking because I haven't looked at all the details there and am curious.)
>

Shiny server based on node.js httpd. A shiny app has a can be embeded in
Tor metrics page with html:
   <iframe src="*URL*"></iframe>



> My library of Tor analytics is small but growing:
> >
> > Exit Nodes <http://rpubs.com/johnbwilliams/nodes>
> > Network Size <http://rpubs.com/johnbwilliams/refactor>
> > Network Size Delta <http://rpubs.com/johnbwilliams/network_size_delta>
> > hidserv-descs-per-hsdir
> > <http://rpubs.com/johnbwilliams/hidserv-descs-per-hsdir>
> >
> > This is all well and good but pales in comparison to what could be done
> in
> > Tor Metrics with an analytics (shiny) server.
> >
> > I see that a local cloud service provider, with whom I have done business
> > from time to time, is hosting a Tor exit address.  I was thinking that
> the
> > Project could make a business case to this provider's management for
> > hosting a shiny server for Tor metrics.
> >
> > Yes, it is possible but not desirable to revert to base R functions from
> > readr - a core package of the tidyverse.  I'll do that as requested.
>
> I can understand that going back to base functions is a bit painful.  My
> main goal here is to start making improvements as soon as possible.  I
> think it would take some time to get everyone involved in dependencies
> on Tor hosts to make a decision here, and I'd rather want to postpone
> that and start coding and reviewing.
>

Understood.  R's packrat package solves the package dependency issue


> Two ideas:
>
>  - We include our own version of a read_csv that internally uses R base
> functions to do the job and that we can later replace by what's in readr.
>

read.csv() likely good enough

>
>  - We try to get readr into Debian backports.  A Debian developer told
> me that it's not that difficult to do that, and he might help with the
> process.  (We could even go one step further and try to get tidyverse
> into Debian backports, but that might be a bigger project.)
>
> I think my preference would be the first option, because that's likely
> the quickest way to get the code updated.  What do you think?
>
>
Willing to help get code updated ASAP, but lack of tidyverse & CRAN
packages likely to be limiting.

Would help to have a list of available R packages in Debian backports

First priority is the 26 plots already identified (1st being
plot_networksize()) ?

> Cheers,
> >
> > John
>
> All the best,
> Karsten
>
>
> >
> > On Mon, May 29, 2017 at 3:12 AM, Karsten Loesing <karsten at torproject.org
> >
> > wrote:
> >
> >> Hi John,
> >>
> >> On 26.05.17 05:02, John Williams wrote:
> >>> something like this?
> >>>
> >>> refactor of network size plot <http://rpubs.com/johnbwilliams/refactor
> >
> >>
> >> Looks great!  A lot shorter and clearer than before.
> >>
> >> By the way, did you change that code after posting here?  I remember
> >> seeing `library(tidyverse)` in the original version, which I couldn't
> >> find in Debian stable or backports.
> >>
> >> The current code doesn't contain `library(tidyverse)` anymore, but it
> >> has `library(readr)` which I couldn't find in Debian stable or backports
> >> either.
> >>
> >> The other libraries are all available in Debian stable or backports as
> >> far as I can see.
> >>
> >> We're trying to depend only on Debian packages for anything running on
> >> the server.  This is not an absolute requirement, and we might not keep
> >> it up anyway if we ever switch to a Shiny server.  But for the moment it
> >> would be great if we could keep this requirement.  Or at least we
> >> shouldn't give up on it too easily.
> >>
> >> Do you think you can change the code to avoid `library(readr)`?
> >>
> >> Thanks!
> >>
> >> All the best,
> >> Karsten
> >>
> >>
> >>> On Wed, May 24, 2017 at 7:57 PM, David Fifield <david at bamsoftware.com>
> >>> wrote:
> >>>
> >>>> On Wed, May 24, 2017 at 07:28:39PM -0400, John Williams wrote:
> >>>>> Thanks, Karsten, but the direct links to CSV files are not working -
> >> get
> >>>>> Oops! Something went wrong here! We encountered a 404 Not Found when
> >>>> processing
> >>>>> your request!
> >>>>
> >>>> The URLs were missing a "/stats".
> >>>>
> >>>> https://metrics.torproject.org/stats/servers.csv
> >>>> https://metrics.torproject.org/stats/bandwidth.csv
> >>>> https://metrics.torproject.org/stats/torperf-1.1.csv
> >>>> https://metrics.torproject.org/stats/connbidirect2.csv
> >>>> https://metrics.torproject.org/stats/advbwdist.csv
> >>>> https://metrics.torproject.org/stats/hidserv.csv
> >>>> https://metrics.torproject.org/stats/webstats.csv
> >>>>
> >>>
> >>
> >>
> >>
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20170529/594f2585/attachment-0001.html>


More information about the metrics-team mailing list