[metrics-team] plots

Mon May 29 13:01:42 UTC 2017

Hi John,

On 29.05.17 13:35, John Williams wrote:
> Hi Karsten
> 
> The modern R tool chain  has the tidyverse
> <https://blog.rstudio.org/2016/09/15/tidyverse-1-0-0/> as it core.  The
> tidyverse library is a convenience library for installing and loading the
> core set of R packages that comprise the tidyverse - ggplot2, dplyr, tidyr,
> readr, purrr & tibble. A dozen other packages are installed but not
> explicitly loaded by tidyverse.
> 
> A high rate of R package innovation currently exists within & beyond
> the tidyverse (e.g., the shiny tool chain) and it would be difficult to
> exclude CRAN <https://cran.r-project.org/>-compliant packages from use.
> 
> I have high confidence in the people & motivations
> <https://rviews.rstudio.com/2016/10/12/interview-with-j-j-allaire/> behind
> the tidyverse & shiny tool chains.
> 
> Speaking of shiny, a shiny server
> <https://www.rstudio.com/products/shiny/shiny-server> installed in the Tor
> metrics subnet will enable rapid innovation in Tor metrics. Instead of
> creating plot code to called by javascript, we'll create interactive shiny
> apps linked by iframe,  and/or standalone, data-centric web apps, just
> using R.

Can you elaborate on that?  How would we use Shiny Server without
JavaScript?  (I'm not asking, because I think it's impossible.  I'm
asking because I haven't looked at all the details there and am curious.)

> My library of Tor analytics is small but growing:
> 
> Exit Nodes <http://rpubs.com/johnbwilliams/nodes>
> Network Size <http://rpubs.com/johnbwilliams/refactor>
> Network Size Delta <http://rpubs.com/johnbwilliams/network_size_delta>
> hidserv-descs-per-hsdir
> <http://rpubs.com/johnbwilliams/hidserv-descs-per-hsdir>
> 
> This is all well and good but pales in comparison to what could be done in
> Tor Metrics with an analytics (shiny) server.
> 
> I see that a local cloud service provider, with whom I have done business
> from time to time, is hosting a Tor exit address.  I was thinking that the
> Project could make a business case to this provider's management for
> hosting a shiny server for Tor metrics.
> 
> Yes, it is possible but not desirable to revert to base R functions from
> readr - a core package of the tidyverse.  I'll do that as requested.

I can understand that going back to base functions is a bit painful.  My
main goal here is to start making improvements as soon as possible.  I
think it would take some time to get everyone involved in dependencies
on Tor hosts to make a decision here, and I'd rather want to postpone
that and start coding and reviewing.

Two ideas:

 - We include our own version of a read_csv that internally uses R base
functions to do the job and that we can later replace by what's in readr.

 - We try to get readr into Debian backports.  A Debian developer told
me that it's not that difficult to do that, and he might help with the
process.  (We could even go one step further and try to get tidyverse
into Debian backports, but that might be a bigger project.)

I think my preference would be the first option, because that's likely
the quickest way to get the code updated.  What do you think?

> Cheers,
> 
> John

All the best,
Karsten

> 
> On Mon, May 29, 2017 at 3:12 AM, Karsten Loesing <karsten at torproject.org>
> wrote:
> 
>> Hi John,
>>
>> On 26.05.17 05:02, John Williams wrote:
>>> something like this?
>>>
>>> refactor of network size plot <http://rpubs.com/johnbwilliams/refactor>
>>
>> Looks great!  A lot shorter and clearer than before.
>>
>> By the way, did you change that code after posting here?  I remember
>> seeing `library(tidyverse)` in the original version, which I couldn't
>> find in Debian stable or backports.
>>
>> The current code doesn't contain `library(tidyverse)` anymore, but it
>> has `library(readr)` which I couldn't find in Debian stable or backports
>> either.
>>
>> The other libraries are all available in Debian stable or backports as
>> far as I can see.
>>
>> We're trying to depend only on Debian packages for anything running on
>> the server.  This is not an absolute requirement, and we might not keep
>> it up anyway if we ever switch to a Shiny server.  But for the moment it
>> would be great if we could keep this requirement.  Or at least we
>> shouldn't give up on it too easily.
>>
>> Do you think you can change the code to avoid `library(readr)`?
>>
>> Thanks!
>>
>> All the best,
>> Karsten
>>
>>
>>> On Wed, May 24, 2017 at 7:57 PM, David Fifield <david at bamsoftware.com>
>>> wrote:
>>>
>>>> On Wed, May 24, 2017 at 07:28:39PM -0400, John Williams wrote:
>>>>> Thanks, Karsten, but the direct links to CSV files are not working -
>> get
>>>>> Oops! Something went wrong here! We encountered a 404 Not Found when
>>>> processing
>>>>> your request!
>>>>
>>>> The URLs were missing a "/stats".
>>>>
>>>> https://metrics.torproject.org/stats/servers.csv
>>>> https://metrics.torproject.org/stats/bandwidth.csv
>>>> https://metrics.torproject.org/stats/torperf-1.1.csv
>>>> https://metrics.torproject.org/stats/connbidirect2.csv
>>>> https://metrics.torproject.org/stats/advbwdist.csv
>>>> https://metrics.torproject.org/stats/hidserv.csv
>>>> https://metrics.torproject.org/stats/webstats.csv
>>>>
>>>
>>
>>
>>
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20170529/0d79b478/attachment.sig>