[metrics-team] plots

Thu Jun 1 09:23:31 UTC 2017

Hi John, please give me another day or four (after the weekend) to
figure out the best way forward.  Karsten

On 29.05.17 18:29, John Williams wrote:
> followup thought .... I know that deploying a shiny server has not been
> your priority, but, replacing the 26 plots with shiny apps will sidestep
> the Debian-R limitation issue , as shiny server runs its own instance of R
> and won't affect legacy code running on the Debian R instance  ... a far
> better solution, IMO, than updating R code with outdated R functions ....
> 
> On Mon, May 29, 2017 at 9:01 AM, Karsten Loesing <karsten at torproject.org>
> wrote:
> 
>> Hi John,
>>
>> On 29.05.17 13:35, John Williams wrote:
>>> Hi Karsten
>>>
>>> The modern R tool chain  has the tidyverse
>>> <https://blog.rstudio.org/2016/09/15/tidyverse-1-0-0/> as it core.  The
>>> tidyverse library is a convenience library for installing and loading the
>>> core set of R packages that comprise the tidyverse - ggplot2, dplyr,
>> tidyr,
>>> readr, purrr & tibble. A dozen other packages are installed but not
>>> explicitly loaded by tidyverse.
>>>
>>> A high rate of R package innovation currently exists within & beyond
>>> the tidyverse (e.g., the shiny tool chain) and it would be difficult to
>>> exclude CRAN <https://cran.r-project.org/>-compliant packages from use.
>>>
>>> I have high confidence in the people & motivations
>>> <https://rviews.rstudio.com/2016/10/12/interview-with-j-j-allaire/>
>> behind
>>> the tidyverse & shiny tool chains.
>>>
>>> Speaking of shiny, a shiny server
>>> <https://www.rstudio.com/products/shiny/shiny-server> installed in the
>> Tor
>>> metrics subnet will enable rapid innovation in Tor metrics. Instead of
>>> creating plot code to called by javascript, we'll create interactive
>> shiny
>>> apps linked by iframe,  and/or standalone, data-centric web apps, just
>>> using R.
>>
>> Can you elaborate on that?  How would we use Shiny Server without
>> JavaScript?  (I'm not asking, because I think it's impossible.  I'm
>> asking because I haven't looked at all the details there and am curious.)
>>
>>> My library of Tor analytics is small but growing:
>>>
>>> Exit Nodes <http://rpubs.com/johnbwilliams/nodes>
>>> Network Size <http://rpubs.com/johnbwilliams/refactor>
>>> Network Size Delta <http://rpubs.com/johnbwilliams/network_size_delta>
>>> hidserv-descs-per-hsdir
>>> <http://rpubs.com/johnbwilliams/hidserv-descs-per-hsdir>
>>>
>>> This is all well and good but pales in comparison to what could be done
>> in
>>> Tor Metrics with an analytics (shiny) server.
>>>
>>> I see that a local cloud service provider, with whom I have done business
>>> from time to time, is hosting a Tor exit address.  I was thinking that
>> the
>>> Project could make a business case to this provider's management for
>>> hosting a shiny server for Tor metrics.
>>>
>>> Yes, it is possible but not desirable to revert to base R functions from
>>> readr - a core package of the tidyverse.  I'll do that as requested.
>>
>> I can understand that going back to base functions is a bit painful.  My
>> main goal here is to start making improvements as soon as possible.  I
>> think it would take some time to get everyone involved in dependencies
>> on Tor hosts to make a decision here, and I'd rather want to postpone
>> that and start coding and reviewing.
>>
>> Two ideas:
>>
>>  - We include our own version of a read_csv that internally uses R base
>> functions to do the job and that we can later replace by what's in readr.
>>
>>  - We try to get readr into Debian backports.  A Debian developer told
>> me that it's not that difficult to do that, and he might help with the
>> process.  (We could even go one step further and try to get tidyverse
>> into Debian backports, but that might be a bigger project.)
>>
>> I think my preference would be the first option, because that's likely
>> the quickest way to get the code updated.  What do you think?
>>
>>> Cheers,
>>>
>>> John
>>
>> All the best,
>> Karsten
>>
>>
>>>
>>> On Mon, May 29, 2017 at 3:12 AM, Karsten Loesing <karsten at torproject.org
>>>
>>> wrote:
>>>
>>>> Hi John,
>>>>
>>>> On 26.05.17 05:02, John Williams wrote:
>>>>> something like this?
>>>>>
>>>>> refactor of network size plot <http://rpubs.com/johnbwilliams/refactor
>>>
>>>>
>>>> Looks great!  A lot shorter and clearer than before.
>>>>
>>>> By the way, did you change that code after posting here?  I remember
>>>> seeing `library(tidyverse)` in the original version, which I couldn't
>>>> find in Debian stable or backports.
>>>>
>>>> The current code doesn't contain `library(tidyverse)` anymore, but it
>>>> has `library(readr)` which I couldn't find in Debian stable or backports
>>>> either.
>>>>
>>>> The other libraries are all available in Debian stable or backports as
>>>> far as I can see.
>>>>
>>>> We're trying to depend only on Debian packages for anything running on
>>>> the server.  This is not an absolute requirement, and we might not keep
>>>> it up anyway if we ever switch to a Shiny server.  But for the moment it
>>>> would be great if we could keep this requirement.  Or at least we
>>>> shouldn't give up on it too easily.
>>>>
>>>> Do you think you can change the code to avoid `library(readr)`?
>>>>
>>>> Thanks!
>>>>
>>>> All the best,
>>>> Karsten
>>>>
>>>>
>>>>> On Wed, May 24, 2017 at 7:57 PM, David Fifield <david at bamsoftware.com>
>>>>> wrote:
>>>>>
>>>>>> On Wed, May 24, 2017 at 07:28:39PM -0400, John Williams wrote:
>>>>>>> Thanks, Karsten, but the direct links to CSV files are not working -
>>>> get
>>>>>>> Oops! Something went wrong here! We encountered a 404 Not Found when
>>>>>> processing
>>>>>>> your request!
>>>>>>
>>>>>> The URLs were missing a "/stats".
>>>>>>
>>>>>> https://metrics.torproject.org/stats/servers.csv
>>>>>> https://metrics.torproject.org/stats/bandwidth.csv
>>>>>> https://metrics.torproject.org/stats/torperf-1.1.csv
>>>>>> https://metrics.torproject.org/stats/connbidirect2.csv
>>>>>> https://metrics.torproject.org/stats/advbwdist.csv
>>>>>> https://metrics.torproject.org/stats/hidserv.csv
>>>>>> https://metrics.torproject.org/stats/webstats.csv
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20170601/157d0f6e/attachment.sig>