[metrics-team] plots

Karsten Loesing karsten at torproject.org
Sun Jun 4 19:35:03 UTC 2017 

Hi John,

I'm very sorry to say this, but it looks like I won't find the time to
support this project, at least not in June and possibly not in July.

The main reason is that I have external commitments ending on June 30,
and after that I'll be distracted for another two or three weeks to
write a report, find new funding, and define goals for Q3/2017.

If you want to create your own visualizations of Tor network data, by
all means, please do that and let us know.  I promise to take a look,
because I'm curious what you come up with, as are others on this list.
But I simply can't promise to give helpful feedback or merge your code.

If you have questions about the data, in particular because
documentations are unclear, please just ask!

And if you're still interested in helping with plots in, say, eight
weeks from now, you might find our guidelines for developing services to
be run on Tor machines helpful:

https://trac.torproject.org/projects/tor/wiki/org/operations/Guidelines

Sorry again!

All the best,
Karsten


On 02.06.17 14:06, John Williams wrote:
> Hi Karsten
> 
> Another thought on the value of Tor shiny server
> 
> Updating metrics is just the start for a shiny server.   The big benefit is
> enabling a Tor data warehouse for privacy intelligence.      I'm willing to
> do the metrics work for the oportunity to implement a privacy warehouse.
> 
> Worth mentioning again the benefit of maintaining current Debian R packages
> --  changes could evolve into a big can of worms with no apparent upside.
> 
> Fyi & Cheers,
> John
> 
> 
> 
> 
> 
> 
> 
> 
> On Thu, Jun 1, 2017 at 5:23 AM, Karsten Loesing <karsten at torproject.org>
> wrote:
> 
>> Hi John, please give me another day or four (after the weekend) to
>> figure out the best way forward.  Karsten
>>
>>
>> On 29.05.17 18:29, John Williams wrote:
>>> followup thought .... I know that deploying a shiny server has not been
>>> your priority, but, replacing the 26 plots with shiny apps will sidestep
>>> the Debian-R limitation issue , as shiny server runs its own instance of
>> R
>>> and won't affect legacy code running on the Debian R instance  ... a far
>>> better solution, IMO, than updating R code with outdated R functions ....
>>>
>>> On Mon, May 29, 2017 at 9:01 AM, Karsten Loesing <karsten at torproject.org
>>>
>>> wrote:
>>>
>>>> Hi John,
>>>>
>>>> On 29.05.17 13:35, John Williams wrote:
>>>>> Hi Karsten
>>>>>
>>>>> The modern R tool chain  has the tidyverse
>>>>> <https://blog.rstudio.org/2016/09/15/tidyverse-1-0-0/> as it core.
>> The
>>>>> tidyverse library is a convenience library for installing and loading
>> the
>>>>> core set of R packages that comprise the tidyverse - ggplot2, dplyr,
>>>> tidyr,
>>>>> readr, purrr & tibble. A dozen other packages are installed but not
>>>>> explicitly loaded by tidyverse.
>>>>>
>>>>> A high rate of R package innovation currently exists within & beyond
>>>>> the tidyverse (e.g., the shiny tool chain) and it would be difficult to
>>>>> exclude CRAN <https://cran.r-project.org/>-compliant packages from
>> use.
>>>>>
>>>>> I have high confidence in the people & motivations
>>>>> <https://rviews.rstudio.com/2016/10/12/interview-with-j-j-allaire/>
>>>> behind
>>>>> the tidyverse & shiny tool chains.
>>>>>
>>>>> Speaking of shiny, a shiny server
>>>>> <https://www.rstudio.com/products/shiny/shiny-server> installed in the
>>>> Tor
>>>>> metrics subnet will enable rapid innovation in Tor metrics. Instead of
>>>>> creating plot code to called by javascript, we'll create interactive
>>>> shiny
>>>>> apps linked by iframe,  and/or standalone, data-centric web apps, just
>>>>> using R.
>>>>
>>>> Can you elaborate on that?  How would we use Shiny Server without
>>>> JavaScript?  (I'm not asking, because I think it's impossible.  I'm
>>>> asking because I haven't looked at all the details there and am
>> curious.)
>>>>
>>>>> My library of Tor analytics is small but growing:
>>>>>
>>>>> Exit Nodes <http://rpubs.com/johnbwilliams/nodes>
>>>>> Network Size <http://rpubs.com/johnbwilliams/refactor>
>>>>> Network Size Delta <http://rpubs.com/johnbwilliams/network_size_delta>
>>>>> hidserv-descs-per-hsdir
>>>>> <http://rpubs.com/johnbwilliams/hidserv-descs-per-hsdir>
>>>>>
>>>>> This is all well and good but pales in comparison to what could be done
>>>> in
>>>>> Tor Metrics with an analytics (shiny) server.
>>>>>
>>>>> I see that a local cloud service provider, with whom I have done
>> business
>>>>> from time to time, is hosting a Tor exit address.  I was thinking that
>>>> the
>>>>> Project could make a business case to this provider's management for
>>>>> hosting a shiny server for Tor metrics.
>>>>>
>>>>> Yes, it is possible but not desirable to revert to base R functions
>> from
>>>>> readr - a core package of the tidyverse.  I'll do that as requested.
>>>>
>>>> I can understand that going back to base functions is a bit painful.  My
>>>> main goal here is to start making improvements as soon as possible.  I
>>>> think it would take some time to get everyone involved in dependencies
>>>> on Tor hosts to make a decision here, and I'd rather want to postpone
>>>> that and start coding and reviewing.
>>>>
>>>> Two ideas:
>>>>
>>>>  - We include our own version of a read_csv that internally uses R base
>>>> functions to do the job and that we can later replace by what's in
>> readr.
>>>>
>>>>  - We try to get readr into Debian backports.  A Debian developer told
>>>> me that it's not that difficult to do that, and he might help with the
>>>> process.  (We could even go one step further and try to get tidyverse
>>>> into Debian backports, but that might be a bigger project.)
>>>>
>>>> I think my preference would be the first option, because that's likely
>>>> the quickest way to get the code updated.  What do you think?
>>>>
>>>>> Cheers,
>>>>>
>>>>> John
>>>>
>>>> All the best,
>>>> Karsten
>>>>
>>>>
>>>>>
>>>>> On Mon, May 29, 2017 at 3:12 AM, Karsten Loesing <
>> karsten at torproject.org
>>>>>
>>>>> wrote:
>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>> On 26.05.17 05:02, John Williams wrote:
>>>>>>> something like this?
>>>>>>>
>>>>>>> refactor of network size plot <http://rpubs.com/
>> johnbwilliams/refactor
>>>>>
>>>>>>
>>>>>> Looks great!  A lot shorter and clearer than before.
>>>>>>
>>>>>> By the way, did you change that code after posting here?  I remember
>>>>>> seeing `library(tidyverse)` in the original version, which I couldn't
>>>>>> find in Debian stable or backports.
>>>>>>
>>>>>> The current code doesn't contain `library(tidyverse)` anymore, but it
>>>>>> has `library(readr)` which I couldn't find in Debian stable or
>> backports
>>>>>> either.
>>>>>>
>>>>>> The other libraries are all available in Debian stable or backports as
>>>>>> far as I can see.
>>>>>>
>>>>>> We're trying to depend only on Debian packages for anything running on
>>>>>> the server.  This is not an absolute requirement, and we might not
>> keep
>>>>>> it up anyway if we ever switch to a Shiny server.  But for the moment
>> it
>>>>>> would be great if we could keep this requirement.  Or at least we
>>>>>> shouldn't give up on it too easily.
>>>>>>
>>>>>> Do you think you can change the code to avoid `library(readr)`?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> All the best,
>>>>>> Karsten
>>>>>>
>>>>>>
>>>>>>> On Wed, May 24, 2017 at 7:57 PM, David Fifield <
>> david at bamsoftware.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Wed, May 24, 2017 at 07:28:39PM -0400, John Williams wrote:
>>>>>>>>> Thanks, Karsten, but the direct links to CSV files are not working
>> -
>>>>>> get
>>>>>>>>> Oops! Something went wrong here! We encountered a 404 Not Found
>> when
>>>>>>>> processing
>>>>>>>>> your request!
>>>>>>>>
>>>>>>>> The URLs were missing a "/stats".
>>>>>>>>
>>>>>>>> https://metrics.torproject.org/stats/servers.csv
>>>>>>>> https://metrics.torproject.org/stats/bandwidth.csv
>>>>>>>> https://metrics.torproject.org/stats/torperf-1.1.csv
>>>>>>>> https://metrics.torproject.org/stats/connbidirect2.csv
>>>>>>>> https://metrics.torproject.org/stats/advbwdist.csv
>>>>>>>> https://metrics.torproject.org/stats/hidserv.csv
>>>>>>>> https://metrics.torproject.org/stats/webstats.csv
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20170604/d51de674/attachment.sig>

More information about the metrics-team mailing list