[metrics-team] plots

Sun Jun 4 19:37:47 UTC 2017

No worries, Karsten,  8 weeks is right round the corner -  lets reconveve
then

Best
John

On Sun, Jun 4, 2017 at 3:35 PM, Karsten Loesing <karsten at torproject.org>
wrote:

> Hi John,
>
> I'm very sorry to say this, but it looks like I won't find the time to
> support this project, at least not in June and possibly not in July.
>
> The main reason is that I have external commitments ending on June 30,
> and after that I'll be distracted for another two or three weeks to
> write a report, find new funding, and define goals for Q3/2017.
>
> If you want to create your own visualizations of Tor network data, by
> all means, please do that and let us know.  I promise to take a look,
> because I'm curious what you come up with, as are others on this list.
> But I simply can't promise to give helpful feedback or merge your code.
>
> If you have questions about the data, in particular because
> documentations are unclear, please just ask!
>
> And if you're still interested in helping with plots in, say, eight
> weeks from now, you might find our guidelines for developing services to
> be run on Tor machines helpful:
>
> https://trac.torproject.org/projects/tor/wiki/org/operations/Guidelines
>
> Sorry again!
>
> All the best,
> Karsten
>
>
> On 02.06.17 14:06, John Williams wrote:
> > Hi Karsten
> >
> > Another thought on the value of Tor shiny server
> >
> > Updating metrics is just the start for a shiny server.   The big benefit
> is
> > enabling a Tor data warehouse for privacy intelligence.      I'm willing
> to
> > do the metrics work for the oportunity to implement a privacy warehouse.
> >
> > Worth mentioning again the benefit of maintaining current Debian R
> packages
> > --  changes could evolve into a big can of worms with no apparent upside.
> >
> > Fyi & Cheers,
> > John
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Jun 1, 2017 at 5:23 AM, Karsten Loesing <karsten at torproject.org>
> > wrote:
> >
> >> Hi John, please give me another day or four (after the weekend) to
> >> figure out the best way forward.  Karsten
> >>
> >>
> >> On 29.05.17 18:29, John Williams wrote:
> >>> followup thought .... I know that deploying a shiny server has not been
> >>> your priority, but, replacing the 26 plots with shiny apps will
> sidestep
> >>> the Debian-R limitation issue , as shiny server runs its own instance
> of
> >> R
> >>> and won't affect legacy code running on the Debian R instance  ... a
> far
> >>> better solution, IMO, than updating R code with outdated R functions
> ....
> >>>
> >>> On Mon, May 29, 2017 at 9:01 AM, Karsten Loesing <
> karsten at torproject.org
> >>>
> >>> wrote:
> >>>
> >>>> Hi John,
> >>>>
> >>>> On 29.05.17 13:35, John Williams wrote:
> >>>>> Hi Karsten
> >>>>>
> >>>>> The modern R tool chain  has the tidyverse
> >>>>> <https://blog.rstudio.org/2016/09/15/tidyverse-1-0-0/> as it core.
> >> The
> >>>>> tidyverse library is a convenience library for installing and loading
> >> the
> >>>>> core set of R packages that comprise the tidyverse - ggplot2, dplyr,
> >>>> tidyr,
> >>>>> readr, purrr & tibble. A dozen other packages are installed but not
> >>>>> explicitly loaded by tidyverse.
> >>>>>
> >>>>> A high rate of R package innovation currently exists within & beyond
> >>>>> the tidyverse (e.g., the shiny tool chain) and it would be difficult
> to
> >>>>> exclude CRAN <https://cran.r-project.org/>-compliant packages from
> >> use.
> >>>>>
> >>>>> I have high confidence in the people & motivations
> >>>>> <https://rviews.rstudio.com/2016/10/12/interview-with-j-j-allaire/>
> >>>> behind
> >>>>> the tidyverse & shiny tool chains.
> >>>>>
> >>>>> Speaking of shiny, a shiny server
> >>>>> <https://www.rstudio.com/products/shiny/shiny-server> installed in
> the
> >>>> Tor
> >>>>> metrics subnet will enable rapid innovation in Tor metrics. Instead
> of
> >>>>> creating plot code to called by javascript, we'll create interactive
> >>>> shiny
> >>>>> apps linked by iframe,  and/or standalone, data-centric web apps,
> just
> >>>>> using R.
> >>>>
> >>>> Can you elaborate on that?  How would we use Shiny Server without
> >>>> JavaScript?  (I'm not asking, because I think it's impossible.  I'm
> >>>> asking because I haven't looked at all the details there and am
> >> curious.)
> >>>>
> >>>>> My library of Tor analytics is small but growing:
> >>>>>
> >>>>> Exit Nodes <http://rpubs.com/johnbwilliams/nodes>
> >>>>> Network Size <http://rpubs.com/johnbwilliams/refactor>
> >>>>> Network Size Delta <http://rpubs.com/johnbwilliams/network_size_
> delta>
> >>>>> hidserv-descs-per-hsdir
> >>>>> <http://rpubs.com/johnbwilliams/hidserv-descs-per-hsdir>
> >>>>>
> >>>>> This is all well and good but pales in comparison to what could be
> done
> >>>> in
> >>>>> Tor Metrics with an analytics (shiny) server.
> >>>>>
> >>>>> I see that a local cloud service provider, with whom I have done
> >> business
> >>>>> from time to time, is hosting a Tor exit address.  I was thinking
> that
> >>>> the
> >>>>> Project could make a business case to this provider's management for
> >>>>> hosting a shiny server for Tor metrics.
> >>>>>
> >>>>> Yes, it is possible but not desirable to revert to base R functions
> >> from
> >>>>> readr - a core package of the tidyverse.  I'll do that as requested.
> >>>>
> >>>> I can understand that going back to base functions is a bit painful.
> My
> >>>> main goal here is to start making improvements as soon as possible.  I
> >>>> think it would take some time to get everyone involved in dependencies
> >>>> on Tor hosts to make a decision here, and I'd rather want to postpone
> >>>> that and start coding and reviewing.
> >>>>
> >>>> Two ideas:
> >>>>
> >>>>  - We include our own version of a read_csv that internally uses R
> base
> >>>> functions to do the job and that we can later replace by what's in
> >> readr.
> >>>>
> >>>>  - We try to get readr into Debian backports.  A Debian developer told
> >>>> me that it's not that difficult to do that, and he might help with the
> >>>> process.  (We could even go one step further and try to get tidyverse
> >>>> into Debian backports, but that might be a bigger project.)
> >>>>
> >>>> I think my preference would be the first option, because that's likely
> >>>> the quickest way to get the code updated.  What do you think?
> >>>>
> >>>>> Cheers,
> >>>>>
> >>>>> John
> >>>>
> >>>> All the best,
> >>>> Karsten
> >>>>
> >>>>
> >>>>>
> >>>>> On Mon, May 29, 2017 at 3:12 AM, Karsten Loesing <
> >> karsten at torproject.org
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi John,
> >>>>>>
> >>>>>> On 26.05.17 05:02, John Williams wrote:
> >>>>>>> something like this?
> >>>>>>>
> >>>>>>> refactor of network size plot <http://rpubs.com/
> >> johnbwilliams/refactor
> >>>>>
> >>>>>>
> >>>>>> Looks great!  A lot shorter and clearer than before.
> >>>>>>
> >>>>>> By the way, did you change that code after posting here?  I remember
> >>>>>> seeing `library(tidyverse)` in the original version, which I
> couldn't
> >>>>>> find in Debian stable or backports.
> >>>>>>
> >>>>>> The current code doesn't contain `library(tidyverse)` anymore, but
> it
> >>>>>> has `library(readr)` which I couldn't find in Debian stable or
> >> backports
> >>>>>> either.
> >>>>>>
> >>>>>> The other libraries are all available in Debian stable or backports
> as
> >>>>>> far as I can see.
> >>>>>>
> >>>>>> We're trying to depend only on Debian packages for anything running
> on
> >>>>>> the server.  This is not an absolute requirement, and we might not
> >> keep
> >>>>>> it up anyway if we ever switch to a Shiny server.  But for the
> moment
> >> it
> >>>>>> would be great if we could keep this requirement.  Or at least we
> >>>>>> shouldn't give up on it too easily.
> >>>>>>
> >>>>>> Do you think you can change the code to avoid `library(readr)`?
> >>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>> All the best,
> >>>>>> Karsten
> >>>>>>
> >>>>>>
> >>>>>>> On Wed, May 24, 2017 at 7:57 PM, David Fifield <
> >> david at bamsoftware.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> On Wed, May 24, 2017 at 07:28:39PM -0400, John Williams wrote:
> >>>>>>>>> Thanks, Karsten, but the direct links to CSV files are not
> working
> >> -
> >>>>>> get
> >>>>>>>>> Oops! Something went wrong here! We encountered a 404 Not Found
> >> when
> >>>>>>>> processing
> >>>>>>>>> your request!
> >>>>>>>>
> >>>>>>>> The URLs were missing a "/stats".
> >>>>>>>>
> >>>>>>>> https://metrics.torproject.org/stats/servers.csv
> >>>>>>>> https://metrics.torproject.org/stats/bandwidth.csv
> >>>>>>>> https://metrics.torproject.org/stats/torperf-1.1.csv
> >>>>>>>> https://metrics.torproject.org/stats/connbidirect2.csv
> >>>>>>>> https://metrics.torproject.org/stats/advbwdist.csv
> >>>>>>>> https://metrics.torproject.org/stats/hidserv.csv
> >>>>>>>> https://metrics.torproject.org/stats/webstats.csv
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20170604/8838d103/attachment-0001.html>