New Tor Metrics graph: Tor Browser downloads and updates
Hi everyone, we just put a new graph on Tor Metrics called "Tor Browser downloads and updates": https://metrics.torproject.org/webstats-tb.html This graph is the result of a joint effort of the metrics team (with a lot of input on the underlying code from iwakeh) together with Sebastian (who co-authored the original database schema and who operates the web log sanitizer), gk and boklm (who analyze an earlier graph and provided insights about Tor Browser internals), and others who discussed the graph and earlier prototypes. Please take a look at this graph, play a bit with the two inputs (start and end date), and give us feedback by raising questions about the data or giving answers to previously raised questions. The goal would be to write a blog post about this graph together with an early analysis. Currently open questions are: - gk asks on the ticket (https://trac.torproject.org/projects/tor/ticket/21236#comment:11): "The results still confuse me (like: We have 100,000 new downloads each day but that does not show up in the update pings which basically stay the same? We still have more than 120,000 update requests every day, even after almost 6 weeks after the last release?). But that is probably another story. :)" - hellais asked at yesterday's Vegas meeting: "how many times does a normal tor browser do an update ping every day?" which was answered by mikeperry and gk with "twice" "and on every start". This is probably something we should clarify in the graph description. - yawning asks on the ticket (https://trac.torproject.org/projects/tor/ticket/21236#comment:25): "Does it matter that the Linux sandbox behavior is totally different from everyone else, because it uses a different implementation to download, update check, and update? I suspect the user base isn't that big right now, but I'd hate to throw stats off..." Note that we're mostly hoping for feedback on the data, not on the presentation. We know that there's room for making the graph prettier or for providing more inputs to customize it. But we already spent way more time on this graph than we had planned, so we'd save any feedback on the presentation (except for typos or trivial tweaks) for later in the year. Thanks! All the best, Karsten
Hi Karsten, This is great work and very important for us -- thanks for doing it! I think a number I would be interested in knowing is the total (cumulative) number of downloads and update requests for each version (6.0.0, 6.0.1, etc.). Is that something we can extract from the data? Thanks, Arthur On Fri, Jan 27, 2017 at 7:15 AM, Karsten Loesing <karsten@torproject.org> wrote:
On 27/01/17 21:18, Arthur D. Edelstein wrote:
Hi Karsten,
Hi Arthur,
This is great work and very important for us -- thanks for doing it!
Glad to hear it's useful!
We don't keep version strings when aggregating statistics, but we do keep channel information (stable, alpha, hardened). Maybe have a look at the .csv file and data format to decide whether that works for you, too: https://metrics.torproject.org/stats/webstats.csv https://metrics.torproject.org/stats.html#webstats I'm worried that adding version strings to the .csv file, in particular until patch level, would be too much. Major and minor version might work, but even that would grow the file a lot. Another option would be that you run your own Metrics webstats module that downloads from webstats.torproject.org and puts logs into a local PostgreSQL database. I'd be happy to walk you through the setup process. All the best, Karsten
On 1/29/17 11:19, Karsten Loesing wrote:
quick question! I see we keep locale information for the downloads in the csv file, would be possible - maybe in a future iteration - to filter it by locale? Would be great to know the top languages etc. o/ isabela
Hi Isabela, On 29/01/17 17:59, isabela wrote:
This is possible. Though we'll have to add a separate tab to the page, like "Tor Browser downloads by locale". One more probably doesn't hurt, but we should be careful with that thinking, or we'll make the interface harder to use for everyone. What do you think about the attached graph? All the best, Karsten
On 31/01/17 09:15, Georg Koppen wrote:
Thanks for looking! Isabela also confirms off-list that this graph is what she had in mind. Here's a preview: https://metrics.torproject.org/webstats-tb-locale.html If that looks good and passes team-internal review, I'll merge it. All the best, Karsten
On Fri, 27 Jan 2017 16:15:41 +0100 Karsten Loesing <karsten@torproject.org> wrote:
If the definition of "update ping" is fetch the JSON file at: `extensions.torbutton.versioncheck_url`, then their comments don't match what I saw in the code... https://gitweb.torproject.org/torbutton.git/tree/src/chrome/content/torbutto... const kMinSecsBetweenChecks = 120 * 60; // 2.0 hours Regards, -- Yawning Angel
Probably, `app.update.interval` https://gitweb.torproject.org/tor-browser.git/tree/browser/branding/official...
On Fri, 27 Jan 2017 12:49:01 -0800 Arlo Breault <arlo@torproject.org> wrote:
Ugh. So there's 2 different update check mechanisms that Tor Browser uses, on different intervals, fetching data from different hosts, in different formats. FWIW, the sandbox update check just uses the XML format data on aus1.torproject.org (prefers fetching from the .onion), with the timer set to every 2 hours. Regards, -- Yawning Angel
On 28/01/17 00:31, Yawning Angel wrote:
Okay, we rely on folks telling us which resource strings to look after and how to interpret them. If we should be looking for different resource strings in the logs or change the documentation in any way, please tell us! All the best, Karsten
On Sun, 29 Jan 2017 17:30:23 +0100 Karsten Loesing <karsten@torproject.org> wrote:
Downloading (sandboxed-tor-browser application): https://dist.torproject.org/torbrowser/7.0a1/sandbox-0.0.3-linux64.zip https://dist.torproject.org/torbrowser/7.0a1/sandbox-0.0.3-linux64.zip.asc sandboxed-tor-browser works similarly to tor-browser-launcher in certain regards, and handles downloading/installing an actual bundle. For the purposes of illustration, the `release` channel will be shown, and I will assume that substituting in the right thing is trivial. Installing tor browser (done only once): http://x3nelbld33llasqv.onion/torbrowser/update_2/release/downloads.json OR https://aus1.torproject.org/torbrowser/update_2/release/downloads.json https://dist.torproject.org/torbrowser/6.5/tor-browser-linux64-6.5_en-US.tar... https://dist.torproject.org/torbrowser/6.5/tor-browser-linux64-6.5_en-US.tar... The HS is only used here if a system tor instance is explicitly pre-configured. (This probably shouldn't be hitting up dist.tp.o, but the JSON file is generated by the automated build process, and is beyond my control.) sandboxed-tor-browser also handles checking for updates, and keeping the installed browser up to date, because the sandboxed firefox instance doesn't have sufficient file system privileges to be able to rewrite itself (This is a good thing). Update check (every 2h): http://x3nelbld33llasqv.onion/torbrowser/update_2/release/Linux_x86_64-gcc3/... If the fetch from the HS fails, it will immediately attempt to fetch: https://aus1.torproject.org/torbrowser/update_2/release/Linux_x86_64-gcc3/6.... Any further failures are retried after another 2 hours has elapsed. All other internal update checks present in Tor Browser are disabled, so it will never fetch: https://www.torproject.org/projects/torbrowser/RecommendedTBBVersions If there is an update, it will grab the MAR as specified in the XML response just fetched. This will probably be the main source of inconsistency. Personally I don't see why `RecommendedTBBVersions` still exists at all. nb: If the user closes sandboxed-tor-browser completely, on next launch an update check cycle is forced before firefox is allowed to execute if it has been over 2 hours since the last update check, after the bootstrap process completes. Complete failure then results in a failure to launch with a modal dialog box (Yes, I know some people will consider this a misfeature. No, I don't care). Regards, -- Yawning Angel
On 30/01/17 09:32, Yawning Angel wrote:
I ran a query on the webstats database to see how many downloads of the sandboxed Tor Browser there are: webstats=> SELECT log_date, resource_string, SUM(count) AS count FROM files NATURAL JOIN requests NATURAL JOIN resources WHERE resource_string LIKE '/torbrowser/%/sandbox-%-linux%.zip' AND response_code = 200 AND method = 'GET' AND log_date >= '2017-01-01' GROUP BY 1, 2 ORDER BY 1, 2; log_date | resource_string | count ------------+---------------------------------------------+------- 2017-01-01 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 32 2017-01-01 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 41 2017-01-02 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 35 2017-01-02 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 74 2017-01-03 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 42 2017-01-03 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 75 2017-01-04 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 39 2017-01-04 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 62 2017-01-05 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 52 2017-01-05 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 84 2017-01-06 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 38 2017-01-06 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 48 2017-01-07 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 36 2017-01-07 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 57 2017-01-08 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 35 2017-01-08 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 50 2017-01-09 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 27 2017-01-09 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 58 2017-01-10 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 43 2017-01-10 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 61 2017-01-11 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 59 2017-01-11 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 92 2017-01-12 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 41 2017-01-12 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 48 2017-01-13 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 35 2017-01-13 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 63 2017-01-14 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 34 2017-01-14 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 57 2017-01-15 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 45 2017-01-15 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 102 2017-01-16 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 43 2017-01-16 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 58 2017-01-17 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 30 2017-01-17 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 58 2017-01-18 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 23 2017-01-18 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 64 2017-01-19 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 34 2017-01-19 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 39 2017-01-20 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 38 2017-01-20 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 45 2017-01-21 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 35 2017-01-21 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 49 2017-01-22 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 45 2017-01-22 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 52 2017-01-23 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 52 2017-01-23 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 41 2017-01-24 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 35 2017-01-24 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 46 2017-01-24 | /torbrowser/7.0a1/sandbox-0.0.3-linux64.zip | 1 2017-01-25 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 2 2017-01-25 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 2 2017-01-25 | /torbrowser/7.0a1/sandbox-0.0.3-linux64.zip | 99 2017-01-26 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 1 2017-01-26 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 3 2017-01-26 | /torbrowser/7.0a1/sandbox-0.0.3-linux64.zip | 76 2017-01-27 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 1 2017-01-27 | /torbrowser/7.0a1/sandbox-0.0.3-linux64.zip | 75 2017-01-28 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 1 2017-01-28 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 1 2017-01-28 | /torbrowser/7.0a1/sandbox-0.0.3-linux64.zip | 80 2017-01-29 | /torbrowser/7.0a1/sandbox-0.0.3-linux64.zip | 61 2017-01-30 | /torbrowser/6.5a6/sandbox-0.0.2-linux32.zip | 1 2017-01-30 | /torbrowser/6.5a6/sandbox-0.0.2-linux64.zip | 1 2017-01-30 | /torbrowser/7.0a1/sandbox-0.0.3-linux64.zip | 75 (64 rows) And one more with totals per day: webstats=> SELECT log_date, SUM(count) AS count FROM files NATURAL JOIN requests NATURAL JOIN resources WHERE resource_string LIKE '/torbrowser/%/sandbox-%-linux%.zip' AND response_code = 200 AND method = 'GET' AND log_date >= '2017-01-01' GROUP BY 1 ORDER BY 1; log_date | count ------------+------- 2017-01-01 | 73 2017-01-02 | 109 2017-01-03 | 117 2017-01-04 | 101 2017-01-05 | 136 2017-01-06 | 86 2017-01-07 | 93 2017-01-08 | 85 2017-01-09 | 85 2017-01-10 | 104 2017-01-11 | 151 2017-01-12 | 89 2017-01-13 | 98 2017-01-14 | 91 2017-01-15 | 147 2017-01-16 | 101 2017-01-17 | 88 2017-01-18 | 87 2017-01-19 | 73 2017-01-20 | 83 2017-01-21 | 84 2017-01-22 | 97 2017-01-23 | 93 2017-01-24 | 82 2017-01-25 | 103 2017-01-26 | 80 2017-01-27 | 76 2017-01-28 | 82 2017-01-29 | 61 2017-01-30 | 77 (30 rows) When these numbers go up we should include them in the graph. But for now, they wouldn't change the totals much, and the SQL query is complex enough already, so I'd rather not want to add more special cases to it. All the best, Karsten
On Thu, 2 Feb 2017 10:30:37 +0100 Karsten Loesing <karsten@torproject.org> wrote:
This seems reasonable for now, thanks and sorry for the extra work. I have no good solution for the updater behavior being different. I'd rather not use a split update check mechanism like Tor Browser, but I suppose the volume isn't high enough to matter. Regards, -- Yawning Angel
Awesome stuff! On 1/27/17 10:15, Karsten Loesing wrote:
Thanks everyone!
+ 1
Would be possible to have the sandbox stats and also have a total stats that ads it up to all TBB downloads? (I +1 Arthur's suggestion for a break down by release and cumulative data)
Note that we're mostly hoping for feedback on the data, not on the presentation.
I wonder if we could add the metrics from google play store on orfox and orbot. If there is an api or something and if that would be too much work. !!! thanks a lot for this !!! Isabela
Hi Isabela, On 27/01/17 22:34, isabela wrote:
Awesome stuff!
Yay!
(Agreed. Let's collect a few more items that need clarification and then edit the description.)
What requests do users make to download the Linux sandbox thing, and how does it check for updates and request them? For comparison, here are the patterns we're currently using: https://metrics.torproject.org/stats.html#webstats And for even more detail, here's the SQL: https://gitweb.torproject.org/metrics-web.git/tree/modules/webstats/src/main...
(I +1 Arthur's suggestion for a break down by release and cumulative data)
(See my response to Arthur.)
I'm afraid that adding another data source than our webstats is out of scope for the moment. It's certainly interesting for the future, and we should keep it in mind. But it's out of reach if we also want to fulfill our many other commitments. Sorry.
!!! thanks a lot for this !!!
Glad it's useful. :)
Isabela
All the best, Karsten
participants (6)
-
Arlo Breault -
Arthur D. Edelstein -
Georg Koppen -
isabela -
Karsten Loesing -
Yawning Angel