[metrics-team] Consumed Bandwidth and User Count anomaly

[Karsten Loesing]

On 2018-12-03 10:00, Karsten Loesing wrote:
> On 2018-12-01 16:49, Matthew Finkel wrote:
>> Hi all!
> 
> Hi Matt,
> 
>> I noticed the bandwidth history data dropped to ~0 between 21-Nov and
>> 23-Nov, with a noticable decrease beginning on 20-Nov [0]. The available
>> data from 24 and 25 Nov show an overall decrease in throughput compared
>> with 19-Nov. Data aren't availabe on 26 or 27 Nov (at this time). Has
>> anyone looked into this and discovered what caused the drop?
> 
> I noticed the same thing on Saturday, roughly around the time you sent
> this message. I looked into the metrics host and found four concurrent
> runs updating statistics, which was not supposed to happen. I killed
> them all, changed the cronjob from running twice per day to running once
> per day, and started a fresh update run.
> 
> Since then the update runtime has increased from 5 hours to 18 hours! In
> particular the onion-service statistics now have an execution time of 14
> hours. I don't know the reason for this, but it's worrisome. I had
> opened #25924 seven months ago, and maybe it has just become more urgent.
> 
> Regarding the temporary drop, that is caused by the aborted runs. I'd
> like to wait a few more days to see if runs at least stabilize at the
> higher execution times. Then I'd re-import the missing data.
> 
>> Similarly, and maybe relatedly, the number of users jumped beginning on
>> 26-Nov [1]. This jump is reflected in all of the top-10 countries by relay
>> users - except UAE [2]. There is a similar jump in bridge users[3].
> 
> I don't think this is real. There's a brief increase on November 29,
> also caused to missing some data, and a one-day gap on November 30. We
> have data for December 1, but it's not yet plotted because of the gap on
> November 30 and the need for two points to draw a line. As soon as we
> have data for December 2, there will be a line again.
> 
> Here are the raw numbers, which are also available via the CSV link:
> 
> 2018-11-20,,1901624,,,99
> 2018-11-21,,1959206,,,98
> 2018-11-22,,1917530,,,100
> 2018-11-23,,1874480,,,99
> 2018-11-24,,1801991,,,98
> 2018-11-25,,1805516,,,98
> 2018-11-26,,1918386,,,99
> 2018-11-27,,1946122,,,99
> 2018-11-28,,1969536,,,94
> 2018-11-29,,2105619,,,67
> 2018-12-01,,1960112,,,65
> 
>> Were there any changes implemented on the Metrics infrastructure around
>> this time?
> 
> I did make some updates. But none of them were related to the
> onion-service statistics module acting up now. I don't yet see the
> connection.

Finally, these issues are now fixed.

There were several issues, but it started with the issue in the bwhist
module that I fixed here:

https://gitweb.torproject.org/metrics-web.git/commit/?id=9dd35e29084ed9380cb374c80a4f9bfb0d9a91e2

As a result, processing got slower and slower, and daily runs started
overlapping.

Anyway, things are fixed now. Just in time for the holiday break.

All the best,
Karsten


> 
>> Thanks,
>> Matt
> 
> Thanks for the report!
> 
> All the best,
> Karsten
> 
> 
> 
>>
>> [0]
>> https://metrics.torproject.org/bandwidth.html?start=2018-11-18&end=2018-12-01
>> [1]
>> https://metrics.torproject.org/userstats-relay-country.html?start=2018-11-15&end=2018-12-01&country=all&events=off
>> [2] https://metrics.torproject.org/userstats-relay-table.html
>> [3]
>> https://metrics.torproject.org/userstats-bridge-country.html?start=2018-11-20&end=2018-12-01&country=all
>> _______________________________________________
>> metrics-team mailing list
>> metrics-team at lists.torproject.org
>> https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team
>>
> 
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 528 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20181221/7f8a5921/attachment.sig>