[metrics-team] did the recent onionoo outage cause onionoo to loose data?

[teor]

> On 6 Aug 2018, at 18:05, Karsten Loesing <karsten at torproject.org> wrote:
> 
> On 2018-08-06 09:56, teor wrote:
>> 
>>> On 6 Aug 2018, at 17:48, Karsten Loesing <karsten at torproject.org> wrote:
>>> 
>>> indeed, it looks like we lost about 1 day of data between 2018-08-04
>>> 19:00 and 2018-08-05 17:00.
>>> 
>>> Unfortunately, we're not going to get that data back. It's unrelated to
>>> Onionoo but related to CollecTor.
>> 
>> Eventually, we want Tor authorities to serve recent consensuses, votes,
>> descriptors, and bandwidth files, to help with outages like this one:
>> https://trac.torproject.org/projects/tor/ticket/27047#comment:1
>> 
>> (Authorities already keep recent consensuses and descriptors, but we
>> don’t have an easy way to retrieve them.)
> 
> Do you think we could use these diffs together with a recent consensus
> to reconstruct the consensuses that we missed?

Background

The diffs are one-way, from the old consensus to the most recent
consensus. So each relay also stores up to 72 hours of recent consensuses
in its diff-cache directory:

https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n1961
https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n3469


Some Missing Consensuses

Here are copies of 55 and 66 recent consensuses from some of my relays:

https://people.torproject.org/~teor/volatile/recent-consensuses-55-2018-08-06.tgz
https://people.torproject.org/~teor/volatile/recent-consensuses-66-2018-08-06.tgz

They are probably the consensuses from Mon 5 Aug 2018 00:00 to
Mon 6 Aug 2018 08:00 UTC.


The Remaining Consensuses

There aren’t 2*72 = 144 consensuses, because the cache is limited to 256
files, including consensuses, diffs, and compressed diffs. I’ve asked
other relay operators for their consensuses as well:

https://lists.torproject.org/pipermail/tor-relays/2018-August/015850.html


Processing The Files

You’ll need to strip off the headers before uploading them to collector:

CacheDirectory/diff-cache

Directory cache only. Holds older consensuses, and diffs from older
consensuses to the most recent consensus of each type, compressed in
various ways. Each file contains a set of key-value arguments describing
its contents, followed by a single NUL byte, followed by the main file
contents.

https://www.torproject.org/docs/tor-manual.html.en


Getting Descriptors

If you upload these consensuses to collector in the next 5 days, all the
corresponding (micro)descriptors should still be available on the directory
authorities. Any directory cache should also have a copy.

(For the record, votes are not cached.)

T

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20180806/9a5562b2/attachment.sig>