[tor-dev] Guardiness: Yet another external dirauth script

[George Kadianakis]

Sebastian Hahn <sebastian at torproject.org> writes:

> On 16 Sep 2014, at 16:15, George Kadianakis <desnacked at riseup.net> wrote:
>> ====How guardiness works====
>> The idea was that the guardiness script will be an external script
>> that is run by Tor in a similar fashion to the bandwidth auth
>> scripts. We chose that because we could write the script in a
>> high-level language and because it could be modular and we could
>> change the algorithm in the future if we wanted. Unfortunately, it
>> seems that external scripts in dirauths are a PITA to maintain as can
>> be seen by the lack of bwauth operators.
>
> The problem isn't so much that this is an external script, the problem
> is that there are never dedicated maintainers for these things (tho
> it being an external script is additionally problematic). If they
> are in Tor proper, we have someone who cares about them when new stuff
> gets introduced, and it gets updated with the rest of Tor, etc.
>

I understand and I can maintain the guardiness script.

However, I'm also hoping for other people to review the script before
deployment (#13125). It's a pretty small script so it shouldn't be too wrong.

>> Summary files are designed to be quick to parse (even with Python) and
>> contain all the necessary information for guardiness. For example,
>> parsing 2000 summary files in my laptop takes about 10 seconds.
>
> Does this scale linearly? 9 months would be ~6500 files.
>

Yes, it should scale linearly.
6500 files should take about 30 to 40 seconds.

(Each consensus parsing is an independent event.) 

>> FWIW, the guardiness scripts are ready for review and can be found here:
>> https://gitweb.torproject.org/user/asn/hax.git/shortlog/refs/heads/guardiness
>
>
>> 
>> ====How the guardiness script will be deployed====
>> 
>> The idea is that dirauths will add another script to their crontab
>> that is called every hour (before or after the bwauth scripts).
>
> Only 4/9 have such scripts, if at all - it is possible to run a
> bwauth on a different host, and scp the file over. I don't know
> if any of the dirauth ops actually do this currently.
>

This should also be possible with the guardiness stuff, as long as you
can arrange fresh consensuses to arrive on the second host.

>> The script first calls the summarizer script, which goes to the
>> consensus/ directory and summarizes all consensuses it finds and puts
>> them in the summary/ directory. The summarizer script then deletes all
>> the consensuses that got summarized.
>
> You must not delete files which you did not create. It's not cool
> to delete consensuses which tor decided to put somewhere if you're
> not tor. This is relevant to the plan to use mvdan's script.
>

Yes, you are absolutely right on that. FWIW, the --delete-summarized
switch of the summarizer script is disabled by default.

If we go ahead with using Daniel's tor feature, we should _not_ delete
those consensuses.

>> Then the script calls the the guardiness script, which goes to the
>> summary/ directory, parses all summary files it finds, and outputs a
>> guardiness output file that gets parsed by the dirauth prior to voting.
>> 
>> That should be all. Easy, eh? :)
>
> What are the failure modes? Are there version strings included, does
> Tor notice if the guardiness file is outdated, etc? What happens when
> different dirauths use different versions of the guardiness file
> generation script?
>

There are various failure modes I think.

Failure modes include dirauths considering 5 consensuses when they
should be considering 3000 of them. Or dirauths considering 12000
consensuses when they should be considering 3000.

As you pointed out, in the future if the guardiness algorithm changes,
failure modes could also include dirauths using different guardiness
versions.

Based on your comment, I think it might be a good idea to publish the
guardiness script version somewhere in the dirauth votes. Also the
number of consensuses considered, and the maximum consesus age
considered.

It might even be smart to make a consensus health module that ensures
that guardiness versions are the same in all dirauths, and they all
derive approximately the same guardiness for all relays (contrary to
bwauths, two perfectly configured dirauths should derive the exact
same guardiness value).