Hello David,
thank you for doing and sharing this.
running active scans is fun. building longer term solutions is fun.
Since you mention the design problem of potentially including relays that are no longer in consensus: Did you review the list of relays failing _ALL_ circuits against the consensus to check if they just went offline (or reset their uptime) during the scan? How much time did the scan take? How much time is usually between each of these 99 attempts?
No, I did not check if relays in the scan consensus were missing from subsequent consensus files. I agree it's a good idea to check. The scan with those parameters takes 45 minutes. The duration of time between circuit build attempts is .25 seconds.
detect_partitions.py --tor-control tcp:127.0.0.1:9051 --log-dir ./ --status-log ./status_log \ --relay-list top100.relays --secret secret --partitions 1 --this-partition 0 \ --build-duration .25 --circuit-timeout 60 --log-chunk-size 1000 --max-concurrency 100
Can I download your raw scan results (logs with timestamps) somewhere so I could check if these 'failed ALL circuits' relays reset their uptime or went offline during the scan?
No... but you can run your own scan and obtain the results in minutes.
Did you also play with teor's scanner? https://github.com/teor2345/onion-graph
No.
I've started a related (but more long-term) thread on tor-dev on collecting this kind of data: https://lists.torproject.org/pipermail/tor-dev/2017-October/012502.html
Cool. Yes and Peer Flow should be an excellent change for the tor network since it's a better bandwidth authority system than torflow:
https://petsymposium.org/2017/papers/issue2/paper12-2017-2-source.pdf