Hello David,
thank you for doing and sharing this.
The other failure in this approach to scanning is that it uses a fixed set of relays... and therefore the Tor consensus file used will become old and contain relays no longer in the consensus well before the scan is complete.
[...]
Show me all the tor relays that failed ALL of their circuit builds:
echo "select first_hop,count(first_hop) from scan_log where status = 'failure' group by second_hop;" | sqlite3 scan2.db | grep 99 $1D3F937E2053E58C18E18D43FA5153E2A9F4DC77|99 $C793AB88565DDD3C9E4C6F15CCB9D8C7EF964CE9|99 $EF6591754F9079DD122EFC2C4B52917F625A8E5B|99 $55C7554AFCEC1062DCBAC93E67B2E03C6F330EFC|99 $87C08DDFD32C62F3C56D371F9774D27BFDBB807B|99 $BD4C647508162F59CB44E4DFC1C2B2B8A9387CCA|99 $EF6591754F9079DD122EFC2C4B52917F625A8E5B|99 $1D3174338A1131A53E098443E76E1103CDED00DC|99 $734EDDC2C04B1C0184178167ABD23AE85413212F|99 $7C0AA4E3B73E407E9F5FEB1912F8BE26D8AA124D|99
Since you mention the design problem of potentially including relays that are no longer in consensus: Did you review the list of relays failing _ALL_ circuits against the consensus to check if they just went offline (or reset their uptime) during the scan? How much time did the scan take? How much time is usually between each of these 99 attempts?
Can I download your raw scan results (logs with timestamps) somewhere so I could check if these 'failed ALL circuits' relays reset their uptime or went offline during the scan?
Did you also play with teor's scanner? https://github.com/teor2345/onion-graph
I've started a related (but more long-term) thread on tor-dev on collecting this kind of data: https://lists.torproject.org/pipermail/tor-dev/2017-October/012502.html
related trac tickets: https://trac.torproject.org/projects/tor/ticket/12131 https://trac.torproject.org/projects/tor/ticket/19068