Hi Damian,
Roger and I discussed the consensus-health web page that the Java version of DocTor produces [0] but your Python version does not. Roger says he uses that page to debug the Tor network. In particular, he says he scans the huge relay flags table to spot differences between relay flags assigned by the directory authorities. I'm pasting the IRC log below, but we should move this discussion to email, because it seems unlikely that we're all present in #tor-dev at the same time to discuss this.
I see four ways how we can make most people mostly happy:
1.) We add votes documents to Onionoo (#9778) and include relay flags contained in votes in relay details pages of Atlas/Globe, so that people can find out what the different directory authorities think about their relay. We write a new Onionoo client that fetches this information for all running relays and produces a long table of relay flags. Before we do all this we make sure that there's nothing else on the consensus-health web page that Roger secretly hoped to keep. Then we shut down the Java DocTor.
2.) We rewrite the website-generating part of DocTor in a language we're happier to maintain long-term. This could be part of your DocTor, or of a new tool. Then we shut down the Java DocTor.
3.) We keep the Java DocTor running to generate the website, but disable the part that sends notifications to the mailing list and to the IRC bot. Or rather, I'll want to move the page away from metrics.tpo to, say, consensus-health.tpo or consensus-explorer.tpo, because it has always been a hack to serve it on metrics.tpo. Then we hope the Java DocTor never breaks.
4.) We do something much smarter which I didn't think of yet.
What do you think?
All the best, Karsten
[0] https://metrics.torproject.org/consensus-health.html
08:04:49 < karsten> armadev: about removing per-relay votes from https://metrics.torproject.org/consensus-health.html. that was intentional, because the page took so long to load and it seemed nobody cared. 08:05:14 < karsten> armadev: but it seems the next step will be to kill that web page entirely.. 08:05:34 < karsten> the reason is that atagar's consensus-health checker will take over very soon. 08:05:56 < karsten> so, what information from the consensus-health page do you still need, either the current one or even the one with per-relay votes? 08:06:10 < karsten> we should try to rescue that information then. 08:07:16 < armadev> karsten: is atagar's consensus health checker up yet? 08:07:22 < karsten> yes, it is. 08:07:26 < armadev> url? 08:07:33 < karsten> it's sending mail to tor-consensus-health@ and to #tor-bots. 08:07:36 < karsten> it has no website 08:07:48 < karsten> it's sending out warnings if something goes wrong. 08:07:56 < karsten> when* :) 08:08:54 < armadev> karsten: in this case, what i wanted from the consensus health page was the set of votes per authority for 000000000000myTOR2 08:08:56 < karsten> right now, both consensus-health things send mail to that list. the ones at :03 are from mine, the other ones from atagar's. 08:09:11 < armadev> i pieced it together manually from moria1's v3-status-votes, but i'm special so i can do that 08:09:16 < karsten> right 08:09:25 < karsten> I thought about this a while ago. we could include votes in onionoo. 08:09:34 < karsten> not sure how yatei will like that though. 08:09:50 < karsten> but it's the votes you care about, ok. anything else from the consensus-health page? 08:10:16 < karsten> wfn: ^^ we should talk about this. 08:11:05 < armadev> s/votes/votes about flags/ here 08:11:23 < karsten> right. 08:37:19 < armadev> it seems i used consensus-health as consensus-explorer 10:11:35 < armadev> karsten: can you fwd me a sample mail from the health checker mailing list? 10:12:07 < karsten> armadev: they're archived: https://lists.torproject.org/pipermail/tor-consensus-health/2013-September/0... 10:13:43 < armadev> oh good 10:13:56 < armadev> that's a very short mail. 10:14:02 < armadev> it has, like, nothing from the consensus health checker 10:14:20 < karsten> that's the things that need fixing to make it quiet again. 10:14:42 < armadev> why is "it takes a long time to load" a bug in consensus-health.html ? 10:15:37 < karsten> well, the page is huge, and firefox can't handle that very well. that's not a bug, but an inconvenience. 10:16:09 < armadev> sure 10:16:25 < karsten> I could have split the page into two. 10:16:34 < karsten> but anyway, the entire page is going away soon. 10:16:39 < armadev> if you were planning to drop it entirely, you could just leave it alone? 10:17:12 < karsten> as unmaintained service? 10:17:36 < karsten> I'd rather take the pieces we like and integrate them in maintained services. 10:17:37 < armadev> i guess. or write a howto for how to take votes in and generate the html, and i'll run it on moria 10:17:53 < armadev> though i think it needs more than votes as input 10:18:11 < armadev> i've used most parts of this page over the past few months 10:18:13 < karsten> it downloads votes and consensuses. 10:18:19 < karsten> I see. 10:18:42 < armadev> i have votes and a consensus on moria 10:18:49 < armadev> if that's the input, i can just cron it to run on moria 10:18:58 < armadev> based on the votes i see and the consensus i see 10:19:09 < armadev> unless i guess it requires a massive tomcat engine too :) 10:19:15 < karsten> it doesn't. 10:19:17 < karsten> it requires java. 10:19:29 < karsten> and produces a static html that you can service with whatever. 10:19:55 < karsten> well, if you need it, it's easier to keep it running on yatei. 10:19:57 < armadev> i don't have any java. i guess i could add it. or we could stick it on people.tp.o or something 10:19:59 < armadev> right 10:20:09 < karsten> and just turn off the email notifications, because atagar's thing will send those. 10:20:11 < armadev> i use it for debugging the network 10:20:32 < karsten> ok 10:20:58 < karsten> would #9778 solve the problem with the missing relay flags table? 10:20:59 -zwiebelbot:#tor-dev- [tor#9778: Adding votes documents to Onionoo] 10:21:08 < karsten> assuming atlas or globe would show those? 10:21:32 < armadev> do i have to load each relay one page at a time? 10:22:05 < karsten> probably. 10:22:31 < karsten> so, you're looking through the table to spot problems? 10:22:41 < karsten> or, have been looking, when the table was there. 10:22:41 < armadev> yes. and to get a sense of trends. 10:22:44 < armadev> right 10:22:57 < armadev> i guess in this most recent case i just wanted one relay 10:23:11 < armadev> but after that, i realized that i wanted to know how many relays had a Running flag from those two authorities and no Running flag from any others 10:23:33 < karsten> ok 10:23:42 < armadev> (i still want to know :) 10:24:06 < karsten> let me re-enable the table and think about a better long-term solution.. :) 10:24:07 < armadev> (related to https://trac.torproject.org/projects/tor/ticket/9775 ) 10:24:08 -zwiebelbot:#tor-dev- [tor#9775: Authorities should report when they don't vote Running but some addresses are still reachable] 10:24:48 < armadev> we are dropping an unknown number of relays from the consensus because they enable ipv6 but screw up the reachability 10:25:16 < armadev> hopefully every relay operator who knows what ipv6 is also knows how to check relay-search for themselves 10:28:21 < karsten> the next consensus-health.html will contain the relay flags table again. in roughly 35 mins. 10:29:51 < armadev> thanks. sorry for the troubles. and i understand the drive to get rid of extra things. i wish we had more things that were actually extra. 10:32:10 < karsten> yeah, we have a lot of stuff. and some of it is really old and hasn't seen love for too long.
Hi Karsten. After thinking about this for a bit I'm gonna vote for #4. :)
DocTor's consensus-health page is solely used by authority operators, and only for specific tasks. If its current visualization of this information was truly helpful then I'd agree that we should keep it. However, for the use case Roger mentioned...
"i wanted to know how many relays had a Running flag from those two authorities and no Running flag from any others"
... means he had to either...
a. Read through ~4,000 entries on that page, counting how many fit his critera. b. curl the page and grep the html for what he wanted
Either one sounds terrible, and I sincerely hope we have better uses for Roger's time. If we can persuade him to give us the use cases for what he wants from consensus-health then I think we can come up with better alternatives. Possibly a suite of scripts that give him the answers to common questions (and can be adapted for more exotic ones). For instance, in this case something like the following would've done the trick...
======================================== Compares the Running flag between moria1 and maatuska ========================================
from stem.descriptor import DocumentHandler, remote
# Query each authority with a v3ident for its vote.
downloader = remote.DescriptorDownloader(document_handler = DocumentHandler.DOCUMENT) queries = {}
for authority in remote.get_authorities().values(): if authority.v3ident is not None: queries[authority.nickname] = downloader.get_vote(authority)
# Wait for the votes to finish being downloaded, this produces a dictionary of # authority nicknames to their vote.
votes = dict((nickname, query.run()[0]) for (nickname, query) in queries.items())
# Get the superset of all relay fingerprints in the votes.
all_fingerprints = set()
for vote in votes.values(): all_fingerprints.update(vote.routers.keys())
# For each relay check if it has the 'Running' flag from moria1 but not # maatuska.
for fingerprint in all_fingerprints: moria1_vote = votes['moria1'].routers.get(fingerprint) maatuska_vote = votes['maatuska'].routers.get(fingerprint)
if not moria1_vote and not maatuska_vote: print "both moria1 and maatuska haven't voted about %s" % fingerprint if not moria1_vote: print "moria1 hasn't voted about %s" % fingerprint elif not maatuska_vote: print "maatuska hasn't voted about %s" % fingerprint elif 'Running' in moria1_vote.flags and 'Running' not in maatuska_vote.flags: print "moria1 has the Running flag but maatuska doesn't: %s" % fingerprint elif 'Running' in maatuska_vote.flags and 'Running' not in moria1_vote.flags: print "maatuska has the Running flag but moria1 doesn't: %s" % fingerprint
========================================
atagar@odin:~/Desktop/stem$ python example.py maatuska has the Running flag but moria1 doesn't: 32A38850B20858B0AC2DF5D5FAFB7F3596161067 maatuska has the Running flag but moria1 doesn't: D3FE846B0DD32285B1D5B3FCF8649040BC6D4CC3 maatuska has the Running flag but moria1 doesn't: 6871F682350BA931838C0EC1E4A23044DAE06A73 moria1 has the Running flag but maatuska doesn't: 8047A4D403EAC6BA15D68D4D93626CDF84B0BC4B moria1 hasn't voted about 1D3E139EB1917EBA0B594607D61CD99F733A8CB9 both moria1 and maatuska haven't voted about 834D996C76C73A78E23B8F41CA2AC476776D4CA8 ... etc...
========================================
Cheers! -Damian
On 9/21/13 11:42 PM, Damian Johnson wrote:
Hi Karsten. After thinking about this for a bit I'm gonna vote for #4. :)
DocTor's consensus-health page is solely used by authority operators,
I think Roger uses it in his role as network debugger, not as authority operator. There may be more people wanting to help with network debugging who are not authority operators. And there may be expert relay operators who are wondering why their relay isn't listed or isn't listed with a given flag.
and only for specific tasks. If its current visualization of this information was truly helpful then I'd agree that we should keep it.
The information is at least somewhat helpful. But it's clearly not the best visualization one can imagine.
However, for the use case Roger mentioned...
"i wanted to know how many relays had a Running flag from those two authorities and no Running flag from any others"
... means he had to either...
a. Read through ~4,000 entries on that page, counting how many fit his critera. b. curl the page and grep the html for what he wanted
Either one sounds terrible, and I sincerely hope we have better uses for Roger's time.
Agreed.
If we can persuade him to give us the use cases for what he wants from consensus-health then I think we can come up with better alternatives.
Our best bet is to suggest an interface, ask if Roger (or anybody else) would be missing something, and then implement that interface.
Possibly a suite of scripts that give him the answers to common questions (and can be adapted for more exotic ones). For instance, in this case something like the following would've done the trick...
A command-line tool doesn't sound too crazy. That tool (or suite of scripts) should be able to cache downloads and votes and share them among all queries we might want to ask it.
But is a command-line tool more user-friendly than a web interface? For example, when we wrote Compass, we started with a command-line interface, but then Roger wanted a web interface for it. Maybe, if there's the possibility that we'll want a web interface later, we should start with a web interface right from the start. A web interface has the advantage that people can share query results more easily than with a command-line interface. Also, no installation required. We should probably run a poll for the preferred interface.
But regardless of the interface type, I can imagine this as a fine #4. I even think we can come up with interface options that allow us to represent future queries without coding. For example, Roger's query above could just be (assuming a command-line interface):
$ ./doctor --download $ ./doctor --flag moria1:Running --flag maatuska:Running --noflag consensus:Running --count
All in all, I think your #4 is a fine idea. I'm curious which interface will be preferred by future users.
All the best, Karsten
I think Roger uses it in his role as network debugger, not as authority operator. There may be more people wanting to help with network debugging who are not authority operators. And there may be expert relay operators who are wondering why their relay isn't listed or isn't listed with a given flag.
I would be really surprised if relay operators, even expert ones, find the table useful. That said, on irc Roger mentioned that he finds the table handy for finding trends he previously didn't know to look for. I suspect he's the only person using it for that, but it's both a reasonable use case and one that's frustratingly tricky to provide a good solution for that's also useful to others...
A command-line tool doesn't sound too crazy. That tool (or suite of scripts) should be able to cache downloads and votes and share them among all queries we might want to ask it.
Agreed, this is a capability I'd *love* to have...
https://trac.torproject.org/projects/tor/ticket/9351
One bit I'm not sure how to address is cache invalidation. We can check when a descriptor has expired, but not when it is no longer the freshest.
$ ./doctor --download $ ./doctor --flag moria1:Running --flag maatuska:Running --noflag consensus:Running --count
I think you're misunderstanding my suggestion. My proposal is to *teach* Roger to write or adapt scripts to solve his questions. This would both empower him to answer more interesting questions than he presently can and, in the long run, save him time. I would be more than happy to write the first dozen, which he could then use to both learn stem and adapt to answer similar questions.
I'm not proposing a new tool. Rather, I'm suggesting that I teach a man to fish. ;)
Cheers! -Damian
On 9/25/13 5:45 AM, Damian Johnson wrote:
I think Roger uses it in his role as network debugger, not as authority operator. There may be more people wanting to help with network debugging who are not authority operators. And there may be expert relay operators who are wondering why their relay isn't listed or isn't listed with a given flag.
I would be really surprised if relay operators, even expert ones, find the table useful.
I disagree. It's useful information to know if some authorities don't assign flags or assign extra flags to a relay. That can help explain why the relay sometimes gets the flag if some of the authorities don't vote. Or in case of Named/Unnamed and Exit/BadExit, the flags contained in votes can help explain why a relay gets the Named or Exit flag. There may be more examples. Sure, this is like debug information, and not many relay operators will understand what's going on there. But saying that the information contained in the table is not useful is not correct.
That said, on irc Roger mentioned that he finds the table handy for finding trends he previously didn't know to look for. I suspect he's the only person using it for that, but it's both a reasonable use case and one that's frustratingly tricky to provide a good solution for that's also useful to others...
I have to say that I don't fully understand how skimming through the table provides an even basic understanding of trends. The table that shows the overlap between votes and consensuses would be a better start to discover trends:
https://metrics.torproject.org/consensus-health.html#overlap
(I didn't mention this, but I'd expect a replacement tool, CLI or web-based, to provide such a summary, too.)
A command-line tool doesn't sound too crazy. That tool (or suite of scripts) should be able to cache downloads and votes and share them among all queries we might want to ask it.
Agreed, this is a capability I'd *love* to have...
https://trac.torproject.org/projects/tor/ticket/9351
One bit I'm not sure how to address is cache invalidation. We can check when a descriptor has expired, but not when it is no longer the freshest.
Consensuses and votes contain these lines:
valid-after 2013-09-25 13:00:00 fresh-until 2013-09-25 14:00:00 valid-until 2013-09-25 16:00:00
You could invalidate consensuses and votes after fresh-until. Checking when a server descriptor or extra-info descriptor is not used anymore is slightly more difficult and probably requires looking at dir-spec.txt or the code. (We can move this discussion to the ticket. Feel free to paste this there if it's useful.)
$ ./doctor --download $ ./doctor --flag moria1:Running --flag maatuska:Running --noflag consensus:Running --count
I think you're misunderstanding my suggestion. My proposal is to *teach* Roger to write or adapt scripts to solve his questions. This would both empower him to answer more interesting questions than he presently can and, in the long run, save him time. I would be more than happy to write the first dozen, which he could then use to both learn stem and adapt to answer similar questions.
I'm not proposing a new tool. Rather, I'm suggesting that I teach a man to fish. ;)
Not sure if that's going to swim. Is it really a good use of Roger's time to write or adapt Python scripts? I guess he can answer that better than I (though I'm not sure it's a good use of his time to follow this thread ;)). However, we shouldn't be developing this tool just for Roger, because hopefully other people care about debugging the network, too. Just imagine what Tor devs would want to use whenever somebody shows up on #tor and asks why their relay doesn't have this or that flag, and imagine what's the easiest way to share results with the relay operator.
Thanks for your efforts here!
All the best, Karsten
I disagree. It's useful information to know if some authorities don't assign flags or assign extra flags to a relay. That can help explain why the relay sometimes gets the flag if some of the authorities don't vote. Or in case of Named/Unnamed and Exit/BadExit, the flags contained in votes can help explain why a relay gets the Named or Exit flag. There may be more examples. Sure, this is like debug information, and not many relay operators will understand what's going on there. But saying that the information contained in the table is not useful is not correct.
If we were to surface flag votes on a one-off basis then we should do so in Atlas. Actually, it would be pretty slick if hovering your mouse over the flags showed the votes for each authority. I still disagree that any except a vanishingly small group will find this useful, but if this is a use case we want to support then Atlas is the spot.
Consensuses and votes contain these lines:
valid-after 2013-09-25 13:00:00 fresh-until 2013-09-25 14:00:00 valid-until 2013-09-25 16:00:00
Thanks! Added a note to the ticket. It'll be a while before I can implement it, but definitely on my list of things I'd like to do.
Not sure if that's going to swim. Is it really a good use of Roger's time to write or adapt Python scripts? I guess he can answer that better than I (though I'm not sure it's a good use of his time to follow this thread ;)). However, we shouldn't be developing this tool just for Roger, because hopefully other people care about debugging the network, too. Just imagine what Tor devs would want to use whenever somebody shows up on #tor and asks why their relay doesn't have this or that flag, and imagine what's the easiest way to share results with the relay operator.
Is debugging the tor network a good use of Roger's time? I can't answer that. That said, if Roger is trying to answer exotic questions like "i wanted to know how many relays had a Running flag from those two authorities and no Running flag from any others" then I can tell you that scripting is the right answer. Not a huge table.
Learning to script both empowers Roger to solve harder problems and do so more efficiently. If he wants to answer the questions like the one above then he should either learn to script or ask me to do it.
Cheers! -Damian
On 9/25/13 6:43 PM, Damian Johnson wrote:
I disagree. It's useful information to know if some authorities don't assign flags or assign extra flags to a relay. That can help explain why the relay sometimes gets the flag if some of the authorities don't vote. Or in case of Named/Unnamed and Exit/BadExit, the flags contained in votes can help explain why a relay gets the Named or Exit flag. There may be more examples. Sure, this is like debug information, and not many relay operators will understand what's going on there. But saying that the information contained in the table is not useful is not correct.
If we were to surface flag votes on a one-off basis then we should do so in Atlas. Actually, it would be pretty slick if hovering your mouse over the flags showed the votes for each authority. I still disagree that any except a vanishingly small group will find this useful, but if this is a use case we want to support then Atlas is the spot.
Moved this discussion to ticket #9778:
https://trac.torproject.org/projects/tor/ticket/9778#comment:5
Consensuses and votes contain these lines:
valid-after 2013-09-25 13:00:00 fresh-until 2013-09-25 14:00:00 valid-until 2013-09-25 16:00:00
Thanks! Added a note to the ticket. It'll be a while before I can implement it, but definitely on my list of things I'd like to do.
Sounds good.
Not sure if that's going to swim. Is it really a good use of Roger's time to write or adapt Python scripts? I guess he can answer that better than I (though I'm not sure it's a good use of his time to follow this thread ;)). However, we shouldn't be developing this tool just for Roger, because hopefully other people care about debugging the network, too. Just imagine what Tor devs would want to use whenever somebody shows up on #tor and asks why their relay doesn't have this or that flag, and imagine what's the easiest way to share results with the relay operator.
Is debugging the tor network a good use of Roger's time? I can't answer that. That said, if Roger is trying to answer exotic questions like "i wanted to know how many relays had a Running flag from those two authorities and no Running flag from any others" then I can tell you that scripting is the right answer. Not a huge table.
Learning to script both empowers Roger to solve harder problems and do so more efficiently. If he wants to answer the questions like the one above then he should either learn to script or ask me to do it.
Works for me, in particular if you're going to be around to help with the scripting.
Roger, does this work for you, too?
All the best, Karsten
-
Damian Johnson -
Karsten Loesing