[tor-bugs] #28563 [Core Tor/sbws]: Work out how sbws can report excluded relays in the bandwidth file

[Tor Bug Tracker & Wiki]

#28563: Work out how sbws can report excluded relays in the bandwidth file
---------------------------+-------------------------------------
 Reporter:  teor           |          Owner:  (none)
     Type:  defect         |         Status:  new
 Priority:  Medium         |      Milestone:  sbws 1.0 (MVP must)
Component:  Core Tor/sbws  |        Version:
 Severity:  Normal         |     Resolution:
 Keywords:  tor-bwauth     |  Actual Points:
Parent ID:  #28547         |         Points:
 Reviewer:                 |        Sponsor:
---------------------------+-------------------------------------

Comment (by teor):

 Replying to [comment:1 juga]:
 > I see some inconvenients to do this:
 > - once we figure out why are being relays excluded, we might not want to
 keep the same format.

 I'm not sure what you mean here.
 Do you think we'll change the bandwidth file format?
 I expect that we'll add extra keys to the relay lines that count the
 relays excluded at each stage. Then we will add more keys for the stages
 that exclude a lot of relays.

 > - we need to wait until longclaw update to the code that publish the
 files

 Or you or micah can sync the file to a public web server, like
 people.torproject.org.
 Most of the other directory authority operators sync their bandwidth files
 somewhere public.

 > - it'd add like around 1000 extra relays with some extra data, though
 this might not be a problem.

 I don't think it's a problem.

 > - the delay that implies creating the spec before

 I'm not sure what you mean here.
 Are you concerned that the spec will take too much time?
 It is ok to try a few things in the code, then update the spec.
 And I can work on the spec next week.

 > I was thinking either on something temporal:
 > 1. produce a different file, with the relays excluded and useful data*
 > 2. implement other script to dump the data to a DB. It sounds kind of
 crazy, but it might not be much work and from that it's easier to make
 queries
 > While currently it's only me accessing to the original data, i can
 publish the results of that.
 >
 > *Currently relays excluded, can be because:
 > - circuits timeout, this is already in the raw results file
 > - when scaling, doesn't find 2 measurements that are at least 1 day away
 and 5 days recent
 > - something else we don't know yet

 Let's make a pad and list all the reasons from the code:
 https://pad.riseup.net/p/sbws-exclude-reasons-keep

 I think there are a lot more, see the children of #28547.

 > What about if i try one of the two other approaches before?. Otherwise
 i'm fine with this.

 Relay operators, authority operators, and developers need to be able to
 find out why a relay isn't being measured.

 If we put that information in the bandwidth file, authorities serve the
 bandwidth file, and metrics archives it, then the information is public
 and available.

 Your other options are good for you to do a quick analysis. But they do
 not help everyone else.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/28563#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online