[tor-bugs] #33871 [Core Tor/sbws]: Scale exactly as torflow does?

[Tor Bug Tracker & Wiki]

#33871: Scale exactly as torflow does?
-------------------------------------------------+-------------------------
 Reporter:  juga                                 |          Owner:  juga
     Type:  defect                               |         Status:
                                                 |  assigned
 Priority:  Medium                               |      Milestone:  sbws:
                                                 |  1.1.x-final
Component:  Core Tor/sbws                        |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:  sbws-majority-blocker, sbws-roadmap  |  Actual Points:
Parent ID:  #33775                               |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------

Comment (by teor):

 I agree with what Juga said, but I'm leaving these notes here, in case you
 need them in future.

 Replying to [comment:11 mikeperry]:
 > Torflow *also* uses Tor 0.2.9, which may have different descriptor vs
 microdescriptor fetching behavior.

 Later tor versions might also have different descriptor caching behaviour.
 And there could also be bugs in how sbws gets descriptors from tor, or how
 it handles relays throughout the process.

 But if it turns out that there are still descriptor issues, we can
 diagnose them by comparing:
 * the descriptors downloaded from an authority at "tor/server/all.z" using
 stem: https://github.com/torproject/torspec/blob/master/dir-spec.txt#L4058
 * the descriptors that the local tor instance has
 * the descriptors that sbws is getting from tor, trying to measure,
 successfully measuring, and outputting in the bandwidth file

 Unfortunately, we lose some descriptors at each stage, so the analysis
 won't be easy.

 > I think these are more likely culprits for this specific ticket.
 >
 > Here is the relevant torrc piece, which I think may behave differently
 with Tor 0.2.9:
 > {{{
 > FetchUselessDescriptors 1
 > # Workaround for Tor #24110, tracked in TorFlow #24094
 > UseMicrodescriptors 0
 > __LeaveStreamsUnattached 1
 >
 > # Bad idea? Too much consensus update activity?
 > FetchDirInfoExtraEarly 1
 > FetchDirInfoEarly 1
 > }}}

 As juga said, sbws sets similar options:
 https://github.com/torproject/sbws/blob/master/sbws/globals.py#L20

 For 0.4.0 and later, sbws might also want to set:
 {{{
 DormantClientTimeout 7 days
 DormantCanceledByStartup 1
 }}}
 So that a network outage or temporary sbws hang won't cause tor to stop
 downloading new descriptors.

 But that's unlikely to be the source of your problem, because the default
 DormantClientTimeout is 24 hours, which is a long network outage.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/33871#comment:13>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online