[tor-bugs] #27416 [Core Tor/Tor]: Improve descriptor validation in Tor using Stem

[Tor Bug Tracker & Wiki]

#27416: Improve descriptor validation in Tor using Stem
------------------------------+------------------------------
     Reporter:  teor          |      Owner:  (none)
         Type:  defect        |     Status:  new
     Priority:  Medium        |  Milestone:  Tor: unspecified
    Component:  Core Tor/Tor  |    Version:
     Severity:  Normal        |   Keywords:  tor-dirauth
Actual Points:                |  Parent ID:
       Points:                |   Reviewer:
      Sponsor:                |
------------------------------+------------------------------
 Continuing from
 https://trac.torproject.org/projects/tor/ticket/27367?replyto=14#comment:14

 Replying to [comment:14 atagar]:
 > Related a few past tickets where this has bitten us.

 I moved these tickets to #27414, because relays should do the UTF-8 check
 before they upload.

 But the UTF-8 check won't be enough to catch all parsing errors, so I
 opened #27415 so that relays parse descriptors before they upload them.

 But the parsing check *still* won't be enough to catch all spec-
 implementation divergences. And that's ok. Sometimes we fix the spec,
 sometimes with fix tor, and sometimes we fix other parsers.

 > Just a quick note that dirauths could use Stem as a tool for rejecting
 malformed content. It does stricter validation than the tor binary that
 descriptors are conformant with the spec. I've been performing this check
 through DocTor since 2013, filing tickets each time more bad data makes it
 into the consensus...
 >
 > https://gitweb.torproject.org/doctor.git/tree/descriptor_checker.py
 >
 > Bad data chokes not only Stem, but metrics-lib and anything else that
 ingests it.
 >
 > Clearly in an ideal world the tor binary itself would do better
 validation

 See the tickets I opened and listed above. I think that's a reasonable
 amount of validation for the tor binary to do.

 In an ideal world, all implementations and the spec would match. When
 there's a difference, we need to decide what to fix. That's not something
 we can automate.

 > but in the absence of that if we took advantage of Stem's validator I
 wouldn't need to keep filing tickets every few months. Using Stem on
 dirauths to reject malformed descriptors would prevent these issues
 upfront, saving Karsten and I hassle.

 Running stem as an essential part of dirauth operation is a significant
 change in tor's security model. It would require a proposal.

 It's also a significant change to the consistency of tor's parsing. At the
 moment, tor uses the same parser and validator for:
 * descriptors via authorities
 * descriptors direct to bridge clients
 * onion service descriptors

 > If that's a no-go I could also redirect the DocTor check I mention above
 to email other folks (Nick? teor? Maybe the network team list?) so I don't
 need to file tickets each time this comes up.

 The network team list is administrative and has a closed membership. It's
 not designed for automatic mailing.

 If the checks are safe to make public, then send them to tor-consensus-
 health, or create another list. (They should be safe, because the
 validation code is public.)

 Whatever you decide, let's adopt the same strategy for the fallback
 checks. We should extend them beyond just nick and me.

 And if there are any other DocTor checks that would benefit from being
 extended beyond a few people, let's do them at the same time.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/27416>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online