[tor-bugs] #10796 [BridgeDB]: Bridgedb became unresponsive

[Tor Bug Tracker & Wiki]

#10796: Bridgedb  became unresponsive
--------------------------+----------------------
     Reporter:  sysrqb    |      Owner:  isis
         Type:  defect    |     Status:  accepted
     Priority:  normal    |  Milestone:
    Component:  BridgeDB  |    Version:
   Resolution:            |   Keywords:
Actual Points:            |  Parent ID:
       Points:            |
--------------------------+----------------------
Changes (by isis):

 * owner:   => isis
 * status:  new => accepted


Comment:

 The old BridgeDB was started via the `run-bridgedb` script, which did:

 {{{
 PYTHONPATH=$PYTHONPATH python -m TorBridgeDB \
   -c bridgedb.conf < /dev/null > /dev/null 2>&1 & disown
 }}}

 Which meant that unhandled exceptions in the templating system were
 displayed to the client (#6127) and that any other unhandled exceptions in
 the main reactor loop were piped to `/dev/null`.

 I think I've handled catching any templating error in my branches
 [https://gitweb.torproject.org/user/isis/bridgedb.git/shortlog/refs/heads/fix/6127
 -web-server-tracebacks fix/6127-web-server-tracebacks],
 [https://gitweb.torproject.org/user/isis/bridgedb.git/shortlog/refs/heads/fix/6127
 -render_GET-traceback fix/6127-render_GET-traceback], and
 [https://gitweb.torproject.org/user/isis/bridgedb.git/shortlog/refs/heads/fix/6127
 -simple-error-page fix/6127-simple-error-page]. Though there may still be
 more exceptions that I couldn't foresee.

 We need better exception handling in general in BridgeDB; I've generally
 been working on it at the same time as other tickets, adding catches and
 actions for previously unhandled errors when I find them. We could
 probably modify the above shell script to pipe unhandled errors to another
 file, but I'd rather fix it the real way and also have better tests. Maybe
 a fuzzer would be nice to try to dig these unhandled exceptions out.

 As far as BridgeDB being unresponsive, I don't know. I looked at the
 machine and did some brief poking around to try to assess if something had
 caused this. Nothing I could find. The error was probably piped to
 `/dev/null`, like all the other ones. :(

 To fix it, I redeployed
 [https://gitweb.torproject.org/user/isis/bridgedb.git/tag/3d23096734384938f5aed7fb1b601c8303f80f52
 BridgeDB-0.1.0], but after 55 minutes of BridgeDB running through the
 `addOrUpdateBridgeHistory()` code in `bridgedb.Storage` I realised that if
 BridgeDB took 55 minutes to process descriptors after every SIGHUP, it
 would never come online (see #5232). Then I decided to quickly tag and
 deploy
 [https://gitweb.torproject.org/user/isis/bridgedb.git/tag/446bc967442c2c7bbac23b50e709058c7f502c3f
 BridgeDB-0.1.1], without staging it first, because it contains the patches
 for #10724.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/10796#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online