[tor-bugs] #5232 [BridgeDB]: Import bridges into BridgeDB in a separate thread and database transaction

[Tor Bug Tracker & Wiki]

#5232: Import bridges into BridgeDB in a separate thread and database transaction
-------------------------+-------------------------------------------------
     Reporter:  karsten  |      Owner:  sysrqb
         Type:  defect   |     Status:  needs_revision
     Priority:  major    |  Milestone:
    Component:           |    Version:
  BridgeDB               |   Keywords:  bridgedb-email, bridgedb-db,
   Resolution:           |  bridgedb-https, bridgedb-0.1.x
Actual Points:           |  Parent ID:
       Points:           |
-------------------------+-------------------------------------------------
Changes (by isis):

 * keywords:  important => bridgedb-email, bridgedb-db, bridgedb-https,
     bridgedb-0.1.x
 * cc: sysrqb (added)
 * status:  needs_review => needs_revision


Comment:

 Sweet. I had to deal with a bit of merge conflicts to get it into
 master... do you mind if I separate the additions to the unittest in
 2a21dfcb55e659775fcde9dd4f668b98f41d0fd6 into another unittest? If I do
 it, then you won't have to deal with the merge conflicts too.

 So, this seems to work great, the parsing is done in a separate thread!
 However, the call which takes longer, especially at start up time, is the
 call to `bridgedb.Stability.addOrUpdateBridgeHistory()`. However, after
 start up, the HTTPS distributor continues to function and hand out bridges
 while the new descriptors are being parsed.

 For 10,000 bridge descriptors, with `addOrUpdateBridges()`:
 {{{
  * Starting the servers took:      1h 6m 58s
  * Restarting (SIGHUP) took:          2m 13s
  * Dumping buckets (SIGUSR1) took:       11s
 }}}

 Same test, only 1,000 descriptors:
 {{{
  * Starting the servers took:         3m 02s
  * Restarting (SIGHUP) took:              3s
  * Dumping buckets (SIGUSR1) took:        1s
 }}}

 However, this breaks the email distributor. It doesn't work (at any time)
 because it's still trying to use an old cursor. This error is unhandled,
 and it gets sent to stdout rather than the logfile:
 {{{
 Unhandled Error
 Traceback (most recent call last):
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/twisted/python/log.py", line 88, in callWithLogger
     return callWithContext({"system": lp}, func, *args, **kw)
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/twisted/python/log.py", line 73, in callWithContext
     return context.call({ILogContext: newCtx}, func, *args, **kw)
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/twisted/python/context.py", line 118, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/twisted/python/context.py", line 81, in callWithContext
     return func(*args,**kw)
 --- <exception caught here> ---
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/twisted/internet/posixbase.py", line 601, in _doReadOrWrite
     why = selectable.doRead()
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/twisted/internet/tcp.py", line 215, in doRead
     return self._dataReceived(data)
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/twisted/internet/tcp.py", line 221, in _dataReceived
     rval = self.protocol.dataReceived(data)
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/twisted/protocols/basic.py", line 454, in dataReceived
     self.lineReceived(line)
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/twisted/mail/smtp.py", line 568, in lineReceived
     return getattr(self, 'state_' + self.mode)(line)
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/twisted/mail/smtp.py", line 795, in dataLineReceived
     m.eomReceived() for m in self.__messages
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/bridgedb-0.1.4_34_gc253a64-py2.7.egg/bridgedb/EmailServer.py",
 line 336, in eomReceived
     replyToMail(self.lines, self.ctx)
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/bridgedb-0.1.4_34_gc253a64-py2.7.egg/bridgedb/EmailServer.py",
 line 243, in replyToMail
     sendToUser, response = getMailResponse(lines, ctx)
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/bridgedb-0.1.4_34_gc253a64-py2.7.egg/bridgedb/EmailServer.py",
 line 175, in getMailResponse
     bridgeFilterRules=bridgeFilterRules)
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/bridgedb-0.1.4_34_gc253a64-py2.7.egg/bridgedb/Dist.py", line 471,
 in getBridgesForEmail
     wasWarned = db.getWarnedEmail(emailaddress)
   File "/home/isis/.virtualenvs/bridgedb/local/lib/python2.7/site-
 packages/bridgedb-0.1.4_34_gc253a64-py2.7.egg/bridgedb/Storage.py", line
 365, in getWarnedEmail
     " email = ?", (addr,))
 sqlite3.ProgrammingError: Cannot operate on a closed cursor.
 }}}

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/5232#comment:15>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online