[tor-bugs] #14011 [Stem]: Implement lazy parsing for stem (was: Implement lazy parsing for zoossh (and maybe Stem))

[Tor Bug Tracker & Wiki]

#14011: Implement lazy parsing for stem
-----------------------------+--------------------------------
     Reporter:  phw          |      Owner:  phw
         Type:  enhancement  |     Status:  new
     Priority:  major        |  Milestone:
    Component:  Stem         |    Version:
   Resolution:               |   Keywords:  descriptor, zoossh
Actual Points:               |  Parent ID:
       Points:               |
-----------------------------+--------------------------------

Comment (by atagar):

 Did a proof of concept of this...

 https://gitweb.torproject.org/user/atagar/stem.git/log/?h=lazyload

 It improved the speed at which we can read server descriptors by ~20% (the
 high parse times with validation is mostly due to cryptographic
 validation)...

 {{{
 Before:

 % python scrap.py
 read 6679 descriptors with validation, took 13.67 seconds
 read 6679 descriptors without validation, took 5.47 seconds

 After:

 % python scrap.py
 read 6679 descriptors with validation, took 13.77 seconds
 read 6679 descriptors without validation, took 4.41 seconds
 }}}

 Script used to test...

 {{{
 import time

 from stem.descriptor import parse_file

 start_time = time.time()
 fingerprints = []

 for desc in parse_file('/home/atagar/.tor/cached-descriptors', validate =
 True):
   fingerprints.append(desc.fingerprint)

 print 'read %i descriptors with validation, took %0.2f seconds' %
 (len(fingerprints), time.time() - start_time)

 start_time = time.time()
 fingerprints = []

 for desc in parse_file('/home/atagar/.tor/cached-descriptors', validate =
 False):
   fingerprints.append(desc.fingerprint)

 print 'read %i descriptors without validation, took %0.2f seconds' %
 (len(fingerprints), time.time() - start_time)
 }}}

 We'll probably see a similar ~20% improvement for other descriptor types.
 There's no point in having a zoossh ticket in the stem component, so just
 having this be about this change.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/14011#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online