[tor-dev] Get Stem and zoossh to talk to each other
phw at nymity.ch
Mon Aug 17 16:07:51 UTC 2015
On Sun, Aug 16, 2015 at 02:44:40PM -0700, Damian Johnson wrote:
> >> > Ideally, zoossh should do the heavy lifting as it's implemented in a
> >> > compiled language.
> >> This is assuming zoossh is dramatically faster than Stem by virtue of being
> >> compiled. I know we've discussed this before but I forget the results - with
> >> the latest tip of Stem (ie, with lazy loading) how do they compare? I'd expect
> >> time to be mostly bound by disk IO, so little to no difference.
> > zoossh's test framework says that it takes 36364357 nanoseconds to
> > lazily parse a consensus that is cached in memory (to eliminate the I/O
> > bottleneck). That amounts to approximately 27 consensuses a second.
> > I used the following simple Python script to get a similar number for
> > Stem:
> > with open(file_name) as consensus_file:
> > for router in stem.descriptor.parse_file(consensus_file,
> > 'network-status-consensus-3 1.0',
> > document_handler = stem.descriptor.DocumentHandler.ENTRIES):
> > pass
> > This script manages to parse 24 consensus files in ~13 seconds, which
> > amounts to 1.8 consensuses a second. Let me know if there's a more
> > efficient way to do this in Stem.
> Interesting! First thought is 'wonder if zoossh is even reading the
> file content'. Couple quick things to try are...
> with open(file_name) as consensus_file:
Disk IO is negligible for both tests because the file content is cached
in memory. As expected, consensus_file.read() terminates almost
FWIW, zoossh doesn't parse as much as Stem does, so it's not quite an
apple-to-apple comparison. For example, exit policies are not parsed
and simply stored as strings for now.
More information about the tor-dev