zoossh's test framework says that it takes 36364357 nanoseconds to lazily parse a consensus that is cached in memory (to eliminate the I/O bottleneck). That amounts to approximately 27 consensuses a second.
I used the following simple Python script to get a similar number for [...] This script manages to parse 24 consensus files in ~13 seconds, which amounts to 1.8 consensuses a second. Let me know if there's a more efficient way to do this in Stem.
Interesting! First thought is 'wonder if zoossh is even reading the file content'. Couple quick things to try are...
with open(file_name) as consensus_file: consensus_file.read()
... to see how much time is disk IO verses parsing. Second is to try doing something practical (say, count the number of relays with the exit flag). Stem does some bytes => unicode normalization which might account for some difference but other than that I'm at a loss for what would be taking the time.
Why are you surprised? Python is a very, very slow language. My very simple unoptimized C++ parsing code is more than 10x faster than Stem (parsing from memory). Go has some nice string parsing features, so zoossh should have an easy time to get fast and simple parsing code.
Why are you surprised? Python is a very, very slow language. My very simple unoptimized C++ parsing code is more than 10x faster than Stem (parsing from memory). Go has some nice string parsing features, so zoossh should have an easy time to get fast and simple parsing code.
Because with either implementation (Stem or Zoossh) I expected disk IO to be the lion's share of the time. Both do lazy parsing so the earlier results didn't make sense. Being cached in memory this makes more sense.
One other thought is that the DocumentHandler.ENTRIES is telling Stem to read the content to provide you with individual RouterStatusEntry instances. If zoossh isn't parsing out individual entries then that may skew the comparison. Reading server descriptors would be a better test in this regard.
Cheers! -Damian