Anshul Singhle anshul.singhle@gmail.com writes:

The aim of this project would be to use the protocol-parsing from stem instead of using the "re-implementation" in txtorcon. Since the protocol parsing is synchronous anyway(AFAICT) it makes sense to do this.

So far so good, yes.

One thing to note is that while "most" of the parsing is basically synchronous in txtorcon, some of the replies are huge. The one we care about being "GETINFO ns/all" which gives TorState the initial list of Router objects; there are ~3000 routers, so around 4*3000 lines to parse and that's way too long to pause the reactor. So there is some incremental parsing stuff in txtorcon (a state machine in TorState), and this would likely be the "some modifications to stem" bit: to provide a way to feed some stem object lines of a consensus and get objects back out representing the routers (which would be given to txtorcon.Router instead of the current way of passing some strings to update() in router.py).

Therefore, it would also make sense to identify other synchronous activities being done by txtorcon and use Stem to do those too (since we will be instantiating a stem object anyway).

Yes, very possibly!

So as I see it, the majority of the changes will be in torcontrolprotocol.py(https://github.com/meejah/txtorcon/blob/master/txtorcon/torcontrolprotocol.p...) and in that in the TorControlProtocol and TorProtocolError parts(mainly)

Yes. There will hopefully be some changes in TorState, to do with the Router information I mention above (TorState keeps the list of Router objects, which represent the Tor relays in the consensus).

These classes have some functions that are used by others and some which are for internal use(i guess the ones with _ are internal if I got the naming convention right).

Yes.

So we will just throw away the internal functions(should we?) and keep the external function api the same, the difference being these external functions will now call stem instead of doing the heavy lifting themselves.

Yes.

I have a question about LineOnlyReciever - Are there other types of receivers which txtorcon doesn't implement but stem does?

LineOnlyReceiver is a Twisted thing; it handles buffering etcetera and delivers one line at a time to TorControlProtocol (via lineRecieved). It could be that a different superclass is appropriate. Stem uses the socket stuff directly, in a synchronous (threaded) way, so I wouldn't expect any sharing to happen here.

Aside: note that the camelCase methods are all Twisted overrides and the_underscore_ones are txtorcon's.

So in essence i guess my questions is - What is the primary focus of this project - Is it to bring the protocol specific code in one place or to use Stem wherever possible? or both?

The main thrust of the project is to have one Python implmentation of the protocol parsing code -- namely Stem. The reasoning here being that Stem already parses a lot more things than txtorcon anyway (and I have no intensions of re-implementing all that) and so it makes sense to leverage that in txtorcon. From atagar's perspective, he'd like more people exercising Stem code and we agree it's somewhat silly to have two Python implementations of this, even if some of it is rather simple.

If you can identify other things that make sense for txtorcon to use from Stem, that's bonus. I would say, however, that I would forsee any users who want pieces of functionality from Stem to "just use Stem". That is, I don't want to *add* an API in txtorcon for loading + parsing consensus files, etc. -- users who want that should just use Stem (since there's no Python/Twisted async file APIs anyway). A good example of this is the Twisted version of torperf.

By the same token, there will very likely be users of txtorcon who don't care about stem functionality so txtorcon users shouldn't be *forced* to learn about stem (installing it as a dependency is fine).

I'm getting at the event stuff here: I would like it to remain optional to receive the event text itself versus a stem RouterStatusEntryV3 instance in an event callback (i.e. one registered via TorControlProtocol.add_event_listener). The Twisted version of torperf would make a fine place to "test out" this event stuff somewhere that will actually be used/released (e.g. instead of another example).

Super extra bonus points if we can a) add a zope.interface for the callback (it lacks one currently) and b) use adaptors to figure out which kind of object the listener wants. I haven't thought about this, really, but could be fun (or not possible).

Thanks for the interest, meejah