I now have similar code which is based on stem: https://github.com/NullHypothesis/exitmap
Hi Philipp, sorry about the long delay before responding. ExitMap looks great!
You might want to look into PEP8 [1], Python's de-facto style guide. It's certainly up to you which bits you do/don't like, but coming close will make your code more uniform with the rest of the Python world. PyPI has a slick pep8 script you can run over your codebase [2]. Personally I run this as part of the tests for Stem.
Would you be amenable to changes in that regard from me? This looks like a fun project, so I'm a bit tempted to sink a weekend or two into into seeing if I can simplify the codebase.
Some general code review feedback follows...
======================================== exitmap / circuit.py ========================================
new = Circuit
Not especially pythonic, but works.
======================================== exitmap / circuitpool.py ========================================
while (circuitID is None):
Parentheses aren't necessary.
if len(self.exitRelays) == 0:
Can also be 'if not self.exitRelays:'.
try: circuitID = self.ctrl.new_circuit([const.FIRST_HOP, exitRelay], await_build=awaitBuild) except (stem.InvalidRequest, stem.CircuitExtensionFailed, stem.ControllerError) as error:
Stem's InvalidRequest and CircuitExtensionFailed extend ControllerError, so this can be simplified to...
except stem.ControllerError as error:
Stem's exception hierarchy can be found on...
https://stem.torproject.org/api/control.html#exceptions-and-attribute-enums
Your _addCircuit() is soley used by _fillPool(), so personally I'd probably do this as a generator...
def _generate_circuit(self, await_build): """ Pops the top relay off our exitRelays and generates a circuit through it, going on to the next entry if it fails.
:param bool await_build: block until the circuit is created if **True**
:returns: :class:`~circuit.Circuit` for the circuit we establish """
while self.exitRelays: exit_fingerprint = self.exitRelays.pop(0)
logger.debug("Attempting to create circuit with '%s' as exit " \ "relay." % exit_fingerprint)
try: circuit_id = self.ctrl.new_circuit( [const.FIRST_HOP, exit_fingerprint], await_build = await_build, )
logger.debug("Created circuit #%s with '%s' as exit relay." % (circuit_id, exit_fingerprint))
yield circuit.Circuit(circuit_id) except stem.ControllerError as exc: logger.warning("Could not establish circuit with '%s'. " \ "Skipping to next exit (error=%s)." % (exit_fingerprint, exc))
logger.warning("No more exit relay fingerprints to create circuits with.")
def _fill_pool(self, await_build = False): if len(self.pool) == const.CIRCUIT_POOL_SIZE: return
logger.debug("Attempting to refill the circuit pool to size %d." % const.CIRCUIT_POOL_SIZE)
while self.exitRelays and len(self.pool) != const.CIRCUIT_POOL_SIZE: self.pool.append(self._generate_circuit(awaitBuild))
# go over circuit pool once poolLen = len(self.pool) for idx in xrange(poolLen):
if idx >= len(self.pool): return None logger.debug("idx=%d, poolsize=%d" % (idx, len(self.pool))) circuit = self.pool[idx] # that's a Circuit() object
Honestly I'm finding this class to be overcomplicated. Popping and appending items between lists is making this more confusing than it needs to be.
======================================== exitmap / command.py ========================================
Stem offers a friendlier method of calling commands. Mostly I intended it for the system module's functions, but you might find it useful here too...
https://stem.torproject.org/api/util/system.html#stem.util.system.call
======================================== exitmap / exitselector.py ========================================
for desc in stem.descriptor.parse_file(open(consensus)):
Why read the consensus file directly? If you have a controller then getting it via tor would be the best option. If not then fetching this directly via the authorities is generally the easiest...
https://stem.torproject.org/api/descriptor/remote.html
if not "Exit" in desc.flags: continue
Perfectly fine, though stem does offer enums...
if stem.Flag.EXIT not in desc.flags: continue
for (ip, port) in hosts: if not desc.exit_policy.can_exit_to(ip, port): continue
I don't think this'll actually work. The 'continue' will be for the iteration over hosts.
======================================== exitmap / scanner.py ========================================
stem.connection.authenticate_none(torCtrl)
There is no point in doing this unless you *only* want it to work without authentication. If you opt for authenticate() instead then this will also work for cookie auth.
Cheers! -Damian
[1] http://www.python.org/dev/peps/pep-0008/ [2] https://pypi.python.org/pypi/pep8