[or-cvs] r21772: {arm} Hiding client/exit information to address privacy concerns a (in arm/trunk: . init interface)
Damian Johnson
atagar1 at gmail.com
Sun Feb 28 02:48:24 UTC 2010
Author: atagar
Date: 2010-02-28 02:48:24 +0000 (Sun, 28 Feb 2010)
New Revision: 21772
Added:
arm/trunk/init/prereq.py
Removed:
arm/trunk/init/versionCheck.py
Modified:
arm/trunk/ChangeLog
arm/trunk/README
arm/trunk/TODO
arm/trunk/arm
arm/trunk/init/__init__.py
arm/trunk/init/starter.py
arm/trunk/interface/__init__.py
arm/trunk/interface/bandwidthMonitor.py
arm/trunk/interface/confPanel.py
arm/trunk/interface/connPanel.py
arm/trunk/interface/controller.py
arm/trunk/interface/headerPanel.py
Log:
Hiding client/exit information to address privacy concerns and fixes for numerous issues brought up in irc.
added: scrubbing connection details of possible client and exit connections
change: providing file descriptions in README, updated known issues and future plans in TODO
change: added precision for bandwidth cap and burst if uneven values (requested by mete1989)
fix: HiddenService* parameters fetched via a special option (caught by dun, karsten, and grumpy3)
fix: workaround for os specific torrc validation bug - unfortunately haven't managed to repro yet so no fix (caught by grumpy3, Tas, and dun)
fix: checking for python curses bindings at startup (caught by dun)
fix: import error - TorCtl and socket missing from confPanel.py (caught by grumpy3)
fix: showing external ip in connection panel rather than local nat address (caught by mete1989)
fix: raised minimum width at which graph stats are displayed beside label (caught by dun)
fix: wasn't treating "accept *" and "reject *" as catch-all policies
fix: wasn't resizing graph panel properly in case of a sighup
Modified: arm/trunk/ChangeLog
===================================================================
--- arm/trunk/ChangeLog 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/ChangeLog 2010-02-28 02:48:24 UTC (rev 21772)
@@ -1,10 +1,25 @@
CHANGE LOG
-2/14/10 - version 1.3.2
+2/27/10 - version 1.3.3
+Hiding client/exit information to address privacy concerns and fixes for numerous issues brought up in irc.
+
+ * added: scrubbing connection details of possible client and exit connections
+ * change: providing file descriptions in README, updated known issues and future plans in TODO
+ * change: added precision for bandwidth cap and burst if uneven values (requested by mete1989)
+ * fix: HiddenService* parameters fetched via a special option (caught by dun, karsten, and grumpy3)
+ * fix: workaround for os specific torrc validation bug - unfortunately haven't managed to repro yet so no fix (caught by grumpy3, Tas, and dun)
+ * fix: checking for python curses bindings at startup (caught by dun)
+ * fix: import error - TorCtl and socket missing from confPanel.py (caught by grumpy3)
+ * fix: showing external ip in connection panel rather than local nat address (caught by mete1989)
+ * fix: raised minimum width at which graph stats are displayed beside label (caught by dun)
+ * fix: wasn't treating "accept *" and "reject *" as catch-all policies
+ * fix: wasn't resizing graph panel properly in case of a sighup
+
+2/14/10 - version 1.3.2 (r21646)
Refactoring goodness and bug fixes.
- * change: revised curses utilities to further simplify interfaces
- * change: substantial layout changes (separating into util and init packages) and including a copy of the gpl
+ * change: revised curses utilities to further simplify interface implementations
+ * change: substantial layout changes (adding util and init packages) and including a copy of the gpl
* fix: bug with handing of DST for accounting's 'Time to reset' (patch provided by waltman)
* fix: header and connection panels weren't accounting for having ORListenAddress set (caught by waltman)
* fix: crashing bug when shrank too much for scrollbars to be drawn
Modified: arm/trunk/README
===================================================================
--- arm/trunk/README 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/README 2010-02-28 02:48:24 UTC (rev 21772)
@@ -33,6 +33,8 @@
This is started via 'arm' (use the '--help' argument for usage).
+-------------------------------------------------------------------------------
+
FAQ:
> Why is it called 'arm'?
@@ -87,3 +89,48 @@
as being in a chroot jail) then it's probably failing due to permission issues.
Arm still runs, just no connection listing or ps stats.
+-------------------------------------------------------------------------------
+
+Layout:
+
+./
+ arm - startup script
+
+ ChangeLog - revision history
+ LICENSE - copy of the gpl v3
+ README - um... guess you figured this one out
+ TODO - known issues, future plans, etc
+
+ screenshot_page1.png
+ screenshot_page2.png
+
+ init/
+ __init__.py
+ arm.py - parses and validates commandline parameters
+ prereq.py - checks python version and for required packages
+
+ interface/
+ __init__.py
+ controller.py - main display loop, handling input and layout
+ headerPanel.py - top of all pages, providing general information
+ connResolver.py - (daemon thread) periodic netstat lookups
+ hostnameResolver.py - (daemon thread) nonblocking reverse dns lookups
+
+
+ graphPanel.py - (page 1) presents graphs for data instances
+ bandwidthMonitor.py - (graph data) tracks tor bandwidth usage
+ cpuMemMonitor.py - (graph data) tracks tor cpu and memory usage
+ connCountMonitor.py - (graph data) tracks number of tor connections
+ logPanel.py - displays tor, arm, and torctl events
+ fileDescriptorPopup.py - (popup) displays file descriptors used by tor
+
+ connPanel.py - (page 2) displays information on tor connections
+ descriptorPopup.py - (popup) displays connection descriptor data
+
+ confPanel.py - (page 3) displays torrc and performs validation
+
+ util/
+ __init__.py
+ panel.py - wrapper for safely working with curses subwindows
+ uiTools.py - helper functions for interface
+
Modified: arm/trunk/TODO
===================================================================
--- arm/trunk/TODO 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/TODO 2010-02-28 02:48:24 UTC (rev 21772)
@@ -6,12 +6,64 @@
fallback if an issue's detected.
notify John Case <case at sdf.lonestar.org>
caught by Christopher Davis
+ * torrc validation bug reported (appears to be os specific)
+ Reported instances were with Gentoo, OpenSuse, and OpenBSD. Change should be in:
+ interface/confPanel.py lines 104-130
+ setting up a Gentoo vm proved to be an absurd pain in the ass so gonna
+ try repro in OpenSuse instead.
+ * torrc validation doesn't catch if parameters are missing
+ * revise multikey sort of connections
+ Currently using a pretty ugly hack. Look at:
+ http://www.velocityreviews.com/forums/
+ t356461-sorting-a-list-of-objects-by-multiple-attributes.html
+ and check for performance difference.
+ * header panel isn't properly detecting catch-all exit policies
+ Missing edge cases
+ * avoid hostname lookups of private connections
+ Stripped most of them but suspect there might be others (have assertions
+ check for this in a debug mode?)
+ * exit policy checks aren't handling all inputs
+ Still need to handle masks, private keyword, and prepended policy,
+ currently erroring on the side of caution.
+ * not catching events unexpected by arm
+ Future tor and TorCtl revisions could provide new events - these should
+ be given the "UNKNOWN" type.
+ * regex fails for multiline log entries
+ * when logging no events still showing brackets
* quitting can hang several seconds when there's hostnames left to resolve
Not sure how to address this - problem is that the calls to 'host' can
take a while to time out. Might need another thread to kill the calls?
Or forcefully terminate thread if it's taking too long (might be noisy)?
- Features / Site
+ * rewrite codebase
+ Currently the interface is a bit of a rat's nest (especially the
+ controller). The goal is to use better modularization to both simplify
+ the codebase and make it possible to use smarter caching to improve
+ performance (far too much is done in the ui logic). This work is in
+ progress, having started with the initialization (/init) and now
+ concerning the utilities (/util). Migrating the following to util:
+ - os calls (to provide transparent platform independence)
+ - torrc validation
+ - arm logging (static interface with listener design)
+ - wrapper for tor connection, state, and data parsing (abstracting
+ TorCtl connection should allow for arm to be resumed if tor restarts)
+ * provide performance ARM-DEBUG events
+ Help with diagnosing performance bottlenecks. This is pending the
+ codebase revisions to figure out the low hanging fruit for caching.
+ * condense tor/arm log listing types if they're the same
+ Ie, make default "TOR/ARM NOTICE - ERR"
+ * graph for arm cpu/mem usage
+ Trivial to implement but not sure if this would be helpful.
+ * startup option to restrict resource usage or set refresh rate
+ * audit tor connections
+ Provide warnings if tor misbehaves, checks possibly including:
+ - ensuring ExitPolicyRejectPrivate is being obeyed
+ - check that ExitPolicy violations don't occure (not possible yet since
+ not all relays aren't identified)
+ - check that all connections are properly related to a circuit, for
+ instance no outbound connections without a corresponding inbound (not
+ possible yet due to being unable to correlate connections to circuts)
* abstract away netstat calls
In preparation for drop in replacement of lsof or calls to tor's
GETINFO.
@@ -35,28 +87,21 @@
if set and there's extra room available show 'MaxAdvertisedBandwidth'
* check family connections to see if they're alive (VERSION cell handshake?)
* update site's screenshots (pretty out of date...)
+ * look into providing UPnP support
+ This might be provided by tor itself so wait and see...
- Ideas (low priority)
- * write up a proposal for the control protocol wishlist
- * look into providing UPnP support
+ * python 3 compatability
+ Currently blocked on TorCtl support.
* bundle script that dumps relay stats to stdout
Django has a small terminal coloring module that could be nice for
formatting. Could possibly include:
- desc / ns information for our relay
- ps / netstat stats like load, uptime, and connection counts, etc
derived from an idea by StrangeCharm
- * provide performance ARM-DEBUG events
- Might help with debugging bottlenecks. This requires that there's more
- refined controls for selecting logged arm runlevel.
* show qos stats
Take a look at 'linux-tor-prio.sh' to see if any of the stats are
available and interesting.
- * get a test environment for Mac OSX or BSD
- Set up a vm for FreeBSD but found working in it to be... painful (wasted
- five hours and gave up when even asking for a working copy of vim was
- too much to ask). As for OSX seems that getting a test environment would
- cost quite a bit. Hence mothballing this - someone that actually uses
- these platforms will need to resolve portability issues if they arise.
* localization
Abstract strings from code and provide on translation portal. Thus far
there hasn't been any requests for this.
@@ -69,59 +114,10 @@
submit at bugs.debian.org with subject "RFP: arm" and starting with a line
"Package: wnpp".
requested by helmut
-
-- Control Protocol Wishlist (low priority)
- * listing of tor's current connections (netstat / lsof replacement)
- Keeping the netstat available would be good for auditing (external view
- of tor and more likely monitored by host based IDS) but tor's listing
- would probably be more effecient, accurate, and could contain additional
- details making it a preferable default.
- * bandwidth usage per connection
- This would need to be rounded and averaged over time to avoid
- correlation problems. Probably the most interesting stat arm currently
- doesn't have since for most purposes (like security threats) especially
- active connections are of most interest.
- * identification of hop type
- Identification if the first, middle or last hop. When this is available
- I'll hide exit connections by default. Another interesting distinction
- would be when we're serving directory data verses acting as a relay.
- * associate connections to circuits
- Currently listing is connection based rather than circuit, ie it lists:
- previous hop -> localhost
- previous hop -> localhost
- localhost -> next hop
+ * follow up on control-spec proposal
+ Proposal and related information is available at:
+ http://www.atagar.com/arm/controlSpecProposal.txt
- rather than:
- previous hop -> localhost -> next hop
- previous hop -> localhost -> *unestablished*
-
- From a debugging and secuirty standpoint this could highlight potential
- issues, for instance relays really shouldn't have any non-client
- connections like:
- *unestablished* -> localhost -> next hop
-
- and entries like:
- previous hop -> localhost -> *extension failed (error X)*
-
- might indicate a firewall blocking tor outbound connections. This would
- be especially helpful if paired with server related circuit status
- events (which would note attempted extensions, failures, etc). We could
- also note other circuit based stats like the amount of buffered data.
- * mapping of ip/port to fingerprint
- Currently inferring the mappings but this only has around a 90% success
- rate (not sure why it fails...). Tor has an internal connection
- identifier so what would probably be best is bidirectional translation
- functions with that, ie getting fingerprint would be done via:
- ip/port -> connection id -> fingerprint
-
- In theory this should be able to tell us if the connection is the first
- or last hop (since in those cases the foreign address doesn't have a
- fingerprint).
- * additional get_info data
- effective relay bandwidth / burst - currently internally mimicing the
- logic of tor (which is RelayBandwidthRate/Burst if set, otherwise
- BandwidthRate/Burst)
- list of directory authorities recognized by that instance of tor
- total data relayed by tor - this is already kinda tracked for accounting
- file descriptor limit (return value of the getrlimit() function)
+ Unfortunatley this doesn't seem to be going anywhere so mothballed for
+ now.
Modified: arm/trunk/arm
===================================================================
--- arm/trunk/arm 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/arm 2010-02-28 02:48:24 UTC (rev 21772)
@@ -1,5 +1,5 @@
#!/bin/sh
-python init/versionCheck.py
+python init/prereq.py
if [ $? = 0 ]
then
Modified: arm/trunk/init/__init__.py
===================================================================
--- arm/trunk/init/__init__.py 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/init/__init__.py 2010-02-28 02:48:24 UTC (rev 21772)
@@ -2,5 +2,5 @@
Scripts involved in validating user input, system state, and initializing arm.
"""
-__all__ = ["starter", "versionCheck"]
+__all__ = ["starter", "prereq"]
Copied: arm/trunk/init/prereq.py (from rev 21710, arm/trunk/init/versionCheck.py)
===================================================================
--- arm/trunk/init/prereq.py (rev 0)
+++ arm/trunk/init/prereq.py 2010-02-28 02:48:24 UTC (rev 21772)
@@ -0,0 +1,23 @@
+"""
+Provides a warning and error code if python version isn't compatible.
+"""
+
+import sys
+
+if __name__ == '__main__':
+ majorVersion = sys.version_info[0]
+ minorVersion = sys.version_info[1]
+
+ if majorVersion > 2:
+ print("arm isn't compatible beyond the python 2.x series\n")
+ sys.exit(1)
+ elif majorVersion < 2 or minorVersion < 5:
+ print("arm requires python version 2.5 or greater\n")
+ sys.exit(1)
+
+ try:
+ import curses
+ except ImportError:
+ print("arm requires curses - try installing the python-curses package\n")
+ sys.exit(1)
+
Modified: arm/trunk/init/starter.py
===================================================================
--- arm/trunk/init/starter.py 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/init/starter.py 2010-02-28 02:48:24 UTC (rev 21772)
@@ -17,8 +17,8 @@
from TorCtl import TorCtl, TorUtil
from interface import controller, logPanel
-VERSION = "1.3.2"
-LAST_MODIFIED = "Feb 14, 2010"
+VERSION = "1.3.3"
+LAST_MODIFIED = "Feb 27, 2010"
DEFAULT_CONTROL_ADDR = "127.0.0.1"
DEFAULT_CONTROL_PORT = 9051
Deleted: arm/trunk/init/versionCheck.py
===================================================================
--- arm/trunk/init/versionCheck.py 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/init/versionCheck.py 2010-02-28 02:48:24 UTC (rev 21772)
@@ -1,17 +0,0 @@
-"""
-Provides a warning and error code if python version isn't compatible.
-"""
-
-import sys
-
-if __name__ == '__main__':
- majorVersion = sys.version_info[0]
- minorVersion = sys.version_info[1]
-
- if majorVersion > 2:
- print("arm isn't compatible beyond the python 2.x series\n")
- sys.exit(1)
- elif majorVersion < 2 or minorVersion < 5:
- print("arm requires python version 2.5 or greater\n")
- sys.exit(1)
-
Modified: arm/trunk/interface/__init__.py
===================================================================
--- arm/trunk/interface/__init__.py 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/interface/__init__.py 2010-02-28 02:48:24 UTC (rev 21772)
@@ -0,0 +1,6 @@
+"""
+Panels, popups, and handlers comprising the arm user interface.
+"""
+
+__all__ = ["bandwidthMonitor", "confPanel", "connCountMonitor", "connPanel", "connResolver", "controller", "cpuMemMonitor", "descriptorPopup", "fileDescriptorPopup", "graphPanel", "headerPanel", "hostnameResolver", "logPanel"]
+
Modified: arm/trunk/interface/bandwidthMonitor.py
===================================================================
--- arm/trunk/interface/bandwidthMonitor.py 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/interface/bandwidthMonitor.py 2010-02-28 02:48:24 UTC (rev 21772)
@@ -14,7 +14,7 @@
# width at which panel abandons placing optional stats (avg and total) with
# header in favor of replacing the x-axis label
-COLLAPSE_WIDTH = 120
+COLLAPSE_WIDTH = 135
class BandwidthMonitor(graphPanel.GraphStats, TorCtl.PostEventListener):
"""
@@ -47,8 +47,14 @@
bwStats = self.conn.get_option(['BandwidthRate', 'BandwidthBurst'])
relayStats = self.conn.get_option(['RelayBandwidthRate', 'RelayBandwidthBurst'])
- self.bwRate = uiTools.getSizeLabel(int(bwStats[0][1] if relayStats[0][1] == "0" else relayStats[0][1]))
- self.bwBurst = uiTools.getSizeLabel(int(bwStats[1][1] if relayStats[1][1] == "0" else relayStats[1][1]))
+ self.bwRate = uiTools.getSizeLabel(int(bwStats[0][1] if relayStats[0][1] == "0" else relayStats[0][1]), 1)
+ self.bwBurst = uiTools.getSizeLabel(int(bwStats[1][1] if relayStats[1][1] == "0" else relayStats[1][1]), 1)
+
+ # if both are using rounded values then strip off the ".0" decimal
+ if ".0" in self.bwRate and ".0" in self.bwBurst:
+ self.bwRate = self.bwRate.replace(".0", "")
+ self.bwBurst = self.bwBurst.replace(".0", "")
+
except (ValueError, socket.error, TorCtl.ErrorReply, TorCtl.TorCtlClosed):
pass # keep old values
@@ -87,7 +93,7 @@
panel.addstr(11, 2, "%s / %s" % (self.accountingInfo["read"], self.accountingInfo["readLimit"]), uiTools.getColor(self.primaryColor))
panel.addstr(11, 37, "%s / %s" % (self.accountingInfo["written"], self.accountingInfo["writtenLimit"]), uiTools.getColor(self.secondaryColor))
else:
- panel.addfstr(10, 0, "<b>Accounting:</b> Shutting Down...")
+ panel.addfstr(10, 0, "<b>Accounting:</b> Connection Closed...")
def getTitle(self, width):
# provides label, dropping stats if there's not enough room
Modified: arm/trunk/interface/confPanel.py
===================================================================
--- arm/trunk/interface/confPanel.py 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/interface/confPanel.py 2010-02-28 02:48:24 UTC (rev 21772)
@@ -4,7 +4,9 @@
import math
import curses
+import socket
+from TorCtl import TorCtl
from util import panel, uiTools
# torrc parameters that can be defined multiple times without overwriting
@@ -12,6 +14,10 @@
# last updated for tor version 0.2.1.19
MULTI_LINE_PARAM = ["AlternateBridgeAuthority", "AlternateDirAuthority", "AlternateHSAuthority", "AuthDirBadDir", "AuthDirBadExit", "AuthDirInvalid", "AuthDirReject", "Bridge", "ControlListenAddress", "ControlSocket", "DirListenAddress", "DirPolicy", "DirServer", "DNSListenAddress", "ExitPolicy", "HashedControlPassword", "HiddenServiceDir", "HiddenServiceOptions", "HiddenServicePort", "HiddenServiceVersion", "HiddenServiceAuthorizeClient", "HidServAuth", "Log", "MapAddress", "NatdListenAddress", "NodeFamily", "ORListenAddress", "ReachableAddresses", "ReachableDirAddresses", "ReachableORAddresses", "RecommendedVersions", "RecommendedClientVersions", "RecommendedServerVersions", "SocksListenAddress", "SocksPolicy", "TransListenAddress", "__HashedControlSessionPassword"]
+# hidden service options need to be fetched with HiddenServiceOptions
+HIDDEN_SERVICE_PARAM = ["HiddenServiceDir", "HiddenServiceOptions", "HiddenServicePort", "HiddenServiceVersion", "HiddenServiceAuthorizeClient"]
+HIDDEN_SERVICE_FETCH_PARAM = "HiddenServiceOptions"
+
# size modifiers allowed by config.c
LABEL_KB = ["kb", "kbyte", "kbytes", "kilobyte", "kilobytes"]
LABEL_MB = ["m", "mb", "mbyte", "mbytes", "megabyte", "megabytes"]
@@ -98,13 +104,30 @@
# check validity against tor's actual state
try:
actualValues = []
- for key, val in self.conn.get_option(command):
- actualValues.append(val)
+ if command in HIDDEN_SERVICE_PARAM:
+ # hidden services are fetched via a special command
+ hsInfo = self.conn.get_option(HIDDEN_SERVICE_FETCH_PARAM)
+ for entry in hsInfo:
+ if entry[0] == command:
+ actualValues.append(entry[1])
+ break
+ else:
+ # general case - fetch all valid values
+ for key, val in self.conn.get_option(command):
+ actualValues.append(val)
if not argument in actualValues:
- self.corrections[lineNumber + 1] = argument + " - " + ", ".join(actualValues)
- except (socket.error, TorCtl.ErrorReply, TorCtl.TorCtlClosed):
- pass # unable to load tor parameter to validate... weird
+ self.corrections[lineNumber + 1] = ", ".join(actualValues)
+ except (TypeError, socket.error, TorCtl.ErrorReply, TorCtl.TorCtlClosed):
+ # TODO: for some reason the above provided:
+ # TypeError: sequence item 0: expected string, NoneType found
+ #
+ # for the corrections setting. This issue seems to be specific to
+ # Gentoo, OpenSuse, and OpenBSD but haven't yet managed to
+ # reproduce. Catching the TypeError to just drop the torrc
+ # validation for those systems
+
+ self.logger.monitor_event("WARN", "Unable to validate torrc")
# logs issues that arose
if self.irrelevantLines:
Modified: arm/trunk/interface/connPanel.py
===================================================================
--- arm/trunk/interface/connPanel.py 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/interface/connPanel.py 2010-02-28 02:48:24 UTC (rev 21772)
@@ -11,6 +11,17 @@
import hostnameResolver
from util import panel, uiTools
+# Scrubs private data from any connection that might belong to client or exit
+# traffic. This is a little overly conservative, hiding anything that isn't
+# identified as a relay and meets the following criteria:
+# - Connection is inbound and relay's either a bridge (BridgeRelay is set) or
+# guard (making it a probable client connection)
+# - Outbound connection permitted by the exit policy (probable exit connection)
+#
+# Note that relay etiquette says these are bad things to look at (ie, DON'T
+# CHANGE THIS UNLESS YOU HAVE A DAMN GOOD REASON!)
+SCRUB_PRIVATE_DATA = True
+
# directory servers (IP, port) for tor version 0.2.2.1-alpha-dev
DIR_SERVERS = [("86.59.21.38", "80"), # tor26
("128.31.0.34", "9031"), # moria1
@@ -29,7 +40,7 @@
TYPE_WEIGHTS = {"inbound": 0, "outbound": 1, "client": 2, "directory": 3, "control": 4, "family": 5, "localhost": 6} # defines ordering
# enums for indexes of ConnPanel 'connections' fields
-CONN_TYPE, CONN_L_IP, CONN_L_PORT, CONN_F_IP, CONN_F_PORT, CONN_COUNTRY, CONN_TIME = range(7)
+CONN_TYPE, CONN_L_IP, CONN_L_PORT, CONN_F_IP, CONN_F_PORT, CONN_COUNTRY, CONN_TIME, CONN_PRIVATE = range(8)
# labels associated to 'connectionCount'
CONN_COUNT_LABELS = ["inbound", "outbound", "client", "directory", "control"]
@@ -137,12 +148,16 @@
# mapping of ip/port to fingerprint of family entries, used in hack to short circuit (ip / port) -> fingerprint lookups
self.familyResolutions = {}
+ self.address = ""
self.nickname = ""
self.listenPort = "0" # port used to identify inbound/outbound connections (from ORListenAddress if defined, otherwise ORPort)
self.orPort = "0"
self.dirPort = "0"
self.controlPort = "0"
self.family = [] # fingerpints of family entries
+ self.isBridge = False # true if BridgeRelay is set
+ self.exitPolicy = ""
+ self.exitRejectPrivate = True # true if ExitPolicyRejectPrivate is 0
self.resetOptions()
@@ -160,6 +175,7 @@
self.familyResolutions = {}
try:
+ self.address = ""
self.nickname = self.conn.get_option("Nickname")[0][1]
self.orPort = self.conn.get_option("ORPort")[0][1]
@@ -176,6 +192,14 @@
familyEntry = self.conn.get_option("MyFamily")[0][1]
if familyEntry: self.family = [entry[1:] for entry in familyEntry.split(",")]
else: self.family = []
+
+ self.isBridge = self.conn.get_option("BridgeRelay")[0][1] == "1"
+ self.exitPolicy = self.conn.get_option("ExitPolicy")[0][1]
+
+ if self.exitPolicy: self.exitPolicy += "," + self.conn.get_info("exit-policy/default")["exit-policy/default"]
+ else: self.exitPolicy = self.conn.get_info("exit-policy/default")["exit-policy/default"]
+
+ self.exitRejectPrivate = self.conn.get_option("ExitPolicyRejectPrivate")[0][1] == "1"
except (socket.error, TorCtl.ErrorReply, TorCtl.TorCtlClosed):
self.nickname = ""
self.listenPort = None
@@ -183,6 +207,9 @@
self.dirPort = "0"
self.controlPort = "0"
self.family = []
+ self.isBridge = False
+ self.exitPolicy = ""
+ self.exitRejectPrivate = True
# change in client circuits
def circ_status_event(self, event):
@@ -242,6 +269,11 @@
Reloads netstat results.
"""
+ # inaccessable during startup so might need to be refetched
+ try:
+ if not self.address: self.address = self.conn.get_info("address")["address"]
+ except (socket.error, TorCtl.ErrorReply, TorCtl.TorCtlClosed): pass
+
self.connectionsLock.acquire()
self.clientConnectionLock.acquire()
@@ -250,7 +282,14 @@
connectionCountTmp = [0] * 5
familyResolutionsTmp = {}
+ # used (with isBridge) to determine if inbound connections should be scrubbed
+ isGuard = False
try:
+ myFingerprint = self.conn.get_info("fingerprint")
+ isGuard = "Guard" in self.conn.get_network_status("id/%s" % myFingerprint)[0].flags
+ except (socket.error, TorCtl.ErrorReply, TorCtl.TorCtlClosed): pass
+
+ try:
if self.clientConnectionCache == None:
# client connection cache was invalidated
self.clientConnectionCache = _getClientConnections(self.conn)
@@ -268,15 +307,17 @@
local, foreign = param[3], param[4]
localIP, foreignIP = local[:local.find(":")], foreign[:foreign.find(":")]
localPort, foreignPort = local[len(localIP) + 1:], foreign[len(foreignIP) + 1:]
+ fingerprint = self.getFingerprint(foreignIP, foreignPort)
+ isPrivate = False
if localPort in (self.listenPort, self.dirPort):
type = "inbound"
connectionCountTmp[0] += 1
+ if SCRUB_PRIVATE_DATA and foreignIP not in self.fingerprintMappings.keys(): isPrivate = isGuard or self.isBridge
elif localPort == self.controlPort:
type = "control"
connectionCountTmp[4] += 1
else:
- fingerprint = self.getFingerprint(foreignIP, foreignPort)
nickname = self.getNickname(foreignIP, foreignPort)
isClient = False
@@ -294,7 +335,11 @@
else:
type = "outbound"
connectionCountTmp[1] += 1
+ if SCRUB_PRIVATE_DATA and foreignIP not in self.fingerprintMappings.keys(): isPrivate = isExitAllowed(foreignIP, foreignPort, self.exitPolicy, self.exitRejectPrivate, self.logger)
+ # replace nat address with external version if available
+ if self.address and type != "control": localIP = self.address
+
try:
countryCodeQuery = "ip-to-country/%s" % foreign[:foreign.find(":")]
countryCode = self.conn.get_info(countryCodeQuery)[countryCodeQuery]
@@ -307,26 +352,25 @@
if (foreignIP, foreignPort) in connTimes: connTime = connTimes[(foreignIP, foreignPort)]
else: connTime = time.time()
- connectionsTmp.append((type, localIP, localPort, foreignIP, foreignPort, countryCode, connTime))
+ connectionsTmp.append((type, localIP, localPort, foreignIP, foreignPort, countryCode, connTime, isPrivate))
# appends localhost connection to allow user to look up their own consensus entry
- selfAddress, selfFingerprint = None, None
+ selfFingerprint = None
try:
- selfAddress = self.conn.get_info("address")["address"]
selfFingerprint = self.conn.get_info("fingerprint")["fingerprint"]
except (socket.error, TorCtl.ErrorReply, TorCtl.TorCtlClosed): pass
- if selfAddress and selfFingerprint:
+ if self.address and selfFingerprint:
try:
- countryCodeQuery = "ip-to-country/%s" % selfAddress
+ countryCodeQuery = "ip-to-country/%s" % self.address
selfCountryCode = self.conn.get_info(countryCodeQuery)[countryCodeQuery]
except (socket.error, TorCtl.ErrorReply, TorCtl.TorCtlClosed):
selfCountryCode = "??"
- if (selfAddress, self.orPort) in connTimes: connTime = connTimes[(selfAddress, self.orPort)]
+ if (self.address, self.orPort) in connTimes: connTime = connTimes[(self.address, self.orPort)]
else: connTime = time.time()
- self.localhostEntry = (("localhost", selfAddress, self.orPort, selfAddress, self.orPort, selfCountryCode, connTime), selfFingerprint)
+ self.localhostEntry = (("localhost", self.address, self.orPort, self.address, self.orPort, selfCountryCode, connTime, False), selfFingerprint)
connectionsTmp.append(self.localhostEntry[0])
else:
self.localhostEntry = None
@@ -346,12 +390,12 @@
else: connTime = time.time()
familyResolutionsTmp[(familyAddress, familyPort)] = fingerprint
- connectionsTmp.append(("family", familyAddress, familyPort, familyAddress, familyPort, familyCountryCode, connTime))
+ connectionsTmp.append(("family", familyAddress, familyPort, familyAddress, familyPort, familyCountryCode, connTime, False))
except (socket.error, TorCtl.ErrorReply):
# use dummy entry for sorting - the draw function notes that entries are unknown
portIdentifier = str(65536 + tmpCounter)
familyResolutionsTmp[("256.255.255.255", portIdentifier)] = fingerprint
- connectionsTmp.append(("family", "256.255.255.255", portIdentifier, "256.255.255.255", portIdentifier, "??", time.time()))
+ connectionsTmp.append(("family", "256.255.255.255", portIdentifier, "256.255.255.255", portIdentifier, "??", time.time(), False))
tmpCounter += 1
except TorCtl.TorCtlClosed:
pass # connections aren't shown when control port is unavailable
@@ -458,6 +502,7 @@
for entry in self.connections:
if lineNum >= 1:
type = entry[CONN_TYPE]
+ isPrivate = entry[CONN_PRIVATE]
color = TYPE_COLORS[type]
# adjustments to measurements for 'xOffset' are to account for scroll bar
@@ -465,6 +510,11 @@
# base data requires 73 characters
src = "%s:%s" % (entry[CONN_L_IP], entry[CONN_L_PORT])
dst = "%s:%s %s" % (entry[CONN_F_IP], entry[CONN_F_PORT], "" if type == "control" else "(%s)" % entry[CONN_COUNTRY])
+
+ if isPrivate:
+ if type == "inbound": src = "<scrubbed>"
+ elif type == "outbound": dst = "<scrubbed>"
+
src, dst = "%-21s" % src, "%-26s" % dst
etc = ""
@@ -492,7 +542,10 @@
if self.maxX > 102 + xOffset:
# shows ip/locale (column width: 22 characters)
foreignHostnameSpace -= 22
- etc += "%-20s " % ("%s %s" % (entry[CONN_F_IP], "" if type == "control" else "(%s)" % entry[CONN_COUNTRY]))
+
+ if isPrivate: ipEntry = "<scrubbed>"
+ else: ipEntry = "%s %s" % (entry[CONN_F_IP], "" if type == "control" else "(%s)" % entry[CONN_COUNTRY])
+ etc += "%-20s " % ipEntry
if self.maxX > 134 + xOffset:
# show fingerprint (column width: 42 characters)
@@ -508,14 +561,17 @@
if len(nickname) > nicknameSpace: nickname = "%s..." % nickname[:nicknameSpace - 3]
etc += ("%%-%is " % nicknameSpace) % nickname
- hostname = self.resolver.resolve(entry[CONN_F_IP])
+ if isPrivate: dst = "<scrubbed>"
+ else:
+ hostname = self.resolver.resolve(entry[CONN_F_IP])
+
+ # truncates long hostnames
+ portDigits = len(str(entry[CONN_F_PORT]))
+ if hostname and (len(hostname) + portDigits) > foreignHostnameSpace - 1:
+ hostname = hostname[:(foreignHostnameSpace - portDigits - 4)] + "..."
+
+ dst = "%s:%s" % (hostname if hostname else entry[CONN_F_IP], entry[CONN_F_PORT])
- # truncates long hostnames
- portDigits = len(str(entry[CONN_F_PORT]))
- if hostname and (len(hostname) + portDigits) > foreignHostnameSpace - 1:
- hostname = hostname[:(foreignHostnameSpace - portDigits - 4)] + "..."
-
- dst = "%s:%s" % (hostname if hostname else entry[CONN_F_IP], entry[CONN_F_PORT])
dst = ("%%-%is" % foreignHostnameSpace) % dst
elif self.listingType == LIST_FINGERPRINT:
# base data requires 75 characters
@@ -534,7 +590,9 @@
if self.maxX > 125 + xOffset:
# shows ip/port/locale (column width: 28 characters)
- etc += "%-26s " % ("%s:%s %s" % (entry[CONN_F_IP], entry[CONN_F_PORT], "" if type == "control" else "(%s)" % entry[CONN_COUNTRY]))
+ if isPrivate: ipEntry = "<scrubbed>"
+ else: ipEntry = "%s:%s %s" % (entry[CONN_F_IP], entry[CONN_F_PORT], "" if type == "control" else "(%s)" % entry[CONN_COUNTRY])
+ etc += "%-26s " % ipEntry
else:
# base data uses whatever extra room's available (using minimun of 50 characters)
src = self.nickname
@@ -553,7 +611,10 @@
if self.maxX > 120 + xOffset:
# shows ip/port/locale (column width: 28 characters)
foreignNicknameSpace -= 28
- etc += "%-26s " % ("%s:%s %s" % (entry[CONN_F_IP], entry[CONN_F_PORT], "" if type == "control" else "(%s)" % entry[CONN_COUNTRY]))
+
+ if isPrivate: ipEntry = "<scrubbed>"
+ else: ipEntry = "%s:%s %s" % (entry[CONN_F_IP], entry[CONN_F_PORT], "" if type == "control" else "(%s)" % entry[CONN_COUNTRY])
+ etc += "%-26s " % ipEntry
dst = ("%%-%is" % foreignNicknameSpace) % dst
@@ -786,3 +847,58 @@
return clients
+def isExitAllowed(ip, port, exitPolicy, isPrivateRejected, logger):
+ """
+ Determines if a given connection is a permissable exit with the given
+ policy or not (True if it's allowed to be an exit connection, False
+ otherwise).
+
+ NOTE: this is a little tricky and liable to need some tweaks
+ """
+
+ # might not be set when first starting up
+ if not exitPolicy: return True
+
+ # TODO: move into a utility and craft some unit tests (this is very error
+ # prone...)
+
+ # TODO: currently doesn't consider ExitPolicyRejectPrivate (which prevents
+ # connections to private networks and local ip)
+ for entry in exitPolicy.split(","):
+ entry = entry.strip()
+
+ isAccept = entry.startswith("accept")
+ entry = entry[7:] # strips off "accept " or "reject "
+
+ # parses ip address (with mask if provided) and port
+ if ":" in entry:
+ entryIP = entry[:entry.find(":")]
+ entryPort = entry[entry.find(":") + 1:]
+ else:
+ entryIP = entry
+ entryPort = "*"
+
+ #raise AssertionError(str(exitPolicy) + " - " + entryIP + ":" + entryPort)
+ isIPMatch = entryIP == ip or entryIP[0] == "*"
+
+ if not "-" in entryPort:
+ # single port
+ isPortMatch = entryPort == str(port) or entryPort[0] == "*"
+ else:
+ # port range
+ minPort = int(entryPort[:entryPort.find("-")])
+ maxPort = int(entryPort[entryPort.find("-") + 1:])
+ isPortMatch = port >= minPort and port <= maxPort
+
+ # TODO: Currently being lazy and considering subnet masks or 'private'
+ # keyword to be equivilant to wildcard if it would reject, and none
+ # if it would accept (ie, being conservative with acceptance). Would be
+ # nice to fix at some point.
+ if not isAccept: isIPMatch |= "/" in entryIP or entryIP == "private"
+
+ if isIPMatch and isPortMatch: return isAccept
+
+ # we shouldn't ever fall through due to default exit policy
+ logger.monitor_event("WARN", "Exit policy left connection uncategorized: %s:%i" % (ip, port))
+ return False
+
Modified: arm/trunk/interface/controller.py
===================================================================
--- arm/trunk/interface/controller.py 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/interface/controller.py 2010-02-28 02:48:24 UTC (rev 21772)
@@ -372,15 +372,15 @@
if sighupTracker.isReset:
panels["header"]._updateParams(True)
- # if bandwidth graph is being shown then height might have changed
- if panels["graph"].currentDisplay == "bandwidth":
- panels["graph"].height = panels["graph"].stats["bandwidth"].height
-
# other panels that use torrc data
panels["conn"].resetOptions()
if not isBlindMode: panels["graph"].stats["connections"].resetOptions(conn)
panels["graph"].stats["bandwidth"].resetOptions()
+ # if bandwidth graph is being shown then height might have changed
+ if panels["graph"].currentDisplay == "bandwidth":
+ panels["graph"].height = panels["graph"].stats["bandwidth"].height
+
panels["torrc"].reset()
sighupTracker.isReset = False
@@ -801,6 +801,7 @@
selectedIp = selection[connPanel.CONN_F_IP]
selectedPort = selection[connPanel.CONN_F_PORT]
+ selectedIsPrivate = selection[connPanel.CONN_PRIVATE]
addrLabel = "address: %s:%s" % (selectedIp, selectedPort)
@@ -808,9 +809,11 @@
# unresolved family entry - unknown ip/port
addrLabel = "address: unknown"
- hostname = resolver.resolve(selectedIp)
+ if selectedIsPrivate: hostname = None
+ else: hostname = resolver.resolve(selectedIp)
+
if hostname == None:
- if resolver.isPaused: hostname = "DNS resolution disallowed"
+ if resolver.isPaused or selectedIsPrivate: hostname = "DNS resolution disallowed"
elif selectedIp not in resolver.resolvedCache.keys():
# if hostname is still being resolved refresh panel every half-second until it's completed
curses.halfdelay(5)
@@ -822,77 +825,82 @@
# hostname too long - truncate
hostname = "%s..." % hostname[:70 - len(addrLabel)]
- popup.addstr(1, 2, "%s (%s)" % (addrLabel, hostname), format)
-
- locale = selection[connPanel.CONN_COUNTRY]
- popup.addstr(2, 2, "locale: %s" % locale, format)
-
- # provides consensus data for selection (needs fingerprint to get anywhere...)
- fingerprint = panels["conn"].getFingerprint(selectedIp, selectedPort)
-
- if fingerprint == "UNKNOWN":
- if selectedIp not in panels["conn"].fingerprintMappings.keys():
- # no consensus entry for this ip address
- popup.addstr(3, 2, "No consensus data found", format)
+ if selectedIsPrivate:
+ popup.addstr(1, 2, "address: <scrubbed> (unknown)", format)
+ popup.addstr(2, 2, "locale: ??", format)
+ popup.addstr(3, 2, "No consensus data found", format)
+ else:
+ popup.addstr(1, 2, "%s (%s)" % (addrLabel, hostname), format)
+
+ locale = selection[connPanel.CONN_COUNTRY]
+ popup.addstr(2, 2, "locale: %s" % locale, format)
+
+ # provides consensus data for selection (needs fingerprint to get anywhere...)
+ fingerprint = panels["conn"].getFingerprint(selectedIp, selectedPort)
+
+ if fingerprint == "UNKNOWN":
+ if selectedIp not in panels["conn"].fingerprintMappings.keys():
+ # no consensus entry for this ip address
+ popup.addstr(3, 2, "No consensus data found", format)
+ else:
+ # couldn't resolve due to multiple matches - list them all
+ popup.addstr(3, 2, "Muliple matches, possible fingerprints are:", format)
+ matchings = panels["conn"].fingerprintMappings[selectedIp]
+
+ line = 4
+ for (matchPort, matchFingerprint, matchNickname) in matchings:
+ popup.addstr(line, 2, "%i. or port: %-5s fingerprint: %s" % (line - 3, matchPort, matchFingerprint), format)
+ line += 1
+
+ if line == 7 and len(matchings) > 4:
+ popup.addstr(8, 2, "... %i more" % len(matchings) - 3, format)
+ break
else:
- # couldn't resolve due to multiple matches - list them all
- popup.addstr(3, 2, "Muliple matches, possible fingerprints are:", format)
- matchings = panels["conn"].fingerprintMappings[selectedIp]
-
- line = 4
- for (matchPort, matchFingerprint, matchNickname) in matchings:
- popup.addstr(line, 2, "%i. or port: %-5s fingerprint: %s" % (line - 3, matchPort, matchFingerprint), format)
- line += 1
+ # fingerprint found - retrieve related data
+ lookupErrored = False
+ if selection in relayLookupCache.keys(): nsEntry, descEntry = relayLookupCache[selection]
+ else:
+ # ns lookup fails, can happen with localhost lookups if relay's having problems (orport not reachable)
+ try: nsData = conn.get_network_status("id/%s" % fingerprint)
+ except (socket.error, TorCtl.ErrorReply, TorCtl.TorCtlClosed): lookupErrored = True
- if line == 7 and len(matchings) > 4:
- popup.addstr(8, 2, "... %i more" % len(matchings) - 3, format)
- break
- else:
- # fingerprint found - retrieve related data
- lookupErrored = False
- if selection in relayLookupCache.keys(): nsEntry, descEntry = relayLookupCache[selection]
- else:
- # ns lookup fails, can happen with localhost lookups if relay's having problems (orport not reachable)
- try: nsData = conn.get_network_status("id/%s" % fingerprint)
- except (socket.error, TorCtl.ErrorReply, TorCtl.TorCtlClosed): lookupErrored = True
+ if not lookupErrored:
+ if len(nsData) > 1:
+ # multiple records for fingerprint (shouldn't happen)
+ panels["log"].monitor_event("WARN", "Multiple consensus entries for fingerprint: %s" % fingerprint)
+
+ nsEntry = nsData[0]
+
+ try:
+ descLookupCmd = "desc/id/%s" % fingerprint
+ descEntry = TorCtl.Router.build_from_desc(conn.get_info(descLookupCmd)[descLookupCmd].split("\n"), nsEntry)
+ relayLookupCache[selection] = (nsEntry, descEntry)
+ except (socket.error, TorCtl.ErrorReply, TorCtl.TorCtlClosed): lookupErrored = True # desc lookup failed
- if not lookupErrored:
- if len(nsData) > 1:
- # multiple records for fingerprint (shouldn't happen)
- panels["log"].monitor_event("WARN", "Multiple consensus entries for fingerprint: %s" % fingerprint)
+ if lookupErrored:
+ popup.addstr(3, 2, "Unable to retrieve consensus data", format)
+ else:
+ popup.addstr(2, 15, "fingerprint: %s" % fingerprint, format)
- nsEntry = nsData[0]
+ nickname = panels["conn"].getNickname(selectedIp, selectedPort)
+ dirPortLabel = "dirport: %i" % nsEntry.dirport if nsEntry.dirport else ""
+ popup.addstr(3, 2, "nickname: %-25s orport: %-10i %s" % (nickname, nsEntry.orport, dirPortLabel), format)
- try:
- descLookupCmd = "desc/id/%s" % fingerprint
- descEntry = TorCtl.Router.build_from_desc(conn.get_info(descLookupCmd)[descLookupCmd].split("\n"), nsEntry)
- relayLookupCache[selection] = (nsEntry, descEntry)
- except (socket.error, TorCtl.ErrorReply, TorCtl.TorCtlClosed): lookupErrored = True # desc lookup failed
+ popup.addstr(4, 2, "published: %-24s os: %-14s version: %s" % (descEntry.published, descEntry.os, descEntry.version), format)
+ popup.addstr(5, 2, "flags: %s" % ", ".join(nsEntry.flags), format)
+
+ exitLine = ", ".join([str(k) for k in descEntry.exitpolicy])
+ if len(exitLine) > 63: exitLine = "%s..." % exitLine[:60]
+ popup.addstr(6, 2, "exit policy: %s" % exitLine, format)
+
+ if descEntry.contact:
+ # clears up some common obscuring
+ contactAddr = descEntry.contact
+ obscuring = [(" at ", "@"), (" AT ", "@"), ("AT", "@"), (" dot ", "."), (" DOT ", ".")]
+ for match, replace in obscuring: contactAddr = contactAddr.replace(match, replace)
+ if len(contactAddr) > 67: contactAddr = "%s..." % contactAddr[:64]
+ popup.addstr(7, 2, "contact: %s" % contactAddr, format)
- if lookupErrored:
- popup.addstr(3, 2, "Unable to retrieve consensus data", format)
- else:
- popup.addstr(2, 15, "fingerprint: %s" % fingerprint, format)
-
- nickname = panels["conn"].getNickname(selectedIp, selectedPort)
- dirPortLabel = "dirport: %i" % nsEntry.dirport if nsEntry.dirport else ""
- popup.addstr(3, 2, "nickname: %-25s orport: %-10i %s" % (nickname, nsEntry.orport, dirPortLabel), format)
-
- popup.addstr(4, 2, "published: %-24s os: %-14s version: %s" % (descEntry.published, descEntry.os, descEntry.version), format)
- popup.addstr(5, 2, "flags: %s" % ", ".join(nsEntry.flags), format)
-
- exitLine = ", ".join([str(k) for k in descEntry.exitpolicy])
- if len(exitLine) > 63: exitLine = "%s..." % exitLine[:60]
- popup.addstr(6, 2, "exit policy: %s" % exitLine, format)
-
- if descEntry.contact:
- # clears up some common obscuring
- contactAddr = descEntry.contact
- obscuring = [(" at ", "@"), (" AT ", "@"), ("AT", "@"), (" dot ", "."), (" DOT ", ".")]
- for match, replace in obscuring: contactAddr = contactAddr.replace(match, replace)
- if len(contactAddr) > 67: contactAddr = "%s..." % contactAddr[:64]
- popup.addstr(7, 2, "contact: %s" % contactAddr, format)
-
popup.refresh()
key = stdscr.getch()
Modified: arm/trunk/interface/headerPanel.py
===================================================================
--- arm/trunk/interface/headerPanel.py 2010-02-28 02:24:50 UTC (rev 21771)
+++ arm/trunk/interface/headerPanel.py 2010-02-28 02:48:24 UTC (rev 21772)
@@ -135,8 +135,9 @@
exitPolicy = self.vals["ExitPolicy"]
# adds note when default exit policy is appended
+ # TODO: the following catch-all policies arne't quite exhaustive
if exitPolicy == None: exitPolicy = "<default>"
- elif not exitPolicy.endswith("accept *:*") and not exitPolicy.endswith("reject *:*"):
+ elif not (exitPolicy.endswith("accept *:*") or exitPolicy.endswith("accept *")) and not (exitPolicy.endswith("reject *:*") or exitPolicy.endswith("reject *")):
exitPolicy += ", <default>"
policies = exitPolicy.split(", ")
More information about the tor-commits
mailing list