[or-cvs] r23243: {arm} added: providing a notice if tor supports event types that a (in arm/trunk: . src/interface src/util)
Damian Johnson
atagar1 at gmail.com
Sun Sep 19 01:41:28 UTC 2010
Author: atagar
Date: 2010-09-19 01:41:28 +0000 (Sun, 19 Sep 2010)
New Revision: 23243
Modified:
arm/trunk/TODO
arm/trunk/armrc.sample
arm/trunk/src/interface/controller.py
arm/trunk/src/interface/logPanel.py
arm/trunk/src/util/torTools.py
Log:
added: providing a notice if tor supports event types that arm doesn't, and logging them as the 'UNKNOWN' type
change: condensing the label for runlevel event ranges further if they're identical for multiple types
fix: ordering of individual runlevel entries were sometimes reversed
Modified: arm/trunk/TODO
===================================================================
--- arm/trunk/TODO 2010-09-19 01:21:49 UTC (rev 23242)
+++ arm/trunk/TODO 2010-09-19 01:41:28 UTC (rev 23243)
@@ -1,6 +1,6 @@
TODO
-- Roadmap for next release (1.3.7)
+- Remaining work for next release (1.3.7)
[ ] refactor panels
Currently the interface is a bit of a rat's nest (especially the
controller). The goal is to use better modularization to both simplify
@@ -8,80 +8,18 @@
performance (far too much is done in the ui logic). This work is in
progress - /init and /util are done and /interface is partly done. Known
bugs are being fixed while refactoring.
- [ ] log panel
- - provide notice if tor supports events that arm doesn't
- getInfo("events/names") provides the space-separated listing
- - check what events TorCtl can provide us, and give notice if any are
- missing
- - log to file, allowing non-runlevel events to be saved (provide both
- a continuous option and snapshots taking into account the current
- filter)
- - condense tor/arm log listing types if they're the same
- Ie, make default "TOR/ARM NOTICE - ERR"
- - provide daily dividers (otherwise there's no indicator for the day)
- - log cropping based on time (idea by voidzero)
- - drop duplicate or overly verbose messages (feature request by asn)
- check if asn wants to implement this when refactoring is done
- [ ] conf panel
- - move torrc validation into util
- - fetch text via getinfo rather than reading directly?
- conn.get_info("config-text")
- [-] conn panel (for version 1.3.8)
- - expand client connections and note location in circuit (entry-exit)
- - for clients list all connections to detect what's going through tor
- and what isn't? If not then netstat calls are unnecessary.
- - check family connections to see if they're alive (VERSION cell
- handshake?)
- - fallback when pid or connection querying via pid is unavailable
- List all connections listed both by netstat and the consensus
- - note when connection times are estimates (color?), ie connection
- was established before arm
- - connection uptime to associate inbound/outbound connections?
- - Identify controller connections (if it's arm, vidalia, etc) with
- special detail page for them
- - provide bridge / client country statistics
- Include bridge related data via GETINFO option (feature request
- by waltman and ioerror).
- - Country data for client connections (requested by ioerror)
- [-] controller (for version 1.3.8)
- [ ] provide performance ARM-DEBUG events
- Help with diagnosing performance bottlenecks. This is pending the
- codebase revisions to figure out the low hanging fruit for caching.
- [ ] email alerts for changes to the relay's status (similar to tor-weather)
- [ ] simple alert if tor shuts down
- [ ] accounting and alerts for if the bandwidth drops to zero
- [ ] daily/weekly/etc alerts for basic status (log output, bandwidth
- history, etc), borrowing from the consensus tracker for some of the
- formatting
- [ ] tor util
- [X] wrapper for accessing torctl
- [ ] allow arm to resume after restarting tor (attaching to a new torctl
- instance)
- [ ] setup scripts for arm
- [X] setup scrpt to add to /usr/bin/arm (requested by ioerror)
- [ ] mac installer
- Couple of options include macport and dmg...
- - macport (http://guide.macports.org/#development)
- Build-from-source distribution method (like BSD portinstall).
- This has been suggested by several people.
-
- - dmg (http://en.wikipedia.org/wiki/Apple_Disk_Image)
- Most conventional method of software distribution on mac. This is
- just a container (no updating/removal support), but could contain
- an icon for the dock that starts a terminal with arm. This might
- include a pkg installer.
-
- - mpkg (http://pypi.python.org/pypi/bdist_mpkg/)
- Plugin for distutils. Like most mac packaging, this can only run
- on a mac. It also requires setuptools:
- http://www.errorhelp.com/search/details/74034/importerror-no-module-named-setuptools
-
- [ ] updater (checks for a new tarball and installs it automatically)
- [ ] look into CAPs to get around permission issues for connection
- listing sudo wrapper for arm to help arm run as the same user as
- tor? Irc suggestions:
- - man capabilities
- - http://www.linuxjournal.com/article/5737
+
+ [ ] log panel
+ - log to file, allowing non-runlevel events to be saved (provide both
+ a continuous option and snapshots taking into account the current
+ filter)
+ - provide daily dividers (otherwise there's no indicator for the day)
+ - log cropping based on time (idea by voidzero)
+ - drop duplicate or overly verbose messages (feature request by asn)
+ [ ] conf panel
+ - move torrc validation into util
+ - fetch text via getinfo rather than reading directly?
+ conn.get_info("config-text")
* release prep
* ask helix about steps for getting a deb and rpm included in the tor repo
* check performance of this version vs last version (general screen refresh
@@ -89,6 +27,36 @@
* pylint --indent-string=" " --disable-msg-cat=CR interface/foo.py | less
* double check __init__.py and README for changes
+- Roadmap for version 1.3.8
+ [ ] refactor panels
+ [ ] conn panel
+ - expand client connections and note location in circuit (entry-exit)
+ - for clients list all connections to detect what's going through tor
+ and what isn't? If not then netstat calls are unnecessary.
+ - check family connections to see if they're alive (VERSION cell
+ handshake?)
+ - fallback when pid or connection querying via pid is unavailable
+ List all connections listed both by netstat and the consensus
+ - note when connection times are estimates (color?), ie connection
+ was established before arm
+ - connection uptime to associate inbound/outbound connections?
+ - Identify controller connections (if it's arm, vidalia, etc) with
+ special detail page for them
+ - provide bridge / client country statistics
+ Include bridge related data via GETINFO option (feature request
+ by waltman and ioerror).
+ [ ] controller and popup panels
+ - country data for client connections (requested by ioerror)
+ - allow arm to resume after restarting tor
+ This requires a full move to the torTools controller.
+ [ ] setup scripts for arm
+ [ ] updater (checks for a new tarball and installs it automatically)
+ [ ] look into CAPs to get around permission issues for connection
+ listing sudo wrapper for arm to help arm run as the same user as
+ tor? Irc suggestions:
+ - man capabilities
+ - http://www.linuxjournal.com/article/5737
+
- Bugs
* util are assuming that tor is running under the default command name
attempt to determine the command name at runtime (if the pid is available
@@ -96,17 +64,12 @@
* util/torTools.py: effective bandwidth rate/burst measurements don't take
SETCONF into consideration, blocked on:
https://trac.torproject.org/projects/tor/ticket/1692
+ * log prepopulation fails to limit entries to the current tor instance if
+ the file isn't logged to at the NOTICE level. A fix is to use the
+ timestamps to see if it belongs to this tor instance. This requires
+ tor's uptime - blocked on implementation of the following proposal:
+ https://gitweb.torproject.org/tor.git/blob/HEAD:/doc/spec/proposals/173-getinfo-option-expansion.txt
- * log panel:
- * not catching events unexpected by arm
- Future tor and TorCtl revisions could provide new events - these should
- be given the "UNKNOWN" type.
- * log prepopulation fails to limit entries to the current tor instance if
- the file isn't logged to at the NOTICE level. Use timestamps as a
- backup method, or just switch entirely?
- * arm is triggering DEBUG level events
- https://trac.torproject.org/projects/tor/ticket/1933
-
* conf panel:
* torrc validation doesn't catch if parameters are missing
* scrolling in the torrc isn't working properly when comments are stripped
@@ -162,6 +125,27 @@
circuits at the exit
* look at vidalia for ideas
* need to solicit for ideas on what would be most helpful to clients
+ * mac installer
+ * Couple of options include macport and dmg...
+ * macport (http://guide.macports.org/#development)
+ Build-from-source distribution method (like BSD portinstall). This has
+ been suggested by several people.
+
+ * dmg (http://en.wikipedia.org/wiki/Apple_Disk_Image)
+ Most conventional method of software distribution on mac. This is just
+ a container (no updating/removal support), but could contain an icon
+ for the dock that starts a terminal with arm. This might include a pkg
+ installer.
+
+ * mpkg (http://pypi.python.org/pypi/bdist_mpkg/)
+ Plugin for distutils. Like most mac packaging, this can only run on a
+ mac. It also requires setuptools:
+ http://www.errorhelp.com/search/details/74034/importerror-no-module-named-setuptools
+ * email alerts for changes to the relay's status, similar to tor-weather
+ * simple alert if tor shuts down
+ * accounting and alerts for if the bandwidth drops to zero
+ * daily/weekly/etc alerts for basic status (log output, bandwidth history,
+ etc), borrowing from the consensus tracker for some of the formatting
* check if batch getInfo/getOption calls provide much performance benefit
* page with details on client circuits, attempting to detect details like
country, ISP, latency, exit policy for the circuit, traffic, etc
@@ -175,9 +159,6 @@
* switch check of ip address validity to regex?
match = re.match("(\d*)\.(\d*)\.(\d*)\.(\d*)", ip)
http://wang.yuxuan.org/blog/2009/4/2/python_script_to_convert_from_ip_range_to_ip_mask
- * look into using Enum2
- TorCtl uses a specail class for enumerations. It's probably better than
- my current range(1, X) approach.
* audit tor connections
Provide warnings if tor misbehaves, checks possibly including:
- ensuring ExitPolicyRejectPrivate is being obeyed
@@ -188,7 +169,6 @@
possible yet due to being unable to correlate connections to circuits)
* check file descriptors being accessed by tor to see if they're outside the
known pattern
- * allow killing of circuits? Probably not useful...
* add page that allows raw control port access
Start with -t (or -c?) option for commandline-only access with help,
syntax highlighting, and other spiffy extras
Modified: arm/trunk/armrc.sample
===================================================================
--- arm/trunk/armrc.sample 2010-09-19 01:21:49 UTC (rev 23242)
+++ arm/trunk/armrc.sample 2010-09-19 01:41:28 UTC (rev 23243)
@@ -85,6 +85,7 @@
log.torCtlPortClosed NOTICE
log.torGetInfo DEBUG
log.torGetConf DEBUG
+log.torEventTypeUnrecognized NOTICE
log.sysCallMade DEBUG
log.sysCallCached NONE
log.sysCallFailed INFO
Modified: arm/trunk/src/interface/controller.py
===================================================================
--- arm/trunk/src/interface/controller.py 2010-09-19 01:21:49 UTC (rev 23242)
+++ arm/trunk/src/interface/controller.py 2010-09-19 01:41:28 UTC (rev 23243)
@@ -42,7 +42,7 @@
["torrc"]]
PAUSEABLE = ["header", "graph", "log", "conn"]
-CONFIG = {"logging.rate.refreshRate": 5, "features.graph.type": 1, "features.graph.bw.prepopulate": True, "log.refreshRate": log.DEBUG, "log.configEntryUndefined": log.NOTICE}
+CONFIG = {"logging.rate.refreshRate": 5, "features.graph.type": 1, "log.torEventTypeUnrecognized": log.NOTICE, "features.graph.bw.prepopulate": True, "log.refreshRate": log.DEBUG, "log.configEntryUndefined": log.NOTICE}
class ControlPanel(panel.Panel):
""" Draws single line label for interface controls. """
@@ -259,6 +259,7 @@
# puzzling results otherwise when trying to discard entries (silently
# returning out of this function!)
events = set(selectedEvents)
+ isLoggingUnknown = "UNKNOWN" in events
# removes special types only used in arm (UNKNOWN, TORCTL, ARM_DEBUG, etc)
toDiscard = []
@@ -267,6 +268,10 @@
for eventType in list(toDiscard): events.discard(eventType)
+ # adds events unrecognized by arm if we're listening to the 'UNKNOWN' type
+ if isLoggingUnknown:
+ events.update(set(logPanel.getMissingEventTypes()))
+
setEvents = torTools.getConn().setControllerEvents(list(events))
# temporary hack for providing user selected events minus those that failed
@@ -409,6 +414,11 @@
#TorUtil.logfile = panels["log"]
#torTools.getConn().addTorCtlListener(panels["log"].tor_ctl_event)
+ # provides a notice about any event types tor supports but arm doesn't
+ missingEventTypes = logPanel.getMissingEventTypes()
+ if missingEventTypes:
+ pluralLabel = "s" if len(missingEventTypes) > 1 else ""
+ log.log(CONFIG["log.torEventTypeUnrecognized"], "arm doesn't recognize the following event type%s: %s (log 'UNKNOWN' events to see them)" % (pluralLabel, ", ".join(missingEventTypes)))
# tells revised panels to run as daemons
panels["header"].start()
Modified: arm/trunk/src/interface/logPanel.py
===================================================================
--- arm/trunk/src/interface/logPanel.py 2010-09-19 01:21:49 UTC (rev 23242)
+++ arm/trunk/src/interface/logPanel.py 2010-09-19 01:41:28 UTC (rev 23243)
@@ -92,6 +92,21 @@
if invalidFlags: raise ValueError(invalidFlags)
else: return expandedEvents
+def getMissingEventTypes():
+ """
+ Provides the event types the current torctl connection supports but arm
+ doesn't. This provides an empty list if no event types are missing, and None
+ if the GETINFO query fails.
+ """
+
+ torEventTypes = torTools.getConn().getInfo("events/names")
+
+ if torEventTypes:
+ torEventTypes = torEventTypes.split(" ")
+ armEventTypes = TOR_EVENT_TYPES.values()
+ return [event for event in torEventTypes if not event in armEventTypes]
+ else: return None # GETINFO call failed
+
def getLogFileEntries(runlevels, readLimit = None, addLimit = None):
"""
Parses tor's log file for past events matching the given runlevels, providing
@@ -611,29 +626,57 @@
# does the following with all runlevel types (tor, arm, and torctl):
# - pulls to the start of the list
# - condenses range if there's three or more in a row (ex. "ARM_INFO - WARN")
+ # - condense further if there's identical runlevel ranges for multiple
+ # types (ex. "NOTICE - ERR, ARM_NOTICE - ERR" becomes "TOR/ARM NOTICE - ERR")
tmpRunlevels = [] # runlevels pulled from the list (just the runlevel part)
+ runlevelRanges = [] # tuple of type, startLevel, endLevel for ranges to be consensed
+
+ # reverses runlevels and types so they're appended in the right order
+ reversedRunlevels = list(RUNLEVELS)
+ reversedRunlevels.reverse()
for prefix in ("TORCTL_", "ARM_", ""):
# blank ending runlevel forces the break condition to be reached at the end
- for runlevel in RUNLEVELS + [""]:
+ for runlevel in reversedRunlevels + [""]:
eventType = prefix + runlevel
- if eventType in eventsList:
+ if runlevel and eventType in eventsList:
# runlevel event found, move to the tmp list
eventsList.remove(eventType)
tmpRunlevels.append(runlevel)
elif tmpRunlevels:
# adds all tmp list entries to the start of eventsList
if len(tmpRunlevels) >= 3:
- # condense sequential runlevels
- startLevel, endLevel = tmpRunlevels[0], tmpRunlevels[-1]
- eventsList.insert(0, "%s%s - %s" % (prefix, startLevel, endLevel))
+ # save condense sequential runlevels to be added later
+ runlevelRanges.append((prefix, tmpRunlevels[-1], tmpRunlevels[0]))
else:
# adds runlevels individaully
- tmpRunlevels.reverse()
for tmpRunlevel in tmpRunlevels:
eventsList.insert(0, prefix + tmpRunlevel)
tmpRunlevels = []
+ # adds runlevel ranges, condensing if there's identical ranges
+ for i in range(len(runlevelRanges)):
+ if runlevelRanges[i]:
+ prefix, startLevel, endLevel = runlevelRanges[i]
+
+ # check for matching ranges
+ matches = []
+ for j in range(i + 1, len(runlevelRanges)):
+ if runlevelRanges[j] and runlevelRanges[j][1] == startLevel and runlevelRanges[j][2] == endLevel:
+ matches.append(runlevelRanges[j])
+ runlevelRanges[j] = None
+
+ if matches:
+ # strips underscores and replaces empty entries with "TOR"
+ prefixes = [entry[0] for entry in matches] + [prefix]
+ for k in range(len(prefixes)):
+ if prefixes[k] == "": prefixes[k] = "TOR"
+ else: prefixes[k] = prefixes[k].replace("_", "")
+
+ eventsList.insert(0, "%s %s - %s" % ("/".join(prefixes), startLevel, endLevel))
+ else:
+ eventsList.insert(0, "%s%s - %s" % (prefix, startLevel, endLevel))
+
# truncates to use an ellipsis if too long, for instance:
attrLabel = ", ".join(eventsList)
if currentPattern: attrLabel += " - filter: %s" % currentPattern
Modified: arm/trunk/src/util/torTools.py
===================================================================
--- arm/trunk/src/util/torTools.py 2010-09-19 01:21:49 UTC (rev 23242)
+++ arm/trunk/src/util/torTools.py 2010-09-19 01:41:28 UTC (rev 23243)
@@ -297,8 +297,8 @@
else: return result
# TODO: This could have client side caching if there were events to indicate
- # SETCONF events. Ask if these can be added to tor (then ask mike if he wants
- # client side caching included in TorCtl?).
+ # SETCONF events. See:
+ # https://trac.torproject.org/projects/tor/ticket/1692
def getOption(self, param, default = None, multiple = False, suppressExc = True):
"""
Queries the control port for the given configuration option, providing the
More information about the tor-commits
mailing list