April 2011 - tor-commits - lists.torproject.org

[arm/master] Backporting fix for parsing multi-line exit policies to current release.
by sebastian＠torproject.org 24 Apr '11

24 Apr '11

commit d50aefb997137c360b4cd51d31faf83d93d9417d Author: Damian Johnson <atagar(a)torproject.org> Date: Sat Aug 7 22:00:08 2010 +0000 Backporting fix for parsing multi-line exit policies to current release. svn:r22820 --- interface/connPanel.py | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/interface/connPanel.py b/interface/connPanel.py index d78c3ed..2fb1d72 100644 --- a/interface/connPanel.py +++ b/interface/connPanel.py @@ -8,7 … [View More]

1 0

[arm/master] Bumping the version for the hotfix.
by sebastian＠torproject.org 24 Apr '11

24 Apr '11

commit cfa0fd24792e7d5c95303d5de10727f04fec99d0 Author: Damian Johnson <atagar(a)torproject.org> Date: Fri Oct 8 16:42:15 2010 +0000 Bumping the version for the hotfix. svn:r23465 --- src/version.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/version.py b/src/version.py index dc41d36..08989e3 100644 --- a/src/version.py +++ b/src/version.py @@ -2,6 +2,6 @@ Provides arm's version and release date. """ -VERSION = '1.3.7' -… [View More]

1 0

[arm/master] Bumping version in debian changelog for hotfix release.
by sebastian＠torproject.org 24 Apr '11

24 Apr '11

commit 76dcdd263dc172c97e46516cecabca321257b5ce Author: Damian Johnson <atagar(a)torproject.org> Date: Fri Oct 8 17:00:14 2010 +0000 Bumping version in debian changelog for hotfix release. svn:r23466 --- debian/changelog | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/debian/changelog b/debian/changelog index 9dd15fa..c3664d8 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +tor-arm (1.3.7-1) unstable; urgency=… [View More]

1 0

[arm/master] More fixes for issues caught by weasel
by sebastian＠torproject.org 24 Apr '11

24 Apr '11

commit c43535aee0c9038fefa7867ffc511dea4e258568 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Dec 12 00:21:28 2010 +0000 More fixes for issues caught by weasel change: noting myself in the license file fix: startup script wasn't hanlding unusual paths, such as with spaces and dashes fix: startup script was forking the final process instad of calling exec fix: initially built diff differed from rebuilt version since it was missing the pycompat … [View More]

1 0

[arm/master] Changing arm's versioning scheme to differentiate it from debian.
by sebastian＠torproject.org 24 Apr '11

24 Apr '11

commit 8dac24227ae6a8eb61da31ea6cb64c0eb3ebc6a2 Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Dec 7 17:42:11 2010 +0000 Changing arm's versioning scheme to differentiate it from debian. svn:r23899 --- src/version.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/version.py b/src/version.py index fc30f0b..4a0b625 100644 --- a/src/version.py +++ b/src/version.py @@ -2,6 +2,6 @@ Provides arm's version and release date. "… [View More]

1 0

[arm/master] Merging to release for version 1.4.0
by sebastian＠torproject.org 24 Apr '11

24 Apr '11

commit f403cccb5119e4290c5dce370249f2cade3b7617 Merge: 76dcdd2 c43410c Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Nov 28 10:58:57 2010 +0000 Merging to release for version 1.4.0 svn:r23873 ChangeLog | 46 ++- README | 19 +- TODO | 103 +++--- arm | 4 +- arm.1 | 73 … [View More]++++ armrc.sample | 228 ++++++++++- debian/MANIFEST | 40 -- debian/arm.1.gz | Bin 1115 -> 0 bytes debian/changelog | 17 - debian/compat | 1 - debian/control | 28 -- debian/copyright | 27 -- debian/make-deb | 21 - debian/make-rpm | 21 - debian/manpages | 1 - debian/pycompat | 1 - debian/rules | 23 - debian/setup.cfg | 3 - debian/source/format | 1 - install | 4 +- setup.py | 60 +++- src/armrc.defaults | 229 ----------- src/interface/__init__.py | 2 +- src/interface/confPanel.py | 292 ------------- src/interface/configPanel.py | 312 ++++++++++++++ src/interface/connPanel.py | 3 +- src/interface/controller.py | 657 +++++++++++++++++++++++------- src/interface/graphing/bandwidthStats.py | 3 +- src/interface/graphing/graphPanel.py | 10 +- src/interface/graphing/psStats.py | 2 +- src/interface/headerPanel.py | 3 +- src/interface/logPanel.py | 126 ++++-- src/interface/torrcPanel.py | 221 ++++++++++ src/settings.cfg | 130 ++++++ src/starter.py | 162 ++++++-- src/uninstall | 2 +- src/util/__init__.py | 2 +- src/util/conf.py | 222 +++++++---- src/util/connections.py | 2 +- src/util/hostnames.py | 10 +- src/util/log.py | 8 +- src/util/panel.py | 96 ++++- src/util/sysTools.py | 20 + src/util/torConfig.py | 659 ++++++++++++++++++++++++++++++ src/util/torTools.py | 170 +++++++- src/util/uiTools.py | 181 ++++++++- src/version.py | 4 +- 47 files changed, 3091 insertions(+), 1158 deletions(-) diff --cc TODO index 6b0fe37,c5fe109..fa9d6ad --- a/TODO +++ b/TODO @@@ -35,17 -28,26 +28,35 @@@ TOD - pick apart applications like iftop and pktstat to see how they get per-connection bandwidth usage. Forum thread discussing it: https://bbs.archlinux.org/viewtopic.php?pid=715906 - - give usage stats for exit port usage (popup?) - - country data for client connections (requested by ioerror) + [ ] control port interpreter (interactive prompt) + Panel and startup option (-t maybe?) for providing raw control port + access along with usability improvements (piggybacking on the arm + connection): + - irc like help (ex "/help GETINFO" could provide a summary of + getinfo commands, partly using the results from + "GETINFO info/names") + - tab completion and up/down for previous commands + - warn and get confirmation if command would disrupt arm (for + instance 'SETEVENTS') + - 'safe' option that restricts to read-only access (start with this) + - issue sighup reset + [ ] low hanging fruit from the "client mode use cases" below + * release prep + * pylint --indent-string=" " --disable=C,R interface/foo.py | less + * double check __init__.py and README for changes + + - Roadmap for version 1.4.2 + [ ] refactor panels + [ ] controller and popup panels + [ ] attempt to clear controller password from memory + - http://www.codexon.com/posts/clearing-passwords-in-memory-with-python + * release prep + * pylint --indent-string=" " --disable=C,R interface/foo.py | less + * double check __init__.py and README for changes + +- Roadmap for version 1.3.9 + [ ] refactor panels + [ ] controller and popup panels - allow arm to resume after restarting tor This requires a full move to the torTools controller. - provide measurements for startup time, and try to improve bottlenecks diff --cc src/version.py index 08989e3,0b32fbf..a35c4ee --- a/src/version.py +++ b/src/version.py @@@ -2,6 -2,6 +2,6 @@@ Provides arm's version and release date. """ - VERSION = '1.3.7-1' - LAST_MODIFIED = "October 7, 2010" -VERSION = '1.4.0_dev' ++VERSION = '1.4.0' + LAST_MODIFIED = "November 27, 2010" [View Less]

1 0

[arm/master] Hotfixes for a couple issues discovered in the release and repairing the rpm build (not quite working yet, though).
by sebastian＠torproject.org 24 Apr '11

24 Apr '11

commit 79b7e13a0d6050f924b56ea8e62ecbf826917dbd Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Dec 1 03:55:25 2010 +0000 Hotfixes for a couple issues discovered in the release and repairing the rpm build (not quite working yet, though). fix: install script was failing to make the temporary directory for the compressed man page fix: a torrc validation log entries had a typo svn:r23882 --- setup.py | 8 +++++++- src/… [View More]

1 0

[arm/master] Reverting change to the license (looks like that isn't allowed) and removing copyright section from the man page.
by sebastian＠torproject.org 24 Apr '11

24 Apr '11

commit a40075d0258738b4620f95c541ca8fd20a2d2bfc Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Dec 12 01:02:54 2010 +0000 Reverting change to the license (looks like that isn't allowed) and removing copyright section from the man page. svn:r23927 --- LICENSE | 2 +- arm.1 | 3 --- 2 files changed, 1 insertions(+), 4 deletions(-) diff --git a/LICENSE b/LICENSE index 1528791..94a9ed0 100644 --- a/LICENSE +++ b/LICENSE @@ -1,7 +1,7 @@ … [View More]

1 0

[arm/master] Hotfix for the graphing issue tomb spotted as a hotfix. Also updating the TODO for kicks and giggles.
by sebastian＠torproject.org 24 Apr '11

24 Apr '11

commit e9d0ac02c1df9239e6dc094bbbd7e9e2b68bcf20 Author: Damian Johnson <atagar(a)torproject.org> Date: Fri Oct 8 16:39:11 2010 +0000 Hotfix for the graphing issue tomb spotted as a hotfix. Also updating the TODO for kicks and giggles. svn:r23464 --- TODO | 226 +++++++++++++++------------------ src/interface/graphing/graphPanel.py | 4 +- 2 files changed, 105 insertions(+), 125 deletions(-) diff --git a/TODO b/TODO index … [View More]3e0dbb0..6b0fe37 100644 --- a/TODO +++ b/TODO @@ -1,6 +1,6 @@ TODO -- Remaining work for next release (1.3.7) +- Roadmap and completed work for next release (1.3.8) [ ] refactor panels Currently the interface is a bit of a rat's nest (especially the controller). The goal is to use better modularization to both simplify @@ -9,50 +9,50 @@ TODO progress - /init and /util are done and /interface is partly done. Known bugs are being fixed while refactoring. - [X] log panel [ ] conf panel - move torrc validation into util - fetch text via getinfo rather than reading directly? - conn.get_info("config-text") + conn.get_info("config-text") - improve parsing failure notice to give line number - just giving "[ARM-WARN] Unable to validate torrc" isn't very - helpful... - * release prep - * ask helix about steps for getting a deb and rpm included in the tor repo - * check performance of this version vs last version (general screen refresh - times) - * stress test under debug level and debug level with large log file - * pylint --indent-string=" " --disable=C,R interface/foo.py | less - * double check __init__.py and README for changes - -- Roadmap for version 1.3.8 - [ ] refactor panels + just giving "[ARM-WARN] Unable to validate torrc" isn't very + helpful... [ ] conn panel - expand client connections and note location in circuit (entry-exit) - for clients list all connections to detect what's going through tor - and what isn't? If not then netstat calls are unnecessary. + and what isn't? If not then netstat calls are unnecessary. - check family connections to see if they're alive (VERSION cell - handshake?) + handshake?) - fallback when pid or connection querying via pid is unavailable - List all connections listed both by netstat and the consensus + List all connections listed both by netstat and the consensus - note when connection times are estimates (color?), ie connection - was established before arm + was established before arm - connection uptime to associate inbound/outbound connections? - Identify controller connections (if it's arm, vidalia, etc) with - special detail page for them + special detail page for them - provide bridge / client country statistics - Include bridge related data via GETINFO option (feature request - by waltman and ioerror). + Include bridge related data via GETINFO option (feature request + by waltman and ioerror). - pick apart applications like iftop and pktstat to see how they get - per-connection bandwidth usage. Forum thread discussing it: - https://bbs.archlinux.org/viewtopic.php?pid=715906 - [ ] controller and popup panels + per-connection bandwidth usage. Forum thread discussing it: + https://bbs.archlinux.org/viewtopic.php?pid=715906 + - give usage stats for exit port usage (popup?) - country data for client connections (requested by ioerror) + [ ] attempt to clear controller password from memory + - http://www.codexon.com/posts/clearing-passwords-in-memory-with-python + * release prep + * pylint --indent-string=" " --disable=C,R interface/foo.py | less + * double check __init__.py and README for changes + +- Roadmap for version 1.3.9 + [ ] refactor panels + [ ] controller and popup panels - allow arm to resume after restarting tor This requires a full move to the torTools controller. - - provide measurements for startup time, and try improving bottlenecks + - provide measurements for startup time, and try to improve bottlenecks [ ] setup scripts for arm [ ] updater (checks for a new tarball and installs it automatically) + - attempt to verify download signature, providing a warning if unable + to do so [ ] look into CAPs to get around permission issues for connection listing sudo wrapper for arm to help arm run as the same user as tor? Irc suggestions: @@ -60,6 +60,7 @@ TODO - http://www.linuxjournal.com/article/5737 - Bugs + * path for sample armrc in man page is wrong * when in client mode and tor stops the header panel doesn't say so * util are assuming that tor is running under the default command name attempt to determine the command name at runtime (if the pid is available @@ -106,12 +107,23 @@ TODO * connection uptimes shouldn't show fractions of a second * connections aren't cleared when control port closes -- Features - * general purpose method of erroring nicely - Some errors cause portions of the display to die, but curses limps along - and overwrites the stacktrace. This has been mostly solved, but all errors - should result in a clean death, with the stacktrace saved and a nice - message for the user. +- Future Features + * control port interpreter (interactive prompt) + Panel and startup option (-t maybe?) for providing raw control port access + along with usability improvements (piggybacking on the arm connection): + * irc like help (ex "/help GETINFO" could provide a summary of getinfo + commands, partly using the results from "GETINFO info/names") + * tab completion and up/down for previous commands + * warn and get confirmation if command would disrupt arm (for instance + 'SETEVENTS') + * 'safe' option that restricts to read-only access (start with this) + * issue sighup reset + * menus + * http://gnosis.cx/publish/programming/charming_python_6.html ? + * additional options: + * make update rates configurable via the ui + * dialog with flag descriptions and other help + * menu with all torrc options (making them editable/toggleable) * client mode use cases * not sure what sort of information would be useful in the header (to replace the orport, fingerprint, flags, etc) @@ -122,21 +134,35 @@ TODO [notice] Opening Socks listener on 127.0.0.1:9050 [notice] Opening Transparent pf/netfilter listener on 127.0.0.1:9040 [notice] Opening DNS listener on 127.0.0.1:53 - * rdns and whois lookups (to find ISP, country, and jurisdiction, etc) - To avoid disclosing connection data to third parties this needs to be - an all-or-nothing operation (ie, needs to fetch information on all - relays or none of them). Plan is something like: + * rdns and whois lookups (to find ISP, country, and jurisdiction, etc) + To avoid disclosing connection data to third parties this needs to be + an all-or-nothing operation (ie, needs to fetch information on all + relays or none of them). Plan is something like: * add resolving/caching capabilities to fetch information on all relays and distil whois entries to just what we care about (hosting provider or ISP), by default updating the cache on a daily basis * construct tarball and make this available for download rather than fetching everything at each client - * possibly make these archives downloadable from peer relays (note: - this is a no-go for clients) via torrents or some dirport like scheme - * special page for client related information, such as ips of our client - circuits at the exit + * possibly make these archives downloadable from peer relays (this is a + no-go for clients) via torrents or some dirport like scheme * look at vidalia for ideas * need to solicit for ideas on what would be most helpful to clients + * general purpose method of erroring nicely + Some errors cause portions of the display to die, but curses limps along + and overwrites the stacktrace. This has been mostly solved, but all errors + should result in a clean death, with the stacktrace saved and a nice + message for the user. + * handle mutiple tor instances + * screen style (dialog for switching between instances) + * extra window with whatever stats can be aggregated over all instances + * option to save the current settings to the config + * provide warning at startup if the armrc doesn't exist, with instructions + for generating it + * email alerts for changes to the relay's status, similar to tor-weather + * simple alert if tor shuts down + * accounting and alerts for if the bandwidth drops to zero + * daily/weekly/etc alerts for basic status (log output, bandwidth history, + etc), borrowing from the consensus tracker for some of the formatting * mac installer * Couple of options include macport and dmg... * macport (http://guide.macports.org/#development) @@ -153,91 +179,45 @@ TODO Plugin for distutils. Like most mac packaging, this can only run on a mac. It also requires setuptools: http://www.errorhelp.com/search/details/74034/importerror-no-module-named-s… - * email alerts for changes to the relay's status, similar to tor-weather - * simple alert if tor shuts down - * accounting and alerts for if the bandwidth drops to zero - * daily/weekly/etc alerts for basic status (log output, bandwidth history, - etc), borrowing from the consensus tracker for some of the formatting - * check if batch getInfo/getOption calls provide much performance benefit - * page with details on client circuits, attempting to detect details like - country, ISP, latency, exit policy for the circuit, traffic, etc - * attempt to clear controller password from memory - http://www.codexon.com/posts/clearing-passwords-in-memory-with-python + * look into additions to the used apis + * curses (python 2.6 extended?): http://docs.python.org/library/curses.html + * new control options (like "desc-annotations/id/<OR identity>")? + * look into better supporting hidden services (what could be useful here?) + * provide option for a consensus page + Shows full consensus with an interface similar to the connection panel. + For this Mike's ConsensusTracker would be helpful (though boost the + startup time by several seconds) + * show qos stats + Take a look at 'linux-tor-prio.sh' to see if any of the stats are + available and interesting. * escaping function for uiTools' formatted strings - * make update rates configurable via the ui - Also provide option for saving these settings to the config - * config option to cap resource usage - * dialog with flag descriptions and other help * switch check of ip address validity to regex? - match = re.match("(\d*)\.(\d*)\.(\d*)\.(\d*)", ip) - http://wang.yuxuan.org/blog/2009/4/2/python_script_to_convert_from_ip_range… - * audit tor connections - Provide warnings if tor misbehaves, checks possibly including: - - ensuring ExitPolicyRejectPrivate is being obeyed - - check that ExitPolicy violations don't occur (not possible yet since - not all relays aren't identified) - - check that all connections are properly related to a circuit, for - instance no outbound connections without a corresponding inbound (not - possible yet due to being unable to correlate connections to circuits) - * check file descriptors being accessed by tor to see if they're outside the - known pattern - * add page that allows raw control port access - Start with -t (or -c?) option for commandline-only access with help, - syntax highlighting, and other spiffy extras - - Piggyback on the arm connection, providing something like an interactive - prompt. In addition, provide: - - irc like help (ex "/help GETINFO" could provide a summary of getinfo - commands, partly using the results from "GETINFO info/names") - - tab completion and up/down populates previous entries - - warn and get confirmation if command would disrupt arm (for instance - 'SETEVENTS') - - 'guard' option that restricts to GETINFO only (start with this) - - issue sighup reset - * menu with all torrc options (making them editable/toggleable) - * Setup wizard for new relays - Setting the password and such for torrc generation (idea by ioerror) - * menus? - http://gnosis.cx/publish/programming/charming_python_6.html - * look into better supporting hidden services (what could be useful here?) - * look into providing UPnP support - This might be provided by tor itself so wait and see... + match = re.match("(\d*)\.(\d*)\.(\d*)\.(\d*)", ip) + http://wang.yuxuan.org/blog/2009/4/2/python_script_to_convert_from_ip_range… + * setup wizard for new relays + Setting the password and such for torrc generation. Maybe a netinstaller + that fetches the right package for the plagform, verifies signatures, etc? + (idea by ioerror) + * audit what tor does + * Provide warnings if tor connections misbehaves, for instance: + * ensuring ExitPolicyRejectPrivate is being obeyed + * check that ExitPolicy violations don't occur (not possible yet since + not all relays aren't identified) + * check that all connections are properly related to a circuit, for + instance no outbound connections without a corresponding inbound (not + possible yet due to being unable to correlate connections to circuits) + * check file descriptors being accessed by tor to see if they're outside a + known pattern + * script that dumps relay stats to stdout + Derived from an idea by StrangeCharm. Django has a small terminal coloring + module that could be nice for formatting. Could possibly include: + * desc / ns information for our relay + * ps / netstat stats like load, uptime, and connection counts, etc + * implement control-spec proposals: + * https://gitweb.torproject.org/tor.git/blob/HEAD:/doc/spec/proposals/172-cir… + * https://gitweb.torproject.org/tor.git/blob/HEAD:/doc/spec/proposals/173-get… * unit tests - Primarily for util, for instance 'addfstr' woudl be a good candidate. - * show qos stats - Take a look at 'linux-tor-prio.sh' to see if any of the stats are - available and interesting. - * handle mutiple tor instances - First multiple tor instances on the same system, then via remote - connections too. - * Investigations of other possible tools: - * look into additions to the used apis - - curses (python 2.6 extended?): http://docs.python.org/library/curses.html - - new control options (like "desc-annotations/id/<OR identity>")? - - look deeper into TorCtl functions (has a resolve function? hu?) - * whois lookup for relays? ISP listing? - * look into what sort of information tcpdump and iptraf provides (probably - can't use for privacy reasons) - * vnstat, nload, mrtg, and traceroute - -- Ideas (low priority) + Primarily for util, for instance 'addfstr' would be a good candidate. * python 3 compatibility - Currently blocked on TorCtl support. - * bundle script that dumps relay stats to stdout - Django has a small terminal coloring module that could be nice for - formatting. Could possibly include: - - desc / ns information for our relay - - ps / netstat stats like load, uptime, and connection counts, etc - derived from an idea by StrangeCharm - * localization - Abstract strings from code and provide on translation portal. Thus far - there hasn't been any requests for this. - * provide option for a consensus page - Shows full consensus with an interface similar to the connection panel. - For this Mike's ConsensusTracker would be helpful (though boost the - startup time by several seconds) - * follow up on control-spec proposals - Proposal and related information is available at: - https://gitweb.torproject.org/tor.git/blob/HEAD:/doc/spec/proposals/172-cir… - https://gitweb.torproject.org/tor.git/blob/HEAD:/doc/spec/proposals/173-get… + Currently blocked on TorCtl support. diff --git a/src/interface/graphing/graphPanel.py b/src/interface/graphing/graphPanel.py index 9a8e1e4..706db75 100644 --- a/src/interface/graphing/graphPanel.py +++ b/src/interface/graphing/graphPanel.py @@ -340,11 +340,11 @@ class GraphPanel(panel.Panel): # creates bar graph (both primary and secondary) for col in range(graphCol): - colCount = param.primaryCounts[self.updateInterval][col + 1] - primaryMinBound + colCount = int(param.primaryCounts[self.updateInterval][col + 1]) - primaryMinBound colHeight = min(self.graphHeight, self.graphHeight * colCount / (max(1, primaryMaxBound) - primaryMinBound)) for row in range(colHeight): self.addstr(self.graphHeight + 1 - row, col + 5, " ", curses.A_STANDOUT | primaryColor) - colCount = param.secondaryCounts[self.updateInterval][col + 1] - secondaryMinBound + colCount = int(param.secondaryCounts[self.updateInterval][col + 1]) - secondaryMinBound colHeight = min(self.graphHeight, self.graphHeight * colCount / (max(1, secondaryMaxBound) - secondaryMinBound)) for row in range(colHeight): self.addstr(self.graphHeight + 1 - row, col + graphCol + 10, " ", curses.A_STANDOUT | secondaryColor) [View Less]

1 0

[arm/master] Removing dependency on python 2.6 for installation and building.
by sebastian＠torproject.org 24 Apr '11

24 Apr '11

commit 66b3c94952d819341fc374326ff3cc7178cbb447 Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Dec 7 17:46:03 2010 +0000 Removing dependency on python 2.6 for installation and building. This was using the gzip name attribute unnecessarily which isn't available in older python versions (caught by weasel). svn:r23900 --- setup.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/setup.py b/setup.py index d11080f..edae307 … [View More]

1 0