- tor-commits - lists.torproject.org

[flashproxy/master] Print "read" log messages only in debugging mode.
by dcf＠torproject.org 24 Jul '11

24 Jul '11

commit 06d95087533353642e8c733d82db64c4ba1193d4 Author: David Fifield <david(a)bamsoftware.com> Date: Sat Jul 23 01:25:01 2011 +0000 Print "read" log messages only in debugging mode. --- ProxyPair.as | 8 +++++++- swfcat.as | 6 +++++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/ProxyPair.as b/ProxyPair.as index 2b66532..10c4263 100644 --- a/ProxyPair.as +++ b/ProxyPair.as @@ -42,6 +42,12 @@ package ui.puts(msg) } + public function logdebug(msg:String):void + { + if (ui.debug) + log(msg); + } + public function set_name(name:String):void { this.name = name; @@ -129,7 +135,7 @@ package s_to.writeBytes(bytes); s_to.flush(); ui.rate_limit.update(n); - log(label + ": read " + bytes.length + "."); + logdebug(label + ": read " + bytes.length + "."); } /* Send as much data as the rate limit currently allows. */ diff --git a/swfcat.as b/swfcat.as index 9f3f967..73f3760 100644 --- a/swfcat.as +++ b/swfcat.as @@ -54,6 +54,8 @@ package private var local_addr:Object; + public var debug:Boolean; + public var rate_limit:RateLimit; public function puts(s:String):void @@ -85,7 +87,9 @@ package private function loaderinfo_complete(e:Event):void { - if (this.loaderInfo.parameters["debug"] || this.loaderInfo.parameters["client"]) { + debug = this.loaderInfo.parameters["debug"]; + + if (debug || this.loaderInfo.parameters["client"]) { output_text = new TextField(); output_text.width = stage.stageWidth; output_text.height = stage.stageHeight;

1 0

r24927: {website} changed ProxyMob to "Proxy Mobile" for more accurate searchi (website/trunk/docs/en)
by Nathan Freitas 23 Jul '11

23 Jul '11

Author: n8fr8 Date: 2011-07-22 20:16:33 +0000 (Fri, 22 Jul 2011) New Revision: 24927 Modified: website/trunk/docs/en/android.wml Log: changed ProxyMob to "Proxy Mobile" for more accurate searching Modified: website/trunk/docs/en/android.wml =================================================================== --- website/trunk/docs/en/android.wml 2011-07-21 15:40:41 UTC (rev 24926) +++ website/trunk/docs/en/android.wml 2011-07-22 20:16:33 UTC (rev 24927) @@ -110,7 +110,7 @@ <li>You must root your device and update the firmware to an iptables-capable ROM for Orbot to transparently proxy all TCP traffic.</li> <li>For non-modified and non-rooted phones, you'll want to manually configure HTTP or SOCKS proxy settings for specific applications.</li> <li>If you root your device and install an iptables-capable ROM (such as <a href="http://cyanogenmod.com">Cyanogen</a>), Orbot can transparently proxy traffic on an app-by-app basis through Tor.</li> - <li>You can also install Firefox for Android from the market with the <a href="https://guardianproject.info/apps/proxymob">ProxyMob Add-on</a> or install the text-only “NDBrowser” from the Android Market. + <li>You can also install Firefox for Android from the market with the <a href="https://guardianproject.info/apps/proxymob">Proxy Mobile Add-on</a> or install the text-only “NDBrowser” from the Android Market. Both of these solutions all routing of web access through Tor on standard, un-rooted devices.</li> <li>For Instant Messaging, try <a href="https://guardianproject.info/apps/gibber">Gibberbot</a>, which includes support for connecting via Tor and Off-the-Record encryption.</li> </ul>

1 0

[obfsproxy/master] Make tor-obfs-howto.txt a bit more user-friendly.
by nickm＠torproject.org 23 Jul '11

23 Jul '11

commit e1a825f7b27fd315d897dab5c0010388fc7ba4c2 Author: George Kadianakis <desnacked(a)gmail.com> Date: Fri Jul 22 20:53:06 2011 +0200 Make tor-obfs-howto.txt a bit more user-friendly. --- doc/tor-obfs-howto.txt | 134 +++++++++++++++++++++++++----------------------- 1 files changed, 69 insertions(+), 65 deletions(-) diff --git a/doc/tor-obfs-howto.txt b/doc/tor-obfs-howto.txt index ed8ba72..324a98c 100644 --- a/doc/tor-obfs-howto.txt +++ b/doc/tor-obfs-howto.txt @@ -1,101 +1,105 @@ -How to set up Tor with Obfsproxy +How to set up Tor with obfsproxy ================================ - Preliminaries: + Instructions: -This isn't the final interface--we should make it much easier. + "I AM A CENSORED FELLOW" -This is experimental code--if it breaks, you get to keep both smouldering -pieces. - - - - Ingredients: - -Client side needs: +[1] You will need: * a copy of obfsproxy: git clone git://git.torproject.org/obfsproxy.git Building it should be easy, just do "./configure && make". * a copy of Tor that supports the Socks5Proxy option. - The git version should do the trick. + Tor 0.2.2.30 - for example - will work. -Bridge side needs: +[2] Set up obfsproxy: -* a copy of obfsproxy: - git clone git://git.torproject.org/obfsproxy.git - Building it should be easy, just do "./configure && make". +This command will setup an obfsproxy client speaking the obfs2 +protocol on localhost:1050: -* a copy of Tor with BridgeRelay capabilities. - Quite easy to find nowadays. +./obfsproxy obfs2 socks 127.0.0.1:1050 - Port setup: +[3] Set up tor: -Our port setup will be like this: +You should put in your torrc file: +--- +SocksPort 5000 +UseBridges 1 +Bridge <bridge address>:1051 # This is provided by the bridge operator. +Socks5Proxy 127.0.0.1:1050 # This points to the obfsproxy client. +--- - 1050 1051 - +-----------+ +------------+ - ----| Proxy |-------------| Server |---- - | +-----------+ +------------+ | - | | -+------------+ +--------------+ -| Tor Client | | Tor Bridge | -+------------+ +--------------+ - 5000 5001 +Of course you should substitute the value of the Bridge line above, +with the information provided to you by a bridge operator. +[4] Done! +Alright, you are done! +Make sure you have followed the above steps on the correct order: +First launch obfsproxy and then launch tor. - Setting up obfsproxies: +Now to test it, launch Firefox and set it up to use a Socks server on +127.0.0.1:5000. Now browse the web. -This command will setup an obfsproxy socks client listening to the -obfs2 protocol on localhost:1050: -./obfsproxy obfs2 socks 127.0.0.1:1050 -This command will setup an obfsproxy server listening to the -obfs2 protocol on localhost:1051: -./obfsproxy obfs2 --dest 127.0.0.1:5001 server 127.0.0.1:1051 + "I AM A BRIDGE OPERATOR THAT WANTS TO HELP CENSORED PEOPLE" - Setting up Tor: +[1] You will need: -Let's create .torrc_client and .torrc_bridge. -Warning: I'm only posting the relevant torrc options. +* a copy of obfsproxy: + git clone git://git.torproject.org/obfsproxy.git + Building it should be easy, just do "./configure && make". -torrc_client: ---- -SocksPort 5000 -UseBridges 1 -Bridge <bridge host>:1051 # This points to the bridge's obfsproxy server. - # You can change <bridge host> to 127.0.0.1 - # for local testing setup. -Socks5Proxy 127.0.0.1:1050 # This points to our obfsproxy client. ---- +* a copy of Tor which can function as a Tor bridge. + +[2] Set up obfsproxy: + +This command will setup an obfsproxy server speaking the obfs2 +protocol on localhost:1051: -torrc_bridge: +./obfsproxy obfs2 --dest 127.0.0.1:5001 server 127.0.0.1:1051 + +Be sure to pass the value of ORPort of your bridge relay to the --dest +argument (see next section). + +[3] Set up tor: + +You should put in your torrc file: --- -ORPort 5001 # Port where bridge will be listening on. +ORPort 5001 # Port on which bridge will be listening on. BridgeRelay 1 ExitPolicy reject *:* --- - Bootstrap sequence: +[4] Done! + +Make sure you have followed the above steps on the correct order. +First launch obfsproxy and then launch tor. +You can use any port numbers you want, just be sure to adjust the +torrc and obfsproxy commands accordingly. + +Don't forget that You might need to set up port forwarding. -Since we are still in PoC state you have to bootstrap the Tors and the -obfsproxies with the correct sequence: +Now you should tell censored people to connect to your IP on port 1051. +So, for example, if your bridge's address is 85.22.13.1, bridge users +should connect to 85.22.13.1:1051. +Pay attention that '1051' is the port where obfsproxy is listening on, +and *not* your bridge's ORPort. -1. Fire up server obfsproxy. -2. Fire up tor with torrc_bridge. -3. Fire up client obfsproxy. -4. Fire up tor with torrc_client. + Network diagram of the above setup: - The End +(If you don't understand this drawing, just ignore this section.) -Now open Firefox on the client side and set it up to use a Socks -server on 127.0.0.1:5000. -Stuff should work now. + 1050 1051 + +-----------+ +------------+ + ----| Proxy |------[internet]------| Server |---- + | +-----------+ +------------+ | + | | ++------------+ +--------------+ +| Tor Client | | Tor Bridge | ++------------+ +--------------+ + 5000 5001 -If you were smart enough to open wireshark and sniff the traffic -before doing all the above, you would notice that the TLS handshake -was not carried out plaintext, effectively obfuscating your Tor -experience. Yes.

1 0

[flashproxy/master] Less sleeping in 01, too lazy
by dcf＠torproject.org 23 Jul '11

23 Jul '11

commit c0701bcd20dd85949e4233056c3c199a7178aeec Author: Jonathan Ellithorpe <jdellit(a)stanford.edu> Date: Tue Jul 19 18:55:30 2011 -0700 Less sleeping in 01, too lazy --- experiments/01 | 6 +----- 1 files changed, 1 insertions(+), 5 deletions(-) diff --git a/experiments/01 b/experiments/01 index 10ee99d..ff0179d 100755 --- a/experiments/01 +++ b/experiments/01 @@ -47,15 +47,10 @@ until [ $i -gt $NUM_CLIENTS ]; do sleep 2 done -# Some connections take a long time to register with cirrus -sleep 15 - # Start the flash proxy, server side echo "starting flash proxy server side..." firefox --remote "openURL(http://localhost:1080/swfcat.swf?facilitator=127.0.0.1:4701)" -sleep 15 - let query_fac_time=NUM_CLIENTS+2 sleep ${query_fac_time} @@ -66,6 +61,7 @@ until [ $i -gt $NUM_CLIENTS ]; do let local_port=2000+i gnome-terminal -e "/bin/bash -c 'wget -t1 http://localhost:${local_port}/dump -O - > /dev/null; exec /bin/bash -i'" let i=i+1 + sleep 2 done # Use this one for delaying the termination of experiment below

1 0

[flashproxy/master] Use new-window to open RTMFP clients.
by dcf＠torproject.org 23 Jul '11

23 Jul '11

commit 4043f75c01779cfb82c003e77c4db23615d1bff0 Author: David Fifield <david(a)bamsoftware.com> Date: Mon Jul 18 06:04:28 2011 +0000 Use new-window to open RTMFP clients. --- RTMFPSocket.as | 9 ++++++--- experiments/00 | 3 +-- experiments/01 | 4 ++-- swfcat.as | 2 +- 4 files changed, 10 insertions(+), 8 deletions(-) diff --git a/RTMFPSocket.as b/RTMFPSocket.as index 9dca232..e91d06d 100644 --- a/RTMFPSocket.as +++ b/RTMFPSocket.as @@ -183,9 +183,12 @@ package public function close():void { - outgoing.close(); - incoming.close(); - nc.close(); + if (outgoing) + outgoing.close(); + if (incoming) + incoming.close(); + if (nc) + nc.close(); } public function readBytes(output:ByteArray, offset:uint = 0, length:uint = 0):void diff --git a/experiments/00 b/experiments/00 index ecc8c48..bc4befb 100755 --- a/experiments/00 +++ b/experiments/00 @@ -60,9 +60,8 @@ echo "getting the dump file..." i=2 until [ $i -gt $NUM_CLIENTS ]; do let local_port=2000+i - gnome-terminal -e "/bin/bash -c 'wget -t1 http://localhost:${local_port}/dump -O - > /dev/null; exec /bin/bash -i'" + xterm -e "/bin/bash -c 'wget -t1 http://localhost:${local_port}/dump -O - > /dev/null; exec /bin/bash -i'" & let i=i+1 - sleep 2 done # Use this one for delaying the termination of experiment below diff --git a/experiments/01 b/experiments/01 index ff0179d..0c273e7 100755 --- a/experiments/01 +++ b/experiments/01 @@ -42,14 +42,14 @@ echo "starting flash proxy client side..." i=1 until [ $i -gt $NUM_CLIENTS ]; do let local_port=9000+i - firefox --remote "openURL(http://localhost:1080/swfcat.swf?local=127.0.0.1:${local_port}&client=1&facilitator=127.0.0.1:4701)" + firefox --remote "openURL(http://localhost:1080/swfcat.swf?local=127.0.0.1:${local_port}&client=1&facilitator=127.0.0.1:4701,new-window)" let i=i+1 sleep 2 done # Start the flash proxy, server side echo "starting flash proxy server side..." -firefox --remote "openURL(http://localhost:1080/swfcat.swf?facilitator=127.0.0.1:4701)" +firefox --remote "openURL(http://localhost:1080/swfcat.swf?facilitator=127.0.0.1:4701,new-window)" let query_fac_time=NUM_CLIENTS+2 sleep ${query_fac_time} diff --git a/swfcat.as b/swfcat.as index c93c154..9f3f967 100644 --- a/swfcat.as +++ b/swfcat.as @@ -31,7 +31,7 @@ package port: 9002 }; - private const MAX_NUM_PROXY_PAIRS:uint = 10; + private const MAX_NUM_PROXY_PAIRS:uint = 100; // Milliseconds. private const FACILITATOR_POLL_INTERVAL:int = 1000;

1 0

[flashproxy/master] Fix a memory leak in RTMFPSocket.
by dcf＠torproject.org 23 Jul '11

23 Jul '11

commit 54bb6795dd0879027498a0a105d976616d2dc8f8 Author: David Fifield <david(a)bamsoftware.com> Date: Mon Jul 18 06:33:51 2011 +0000 Fix a memory leak in RTMFPSocket. There was a ByteArray that always grew. When the array has been emptied, it's safe to collapse it back down to 0 bytes. This was being done before but it was in the wrong place. --- RTMFPSocket.as | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/RTMFPSocket.as b/RTMFPSocket.as index e91d06d..4de3502 100644 --- a/RTMFPSocket.as +++ b/RTMFPSocket.as @@ -157,11 +157,6 @@ package event.bytesLoaded = bytes.bytesAvailable; bytes.readBytes(buffer, buffer.length, bytes.bytesAvailable); - if (bytes.bytesAvailable == 0) { - /* Reclaim memory space. */ - bytes.position = 0; - bytes.length = 0; - } dispatchEvent(event); } @@ -194,6 +189,11 @@ package public function readBytes(output:ByteArray, offset:uint = 0, length:uint = 0):void { buffer.readBytes(output, offset, length); + if (buffer.bytesAvailable == 0) { + /* Reclaim memory space. */ + buffer.position = 0; + buffer.length = 0; + } } public function writeBytes(input:ByteArray, offset:uint = 0, length:uint = 0):void

1 0

r24926: {website} updated perl-globals to properly include languages (website/trunk/include)
by Runa Sandvik 22 Jul '11

22 Jul '11

Author: runa Date: 2011-07-21 15:40:41 +0000 (Thu, 21 Jul 2011) New Revision: 24926 Modified: website/trunk/include/perl-globals.wmi Log: updated perl-globals to properly include languages Modified: website/trunk/include/perl-globals.wmi =================================================================== --- website/trunk/include/perl-globals.wmi 2011-07-21 15:20:43 UTC (rev 24925) +++ website/trunk/include/perl-globals.wmi 2011-07-21 15:40:41 UTC (rev 24926) @@ -5,7 +5,10 @@ my @LANGUAGES = qw{ar ar-se bms cy da de en es et fa fi fr hu id it ja ko nl no pl pt pt-br ru se tr vn zh-cn}; my %LANGUAGES = ( 'ar' => "عربية (Arabiya)", + 'ar-se' => "Arabic (Saudi Arabia)", 'bms' => "Burmese", + 'cy' => "Welsh", + 'da' => "Danish", 'de' => "Deutsch", 'en' => "English", 'es' => "español", @@ -13,6 +16,8 @@ 'fa' => "فارسی (Fārsī)", 'fi' => "suomi", 'fr' => "français", + 'hu' => "Hungarian", + 'id' => "Bahasa Indonesia", 'it' => "Italiano", 'ja' => "日本語 (Nihongo)", 'ko' => "한국어 (Hangul)", @@ -20,9 +25,11 @@ 'no' => "norsk", 'pl' => "polski", 'pt' => "Português", + 'pt-br' => "Portuguese (Brazil)", 'ru' => "Русский (Russkij)", 'se' => "svenska", 'tr' => "Türkçe", + 'vn' => "Vietnamese", 'zh-cn' => "中文(简) (Simplified Chinese)", );

1 0

r24925: {website} update perl-globals.wmi to include more languages (website/trunk/include)
by Runa Sandvik 22 Jul '11

22 Jul '11

Author: runa Date: 2011-07-21 15:20:43 +0000 (Thu, 21 Jul 2011) New Revision: 24925 Modified: website/trunk/include/perl-globals.wmi Log: update perl-globals.wmi to include more languages Modified: website/trunk/include/perl-globals.wmi =================================================================== --- website/trunk/include/perl-globals.wmi 2011-07-21 15:12:53 UTC (rev 24924) +++ website/trunk/include/perl-globals.wmi 2011-07-21 15:20:43 UTC (rev 24925) @@ -2,7 +2,7 @@ <: # Change both. The array is ordered (for pages like translation status, order of flags, etc). - my @LANGUAGES = qw{ar bms de en es et fa fi fr it ja ko nl no pl pt ru se tr zh-cn}; + my @LANGUAGES = qw{ar ar-se bms cy da de en es et fa fi fr hu id it ja ko nl no pl pt pt-br ru se tr vn zh-cn}; my %LANGUAGES = ( 'ar' => "عربية (Arabiya)", 'bms' => "Burmese",

1 0

r24924: {website} include more languages when building the website (website/trunk)
by Runa Sandvik 22 Jul '11

22 Jul '11

Author: runa Date: 2011-07-21 15:12:53 +0000 (Thu, 21 Jul 2011) New Revision: 24924 Modified: website/trunk/Makefile.common Log: include more languages when building the website Modified: website/trunk/Makefile.common =================================================================== --- website/trunk/Makefile.common 2011-07-21 10:00:37 UTC (rev 24923) +++ website/trunk/Makefile.common 2011-07-21 15:12:53 UTC (rev 24924) @@ -18,7 +18,7 @@ -D STABLETAG=$(STABLETAG) #LANGS=ar bms de en es et fa it fi fr ja ko nl no pl pt ru se tr zh-cn -LANGS=en ar es fr ru pl da it fa de fi vn cy zh-cn pt pt-br id nl +LANGS=ar ar-se bms cy da de en es et fa fi fr hu id it ja ko nl no pl pt pt-br ru se tr vn zh-cn WMLFILES=$(wildcard $(patsubst %, %/*.wml, $(LANGS))) WMIFILES=$(wildcard $(patsubst %, %/*.wmi, $(LANGS)) $(WMLBASE)/include/*.wmi ) @@ -35,9 +35,15 @@ %.html.ar: ar/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ +%.html.ar-se: ar-se/%.wml en/%.wml + lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ + %.html.bms: bms/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ +%.html.cy: cy/%.wml en/%.wml + lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ + %.html.es: es/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ @@ -65,6 +71,9 @@ %.html.fr: fr/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ +%.html.hu: hu/%.wml en/%.wml + lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ + %.html.ja: ja/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ @@ -98,6 +107,9 @@ %.html.vi: vi/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ +%.html.vn: vn/%.wml en/%.wml + lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ + %.html.zh-cn: zh-cn/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ @@ -115,6 +127,13 @@ wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $$OUT --depend | tee $$tmpfile > $@ && \ sed -e s',$^[^ ]*$:,.deps/\1.d:,' < $$tmpfile >> $@ && \ rm -f $$tmpfile +.deps/%.html.ar-se.d: ar-se/%.wml .deps/.stamp + tmpfile=`mktemp -t tmp.XXXXXXX` \ + lang=`dirname $<` && \ + OUT=`echo $@ | sed -e 's,\.deps/$.*$\.d$$,\1,'` && \ + wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $$OUT --depend | tee $$tmpfile > $@ && \ + sed -e s',$^[^ ]*$:,.deps/\1.d:,' < $$tmpfile >> $@ && \ + rm -f $$tmpfile .deps/%.html.bms.d: bms/%.wml .deps/.stamp tmpfile=`mktemp -t tmp.XXXXXXX` \ lang=`dirname $<` && \ @@ -192,6 +211,13 @@ wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $$OUT --depend | tee $$tmpfile > $@ && \ sed -e s',$^[^ ]*$:,.deps/\1.d:,' < $$tmpfile >> $@ && \ rm -f $$tmpfile +.deps/%.html.hu.d: hu/%.wml .deps/.stamp + tmpfile=`mktemp -t tmp.XXXXXXX` \ + lang=`dirname $<` && \ + OUT=`echo $@ | sed -e 's,\.deps/$.*$\.d$$,\1,'` && \ + wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $$OUT --depend | tee $$tmpfile > $@ && \ + sed -e s',$^[^ ]*$:,.deps/\1.d:,' < $$tmpfile >> $@ && \ + rm -f $$tmpfile .deps/%.html.ja.d: ja/%.wml .deps/.stamp tmpfile=`mktemp -t tmp.XXXXXXX` \ lang=`dirname $<` && \ @@ -269,6 +295,13 @@ wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $$OUT --depend | tee $$tmpfile > $@ && \ sed -e s',$^[^ ]*$:,.deps/\1.d:,' < $$tmpfile >> $@ && \ rm -f $$tmpfile +.deps/%.html.vn.d: vn/%.wml .deps/.stamp + tmpfile=`mktemp -t tmp.XXXXXXX` \ + lang=`dirname $<` && \ + OUT=`echo $@ | sed -e 's,\.deps/$.*$\.d$$,\1,'` && \ + wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $$OUT --depend | tee $$tmpfile > $@ && \ + sed -e s',$^[^ ]*$:,.deps/\1.d:,' < $$tmpfile >> $@ && \ + rm -f $$tmpfile .deps/%.html.zh-cn.d: zh-cn/%.wml .deps/.stamp tmpfile=`mktemp -t tmp.XXXXXXX` \ lang=`dirname $<` && \

1 0

[metrics-tasks/master] Avoid downloading server descriptor tarballs in #2394.
by karsten＠torproject.org 22 Jul '11

22 Jul '11

commit 7b75c704f5dad573ad6ba4f0a04cb8599b4cdb7a Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jul 21 16:01:14 2011 +0200 Avoid downloading server descriptor tarballs in #2394. --- task-2394/ParseDescriptors.java | 133 ++++++++++++++++++++++----------------- task-2394/README | 17 ++---- 2 files changed, 80 insertions(+), 70 deletions(-) diff --git a/task-2394/ParseDescriptors.java b/task-2394/ParseDescriptors.java index 2197670..641518d 100644 --- a/task-2394/ParseDescriptors.java +++ b/task-2394/ParseDescriptors.java @@ -21,13 +21,13 @@ public class ParseDescriptors { consensuses.put(filename, file); } else if (filename.endsWith("-votes")) { votes.put(filename, file); - } else if (filename.length() == 40) { + } else if (filename.endsWith("-descriptors")) { descriptors.put(filename, file); } } System.out.println("We found " + consensuses.size() - + " consensuses, " + votes.size() + " votes, and " - + descriptors.size() + " server descriptors."); + + " consensus files, " + votes.size() + " vote files, and " + + descriptors.size() + " server descriptor files."); /* Parse consensuses in an outer loop and the referenced votes and * descriptors in inner loops. Write the results to disk as soon as @@ -50,7 +50,6 @@ public class ParseDescriptors { bw.write("," + bandwidthAuthority + "bandwidth"); } bw.write("\n"); - Map<String, String> parsedDescriptors = new HashMap<String, String>(); for (File consensusFile : consensuses.values()) { System.out.println("Parsing consensus " + consensusFile.getName()); BufferedReader brC = new BufferedReader(new FileReader( @@ -58,12 +57,12 @@ public class ParseDescriptors { String lastRLine = null, lastSLine = null; String consensusTimestamp = consensusFile.getName().substring(0, "YYYY-MM-DD-hh-mm-ss".length()); - String votesFilename = consensusTimestamp + "-votes"; Map<String, Map<String, String>> measuredBandwidthsByDirSource = new HashMap<String, Map<String, String>>(); /* Parse votes first, if we have them, and extract measured * bandwidths. */ + String votesFilename = consensusTimestamp + "-votes"; if (votes.containsKey(votesFilename)) { BufferedReader brV = new BufferedReader(new FileReader( votes.get(votesFilename))); @@ -89,6 +88,74 @@ public class ParseDescriptors { brV.close(); } + /* Parse referenced server descriptors to learn about exit policies + * and reported bandwidths. */ + String descriptorsFilename = consensusTimestamp + "-descriptors"; + Map<String, String> parsedDescriptors = + new HashMap<String, String>(); + if (descriptors.containsKey(descriptorsFilename)) { + BufferedReader brD = new BufferedReader(new FileReader( + descriptors.get(descriptorsFilename))); + Set<String> defaultRejects = new HashSet<String>(); + /* Starting with 0.2.1.6-alpha, ports 465 and 587 were allowed + * in the default exit policy again (and therefore removed + * from the default reject lines). */ + Set<String> optionalRejects = new HashSet<String>(); + String lineD, address = null, fingerprint = null, + descriptorBandwidth = null; + boolean defaultPolicy = false, comparePolicies = true; + while ((lineD = brD.readLine()) != null) { + if (lineD.startsWith("router ")) { + address = lineD.split(" ")[2]; + defaultRejects.clear(); + defaultRejects.addAll(Arrays.asList(("0.0.0.0/8:*," + + "169.254.0.0/16:*,127.0.0.0/8:*,192.168.0.0/16:*," + + "10.0.0.0/8:*,172.16.0.0/12:*,$IP:*,*:25,*:119," + + "*:135-139,*:445,*:563,*:1214,*:4661-4666,*:6346-6429," + + "*:6699,*:6881-6999").split(","))); + optionalRejects.clear(); + optionalRejects.addAll(Arrays.asList( + "*:465,*:587".split(","))); + fingerprint = null; + descriptorBandwidth = null; + defaultPolicy = false; + comparePolicies = true; + } else if (lineD.startsWith("opt fingerprint ") || + lineD.startsWith("fingerprint ")) { + fingerprint = lineD.substring(lineD.startsWith("opt ") ? + "opt fingerprint".length() : "fingerprint".length()). + replaceAll(" ", "").toLowerCase(); + } else if (lineD.startsWith("bandwidth ")) { + descriptorBandwidth = lineD.split(" ")[3]; + } else if (lineD.startsWith("reject ") && comparePolicies) { + String rejectPattern = lineD.substring("reject ". + length()); + if (defaultRejects.contains(rejectPattern)) { + defaultRejects.remove(rejectPattern); + } else if (optionalRejects.contains(rejectPattern)) { + optionalRejects.remove(rejectPattern); + } else if (rejectPattern.equals(address + ":*")) { + defaultRejects.remove("$IP:*"); + } else { + comparePolicies = false; + } + } else if (lineD.startsWith("accept ") && comparePolicies) { + if (defaultRejects.isEmpty() && + lineD.equals("accept *:*")) { + defaultPolicy = true; + } + comparePolicies = false; + } else if (lineD.equals("router-signature")) { + if (address != null && fingerprint != null && + descriptorBandwidth != null) { + parsedDescriptors.put(fingerprint, descriptorBandwidth + "," + + (defaultPolicy ? "1" : "0")); + } + } + } + brD.close(); + } + /* Parse r, s, and w lines from the consensus. */ String lineC, validAfter = null; while ((lineC = brC.readLine()) != null) { @@ -110,64 +177,14 @@ public class ParseDescriptors { String consensusBandwidth = lineC.substring(lineC.indexOf( " Bandwidth=") + " Bandwidth=".length()).split(" ")[0]; - /* Parse the referenced server descriptor (if we haven't done so - * before) to learn about the relay's exit policy and reported - * bandwidth. */ + /* Look up whether we parsed this descriptor before. */ boolean parsedDescriptor = false, defaultPolicy = false; String descriptorBandwidth = null; - if (parsedDescriptors.containsKey(descriptor)) { - String parseResults = parsedDescriptors.get(descriptor); + if (parsedDescriptors.containsKey(fingerprint)) { + String parseResults = parsedDescriptors.get(fingerprint); parsedDescriptor = true; defaultPolicy = parseResults.endsWith("1"); descriptorBandwidth = parseResults.split(",")[0]; - } else if (descriptors.containsKey(descriptor)) { - parsedDescriptor = true; - BufferedReader brD = new BufferedReader(new FileReader( - descriptors.get(descriptor))); - Set<String> defaultRejects = new HashSet<String>( - Arrays.asList(("0.0.0.0/8:*,169.254.0.0/16:*," - + "127.0.0.0/8:*,192.168.0.0/16:*,10.0.0.0/8:*," - + "172.16.0.0/12:*,$IP:*,*:25,*:119,*:135-139,*:445," - + "*:563,*:1214,*:4661-4666,*:6346-6429,*:6699," - + "*:6881-6999").split(","))); - /* Starting with 0.2.1.6-alpha, ports 465 and 587 were allowed - * in the default exit policy again (and therefore removed - * from the default reject lines). */ - Set<String> optionalRejects = new HashSet<String>( - Arrays.asList("*:465,*:587".split(","))); - String lineD, address = null; - while ((lineD = brD.readLine()) != null) { - if (lineD.startsWith("router ")) { - address = lineD.split(" ")[2]; - } else if (lineD.startsWith("bandwidth ")) { - descriptorBandwidth = lineD.split(" ")[3]; - } else if (lineD.startsWith("reject ")) { - String rejectPattern = lineD.substring("reject ". - length()); - if (defaultRejects.contains(rejectPattern)) { - defaultRejects.remove(rejectPattern); - } else if (optionalRejects.contains(rejectPattern)) { - optionalRejects.remove(rejectPattern); - } else if (rejectPattern.equals(address + ":*")) { - defaultRejects.remove("$IP:*"); - } else { - break; - } - } else if (lineD.startsWith("accept ")) { - if (defaultRejects.isEmpty() && - lineD.equals("accept *:*")) { - defaultPolicy = true; - } - break; - } - } - brD.close(); - parsedDescriptors.put(descriptor, descriptorBandwidth + "," - + (defaultPolicy ? "1" : "0")); - } else { - System.out.println("We're missing descriptor " + descriptor - + ". Please make sure that all referenced server " - + "descriptors are available. Continuing anyway."); } /* Write everything we know about this relay to disk. */ diff --git a/task-2394/README b/task-2394/README index 5cb47f6..714853e 100644 --- a/task-2394/README +++ b/task-2394/README @@ -1,26 +1,19 @@ Visualize self-reported vs. measured bandwidth of relays ======================================================== - - Download consensus and votes from the metrics website, e.g., + - Download a single consensus and all referenced votes and server + descriptors from the metrics website, e.g., https://metrics.torproject.org/votes?valid-after=2011-07-13-05-00-00 https://metrics.torproject.org/consensus?valid-after=2011-07-13-05-00-00 + https://metrics.torproject.org/serverdesc?valid-after=2011-07-13-05-00-00 - - Also download the server descriptor archives containing the referenced - server descriptors from the metrics website. If the consensus was - published on the 3rd of a month or later, it's sufficient to download a - single tarball. Otherwise, download both tarballs for the current and - previous month: - - https://metrics.torproject.org/data/server-descriptors-2011-07.tar.bz2 - - - Extract the server descriptor tarball(s) into the subdirectory - descriptors/. Also copy the consensus and votes there. With the URLs + - Put all files into a new subdirectory descriptors/. With the URLs stated above, the directory would contain these files: 2011-07-13-05-00-00-consensus 2011-07-13-05-00-00-votes - server-descriptors-2011-07 + 2011-07-13-05-00-00-descriptors - Download Apache Commons Codec 1.4 or higher and put in in this directory.

1 0