July 2011 - tor-commits - lists.torproject.org

[obfsproxy/master] Make tor-obfs-howto.txt a bit more user-friendly.
by nickm＠torproject.org 22 Jul '11

22 Jul '11

commit e1a825f7b27fd315d897dab5c0010388fc7ba4c2 Author: George Kadianakis <desnacked(a)gmail.com> Date: Fri Jul 22 20:53:06 2011 +0200 Make tor-obfs-howto.txt a bit more user-friendly. --- doc/tor-obfs-howto.txt | 134 +++++++++++++++++++++++++----------------------- 1 files changed, 69 insertions(+), 65 deletions(-) diff --git a/doc/tor-obfs-howto.txt b/doc/tor-obfs-howto.txt index ed8ba72..324a98c 100644 --- a/doc/tor-obfs-howto.txt +++ b/doc/tor-obfs-howto.txt @@ -1,101 +1,105 @@ -How to set up Tor with Obfsproxy +How to set up Tor with obfsproxy ================================ - Preliminaries: + Instructions: -This isn't the final interface--we should make it much easier. + "I AM A CENSORED FELLOW" -This is experimental code--if it breaks, you get to keep both smouldering -pieces. - - - - Ingredients: - -Client side needs: +[1] You will need: * a copy of obfsproxy: git clone git://git.torproject.org/obfsproxy.git Building it should be easy, just do "./configure && make". * a copy of Tor that supports the Socks5Proxy option. - The git version should do the trick. + Tor 0.2.2.30 - for example - will work. -Bridge side needs: +[2] Set up obfsproxy: -* a copy of obfsproxy: - git clone git://git.torproject.org/obfsproxy.git - Building it should be easy, just do "./configure && make". +This command will setup an obfsproxy client speaking the obfs2 +protocol on localhost:1050: -* a copy of Tor with BridgeRelay capabilities. - Quite easy to find nowadays. +./obfsproxy obfs2 socks 127.0.0.1:1050 - Port setup: +[3] Set up tor: -Our port setup will be like this: +You should put in your torrc file: +--- +SocksPort 5000 +UseBridges 1 +Bridge <bridge address>:1051 # This is provided by the bridge operator. +Socks5Proxy 127.0.0.1:1050 # This points to the obfsproxy client. +--- - 1050 1051 - +-----------+ +------------+ - ----| Proxy |-------------| Server |---- - | +-----------+ +------------+ | - | | -+------------+ +--------------+ -| Tor Client | | Tor Bridge | -+------------+ +--------------+ - 5000 5001 +Of course you should substitute the value of the Bridge line above, +with the information provided to you by a bridge operator. +[4] Done! +Alright, you are done! +Make sure you have followed the above steps on the correct order: +First launch obfsproxy and then launch tor. - Setting up obfsproxies: +Now to test it, launch Firefox and set it up to use a Socks server on +127.0.0.1:5000. Now browse the web. -This command will setup an obfsproxy socks client listening to the -obfs2 protocol on localhost:1050: -./obfsproxy obfs2 socks 127.0.0.1:1050 -This command will setup an obfsproxy server listening to the -obfs2 protocol on localhost:1051: -./obfsproxy obfs2 --dest 127.0.0.1:5001 server 127.0.0.1:1051 + "I AM A BRIDGE OPERATOR THAT WANTS TO HELP CENSORED PEOPLE" - Setting up Tor: +[1] You will need: -Let's create .torrc_client and .torrc_bridge. -Warning: I'm only posting the relevant torrc options. +* a copy of obfsproxy: + git clone git://git.torproject.org/obfsproxy.git + Building it should be easy, just do "./configure && make". -torrc_client: ---- -SocksPort 5000 -UseBridges 1 -Bridge <bridge host>:1051 # This points to the bridge's obfsproxy server. - # You can change <bridge host> to 127.0.0.1 - # for local testing setup. -Socks5Proxy 127.0.0.1:1050 # This points to our obfsproxy client. ---- +* a copy of Tor which can function as a Tor bridge. + +[2] Set up obfsproxy: + +This command will setup an obfsproxy server speaking the obfs2 +protocol on localhost:1051: -torrc_bridge: +./obfsproxy obfs2 --dest 127.0.0.1:5001 server 127.0.0.1:1051 + +Be sure to pass the value of ORPort of your bridge relay to the --dest +argument (see next section). + +[3] Set up tor: + +You should put in your torrc file: --- -ORPort 5001 # Port where bridge will be listening on. +ORPort 5001 # Port on which bridge will be listening on. BridgeRelay 1 ExitPolicy reject *:* --- - Bootstrap sequence: +[4] Done! + +Make sure you have followed the above steps on the correct order. +First launch obfsproxy and then launch tor. +You can use any port numbers you want, just be sure to adjust the +torrc and obfsproxy commands accordingly. + +Don't forget that You might need to set up port forwarding. -Since we are still in PoC state you have to bootstrap the Tors and the -obfsproxies with the correct sequence: +Now you should tell censored people to connect to your IP on port 1051. +So, for example, if your bridge's address is 85.22.13.1, bridge users +should connect to 85.22.13.1:1051. +Pay attention that '1051' is the port where obfsproxy is listening on, +and *not* your bridge's ORPort. -1. Fire up server obfsproxy. -2. Fire up tor with torrc_bridge. -3. Fire up client obfsproxy. -4. Fire up tor with torrc_client. + Network diagram of the above setup: - The End +(If you don't understand this drawing, just ignore this section.) -Now open Firefox on the client side and set it up to use a Socks -server on 127.0.0.1:5000. -Stuff should work now. + 1050 1051 + +-----------+ +------------+ + ----| Proxy |------[internet]------| Server |---- + | +-----------+ +------------+ | + | | ++------------+ +--------------+ +| Tor Client | | Tor Bridge | ++------------+ +--------------+ + 5000 5001 -If you were smart enough to open wireshark and sniff the traffic -before doing all the above, you would notice that the TLS handshake -was not carried out plaintext, effectively obfuscating your Tor -experience. Yes.

1 0

[flashproxy/master] Less sleeping in 01, too lazy
by dcf＠torproject.org 22 Jul '11

22 Jul '11

commit c0701bcd20dd85949e4233056c3c199a7178aeec Author: Jonathan Ellithorpe <jdellit(a)stanford.edu> Date: Tue Jul 19 18:55:30 2011 -0700 Less sleeping in 01, too lazy --- experiments/01 | 6 +----- 1 files changed, 1 insertions(+), 5 deletions(-) diff --git a/experiments/01 b/experiments/01 index 10ee99d..ff0179d 100755 --- a/experiments/01 +++ b/experiments/01 @@ -47,15 +47,10 @@ until [ $i -gt $NUM_CLIENTS ]; do sleep 2 done -# Some connections take a long time to register with cirrus -sleep 15 - # Start the flash proxy, server side echo "starting flash proxy server side..." firefox --remote "openURL(http://localhost:1080/swfcat.swf?facilitator=127.0.0.1:4701)" -sleep 15 - let query_fac_time=NUM_CLIENTS+2 sleep ${query_fac_time} @@ -66,6 +61,7 @@ until [ $i -gt $NUM_CLIENTS ]; do let local_port=2000+i gnome-terminal -e "/bin/bash -c 'wget -t1 http://localhost:${local_port}/dump -O - > /dev/null; exec /bin/bash -i'" let i=i+1 + sleep 2 done # Use this one for delaying the termination of experiment below

1 0

[flashproxy/master] Use new-window to open RTMFP clients.
by dcf＠torproject.org 22 Jul '11

22 Jul '11

commit 4043f75c01779cfb82c003e77c4db23615d1bff0 Author: David Fifield <david(a)bamsoftware.com> Date: Mon Jul 18 06:04:28 2011 +0000 Use new-window to open RTMFP clients. --- RTMFPSocket.as | 9 ++++++--- experiments/00 | 3 +-- experiments/01 | 4 ++-- swfcat.as | 2 +- 4 files changed, 10 insertions(+), 8 deletions(-) diff --git a/RTMFPSocket.as b/RTMFPSocket.as index 9dca232..e91d06d 100644 --- a/RTMFPSocket.as +++ b/RTMFPSocket.as @@ -183,9 +183,12 @@ package public function close():void { - outgoing.close(); - incoming.close(); - nc.close(); + if (outgoing) + outgoing.close(); + if (incoming) + incoming.close(); + if (nc) + nc.close(); } public function readBytes(output:ByteArray, offset:uint = 0, length:uint = 0):void diff --git a/experiments/00 b/experiments/00 index ecc8c48..bc4befb 100755 --- a/experiments/00 +++ b/experiments/00 @@ -60,9 +60,8 @@ echo "getting the dump file..." i=2 until [ $i -gt $NUM_CLIENTS ]; do let local_port=2000+i - gnome-terminal -e "/bin/bash -c 'wget -t1 http://localhost:${local_port}/dump -O - > /dev/null; exec /bin/bash -i'" + xterm -e "/bin/bash -c 'wget -t1 http://localhost:${local_port}/dump -O - > /dev/null; exec /bin/bash -i'" & let i=i+1 - sleep 2 done # Use this one for delaying the termination of experiment below diff --git a/experiments/01 b/experiments/01 index ff0179d..0c273e7 100755 --- a/experiments/01 +++ b/experiments/01 @@ -42,14 +42,14 @@ echo "starting flash proxy client side..." i=1 until [ $i -gt $NUM_CLIENTS ]; do let local_port=9000+i - firefox --remote "openURL(http://localhost:1080/swfcat.swf?local=127.0.0.1:${local_port}&client=1&facilitator=127.0.0.1:4701)" + firefox --remote "openURL(http://localhost:1080/swfcat.swf?local=127.0.0.1:${local_port}&client=1&facilitator=127.0.0.1:4701,new-window)" let i=i+1 sleep 2 done # Start the flash proxy, server side echo "starting flash proxy server side..." -firefox --remote "openURL(http://localhost:1080/swfcat.swf?facilitator=127.0.0.1:4701)" +firefox --remote "openURL(http://localhost:1080/swfcat.swf?facilitator=127.0.0.1:4701,new-window)" let query_fac_time=NUM_CLIENTS+2 sleep ${query_fac_time} diff --git a/swfcat.as b/swfcat.as index c93c154..9f3f967 100644 --- a/swfcat.as +++ b/swfcat.as @@ -31,7 +31,7 @@ package port: 9002 }; - private const MAX_NUM_PROXY_PAIRS:uint = 10; + private const MAX_NUM_PROXY_PAIRS:uint = 100; // Milliseconds. private const FACILITATOR_POLL_INTERVAL:int = 1000;

1 0

[flashproxy/master] Fix a memory leak in RTMFPSocket.
by dcf＠torproject.org 22 Jul '11

22 Jul '11

commit 54bb6795dd0879027498a0a105d976616d2dc8f8 Author: David Fifield <david(a)bamsoftware.com> Date: Mon Jul 18 06:33:51 2011 +0000 Fix a memory leak in RTMFPSocket. There was a ByteArray that always grew. When the array has been emptied, it's safe to collapse it back down to 0 bytes. This was being done before but it was in the wrong place. --- RTMFPSocket.as | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/RTMFPSocket.as b/RTMFPSocket.as index e91d06d..4de3502 100644 --- a/RTMFPSocket.as +++ b/RTMFPSocket.as @@ -157,11 +157,6 @@ package event.bytesLoaded = bytes.bytesAvailable; bytes.readBytes(buffer, buffer.length, bytes.bytesAvailable); - if (bytes.bytesAvailable == 0) { - /* Reclaim memory space. */ - bytes.position = 0; - bytes.length = 0; - } dispatchEvent(event); } @@ -194,6 +189,11 @@ package public function readBytes(output:ByteArray, offset:uint = 0, length:uint = 0):void { buffer.readBytes(output, offset, length); + if (buffer.bytesAvailable == 0) { + /* Reclaim memory space. */ + buffer.position = 0; + buffer.length = 0; + } } public function writeBytes(input:ByteArray, offset:uint = 0, length:uint = 0):void

1 0

r24926: {website} updated perl-globals to properly include languages (website/trunk/include)
by Runa Sandvik 21 Jul '11

21 Jul '11

Author: runa Date: 2011-07-21 15:40:41 +0000 (Thu, 21 Jul 2011) New Revision: 24926 Modified: website/trunk/include/perl-globals.wmi Log: updated perl-globals to properly include languages Modified: website/trunk/include/perl-globals.wmi =================================================================== --- website/trunk/include/perl-globals.wmi 2011-07-21 15:20:43 UTC (rev 24925) +++ website/trunk/include/perl-globals.wmi 2011-07-21 15:40:41 UTC (rev 24926) @@ -5,7 +5,10 @@ my @LANGUAGES = qw{ar ar-se bms cy da de en es et fa fi fr hu id it ja ko nl no pl pt pt-br ru se tr vn zh-cn}; my %LANGUAGES = ( 'ar' => "عربية (Arabiya)", + 'ar-se' => "Arabic (Saudi Arabia)", 'bms' => "Burmese", + 'cy' => "Welsh", + 'da' => "Danish", 'de' => "Deutsch", 'en' => "English", 'es' => "español", @@ -13,6 +16,8 @@ 'fa' => "فارسی (Fārsī)", 'fi' => "suomi", 'fr' => "français", + 'hu' => "Hungarian", + 'id' => "Bahasa Indonesia", 'it' => "Italiano", 'ja' => "日本語 (Nihongo)", 'ko' => "한국어 (Hangul)", @@ -20,9 +25,11 @@ 'no' => "norsk", 'pl' => "polski", 'pt' => "Português", + 'pt-br' => "Portuguese (Brazil)", 'ru' => "Русский (Russkij)", 'se' => "svenska", 'tr' => "Türkçe", + 'vn' => "Vietnamese", 'zh-cn' => "中文(简) (Simplified Chinese)", );

1 0

r24925: {website} update perl-globals.wmi to include more languages (website/trunk/include)
by Runa Sandvik 21 Jul '11

21 Jul '11

Author: runa Date: 2011-07-21 15:20:43 +0000 (Thu, 21 Jul 2011) New Revision: 24925 Modified: website/trunk/include/perl-globals.wmi Log: update perl-globals.wmi to include more languages Modified: website/trunk/include/perl-globals.wmi =================================================================== --- website/trunk/include/perl-globals.wmi 2011-07-21 15:12:53 UTC (rev 24924) +++ website/trunk/include/perl-globals.wmi 2011-07-21 15:20:43 UTC (rev 24925) @@ -2,7 +2,7 @@ <: # Change both. The array is ordered (for pages like translation status, order of flags, etc). - my @LANGUAGES = qw{ar bms de en es et fa fi fr it ja ko nl no pl pt ru se tr zh-cn}; + my @LANGUAGES = qw{ar ar-se bms cy da de en es et fa fi fr hu id it ja ko nl no pl pt pt-br ru se tr vn zh-cn}; my %LANGUAGES = ( 'ar' => "عربية (Arabiya)", 'bms' => "Burmese",

1 0

r24924: {website} include more languages when building the website (website/trunk)
by Runa Sandvik 21 Jul '11

21 Jul '11

Author: runa Date: 2011-07-21 15:12:53 +0000 (Thu, 21 Jul 2011) New Revision: 24924 Modified: website/trunk/Makefile.common Log: include more languages when building the website Modified: website/trunk/Makefile.common =================================================================== --- website/trunk/Makefile.common 2011-07-21 10:00:37 UTC (rev 24923) +++ website/trunk/Makefile.common 2011-07-21 15:12:53 UTC (rev 24924) @@ -18,7 +18,7 @@ -D STABLETAG=$(STABLETAG) #LANGS=ar bms de en es et fa it fi fr ja ko nl no pl pt ru se tr zh-cn -LANGS=en ar es fr ru pl da it fa de fi vn cy zh-cn pt pt-br id nl +LANGS=ar ar-se bms cy da de en es et fa fi fr hu id it ja ko nl no pl pt pt-br ru se tr vn zh-cn WMLFILES=$(wildcard $(patsubst %, %/*.wml, $(LANGS))) WMIFILES=$(wildcard $(patsubst %, %/*.wmi, $(LANGS)) $(WMLBASE)/include/*.wmi ) @@ -35,9 +35,15 @@ %.html.ar: ar/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ +%.html.ar-se: ar-se/%.wml en/%.wml + lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ + %.html.bms: bms/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ +%.html.cy: cy/%.wml en/%.wml + lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ + %.html.es: es/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ @@ -65,6 +71,9 @@ %.html.fr: fr/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ +%.html.hu: hu/%.wml en/%.wml + lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ + %.html.ja: ja/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ @@ -98,6 +107,9 @@ %.html.vi: vi/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ +%.html.vn: vn/%.wml en/%.wml + lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ + %.html.zh-cn: zh-cn/%.wml en/%.wml lang=`dirname $<` && wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $@ @@ -115,6 +127,13 @@ wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $$OUT --depend | tee $$tmpfile > $@ && \ sed -e s',$^[^ ]*$:,.deps/\1.d:,' < $$tmpfile >> $@ && \ rm -f $$tmpfile +.deps/%.html.ar-se.d: ar-se/%.wml .deps/.stamp + tmpfile=`mktemp -t tmp.XXXXXXX` \ + lang=`dirname $<` && \ + OUT=`echo $@ | sed -e 's,\.deps/$.*$\.d$$,\1,'` && \ + wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $$OUT --depend | tee $$tmpfile > $@ && \ + sed -e s',$^[^ ]*$:,.deps/\1.d:,' < $$tmpfile >> $@ && \ + rm -f $$tmpfile .deps/%.html.bms.d: bms/%.wml .deps/.stamp tmpfile=`mktemp -t tmp.XXXXXXX` \ lang=`dirname $<` && \ @@ -192,6 +211,13 @@ wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $$OUT --depend | tee $$tmpfile > $@ && \ sed -e s',$^[^ ]*$:,.deps/\1.d:,' < $$tmpfile >> $@ && \ rm -f $$tmpfile +.deps/%.html.hu.d: hu/%.wml .deps/.stamp + tmpfile=`mktemp -t tmp.XXXXXXX` \ + lang=`dirname $<` && \ + OUT=`echo $@ | sed -e 's,\.deps/$.*$\.d$$,\1,'` && \ + wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $$OUT --depend | tee $$tmpfile > $@ && \ + sed -e s',$^[^ ]*$:,.deps/\1.d:,' < $$tmpfile >> $@ && \ + rm -f $$tmpfile .deps/%.html.ja.d: ja/%.wml .deps/.stamp tmpfile=`mktemp -t tmp.XXXXXXX` \ lang=`dirname $<` && \ @@ -269,6 +295,13 @@ wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $$OUT --depend | tee $$tmpfile > $@ && \ sed -e s',$^[^ ]*$:,.deps/\1.d:,' < $$tmpfile >> $@ && \ rm -f $$tmpfile +.deps/%.html.vn.d: vn/%.wml .deps/.stamp + tmpfile=`mktemp -t tmp.XXXXXXX` \ + lang=`dirname $<` && \ + OUT=`echo $@ | sed -e 's,\.deps/$.*$\.d$$,\1,'` && \ + wml $(WMLOPT) -I $$lang -I $(WMLBASE)/$$lang -D LANG=$$lang $< -o $$OUT --depend | tee $$tmpfile > $@ && \ + sed -e s',$^[^ ]*$:,.deps/\1.d:,' < $$tmpfile >> $@ && \ + rm -f $$tmpfile .deps/%.html.zh-cn.d: zh-cn/%.wml .deps/.stamp tmpfile=`mktemp -t tmp.XXXXXXX` \ lang=`dirname $<` && \

1 0

[metrics-tasks/master] Avoid downloading server descriptor tarballs in #2394.
by karsten＠torproject.org 21 Jul '11

21 Jul '11

commit 7b75c704f5dad573ad6ba4f0a04cb8599b4cdb7a Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jul 21 16:01:14 2011 +0200 Avoid downloading server descriptor tarballs in #2394. --- task-2394/ParseDescriptors.java | 133 ++++++++++++++++++++++----------------- task-2394/README | 17 ++---- 2 files changed, 80 insertions(+), 70 deletions(-) diff --git a/task-2394/ParseDescriptors.java b/task-2394/ParseDescriptors.java index 2197670..641518d 100644 --- a/task-2394/ParseDescriptors.java +++ b/task-2394/ParseDescriptors.java @@ -21,13 +21,13 @@ public class ParseDescriptors { consensuses.put(filename, file); } else if (filename.endsWith("-votes")) { votes.put(filename, file); - } else if (filename.length() == 40) { + } else if (filename.endsWith("-descriptors")) { descriptors.put(filename, file); } } System.out.println("We found " + consensuses.size() - + " consensuses, " + votes.size() + " votes, and " - + descriptors.size() + " server descriptors."); + + " consensus files, " + votes.size() + " vote files, and " + + descriptors.size() + " server descriptor files."); /* Parse consensuses in an outer loop and the referenced votes and * descriptors in inner loops. Write the results to disk as soon as @@ -50,7 +50,6 @@ public class ParseDescriptors { bw.write("," + bandwidthAuthority + "bandwidth"); } bw.write("\n"); - Map<String, String> parsedDescriptors = new HashMap<String, String>(); for (File consensusFile : consensuses.values()) { System.out.println("Parsing consensus " + consensusFile.getName()); BufferedReader brC = new BufferedReader(new FileReader( @@ -58,12 +57,12 @@ public class ParseDescriptors { String lastRLine = null, lastSLine = null; String consensusTimestamp = consensusFile.getName().substring(0, "YYYY-MM-DD-hh-mm-ss".length()); - String votesFilename = consensusTimestamp + "-votes"; Map<String, Map<String, String>> measuredBandwidthsByDirSource = new HashMap<String, Map<String, String>>(); /* Parse votes first, if we have them, and extract measured * bandwidths. */ + String votesFilename = consensusTimestamp + "-votes"; if (votes.containsKey(votesFilename)) { BufferedReader brV = new BufferedReader(new FileReader( votes.get(votesFilename))); @@ -89,6 +88,74 @@ public class ParseDescriptors { brV.close(); } + /* Parse referenced server descriptors to learn about exit policies + * and reported bandwidths. */ + String descriptorsFilename = consensusTimestamp + "-descriptors"; + Map<String, String> parsedDescriptors = + new HashMap<String, String>(); + if (descriptors.containsKey(descriptorsFilename)) { + BufferedReader brD = new BufferedReader(new FileReader( + descriptors.get(descriptorsFilename))); + Set<String> defaultRejects = new HashSet<String>(); + /* Starting with 0.2.1.6-alpha, ports 465 and 587 were allowed + * in the default exit policy again (and therefore removed + * from the default reject lines). */ + Set<String> optionalRejects = new HashSet<String>(); + String lineD, address = null, fingerprint = null, + descriptorBandwidth = null; + boolean defaultPolicy = false, comparePolicies = true; + while ((lineD = brD.readLine()) != null) { + if (lineD.startsWith("router ")) { + address = lineD.split(" ")[2]; + defaultRejects.clear(); + defaultRejects.addAll(Arrays.asList(("0.0.0.0/8:*," + + "169.254.0.0/16:*,127.0.0.0/8:*,192.168.0.0/16:*," + + "10.0.0.0/8:*,172.16.0.0/12:*,$IP:*,*:25,*:119," + + "*:135-139,*:445,*:563,*:1214,*:4661-4666,*:6346-6429," + + "*:6699,*:6881-6999").split(","))); + optionalRejects.clear(); + optionalRejects.addAll(Arrays.asList( + "*:465,*:587".split(","))); + fingerprint = null; + descriptorBandwidth = null; + defaultPolicy = false; + comparePolicies = true; + } else if (lineD.startsWith("opt fingerprint ") || + lineD.startsWith("fingerprint ")) { + fingerprint = lineD.substring(lineD.startsWith("opt ") ? + "opt fingerprint".length() : "fingerprint".length()). + replaceAll(" ", "").toLowerCase(); + } else if (lineD.startsWith("bandwidth ")) { + descriptorBandwidth = lineD.split(" ")[3]; + } else if (lineD.startsWith("reject ") && comparePolicies) { + String rejectPattern = lineD.substring("reject ". + length()); + if (defaultRejects.contains(rejectPattern)) { + defaultRejects.remove(rejectPattern); + } else if (optionalRejects.contains(rejectPattern)) { + optionalRejects.remove(rejectPattern); + } else if (rejectPattern.equals(address + ":*")) { + defaultRejects.remove("$IP:*"); + } else { + comparePolicies = false; + } + } else if (lineD.startsWith("accept ") && comparePolicies) { + if (defaultRejects.isEmpty() && + lineD.equals("accept *:*")) { + defaultPolicy = true; + } + comparePolicies = false; + } else if (lineD.equals("router-signature")) { + if (address != null && fingerprint != null && + descriptorBandwidth != null) { + parsedDescriptors.put(fingerprint, descriptorBandwidth + "," + + (defaultPolicy ? "1" : "0")); + } + } + } + brD.close(); + } + /* Parse r, s, and w lines from the consensus. */ String lineC, validAfter = null; while ((lineC = brC.readLine()) != null) { @@ -110,64 +177,14 @@ public class ParseDescriptors { String consensusBandwidth = lineC.substring(lineC.indexOf( " Bandwidth=") + " Bandwidth=".length()).split(" ")[0]; - /* Parse the referenced server descriptor (if we haven't done so - * before) to learn about the relay's exit policy and reported - * bandwidth. */ + /* Look up whether we parsed this descriptor before. */ boolean parsedDescriptor = false, defaultPolicy = false; String descriptorBandwidth = null; - if (parsedDescriptors.containsKey(descriptor)) { - String parseResults = parsedDescriptors.get(descriptor); + if (parsedDescriptors.containsKey(fingerprint)) { + String parseResults = parsedDescriptors.get(fingerprint); parsedDescriptor = true; defaultPolicy = parseResults.endsWith("1"); descriptorBandwidth = parseResults.split(",")[0]; - } else if (descriptors.containsKey(descriptor)) { - parsedDescriptor = true; - BufferedReader brD = new BufferedReader(new FileReader( - descriptors.get(descriptor))); - Set<String> defaultRejects = new HashSet<String>( - Arrays.asList(("0.0.0.0/8:*,169.254.0.0/16:*," - + "127.0.0.0/8:*,192.168.0.0/16:*,10.0.0.0/8:*," - + "172.16.0.0/12:*,$IP:*,*:25,*:119,*:135-139,*:445," - + "*:563,*:1214,*:4661-4666,*:6346-6429,*:6699," - + "*:6881-6999").split(","))); - /* Starting with 0.2.1.6-alpha, ports 465 and 587 were allowed - * in the default exit policy again (and therefore removed - * from the default reject lines). */ - Set<String> optionalRejects = new HashSet<String>( - Arrays.asList("*:465,*:587".split(","))); - String lineD, address = null; - while ((lineD = brD.readLine()) != null) { - if (lineD.startsWith("router ")) { - address = lineD.split(" ")[2]; - } else if (lineD.startsWith("bandwidth ")) { - descriptorBandwidth = lineD.split(" ")[3]; - } else if (lineD.startsWith("reject ")) { - String rejectPattern = lineD.substring("reject ". - length()); - if (defaultRejects.contains(rejectPattern)) { - defaultRejects.remove(rejectPattern); - } else if (optionalRejects.contains(rejectPattern)) { - optionalRejects.remove(rejectPattern); - } else if (rejectPattern.equals(address + ":*")) { - defaultRejects.remove("$IP:*"); - } else { - break; - } - } else if (lineD.startsWith("accept ")) { - if (defaultRejects.isEmpty() && - lineD.equals("accept *:*")) { - defaultPolicy = true; - } - break; - } - } - brD.close(); - parsedDescriptors.put(descriptor, descriptorBandwidth + "," - + (defaultPolicy ? "1" : "0")); - } else { - System.out.println("We're missing descriptor " + descriptor - + ". Please make sure that all referenced server " - + "descriptors are available. Continuing anyway."); } /* Write everything we know about this relay to disk. */ diff --git a/task-2394/README b/task-2394/README index 5cb47f6..714853e 100644 --- a/task-2394/README +++ b/task-2394/README @@ -1,26 +1,19 @@ Visualize self-reported vs. measured bandwidth of relays ======================================================== - - Download consensus and votes from the metrics website, e.g., + - Download a single consensus and all referenced votes and server + descriptors from the metrics website, e.g., https://metrics.torproject.org/votes?valid-after=2011-07-13-05-00-00 https://metrics.torproject.org/consensus?valid-after=2011-07-13-05-00-00 + https://metrics.torproject.org/serverdesc?valid-after=2011-07-13-05-00-00 - - Also download the server descriptor archives containing the referenced - server descriptors from the metrics website. If the consensus was - published on the 3rd of a month or later, it's sufficient to download a - single tarball. Otherwise, download both tarballs for the current and - previous month: - - https://metrics.torproject.org/data/server-descriptors-2011-07.tar.bz2 - - - Extract the server descriptor tarball(s) into the subdirectory - descriptors/. Also copy the consensus and votes there. With the URLs + - Put all files into a new subdirectory descriptors/. With the URLs stated above, the directory would contain these files: 2011-07-13-05-00-00-consensus 2011-07-13-05-00-00-votes - server-descriptors-2011-07 + 2011-07-13-05-00-00-descriptors - Download Apache Commons Codec 1.4 or higher and put in in this directory.

1 0

[metrics-web/master] Wait, we have a ChangeLog file. Let's use it!
by karsten＠torproject.org 21 Jul '11

21 Jul '11

commit d1a80b31838d3678f48b8ce29cad3df2d847ccd8 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jul 21 16:04:25 2011 +0200 Wait, we have a ChangeLog file. Let's use it! --- ChangeLog | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/ChangeLog b/ChangeLog index 42032ca..d9829d0 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,8 +1,10 @@ Metrics database and website change log: Changes in version 0.0.2 - 2011-??-?? - - Let users download votes from a given valid-after time. + - Let users download votes and server descriptors referenced from a + consensus by specifying its valid-after time. - Allow importing descriptors from a local Tor directory. + - Flush and close output streams of some servlets. Changes in version 0.0.1 - 2011-06-08 - Initial release

1 0

[metrics-web/master] Flush and close output streams.
by karsten＠torproject.org 21 Jul '11

21 Jul '11

commit 57a5a9994e6b62f03f1f6c7bff69637caaed066d Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jul 21 14:46:21 2011 +0200 Flush and close output streams. Apparently, Tomcat 5 didn't care about this. But Tomcat 6 does. --- src/org/torproject/ernie/web/ConsensusServlet.java | 2 ++ .../ernie/web/ExtraInfoDescriptorServlet.java | 2 ++ .../torproject/ernie/web/GraphImageServlet.java | 1 + .../ernie/web/ServerDescriptorServlet.java | 2 ++ src/org/torproject/ernie/web/VotesServlet.java | 2 ++ 5 files changed, 9 insertions(+), 0 deletions(-) diff --git a/src/org/torproject/ernie/web/ConsensusServlet.java b/src/org/torproject/ernie/web/ConsensusServlet.java index eade324..d9a6ae1 100644 --- a/src/org/torproject/ernie/web/ConsensusServlet.java +++ b/src/org/torproject/ernie/web/ConsensusServlet.java @@ -96,6 +96,8 @@ public class ConsensusServlet extends HttpServlet { BufferedOutputStream output = new BufferedOutputStream( response.getOutputStream()); output.write(rawDescriptor); + output.flush(); + output.close(); } finally { /* Nothing to do here. */ } diff --git a/src/org/torproject/ernie/web/ExtraInfoDescriptorServlet.java b/src/org/torproject/ernie/web/ExtraInfoDescriptorServlet.java index 855801e..8ba33be 100644 --- a/src/org/torproject/ernie/web/ExtraInfoDescriptorServlet.java +++ b/src/org/torproject/ernie/web/ExtraInfoDescriptorServlet.java @@ -85,6 +85,8 @@ public class ExtraInfoDescriptorServlet extends HttpServlet { BufferedOutputStream output = new BufferedOutputStream( response.getOutputStream()); output.write(rawDescriptor); + output.flush(); + output.close(); } finally { /* Nothing to do here. */ } diff --git a/src/org/torproject/ernie/web/GraphImageServlet.java b/src/org/torproject/ernie/web/GraphImageServlet.java index db9eaaa..2eb2efd 100644 --- a/src/org/torproject/ernie/web/GraphImageServlet.java +++ b/src/org/torproject/ernie/web/GraphImageServlet.java @@ -96,6 +96,7 @@ public class GraphImageServlet extends HttpServlet { "inline; filename=\"" + imageFilename + "\""); output = new BufferedOutputStream(response.getOutputStream(), 1024); output.write(graphBytes, 0, graphBytes.length); + output.flush(); output.close(); } } diff --git a/src/org/torproject/ernie/web/ServerDescriptorServlet.java b/src/org/torproject/ernie/web/ServerDescriptorServlet.java index c8b45be..960bd75 100644 --- a/src/org/torproject/ernie/web/ServerDescriptorServlet.java +++ b/src/org/torproject/ernie/web/ServerDescriptorServlet.java @@ -86,6 +86,8 @@ public class ServerDescriptorServlet extends HttpServlet { BufferedOutputStream output = new BufferedOutputStream( response.getOutputStream()); output.write(rawDescriptor); + output.flush(); + output.close(); } finally { /* Nothing to do here. */ } diff --git a/src/org/torproject/ernie/web/VotesServlet.java b/src/org/torproject/ernie/web/VotesServlet.java index 3af9705..6755495 100644 --- a/src/org/torproject/ernie/web/VotesServlet.java +++ b/src/org/torproject/ernie/web/VotesServlet.java @@ -102,6 +102,8 @@ public class VotesServlet extends HttpServlet { for (byte[] rawDescriptor : rawDescriptors) { output.write(rawDescriptor); } + output.flush(); + output.close(); } finally { /* Nothing to do here. */ }

1 0