- tor-dev - lists.torproject.org

Next OONI weekly meetings
by Arturo Filastò 24 Nov '14

24 Nov '14

Hello Oonitarians, Sorry for not sending this email sooner, but this week has been a bit hectic. A couple of people have said that they cannot attend the weekly meeting on thursday at 18:00, because it conflicts with another recurring appointment they have. For this reason I am going to suggest we reschedule the next OONI meetings to a date that works for everybody. Monday and Tuesdays seem to be the best option, so I am going to propose we skip next weeks meeting and instead opt for holding next weeks one on either a Monday or Tuesday at 18:00 UTC, 19:00 CET, 13:00 EST, 12:00 CST, 10:00:00 PST. How does this sound for everybody? I personally would prefer Monday over Tuesday, but if you would instead prefer Tuesday please send a reply to the ooni-dev mailing list. If nobody replies I am going to assume every Monday works for you all and the next meeting will be on Monday 24th of November. Have fun! ~ Arturo

2 2

How does a hidden service select introduction points
by Richard Dennis 24 Nov '14

24 Nov '14

Hi guys, i have been doing a bit of reading recently about hidden services, and i am just wondering how exactly a hidden service selects its introduction points, i understand there are two methods, one where the H.S picks its own set of I.P's and one where this is done for the H.S., is this correct? If so is there an algorithm for automatically selecting introduction points for a hidden service and is one method recommended over another, if so why? Finally how many introduction points does a H.S. usually have, can a larger H.S. have more I.P's to deal with the traffic or is this not advised ? Thanks for the help Rich

2 1

More consistant src/or/config.c [patch]
by grarpamp 24 Nov '14

24 Nov '14

--- src/or/config.c.orig 2014-10-19 11:13:04.000000000 -0400 +++ src/or/config.c @@ -852,12 +852,14 @@ "moria1 orport=9101 " "v3ident=D586D18309DED4CD6D57C18FDB97EFA96D330566 " "128.31.0.39:9131 9695 DFC3 5FFE B861 329B 9F1A B04C 4639 7020 CE31", - "tor26 orport=443 v3ident=14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4 " + "tor26 orport=443 " + "v3ident=14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4 " "86.59.21.38:80 847B 1F85 0344 D787 6491 A548 92F9 0493 4E4E B85D", - "dizum orport=443 v3ident=E8A9C45EDE6D711294FADF8E7951F4DE6CA56B58 " + "dizum orport=443 " + "v3ident=E8A9C45EDE6D711294FADF8E7951F4DE6CA56B58 " "194.109.206.212:80 7EA6 EAD6 FD83 083C 538F 4403 8BBF A077 587D D755", - "Tonga orport=443 bridge 82.94.251.203:80 " - "4A0C CD2D DC79 9508 3D73 F5D6 6710 0C8A 5831 F16D", + "Tonga orport=443 bridge " + "82.94.251.203:80 4A0C CD2D DC79 9508 3D73 F5D6 6710 0C8A 5831 F16D", "turtles orport=9090 " "v3ident=27B6B5996C426270A5C95488AA5BCEB6BCC86956 " "76.73.17.194:9030 F397 038A DC51 3361 35E7 B80B D99C A384 4360 292B", @@ -867,7 +869,8 @@ "dannenberg orport=443 " "v3ident=585769C78764D58426B8B52B6651A5A71137189A " "193.23.244.244:80 7BE6 83E6 5D48 1413 21C5 ED92 F075 C553 64AC 7123", - "urras orport=80 v3ident=80550987E1D626E3EBA5E5E75A458DE0626D088C " + "urras orport=80 " + "v3ident=80550987E1D626E3EBA5E5E75A458DE0626D088C " "208.83.223.34:443 0AD3 FA88 4D18 F89E EA2D 89C0 1937 9E0E 7FD9 4417", "maatuska orport=80 " "v3ident=49015F787433103580E3B66A1707A00E60F2D15B "

2 1

Damian's Status Report - November 2014
by Damian Johnson 23 Nov '14

23 Nov '14

Hi all. Next weekend I'm returning to ye olde isle of Vashon to feast with family, so figured I might as well send this out early. I'll be in a Thanksgiving food coma this time next week, after all... My devious scheme for November was to finish rewriting arm's graph panel and, while that certainly made progress, Stem, DocTor, and internal discussions stole the spotlight this month... -------------------------------------------------------------------------------- DocTor -------------------------------------------------------------------------------- * We now notify directory authority operators directly of issues in addition to the tor-consensus-health@ list. This should reduce duration of outages and other issues with the authorities. * We replaced turtles with longclaw. Sebastian deserves major kudos for orchestrating this. [1] * Properly fixed DocTor's OOM issues by not shelling out to send notification emails. [2] * DocTor detected a burst of relays from Google App Engine. These relays lacked any contact information so we dropped them from the network as a potential Sybil attack. [3] If you're the operator of these then please let us know! We'd be delighted to add you back in once there's a proper family and contact information. -------------------------------------------------------------------------------- Stem -------------------------------------------------------------------------------- * Ossi Herrala made delightful improvements so we parse descriptors ~10% faster [4], and now read from the '.new' files in Tor's data directory [5]. Thanks Ossi! * Expanded our descriptor tutorials with an example of persisting descriptors to disk. Thanks to mmcc for the idea [6]... https://stem.torproject.org/tutorials/mirror_mirror_on_the_wall.html#saving… https://stem.torproject.org/tutorials/examples/persisting_a_consensus.html * Replaced quite of bit of boilerplate in Stem with a new @with_default decorator. Mmmmm, deleting code is fun... [7] * Updates to reflect changes in Tor's spec this month, including HS_DESC's new reason attribute, and validating the new consensus parameters. A couple other quick things of note is that I spruced up our volunteer page [8] and attended this month's Seattle TA3M. The later had a nice bit of serendipity in that I met Anna, a UW PhD student who's looking into our Bandwidth Authorities [8]. Best of luck Anna! That's certainly an area that could do with a lot of improvement. ;) Cheers! -Damian [1] https://trac.torproject.org/projects/tor/ticket/13296 [2] https://gitweb.torproject.org/doctor.git/commitdiff/94a2e9d [3] https://lists.torproject.org/pipermail/tor-consensus-health/2014-November/0… [4] https://trac.torproject.org/projects/tor/ticket/13821 [5] https://trac.torproject.org/projects/tor/ticket/13756 [6] https://trac.torproject.org/projects/tor/ticket/13774 [7] https://gitweb.torproject.org/stem.git/commitdiff/56a1f03 [8] https://gitweb.torproject.org/project/web/webwml.git/commitdiff/05c1b60

1 0

botnets in tor
by Enkidu Mo Shiri 22 Nov '14

22 Nov '14

Hi everyone, am new and not very sure about rules here,hope its not off-topic. im master student researcher and i am working on botnet detection. it would be appreciated if anyone can help me with : I. anyway to detect botnet trafic out of normal tor traffic? II. is this solution possible to apply? to attach a script to botnet ( so we pass reverse engineering and no mess with codes of malware) and send it back to network, let the relays and last node which is botmaster recieves the compromised botnet ,and the script report our detector machine of any time it been re-routed in tor network ( hidden service and relays and botmaster) about : ip address of receiver hosts (relays) and their computer information( os,...) and consider last hop as botmaster. after using this technic for few botnets, we can have a good view of how relays (and which relays) been used for attack bt botnet and who is botmaster : it can help tor admin to clean relays, remove that hidden service , blacklist botmaster ip address thank you *Ehsan Moshiri (Enkidu)* *Digital Forensic Student* *H/P:+96164953954 , +961124249769* *Linkedin: http://my.linkedin.com/pub/enkidu-moshiri/59/baa/90b/ <http://my.linkedin.com/pub/enkidu-moshiri/59/baa/90b/>* *Facebook: Enkidu Mo Shi Ri* *wechat: Enkidu-Moshiri* *Line: Enkidu.Moshiri*

1 0

Using Shadow to increase agility of Sponsor R deliverables
by Rob Jansen 21 Nov '14

21 Nov '14

Hi David, Roger, I think it would be great if we could show off some HS performance improvements at the next Sponsor R PI meeting in January, both as a way to show that we are making progress on the performance front and also to help demonstrate our agility and how quickly we can get results on new designs. I’m here to advocate the use of Shadow to help in this regard. So far, the list of deliverable that we could possibly show improvements is quite small: https://trac.torproject.org/projects/tor/wiki/org/sponsors/SponsorR#Tickets From the list of completed tickets, the only one that stands out to me as providing a performance boost is #13211 (Allow optimistic data on connections to hidden services): https://trac.torproject.org/projects/tor/ticket/13211 This seems like a somewhat small change; despite that, do we think this may be something worth simulating to verify it works as expected and understand the extent to which it reduces time to first byte for HS downloads? Are there other HS performance improvements that we think may be ready by January? Best, Rob

2 6

Stem roadmap, and ideas for the interpreter?
by Damian Johnson 21 Nov '14

21 Nov '14

Hi Nick. While sipping my morning coffee I just realized that you might not be aware of my sinister long schemes, and that might be to the detriment of us both. The roadmap I have for myself goes something like... Step 1: Write Stem so we have a well tested, stable controller library. (Done [1]) Step 2: Write a control interpreter to greatly simplify low level controller troubleshooting. (Done [2]) Step 3: Rewrite arm, both simplifying the codebase and smoothing the rough edges that have confused users for years. (In progress, gonna keep me busy for the better part of a year) Step 4: Write a web interface similar to arm, so relay operators can have a nice monitor by going to localhost. [3] I bring this up because honestly we don't communicate much. I don't know core tor's long term plans, and I get the impression you don't know much about Stem either. My hope had been that at step #3 we'd start bundling arm and Stem with the Tor relaying bundle, providing users both a terminal monitor and the tor-prompt command. But since we axed that with the removal of Vidalia I suppose I'll need to hit step #4 before proposing that we re-introduce those bundles (... and by extension making it so Windows and OSX users can once have an easy option for relaying). Have you ever used Stem's tor-prompt before? https://stem.torproject.org/tutorials/down_the_rabbit_hole.html If not then I'd appreciate input so this can be a useful tool to you before proposing that we make it more widely available. Cheers! -Damian PS. More general ideas on directions to take Stem, the tor-prompt, and this whole space are also welcome both from you and the list! [1] https://stem.torproject.org/ [2] https://stem.torproject.org/tutorials/down_the_rabbit_hole.html [3] https://www.torproject.org/getinvolved/volunteer.html.en#relayWebPanel

1 0

Hidden Service authorization UI
by George Kadianakis 21 Nov '14

21 Nov '14

Hidden Service authorization is a pretty obscure feature of HSes, that can be quite useful for small-to-medium HSes. Basically, it allows client access control during the introduction step. If the client doesn't prove itself, the Hidden Service will not poroceed to the rendezvous step. This allows HS operators to block access in a lower level than the application-layer. It also prevents guard discovery attacks since the HS will not show up in the rendezvous. It's also a way for current HSes to hide their address and list of IPs from the HSDirs (we get this for free in rend-spec-ng.txt). In the current HS implementation there are two ways to do authorization: https://gitweb.torproject.org/torspec.git/blob/HEAD:/rend-spec.txt#l768 both have different threat models. In the future "Next Generation Hidden Services" specification there are again two ways to do authorization: https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/224-rend-spe… One way is with a password and the other is with a public key. I suspect that HS authorization is very rare in the current network, and if we believe it's a useful tool, it might be worthwhile to make it more useable by people. For example, it would be interesting if TBB would allow people to input a password/pubkey upon visiting a protected HS. Protected HSes can be recognized by looking at the "authentication-required" field of the HS descriptor. Typing your password on the browser is much more useable than editing a config file. Furthermore on the server-side, like meejah recently suggested [0], it would be nice if there was a way for HSes to be able to dynamically add/remove authorized clients using the control port. [0]: https://lists.torproject.org/pipermail/tor-dev/2014-October/007693.html

12 14

tor-0.2.5.10: weird test failure
by Thomas Klausner 21 Nov '14

21 Nov '14

Hi! I'm currently trying to update tor in pkgsrc (www.pkgsrc.org) to 0.2.5.10. When I run the self tests with 'make check', I see: .../work/tor-0.2.5.10> make check make check-TESTS check-local PASS: src/test/test make all-am ============================================================================ Testsuite summary for tor 0.2.5.10 ============================================================================ # TOTAL: 1 # PASS: 1 # SKIP: 0 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ python2.7 ./src/test/test_cmdline_args.py ./src/or/tor "." ............ ---------------------------------------------------------------------- Ran 12 tests in 0.299s OK python2.7 ./src/test/ntor_ref.py test-tor OK python2.7 ./src/test/ntor_ref.py self-test OK ./src/test/test-bt-cl assert | python2.7 ./src/test/bt_test.py BAD ./src/test/test-bt-cl crash | python2.7 ./src/test/bt_test.py BAD [1] Segmentation fault (core dumped) ./src/test/test-... | Done python2.7 ./src/... Running src/test/test directly works fine though: .../work/tor-0.2.5.10> ./src/test/test ... util/fgets_eagain: SKIPPED ... 222 tests ok. (1 skipped) Any ideas where to look first? Thomas

1 0

torsocks-2.0: NetBSD mmap patch
by Thomas Klausner 21 Nov '14

21 Nov '14

Hi! Please apply the attached patch to fix torsocks-2.0 on NetBSD. Thanks, Thomas

1 0

Re: [tor-dev] How to distribute Tor with other software
by David Stainton 21 Nov '14

21 Nov '14

Greetings, olde thread resurrection: Earlier Meejah pointed out that the tor control port can be used to create tor hidden services. Now that tor trac ticket #11291 (https://trac.torproject.org/projects/tor/ticket/11291) is resolved this will actually be usable? There are deployment issues. The program must run as a user who is a member of the tor group in order to read the hostname file and thus retrieve the onion address. With the current upstream master tor you can use the HiddenServiceDirGroupReadable option, like this: SETCONF hiddenservicedir=/var/lib/tor-alpha-hidden-services/hiddenService01 hiddenserviceport="80 127.0.0.1:8080" HiddenServiceDirGroupReadable=1 This should result in the hostname file being readable by the tor group. Is anyone else working on tor hidden service file sharing and or tor controller written in golang? I've implemented a golang api for creating and destroying tor hidden services using the control port... and a golang commandline tool for sharing directories via http + tor hidden service similar to carml pastebin or onionshare. If we write these tools in golang then we can use a deterministic build system much like the recent gitian golang deterministic builds for golang pluggable transports. Cheers, David

1 0

Pluggable transports meeting tomorrow (16:00UTC Wednesday 19th of November 2014)
by Yawning Angel 18 Nov '14

18 Nov '14

Hello! just wanted to remind you that the regular biweekly pluggable transports meeting is going to occur tomorrow at 16:00 UTC. Place is the #tor-dev IRC channel in the OFTC network. Thanks for your attention! -- Yawning Angel

1 0

User experience issue
by Adam Shostack 15 Nov '14

15 Nov '14

Hi, I just had the fun experience of trying to walk someone who's not very technical and not familiar with Tor through downloading and installing the TBB. One element which came out was she was confused by the relationship between Tor, the Tor Browser Bundle, and Vidalia, and became frustrated by that experience. ("What's this Vidalia thing? I thought we were installing Tor!") I'm unsure if it would be appropriate/useful to file a bug, and so would like advice on if/how to tackle this. Adam

2 2

Re: [tor-dev] Of CA-signed certs and .onion URIs
by Jeremy Rowley 14 Nov '14

14 Nov '14

Great summary Tom, >From my perspective, getting .onion reserved is a pretty high priority. Once reserved, we can really eliminate it as an internal name and get onion listed as part of the PSL. I'm happy to help with this part of the project if I can. >Syrup-tan had an idea on irc: Have a DV certificate sign a certificate >that is valid for the .onion URL, and display the URL of the DV >certificate. This doesn't eliminate phishing - I can register >facebok.com and then get that displayed. But doing bootstapping off >DNS and DV certificates is a fairly low bar in terms of the cost to a >.onion operator. (There are other concerns here, I'm not completely >comfortable with repurposing the EV indicator in this way. Asa on irc >had the good point that if we did this, maybe we'd want to change the >EV green to another color just to be a little bit different. Not that >I really expect users to notice that though...) This is similar to what I was thinking by proposing that CAs have both a non-onion and onion name in it. If you do that and both are validated as part of the same certificate order, you could give users an indicator that the non-onion name is related to the onion name in the certificate. >Allowing an organization to purchase an EV certificate from a CA, and >display the organization's name in the address bar, is another way - >albeit a very high bar in terms of cost to an onion operator. I'm hoping the cost won't be high - I'm interested in this solely as something that supports Tor and companies like Facebook who want to give users greater privacy. Like you, I'm not looking to make EV certs required for onion operators. Instead, I'd like to see them permitted under the industry standards for companies looking to prominently promote their onion services. The biggest penalty of requiring EV is that it locks out individuals - which is very bad. I've proposed EV for individuals several times on the forum. Probably time to bring it up again. >But there should be at least one more solution in >the short to long term (e.g. a petname approach). Unfortunately, if >the time between now and the 'long term' solution is too long, it >locks out everyone who can't get an EV cert - which is a legitimate >concern. Perhaps after there's a spec Tor likes, some large >organization concerned about preventing phishing could throw some >engineering time at the problem. Jeremy

1 0

OONI weekly dev meeting reminder
by Arturo Filastò 13 Nov '14

13 Nov '14

Hello Oonitarians, This is reminder that today (Thursday 6th of November 2014) there will be the weekly ooni dev meeting in #ooni on irc.oftc.net. The time will be: 18:00 UTC 19:00 CET 13:00 EST 12:00 CST 10:00:00 PST You are all invited to attend! ~ Arturo

1 1

Defending against guard discovery attacks by pinning middle nodes
by George Kadianakis 13 Nov '14

13 Nov '14

Hey Nick, this mail is about the schemes we were discussing during the dev meeting on how to protect HSes against guard discovery attacks (#9001). I think we have some ideas on how to offer better protection against such attacks, mainly by keeping our middle nodes more static than we do currently. For example, we could keep our middle nodes for 3-4 days instead of choosing new ones for every circuit. As Roger has suggested, maybe we don't even need to write the static middle nodes on the state file, just use new ones if Tor has restarted. Keeping middle nodes around for longer will make those attacks much slower (it restricts them to one attack attempt every 3-4 days), but are there any serious negative implications? For example, if you were unlucky and you picked an evil middle node, and you keep it for 3-4 days, that middle node will always see your traffic coming through your guard (assuming a single guard per client). If we assume you use a non-popular guard node (with only a few clients using it), the middle guard might be able to think "Ah, the circuit that comes from that guard node is always user X" making your circuits a bit linkable from the PoV of your middle node. What other attacks should we be wary about? Maybe partitioning attacks based on client behavior? And how should we move this forward if we decide it's worth it? Should we start writing a Tor proposal? Thanks!

8 26

Gap between advertised and utilized bandwidth
by Karsten Loesing 12 Nov '14

12 Nov '14

Hi George, I found this in my IRC backlog from November 7, 2014: 20:34 #tor-dev: < asn> karsten: how should one read these graphs https://metrics.torproject.org/bandwidth.html#bandwidth-flags ? 20:34 #tor-dev: < asn> karsten: can the gap between two lines be interpreted as available (unused) bw? 20:35 #tor-dev: < asn> karsten: so the gap between the yellow lines is the available guard bw? 20:35 #tor-dev: < asn> karsten: but there is not that much available exit bw, right? so it's not entirely accurate. These are all fine questions. First of all, the gap between two lines is *supposed* to be interpreted as available (unused) bandwidth. Advertised bandwidth is the number of bytes that all relay together were willing and capable to handle. (It's the minimum of the three bandwidth numbers contained in server descriptors.) Bandwidth history is the number of bytes that all relays together actually handled. (It's the sum of byte histories contained in extra-info descriptors.) So, the difference of the two should be the number of bytes that was left unused, in theory. However, I can't say how accurate this comparison is. If I had the time to investigate, I'd start by downloading the latest descriptor archives from CollecTor and comparing advertised bandwidths and bandwidth histories per relay. If you or somebody else wants to do that I might be able to give them a hand. All the best, Karsten

2 1

tor packet handling
by Mohiuddin Ebna Kawsar 12 Nov '14

12 Nov '14

Hi, I want to develop extension(intrusion detection) for tor. for that i have to extract TCP and IP header from packet. I need to know where and how tor handle packet(TCP/IP). is it in buffer.c / connection.c ? Regards Kawsar

2 1

Re: [tor-dev] Pluggable-transport implementations of your website fingerprinting defenses
by David Fifield 10 Nov '14

10 Nov '14

On Sun, Nov 09, 2014 at 08:23:33PM -0500, Xiang Cai wrote: > I started to work on csbuflo code a long time ago, and I wasn’t using any > version control software back then, so I don’t have file commit history either… > Sorry about that. > > However, I only modified several core files based on openssh-5.9p1 source code: > clientloop.c > serverloop.c > packet.c > misc.c > and related header files. A simple diff between these files and the original > ssh code will tell you what I modified. > > I am not sure if the code is directly useable for your purpose, but I’ll > briefly talk about what my code does, and hopefully, it will give you some help > when reading the code. > > The code actually implements the Glove system, which requires that both the > client and server have a transcript of “super traces”. — I believe the location > of the transcript is hardcoded as ‘/var/tmp/st.txt’ in my code … > When visiting a website that is not shown in the transcript, the system falls > back to use CSBuFLO scheme. I see. Thanks for the references. Building this code into a pluggable transport would be more work than I had originally supposed. If there were a minimal network client and server, only implementing the website fingerprinting defense, then I wouldn't mind spending half a day to make them into a pluggable transport. But as it stands, it looks like it will be more work, essentially a reimplementation. David Fifield

1 0

Re: [tor-dev] Hidden Service authorization UI
by Nathan Freitas 10 Nov '14

10 Nov '14

On Sun, Nov 9, 2014, at 07:50 AM, George Kadianakis wrote: > Hidden Service authorization is a pretty obscure feature of HSes, that > can be quite useful for small-to-medium HSes. ... > For example, it would be interesting if TBB would allow people to > input a password/pubkey upon visiting a protected HS. Protected HSes > can be recognized by looking at the "authentication-required" field of > the HS descriptor. Typing your password on the browser is much more > useable than editing a config file. We have been working on implementing an OnionShare feature for Orbot, as a plugin/add-on. Since the client and the server are both one simple app, it seems like we could easily implement the HS Authorization feature. Since the goal is to share a file with a small audience, the second "client key" approach seems to be the best, most secure approach to me. Any thoughts?

1 0

Re: [tor-dev] HSDir Auth and onion descriptor scraping - actual stats
by Gareth Owen 10 Nov '14

10 Nov '14

OK. curiosity got the better of me. I took a random sample of 20,368 HS descriptors and just 131 were authenticated - that's about 0.6%. The code I used is here: https://github.com/drgowen/tor-research-framework/blob/master/src/main/java… Best Gareth PS - I only took a sample (rather than the whole batch) because each HS descriptor is in its own file and it takes ages to process - nevertheless, because of the way its collected the sample should be representative. -- Dr Gareth Owen Senior Lecturer Forensic Computing Course Leader School of Computing, University of Portsmouth *Office:* BK1.25 *Tel:* +44 (0)2392 84 (6423) *Web*: ghowen.me

1 0

HSDir Auth and onion descriptor scraping
by grarpamp 10 Nov '14

10 Nov '14

> George K: > I suspect that HS authorization is very rare in the current network, > and if we believe it's a useful tool, it might be worthwhile to make > it more useable by people. Is anyone making their HSDir onion descriptor scraping patches available somewhere? I'd suspect the rarity of HS authorization could also be determined with that since some fields would be obfuscated and thus not match patterns. s/scraping/logging/ rend--spec.txt: 2. Authentication and authorization. 2.1. Service with large-scale client authorization 2.2. Authorization for limited number of clients 2.3. Hidden service configuration 2.4. Client configuration

3 4

Running a Separate Tor Network
by Tom Ritter 10 Nov '14

10 Nov '14

Hi all, Not content to let you have all the fun, I decided to run my own Tor network! Kidding ;) But the Directory Authorities, the crappy experiment leading up to Black Hat, and the promise that one can recreate the Tor Network in the event of some catastrophe interests me enough that I decided to investigate it. I'm aware of Chutney and Shadow, but I wanted it to feel as authentic as possible, so I forwent those, and just ran full-featured independent tor daemons. I explicitly wanted to avoid setting TestingTorNetwork. I did have to edit a few other parameters, but very few. [0] I plan on doing a blog post, giving a HOWTO, but I thought I'd write about my experience so far. I've found a number of interesting issues that arise in the bootstrapping of a non-TestingTorNetwork, mostly around reachability testing. ----- One of the first things I ran into was a problem where I could not get any routers to upload descriptors. An OR checks itself to determine reachability before uploading a descriptor by building a circuit - bypassed with AssumeReachable or TestingTorNetwork. This works fine for Chutney and Shadow, as they reach into the OR and set AssumeReachable. But if the Tor Network were to be rebooted... most nodes out there would _not_ have AssumeReachable, and they would not be able to perform self-testing with a consensus consisting of just Directory Authorities. I think nodes left running would be okay, but nodes restarted would be stuck in a startup loop. I imagine what would actually happen is Noisebridge and TorServers and a few other close friends would set the flag, they would get into the consensus, and then the rest of the network would start coming back... (Or possibly a few nodes could anticipate this problem ahead of time, and set it now.) What I had to do was make one of my Directory Authorities an exit - this let the other nodes start building circuits through the authorities and upload descriptors. Maybe an OR should have logic that if it has a valid consensus with no Exit nodes, it should assume it's reachable and send a descriptor - and then let the Directory Authorities perform reachability tests for whether or not to include it? From the POV of an intentional DoS - an OR doesn't have to obey the reachability test of course, so no change there. It could potentially lead to an unintentional DoS where all several thousand routers start slamming the DirAuths as soon as a usable-but-blank consensus is found... but AFAIK routers probe for a consensus based on semi-random timing anyway, so that may mitigate that? ----- Another problem I ran into was that nodes couldn't conduct reachability tests when I had exits that were only using the Reduced Exit Policy - because it doesn't list the ORPort/DirPort! (I was using nonstandard ports actually, but indeed the reduced exit policy does not include 9001 or 9030.) Looking at the current consensus, there are 40 exits that exit to all ports, and 400-something exits that use the ReducedExitPolicy. It seems like 9001 and 9030 should probably be added to that for reachability tests? ----- Continuing in this thread, another problem I hit was that (I believe) nodes expect the 'Stable' flag when conducting certain reachability tests. I'm not 100% certain - it may not prevent the relay from uploading a descriptor, but it seems like if no acceptable exit node is Stable - some reachability tests will be stuck. I see these sorts of errors when there is no stable Exit node (the node generating the errors is in fact a Stable Exit though, so it clearly uploaded its descriptor and keeps running): Oct 13 14:49:46.000 [warn] Making tunnel to dirserver failed. Oct 13 14:49:46.000 [warn] We just marked ourself as down. Are your external addresses reachable? Oct 13 14:50:47.000 [notice] No Tor server allows exit to [scrubbed]:25030. Rejecting. Since ORPort/DirPort are not in the ReducedExitPolicy, this (may?) restrict the number of nodes available for conducting a reachability test. I think the Stable flag is calculated off the average age of the network though, so the only time when this would cause a big problem is when the network (DirAuths) have been running for a little bit and a full exit node hasn't been added - it would have to wait longer for the full exit node to get the Stable flag. ----- Getting a BWAuth running was... nontrivial. Some of the things I found: - SQLAlchemy 0.7.x is no longer supported. 0.9.x does not work, nor 0.8.x. 0.7.10 does. - Several quasi-bugs with the code/documentation (the earliest three commits here: https://github.com/tomrittervg/torflow/commits/tomedits) - The bandwidth scanner actively breaks in certain situations of divide-by-zero (https://github.com/tomrittervg/torflow/commit/053dfc17c0411dac0f6c4e43954f9…) - The scanner will be perpetually stuck if you're sitting on the same /16 and you don't perform the equivalent of EnforceDistinctSubnets 0 [1] Ultimately, while I successfully produced a bandwidth file [2], I wasn't convinced it was meaningful. There is a tremendous amount of code complexity buried beneath the statement 'Scan the nodes and see how fast they are', and a tremendous amount of informational complexity behind 'Weight the nodes so users can pick a good stream'. ----- I tested what it would look like if an imposter DirAuth started trying to participate in the consensus. It generated the warning you would expect: Oct 14 00:04:31.000 [warn] Got a vote from an authority (nickname authimposter, address W.X.Y.Z) with authority key ID Z. This key ID is not recognized. Known v3 key IDs are: A, B, C, D But it also generated a warning you would not expect, and sent me down a rabbit hole for a while: Oct 10 21:44:56.000 [debug] directory_handle_command_post(): Received POST command. Oct 10 21:44:56.000 [debug] directory_handle_command_post(): rewritten url as '"/tor/post/consensus-signature"'. Oct 10 21:44:56.000 [notice] Got a signature from W.X.Y.Z. Adding it to the pending consensus. Oct 10 21:44:56.000 [info] dirvote_add_signatures_to_pending_consensus(): Have 1 signatures for adding to ns consensus. Oct 10 21:44:56.000 [info] dirvote_add_signatures_to_pending_consensus(): Added -1 signatures to consensus. Oct 10 21:44:56.000 [info] dirvote_add_signatures_to_pending_consensus(): Have 1 signatures for adding to microdesc consensus. Oct 10 21:44:56.000 [info] dirvote_add_signatures_to_pending_consensus(): Added -1 signatures to consensus. Oct 10 21:44:56.000 [warn] Unable to store signatures posted by W.X.Y.Z: Mismatched digest. Over on the imposter: Oct 14 00:19:32.000 [warn] http status 400 ("Mismatched digest.") response after uploading signatures to dirserver 'W.X.Y.Z:15030'. Please correct. The imposter DirAuth is sending up a signature for a consensus that is not the same consensus that the rest of the DirAuths computed. Specifically, the imposter DirAuth lists itself as a dir-source and the signature covers this line. (Everything else matches because the imposter has been outvoted and respects that.) I guess the lesson is, if you see the "Mismatched digest" warning in conjunction with the unrecognized key ID - it's just one issue, not two. ----- The notion and problems of an imposter DirAuth also come up during how the network behaves when adding a DirAuth. I started with 4 authorities, then started a fifth (auth5). Not interesting - it behaved as the imposter scenario. I then added auth5 to a single DirAuth (auth1) as a trusted DirAuth. This resulted in a consensus with 3 signatures, as auth1 did not sign the consensus. On auth1 I got warn messages: A consensus needs 3 good signatures from recognized authorities for us to accept it. This one has 2 (auth1 auth5). 3 (auth2 auth3 auth4) of the authorities we know didn't sign it. I then added auth5 to a second DirAuth (auth2) as a trusted DirAuth. This results in a consensus for auth1, auth2, and auth5 - but auth3 and auth4 did not sign it or produce a consensus. Because the consensus was only signed by 2 of the 4 Auths (e.g., not a majority) - it was rejected by the relays (which did not list auth5). At this point something interesting and unexpected happened: The other 2 DirAuths (not knowing about auth5) did not have a consensus. This tricked dirvote_recalculate_timing into thinking we should use the TestingV3AuthInitialVotingInterval parameters, so they got out of sync with the other 3 DirAuths (that did know about auth5). That if/else statement seems very odd, and the parameters seem odd as well. First off, I'm not clear what the parameters are intended to represent. The man page says: TestingV3AuthInitialVotingInterval N minutes|hours Like V3AuthVotingInterval, but for initial voting interval before the first consensus has been created. Changing this requires that TestingTorNetwork is set. (Default: 30 minutes) TestingV3AuthInitialVoteDelay N minutes|hours Like TestingV3AuthInitialVoteDelay, but for initial voting interval before the first consensus has been created. Changing this requires that TestingTorNetwork is set. (Default: 5 minutes) TestingV3AuthInitialDistDelay N minutes|hours Like TestingV3AuthInitialDistDelay, but for initial voting interval before the first consensus has been created. Changing this requires that TestingTorNetwork is set. (Default: 5 minutes) Notice that the first says "Like V3AuthVotingInterval", but the other two just repeat their name? And how there _is no_ V3AuthInitialVotingInterval? And that you can't modify these parameters without turning on TestingTorParameters (despite the fact that they will be used without TestingTorNetwork?) And also, unrelated to the naming, these parameters are a fallback case for when we don't have a consensus, but if they're not kept in sync with V3AuthVotingInterval and their kin - the DirAuth can wind up completely out of sync and be unable to recover (except by luck). It seems like these parameters should be renamed to V3AuthInitialXXX, keep their existing defaults, remove the requirement on TestingTorNetwork, and be documented as needing to be divisors of the V3AuthVotingXXX parameter, to allow a DirAuth who has tripped into them to be able to recover. I have a number of other situations I want to test around adding, subtracting, and manipulating traffic to a DirAuth to see if there are other strange situations that can arise. ----- Other notes: - I was annoyed by TestingAuthDirTimeToLearnReachability several times (as I refused to turn on TestingTorNetwork) - I wanted to override it. I thought maybe that should be an option, but ultimately convinced myself that in the event of a network reboot, the 30 minutes would likely still be needed. - The Directory Authority information is a bit out of date. Specifically, I was most confused by V1 vs V2 vs V3 Directories. I am not sure if the actual network's DirAuths set V1AuthoritativeDirectory or V2AuthoritativeDirectory - but I eventually convinced myself that only V3AuthoritativeDirectory was needed. - It seems like an Authority will not vote for itself as an HSDir or Stable... but I could't find precisely where that was in the code. (It makes sense to not vote itself Stable, but I'm not sure why HSDir...) - The networkstatus-bridges file is not included in the tor man page - I feel like the log message "Consensus includes unrecognized authority" (currently info) is worthy of being upgraded to notice. - While debugging, I feel this patch would be helpful. [3] - I've had my eye on Proposal 164 for a bit, so I'm keeping that in mind - I wanted the https://consensus-health.torproject.org/ page for my network, but didn't want to run the java code, so I ported it to python. This project is growing, and right now I've been editing consensus_health_checker.py as well. https://github.com/tomrittervg/doctor/commits/python-website I have a few more TODOs for it (like download statistics), but it's coming along. ----- Finally, something I wanted to ask after was the idea of a node (an OR, not a client) belonging to two or more Tor networks. From the POV of the node operator, I would see it as a node would add some config lines (maybe 'AdditionalDirServer' to add to, rather than redefining, the default DirServers), and it would upload its descriptors to those as well, fetch a consensus from all AdditionalDirServers, and allow connections from and to nodes in either. I'm still reading through the code to see which areas would be particularly confusing in the context of multiple consensuses, but I thought I'd throw it out there. -tom ----- [0] AuthoritativeDirectory 1 V3AuthoritativeDirectory 1 VersioningAuthoritativeDirectory 1 RecommendedClientVersions [stuff] RecommendedServerVersions [stuff] ConsensusParams [stuff] AuthDirMaxServersPerAddr 0 AuthDirMaxServersPerAuthAddr 0 V3AuthVotingInterval 5 minutes V3AuthVoteDelay 30 seconds V3AuthDistDelay 30 seconds V3AuthNIntervalsValid 3 MinUptimeHidServDirectoryV2 1 hour ----- [1] diff --git a/NetworkScanners/BwAuthority/bwauthority_child.py b/NetworkScanners/BwAuthority/bwauthority_child.py index 28b89c2..e07718f 100755 --- a/NetworkScanners/BwAuthority/bwauthority_child.py +++ b/NetworkScanners/BwAuthority/bwauthority_child.py @@ -60,7 +60,7 @@ __selmgr = PathSupport.SelectionManager( percent_fast=100, percent_skip=0, min_bw=1024, - use_all_exits=False, + use_all_exits=True, uniform=True, use_exit=None, use_guards=False, ----- [2] node_id=$C447A9E99C66A96E775A5EF7A8B0DF96C414D0FE bw=37 nick=relay4 measured_at=1413257649 updated_at=1413257649 pid_error=0.0681488657221 pid_error_sum=0 pid_bw=59064 pid_delta=0 circ_fail=0.0 node_id=$70145044B8C20F46F991B7A38D9F27D157B1CB9D bw=37 nick=relay5 measured_at=1413257649 updated_at=1413257649 pid_error=0.0583026021009 pid_error_sum=0 pid_bw=59603 pid_delta=0 circ_fail=0.0 node_id=$7FAC1066DCCC0C62984B8E579C5AABBBAE8146B2 bw=37 nick=exit2 measured_at=1413257649 updated_at=1413257649 pid_error=0.0307144741235 pid_error_sum=0 pid_bw=55938 pid_delta=0 circ_fail=0.0 node_id=$9838F41EB01BA62B7AA67BDA942AC4DC3B2B0F98 bw=37 nick=exit3 measured_at=1413257649 updated_at=1413257649 pid_error=0.0124944051714 pid_error_sum=0 pid_bw=55986 pid_delta=0 circ_fail=0.0 node_id=$49090AC6DB52AD8FFF95AF1EC1E898126A9E5CA6 bw=37 nick=relay3 measured_at=1413257649 updated_at=1413257649 pid_error=0.0030073731241 pid_error_sum=0 pid_bw=56489 pid_delta=0 circ_fail=0.0 node_id=$F5C43BB6AD2256730197533596930A8DD7BEC367 bw=37 nick=exit1 measured_at=1413257649 updated_at=1413257649 pid_error=-0.0032777385693 pid_error_sum=0 pid_bw=55114 pid_delta=0 circ_fail=0.0 node_id=$3D53FF771CC3CB9DE0A55C33E5E8DA4238C96AB5 bw=37 nick=relay2 measured_at=1413257649 updated_at=1413257649 pid_error=-0.0418210520821 pid_error_sum=0 pid_bw=51021 pid_delta=0 circ_fail=0.0 ----- [3] diff --git a/src/or/networkstatus.c b/src/or/networkstatus.c index 890da0a..4d72add 100644 --- a/src/or/networkstatus.c +++ b/src/or/networkstatus.c @@ -1442,6 +1442,8 @@ networkstatus_note_certs_arrived(void) waiting_body, networkstatus_get_flavor_name(i), NSSET_WAS_WAITING_FOR_CERTS)) { + log_info(LD_DIR, "After fetching certificates, we were able to " + "accept the consensus."); tor_free(waiting_body); } }

3 3

Pluggable-transport implementations of your website fingerprinting defenses
by David Fifield 09 Nov '14

09 Nov '14

NB I'm copying the tor-dev mailing list on this message. At CCS I saw Rishab present these papers: "CS-BuFLO: A Congestion Sensitive Website Fingerprinting Defense" http://www3.cs.stonybrook.edu/~rnithyanand/pubs/wpes2014-csb.pdf "Glove: A Bespoke Website Fingerprinting Defense" http://www3.cs.stonybrook.edu/~rnithyanand/pubs/wpes2014-glove.pdf "A Systematic Approach to Developing and Evaluating Website Fingerprinting Defenses" http://www3.cs.stonybrook.edu/~rnithyanand/pubs/ccs2014.pdf I spoke quite a lot to Rishab and suggested, since these schemes have source code, that they be wrapped in a pluggable transports interface so they can be easily used by tor clients. It is not super difficult to turn a network program into a pluggable transport, as there are libraries like pyptlib and liballium that implement the internal pluggable transports protocol. You might be familiar with wfpadtools (https://bitbucket.org/mjuarezm/obfsproxy-wfpadtools) which aims to make it easy to prototype fingerprinting defenses by specifying them in terms of (e.g. padding) primitives. I looked for the source code mentioned in the papers and I wasn't sure what would be best to use. I found https://crysp.uwaterloo.ca/software/webfingerprint/ which links to some short Python files like https://crysp.uwaterloo.ca/software/webfingerprint/tamaraw.py There is also https://github.com/xiang-cai/CSBuFLO but it doesn't appear to be directly usable. It appears to be a modified OpenSSH, without a commit history showing what was modified. What source code do you recommend for an implementation? David Fifield

1 0

Hi everyone!
by Conny Hermansson 09 Nov '14

09 Nov '14

Hi! I´m new to the Tor project and I´m looking for some easy project to get me started. preferebly in Java, I can do debugging or write some code. Conny

2 1