- tor-dev - lists.torproject.org

Proposal 246: Defending Against Guard Discovery Attacks using Vanguards
by George Kadianakis 19 Jul '15

19 Jul '15

Hello, I'm attaching a proposal draft that should help us defend against guard discovery attacks. There are a few pieces left unfinished (see the XXXs) but I decided to release early and release often for the sake of moving forward with this. I consider this issue very important and any feedback is greatly appreciated. Thanks! ---- Filename: 246-hs-guard-discovery.txt Title: Defending Against Guard Discovery Attacks using Vanguards Author: George Kadianakis Created: 2015-07-10 Status: Draft 0. Motivation A guard discovery attack allow attackers to determine the guard node of a Tor client. The hidden service rendezvous protocol provides an attack vector for a guard discovery attack since anyone can force an HS to construct a 3-hop circuit to a relay (#9001). Following the guard discovery attack with a compromise or coercion of the guard node can lead to the deanonymization of a hidden service. 1. Overview This document tries to make the above guard discovery + coersion attack harder to launch. It introduces an optional configuration option which makes the hidden service also pin the second and third hops of its circuits for a longer duration. With this new path selection, we force the adversary to perform a Sybil attack and two coercion attacks before succeeding. This is an improvement over the current state where the Sybil attack is trivial to pull off, and only a single coercion attack is required. With this new path selection, an attacker is forced to do a coercion attack before learning the guard node of a hidden service. This increases the uncertainty of the attacker, since he will need to perform a coercion attack before he learns the identity of the guard node and whether he can compromise it. Coercion attacks are costly and potentially detectable, so an attacker will have to think twice before beginning a chain of coercion attacks that he might not be able to complete. 1.1. Visuals Here is how a hidden service rendezvous circuit currently looks like: -> middle_1 -> middle_A -> middle_2 -> middle_B -> middle_3 -> middle_C -> middle_4 -> middle_D HS -> guard -> middle_5 -> middle_E -> Rendezvous Point -> middle_6 -> middle_F -> middle_7 -> middle_G -> middle_8 -> middle_H -> ... -> ... -> middle_n -> middle_n this proposal pins the two middles nodes to a much more restricted set, as follows: -> guard_3_A -> guard_3_B HS -> guard_1 -> guard_2_A -> guard_3_C -> Rendezvous Point -> guard_2_B -> guard_3_D -> guard_3_E -> guard_3_F 2. Design This feature requires the HiddenServiceGuardDiscovery torrc option to be enabled. When a hidden service picks its guard nodes, it also picks two additional sets of guard nodes `second_guard_set` and `third_guard_set` of size NUM_SECOND_GUARDS and NUM_THIRD_GUARDS respectively. When a hidden service needs to establish a circuit to an HSDir, introduction point or a rendezvous point, it uses nodes from `second_guard_set` as the second hop of the circuit and nodes from `third_guard_set` as third hops of the circuit. A hidden service rotates 'second_guard_set' every SECOND_GUARD_ROTATION hours, and 'third_guard_set' every THIRD_GUARD_ROTATION hours. These extra guard nodes should be picked with the same path selection procedure that is used for regular guard nodes. Care should be taken such that guard sets do not share any common relays. XXX or simply that they are not used in the same circuit? XXX maybe pick the second and third layer guards from the set of middle nodes but also enforce some kind of uptime requirement? that should greatly help our load balancing. XXX maybe we should also introduce consensus flags for the extra guard layers? Vanguard? XXX how should proposal 241 ("Resisting guard-turnover attacks") be applied here? 2.1. Security parameters We set NUM_SECOND_GUARDS to 2 nodes and NUM_THIRD_GUARDS to 6 nodes. We set SECOND_GUARD_ROTATION to 2 weeks and THIRD_GUARD_ROTATION to 1 day. See the discussion section for more analysis on these constants. 3. Discussion 3.1 How were these security parameters chosen? Consider an adversary with the following powers: - Can launch a Sybil guard discovery attack against any node of a rendezvous circuit. The slower the rotation period of the node, the longer the attack takes. - Can compromise any node on the network. We assume that the adversary cannot compromise too many nodes, otherwise Tor's security would be breached anyhow. We now calculate the time it takes for the adversary to launch a Sybil attack with 50% success assuming 5% network control. This depends solely on how frequently the hidden service rotates that node: +-------------------+-------------------------------+------------------------+----------------------------+ | Rotation period | Sybil attack with 50% success | Sybil attack (2 guards)| Sybil attack (6 guards) | +-------------------+-------------------------------+------------------------+----------------------------+ | 1 hour | 14 hours | 7 hours | 2.5 hours | | 1 day | 14 days | 7 days | 2.5 days | | 1 week | 3.5 months | 6 weeks | 2.5 weeks | | 2 weeks | 7 months | 3.5 months | 5 weeks | | 1 month | 1 year and 2 months | 6 months | 2.5 months | | 3 months | 3.5 years | 1.7 years | 6 months | +-------------------+-------------------------------+------------------------+----------------------------+ Required time for Sybil attack by a 5% adversary Our security parameters were selected so that the first two layers of guards should be hard to attack using a Sybil guard discovery attack and hence require a coercion attack. On the other hand, the outmost layer of guards should rotate fast enough to _require_ a Sybil attack. XXX does this security model even make sense? what about a network adversary, or an adversary that can launch congestion attacks etc.???? 3.2. Distinguishing new HS circuits from normal HS circuits By pinning the middle nodes of rendezvous circuits, we make it easier for all hops of the circuit to detect that they are part of a special hidden service circuit. XXX hm how does the outermost guard knows? Compare this to the current Tor network, where only the guard and the third hop of the HS circuit can trivially distinguish whether it's part of an HS circuit. 3.3. Circuit nodes can now be linked to specific hidden services With this proposal the hops of hidden service circuits will be static, and hence an adversrary will be able to map them to specific hidden services. For example, a middle node that sees two hidden service circuits with the same guard node in different times, can assume with non-negligible probability that both circuits correspond to the same hidden service. That said, this is also partially the case for the current Tor network, where the middle node can associate a guard with a specific hidden service. 3.4 Why is the torrc setting disabled by default, if it's so good? We suggest this torrc option to be optional because it puts additional load on guard nodes and we are not sure if the network will be able to handle it. However, by having this setting be disabled by default, we make hidden services who use it stand out a lot. For this reason, we should eventually fix our hidden service circuit load balancing so that we can enable this globally. XXX But hidden services traffic is only 6% of the total network, so maybe it's not that big of a problem and we should just enable this feature by default anyway 3.4.1. How should we load balance to enable this feature globally? The load balancing issue with this feature is that hidden service traffic that would otherwise be passing through middle nodes, will now be passing through guard nodes. Furthermore, this additional traffic is not accounted for in our bandwidth weights. This means that a guard node that had 1% probability of being chosen as a guard for normal circuits, will still have the same probability of being chosen as a guard even though the hidden service traffic that it pushes increases. To improve the load balancing here, we could have each relay report the amount of hidden service traffic it pushes every day (#15254), and have the authorities take this into account when they calculate bandwidth weights. The idea here would be that the dirauths would know that N% of the network is hidden services traffic, hence they would tweak the bandwidth weights such that guards would reserve some N% of their bandwidth for hidden service purposes. 4. Future directions Here are some more ideas for improvements that should be done sooner or later: - Maybe we should make the size and rotation period of secondary/third guard sets to be configurable by the user. - To make it harder for an adversary, a hidden service MAY extend the path length of its circuits by an additional static hop. This forces the adversary to use another coercion attack to walk the chain up to the hidden service. 5. Acknowledgements Thanks to Aaron Johnson, John Brooks, Mike Perry and everyone else who helped with this idea.

7 12

Some statistics on introduction point stability and correctness
by George Kadianakis 18 Jul '15

18 Jul '15

Hello, during the past months we have been working on evaluating and confirming the stability and correctness of Tor hidden services. The hidden services protocol has multiple steps, and its soundness depends on various components of the Tor network. Throughout this document, we assume that the reader is familiar with the hidden services protocol. For people who want to get started now, we suggest you read the hidden services protocol description on our website [0]. During our evaluation, we continuously fetched the descriptors of a few hidden services, and collected statistics on the introduction points they picked. The hidden services we tested were selected arbitrarily so that they have decent uptime and are moderately used. We have labeled them alphabetically from (a) to (e). The data was collected over a period of 90 days using tor-hs-descriptor-fetcher [1]. With our experiment we were trying to answer questions like: Q: Do hidden services publish descriptors correctly and in the intended time schedule? Q: Is the introduction point codebase working properly? Are hidden service introduction points as stable as we want them to be? Q: How many introduction points do hidden services expose themselves to? Q: Normally, hidden services will use 3 intro points. However if they think they are getting too much traffic they will dynamically adjust the number of their introduction points according to a self-evaluation of their popularity. In this case, very popular hidden services may use up to 10 introduction points. Does this algorithm work as intended? Let's start our analysis by looking at some graphs: ----- [*] https://people.torproject.org/~asn/desc_stats/lifetimes-2015-07-14-b.png This graph shows the number of descriptors published per hour by each hidden service. Including descriptor replicas, a normal hidden service would publish 2 descriptors per hour which indeed seems to be the case most of the time according to the graphs. This is good since it shows that our system works properly. However, we also observe that all HSes will publish more than 2 descriptors every hour at least 15% of the time. We believe this occurs when they republish their descriptor because of an expired or dead intro point. ---- [*] https://people.torproject.org/~asn/desc_stats/lifetimes-2015-07-14-a.png Normally hidden services will keep their introduction points for a random lifetime between 18 and 24 hours. This graph shows how long the measured hidden services kept their introduction points for. Looking at the three hidden services (a), (e) and (f) it seems that introduction points indeed rotate as intended most of the time. Specifically, we see that about 75% of the intro points of those three hidden services indeed stay up for more than 18 hours. This reassured us that the introduction point rotation code works well. However, we can see that hidden services (b), (c) and (d) have lower intro point lifetimes. We believe that this is caused by the dynamic intro point formula which adjusts the number of introduction points for popular hidden services. Consider a hidden service with 9 intro points that starts getting less traffic and needs to go down to 5 intro points; in that case, the HS will discard 4 IPs reducing the average intro circuit lifetime. If the above procedure happens multiple times, it will drastically reduce the average lifetime of intro circuits. ---- [*] https://people.torproject.org/~asn/desc_stats/lifetimes-2015-07-14-g.png In this graph we present the number of introduction points of the hidden services over time. As mentioned previously, hidden services normally use 3 intro points, but they may increase that number if they believe they are too popular. Although this self-evaluation sounds like a neat feature, it also means that an attacker can estimate the popularity of a hidden service just by looking at the number of its introduction points. While this might not sound as a dangerous information leak, we think that there are legitimate use cases where the popularity of a hidden service should be hidden [2]. Because we want to avoid this popularity leak and also because we think that the number of introduction points is not that important for scalability, we have disabled the dynamic intro point formula completely (#4862). Now, hidden services establish 3 intro points by default but operators have the option to tune that number using a torrc parameter. ---- [*] https://people.torproject.org/~asn/desc_stats/lifetimes-2015-07-14-h.png In this graph, we see the total number of relays that were used as the introduction points of each hidden service over the whole measurement period. This data is interesting because introduction points can estimate the popularity of their hidden services, so a service should ideally not expose itself to too many of them. Looking at the last graph we see that the three normal hidden services have used approximately 250 distinct relays as intro points. In other words, about 250 relays had the chance to measure the popularity of those hidden services. This seems reasonable given the 90 days measurement period and if we assume an average lifetime of 20 hours per introduction point and about 3 intro points per HS, which gives us some confidence for the correctness of the whole system. In the meanwhile, hidden service (b) used about 2000 relays as IPs! This was again caused by the dynamic intro point formula which forced it to rotate introduction continuously. We have also heard rumors that (b) got attacked by a DoS in the beginning of our measurement period, which caused it to rotate introduction points even more. We also see that hidden service (e) starts using more introduction points from the 15th of May and onwards. This seems to be caused because that hidden service started using two hidden service instances for load balancing and each instance advertises a different introduction point set. With regards to load balancing, it's worth mentioning that we are currently developing a tool called 'onionbalance' which will become the better way of load-balancing hidden services. Donncha, the author, released an alpha version just a few weeks ago which is worth trying [3]. ---- And this sums up our short analysis for today. All in all, it seems that the system works properly most of the time. Descriptors get published in the intended frequency and introduction points get rotated as they should. The popularity leakage we found for popular hidden services was also interesting, and we are happy that it has since been fixed. We hope you had fun reading our analysis, and please let us know if there are any other privacy-preserving experiments you would like to see on the hidden services world. Footnotes: [0]: https://www.torproject.org/docs/hidden-services.html.en [1]: https://github.com/DonnchaC/tor-hs-descriptor-fetcher [2]: https://lists.torproject.org/pipermail/tor-dev/2015-April/008597.html [3]: https://github.com/DonnchaC/onionbalance/ https://lists.torproject.org/pipermail/tor-talk/2015-July/038312.html

1 0

Finding location metadata in large "dark market" datasets
by Griffin Boyce 18 Jul '15

18 Jul '15

Hello all, I came across a blog post that might interest you all. @techdad did a quick analysis of public images from online black markets (such as Silk Road et al)[2] from 2011-2015, and came to the following conclusion: "After parsing hundreds of thousands of images, I came across about 37 unique images that were not properly sanitized."[1] That's surprisingly low -- 0.00037% if one assumes 100k images analyzed. Given the number of high-profile cases [4] where this location information led to arrests, it's not very surprising that some people likely took the time to remove the EXIF data, but I'm curious whether a given website may have stripped the metadata for uploaded images. The images that tested positive are shown on the blog post, and 8/37 were clearly from the same individual. When mapped out, the location data is primarily in the US (5 locations), along with 1 location in France and Australia. Incidentally, the full 1.6TB dataset from 2011-2015 is available on the Internet Archive [3], just in case the Hacking Team disclosures haven't used up all your hard drive space. ;-) This data on its own is a rather interesting look into the workings of black markets -- many of which no longer exist. Curious to see what you all think and what analyses you'd like to see from this kind of data. best, Griffin [1] http://atechdad.com/Deanonymizing-Darknet-Data/ [2] http://www.gwern.net/Black-market%20archives [3] https://archive.org/details/dnmarchives [4] https://www.eff.org/deeplinks/2012/04/picture-worth-thousand-words-includin…

1 0

Linting the ' iff ' ism...
by grarpamp 16 Jul '15

16 Jul '15

There are about 350 places in tor and torspec that are afflicted by this ism, other locations surely exist... find . -type f -exec egrep -i ' ' {} \+

3 3

Fwd: Changes to ExoneraTor for this week's TWN
by Karsten Loesing 15 Jul '15

15 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi everyone, Jens asked me to forward this email here, so that other people can follow the recent changes we made to ExoneraTor. https://exonerator.torproject.org/ Harmony did a wonderful job summarizing these changes even more in this week's TWN issue. Thanks, harmony! Of course, if you're curious about the details, be sure to check out the commits and/or tickets: https://gitweb.torproject.org/exonerator.git/ https://trac.torproject.org/projects/tor/query?status=!closed&component=Exo… All the best, Karsten - -------- Forwarded Message -------- Subject: Changes to ExoneraTor for this week's TWN Date: Wed, 15 Jul 2015 12:51:17 +0200 From: Karsten Loesing <karsten(a)torproject.org> To: harmony <harmony01(a)riseup.net> CC: Tor Assistants <tor-assistants(a)lists.torproject.org> Hi harmony, here's a quick overview of improvements made to ExoneraTor in the past days in no specific order: - Restrict searches to dates: Rather than allowing searches by timestamp we restrict searches to full dates. Most people don't understand timezones nor that all our dates and times are in UTC. Letting them specify a precise time makes it more likely that they'll search for the wrong thing. The timezone problem doesn't go away by limiting searches to full days (if their incident happened on February 17 at 9pm EST, would they know that it was already February 18 in UTC?). But we can easily widen searches to +/- 1 day. The technical details still contain hours. - No raw descriptors anymore: We stopped processing server descriptors and linking raw descriptors in the latest version of ExoneraTor. The idea was to simplify the service and make it useful for the average user who doesn't care about this level of detail. And it's a good opportunity to prevent the database from growing even larger. - No target addresses and ports anymore: If there's a result to the question "Was there a Tor relay running on this IP address?", do not offer a subsequent form to answer the question "Was this relay configured to permit exiting to a given target?". The second search increases complexity of searches a bit, and it's unlikely that people would interpret results correctly. The technical details now contain an "Exit" column that indicates whether a relay permitted transit of Tor traffic to the open Internet at that time. - New non-technical explanations of Tor and ExoneraTor: We rewrote the text that explains what Tor is and what ExoneraTor does, and we tried to make it as non-technical and still accurate as possible. - Translations: There's now a translation to German, and we're currently making plans for adding more translations. The difficulty is that we'll want those translations to also be as non-technical and still accurate as possible. - New design: We're now using a basic Bootstrap design which is already much cleaner than the previous design, but which still leaves room for improvement. Hope I didn't miss anything important. Please feel free to pick whatever you find most important, and feel free to rephrase as needed. Sorry, gotta run! Thanks! All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJVplkAAAoJEJD5dJfVqbCrJDkH/3ie20NA7cCgLtwbpPwuJHdR EXgRQ/sBHzPHUNV+NmesG95jcKfUe/XcEDFYisa/ayrwWpTap+cOek7/02uM4RD/ J3Ik0xk7m0Cwo2nxPWhHYOtmF97Yv33zE26rbh6+Fix7b9O5xNgTLJ53k3rvhNKy xRKm5DM6blXYppJpeYdmTP0k1wJdiy/qsETScKWldBGlOtXdmNKwrncxZ+tg5Z2b 57VZvXSUKx0rRTCh2LuOYUcmyLuX9XqpJOKrdwQ+BcATETuaklWLEH55iSJN1d/h GNsVrhPgEz/VwKzJ472s+Xssrgmh7lk80sIoYxGn46J6VLg80a8GJVCo2qznNQ8= =+3gk -----END PGP SIGNATURE-----

1 0

packaging advice needed
by Magnus Hedemark 15 Jul '15

15 Jul '15

Hello, I have set up a Jenkins CI server and have it tracking the Tor project's git repository. I'm going to be building packages for the OmniOS platform, which is not yet supported. I tried searching the site for advice for packagers but haven't found any. If I've overlooked it, I'd appreciate a nudge in the right direction. More specifically, I'd like to better understand the release and patch workflow that is followed so that I can build the right commits at the right time. I'll have one job tracking the master branch for those who want to test potentially unstable software, but I have another job intended to promote "stable" packages. By the way, this is all a precursor to a set of articles showing how to create network-isolated hidden services inside of illumos zones. I think OmniOS has a lot of untapped potential to help hidden service operators. TIA, Magnus Sent from [ProtonMail](https://protonmail.ch), encrypted email based in Switzerland.

4 3

Re: [tor-dev] Tor Browser videos Automation
by Isabela 12 Jul '15

12 Jul '15

+ tor-dev Sorry for the noise, but thought of expanding the audience a little, maybe someone the time to take look into this, or might know what is going on. Any help is appreciated, we are trying to finish this project and this has been blocking us. Thanks! Sherief Alaa: > On Mon, May 18, 2015 at 3:49 PM, Mark Smith <mcs(a)pearlcrescent.com> wrote: > >> On 5/17/15 6:54 AM, Sherief Alaa wrote: >> >>> Hi Georg/Mark, >>> >>> I am working on the new Tor Browser videos this month (or at least a >>> process to produce them in an automated manner). I am contacting you >>> because I am running into a problem that you may know how to solve >>> because I think it's browser related. >>> >>> Karsten insisted that I have to run a local copy of torproject.org >>> <http://torproject.org> using a web server while the automated script >>> runs since we can't estimate or depend on the connection speed. The >>> major blocker in this is that the browser redirects to >>> https://torproject.org/ whenever I try to map 127.0.0.1 to >>> torproject.org <http://torproject.org> (or www.torproject.org >>> <http://www.torproject.org>) in my /etc/hosts file. Now I've tried >>> everything that will come up to mind that may solve the problem, such >>> as: flushing DNS cache, clearing history, disabling all addons and more >>> but nothing worked. >>> >>> I also ran Wireshark to inspect DNS packets but the browser doesn't even >>> try to query any DNS server. >>> >>> I've tested on OS X and Ubuntu and both produce the same results. I've >>> also tested multiple browsers (Safari, Chrome and Firefox). Same results. >>> >>> Trivial note: During testing, I could map google.com >>> <http://google.com>, ign.com <http://ign.com> and basically all .coms >>> but never for .orgs like eff.org <http://eff.org> or torproject.org >>> <http://torproject.org> >>> >>> If you have any idea what's going on, please let me know as soon as >>> possible as I am considering running a local nameserver but I am keeping >>> that as a last resort. >>> >> I do not know what is going on. Most likely it is a caching problem (both >> the OS and browser cache DNS info, as you know). But Kathy (CC'd on this >> reply) and I experimented a little on Mac OS 10.9.5 and found like you did >> that some hosts worked and some did not, even after doing dscacheutil >> -flushcache and using Firefox with a new profile. From the command line, >> ping always seems to respect /etc/hosts but other things such as curl do >> not. Which I guess means it is an OS caching issue, or some applications >> have their own DNS resolver that bypasses the OS libraries. >> >> -Mark >> > Ccing Isabela > > > -- > PM at TorProject.org > gpg fingerprint = 8F2A F9B6 D4A1 4D03 FDF1 B298 3224 4994 1506 4C7B > @isa

3 4

Two new tech reports on website fingerprinting
by t55wang＠cs.uwaterloo.ca 10 Jul '15

10 Jul '15

I'd like to introduce two new works on website fingerprinting I've written with my supervisor, Ian Goldberg. The first is titled ``On Realistically Attacking Tor with Website Fingerprinting''. We talk about methods to allow website fingerprinting to perform under realistically difficult scenarios. We have a tech report here: http://cacr.uwaterloo.ca/techreports/2015/cacr2015-09.pdf The second is titled ``Walkie-Talkie: An Effective and Efficient Defense against Website Fingerprinting''. We propose a new website fingerprinting defense that is practically efficient and provably effective. We have a tech report here: http://cacr.uwaterloo.ca/techreports/2015/cacr2015-08.pdf Our code for both of those works are here: https://crysp.uwaterloo.ca/software/webfingerprint/ In the code we have an implementation of a website fingerprinting attack that can take a long series of web page accesses (as one packet sequence) as input, and output if it has recognized any page in the sequence. We also have an implementation of half-duplex communication for Tor Firefox as the core of Walkie-Talkie. Tao Wang

3 2

Env variables for chutney
by Cory Pruce 10 Jul '15

10 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hey Tim, I see that I may need to set some environment variables as told by the git readme but the error message is on a different env var: cory@Nulix ~/tor/tor =) make test-network make all-am make[1]: Entering directory `/home/cory/tor/tor' make[1]: Leaving directory `/home/cory/tor/tor' ./src/test/test-network.sh test-network.sh: missing 'chutney' in CHUTNEY_PATH (~/tor/chutney/) make: *** [test-network] Error 1 I also tried modifying my .bashrc so that the CHUTNEY_PATH pointed to the executable but no luck. Do I need to set the chutney path? - - Cory -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJVluRoAAoJEB7DuCIJauCgUNUQAICDPM+vQ1YPU36B13beWgsX OUN6wQKdjtnRP9aCvusf49vomr+E8g2JcIvR+KiRKTDtnvVUMijsl1TZCmyTptEh 61opZJBweCP78MGuaeMXjJg3Ijw5vFDV6tszJ+Z/5xY/f2PinKO2Db+JWI1/oSP3 kMERE/Va+ehL8u4Nx/meP9VzVQDFZH1nz381FQFAPFYSv9+JR1x4HFGWLElOH0l8 1+hcGhhtRD5mzYoDLIx6Lr/3jIJxmbJiiOmpNQcfW3ISHTrgOB/Wo2n6oO/OcjEk L66YvFpZNYqWQ8ZXf+ZN3cC/+sTpWvYDOcdKv9LJvEmHNSevmU+i2+nMoeanpXzX z5eYNCE9unfitNg6hPoQ8x7wD+LLpBDEA42QoQVzx26WMCk3ziu9JIsI5/VfZ7Is /Ny+gQ4jv1Jlk+SEu6ZmRVuyN8sipXnkseXGcpqVR9HBWe7Hy+RfUpODhR3cOy7o EBClLrP8s4H0PUb2WYU4IQrIp2xy8cqqCIaDhs4XLhdXgaX2667irM9uEPw6ZMhL vceyU0LM0v5O+FfzFtdUC4QwfLAbzbiwRi+w74c6bneWWFTi/w5RRaNUiv/IZVMW XTh22mRIOkZXQ3dTc2TsI4TjhT2U95oMiO8hFjDBTS9RfZk/w+unToyJ2aed4UxK 9WC/r/vqUpgD4mljBr76 =9Cvw -----END PGP SIGNATURE-----

3 15

CollecTor data: mapping bridge-network-status to bridge-server-descriptor to bridge-extra-info
by David Fifield 09 Jul '15

09 Jul '15

I'm trying to use CollecTor data to find out how much bandwidth is offered by different pluggable transports over time. I.e., I want to be able to say something like, "On July 1, bridges with obfs3 offered X MB/s, bridges with obfs4 offered Y MB/s," etc. To do this, I'm mapping through three types of CollecTor documents: bridge-network-status (where the bandwidth is and which links to router digests) bridge-server-descriptor (which links to extra-info digests) bridge-extra-info (where the transports are) I'm having trouble because sometimes, a router digest listed in a bridge-network-status document is not found in the same tarball. https://collector.torproject.org/archive/bridge-descriptors/bridge-descript… Here is an example of what I'm doing, using the above tarball. bridge-descriptors-2015-07/statuses/04/20150704-000350-4A0CCD2DDC7995083D73F5D667100C8A5831F16D This is a bridge-network-status document. One of its entries is: r starman qgM+62FgGytzEtibYqqiPcPtijQ mdOOBxVOTpw8loBezhSDZxLIcXs 2015-07-03 21:39:31 10.174.163.60 9002 0 s Fast Guard Running Stable Valid w Bandwidth=2646 p reject 1-65535 The second base64-encoded string is the router digest. base64decode("mdOOBxVOTpw8loBezhSDZxLIcXs") = 99D38E07154E4E9C3C96805ECE14836712C8717B bridge-descriptors-2015-07/server-descriptors/9/9/99d38e07154e4e9c3c96805ece14836712c8717b Now we go looking for a bridge-server-descriptor with router digest 99D38E07154E4E9C3C96805ECE14836712C8717B, which is in the above file. It has a line: extra-info-digest D69106C8BAF5C0044F7331F24DF77E85BBF84027 bridge-descriptors-2015-07/extra-infos/d/6/d69106c8baf5c0044f7331f24df77e85bbf84027 Now we find a bridge-extra-info with digest D69106C8BAF5C0044F7331F24DF77E85BBF84027 in the above file. It tells us what transports the bridge supports (there are two, one for IPv4 and one for IPv6): transport meek transport meek Here's an example of where it goes wrong. bridge-descriptors-2015-07/statuses/01/20150701-060138-4A0CCD2DDC7995083D73F5D667100C8A5831F16D r Unnamed ABk0wg4j6BLCdZKleVtmNrfzJGI eGIOW1mGM/Dbw+t5bXnR8jdnsoY 2015-07-01 05:56:14 10.123.124.91 443 0 s Fast Running Stable Valid w Bandwidth=156 p reject 1-65535 We are looking for router digest 78620E5B598633F0DBC3EB796D79D1F23767B286: base64decode("eGIOW1mGM/Dbw+t5bXnR8jdnsoY") = 78620E5B598633F0DBC3EB796D79D1F23767B286 But there is no file bridge-descriptors-2015-07/server-descriptors/7/8/78620e5b598633f0dbc3eb796d79d1f23767b286. However, I did find it in the previous month's tarball, https://collector.torproject.org/archive/bridge-descriptors/bridge-descript… bridge-descriptors-2015-06/server-descriptors/8/3/835a43ff89db9c1be8ddf7536d759875878620e7 It seems rare that the bridge-server-descriptor is missing. In the 2015-07 tarball, it happened for 5891/477496 relays (1.2%). An additional 4/477496 (0.0%) had a bridge-server-descriptor but were missing bridge-extra-info. How do you handle cases like this? I had a browse through the Onionoo source code, but did not quickly understand it. Should I just always include the month preceding the earliest month I want to process?

3 4