- tor-dev - lists.torproject.org

CollecTor data: mapping bridge-network-status to bridge-server-descriptor to bridge-extra-info
by David Fifield 09 Jul '15

09 Jul '15

I'm trying to use CollecTor data to find out how much bandwidth is offered by different pluggable transports over time. I.e., I want to be able to say something like, "On July 1, bridges with obfs3 offered X MB/s, bridges with obfs4 offered Y MB/s," etc. To do this, I'm mapping through three types of CollecTor documents: bridge-network-status (where the bandwidth is and which links to router digests) bridge-server-descriptor (which links to extra-info digests) bridge-extra-info (where the transports are) I'm having trouble because sometimes, a router digest listed in a bridge-network-status document is not found in the same tarball. https://collector.torproject.org/archive/bridge-descriptors/bridge-descript… Here is an example of what I'm doing, using the above tarball. bridge-descriptors-2015-07/statuses/04/20150704-000350-4A0CCD2DDC7995083D73F5D667100C8A5831F16D This is a bridge-network-status document. One of its entries is: r starman qgM+62FgGytzEtibYqqiPcPtijQ mdOOBxVOTpw8loBezhSDZxLIcXs 2015-07-03 21:39:31 10.174.163.60 9002 0 s Fast Guard Running Stable Valid w Bandwidth=2646 p reject 1-65535 The second base64-encoded string is the router digest. base64decode("mdOOBxVOTpw8loBezhSDZxLIcXs") = 99D38E07154E4E9C3C96805ECE14836712C8717B bridge-descriptors-2015-07/server-descriptors/9/9/99d38e07154e4e9c3c96805ece14836712c8717b Now we go looking for a bridge-server-descriptor with router digest 99D38E07154E4E9C3C96805ECE14836712C8717B, which is in the above file. It has a line: extra-info-digest D69106C8BAF5C0044F7331F24DF77E85BBF84027 bridge-descriptors-2015-07/extra-infos/d/6/d69106c8baf5c0044f7331f24df77e85bbf84027 Now we find a bridge-extra-info with digest D69106C8BAF5C0044F7331F24DF77E85BBF84027 in the above file. It tells us what transports the bridge supports (there are two, one for IPv4 and one for IPv6): transport meek transport meek Here's an example of where it goes wrong. bridge-descriptors-2015-07/statuses/01/20150701-060138-4A0CCD2DDC7995083D73F5D667100C8A5831F16D r Unnamed ABk0wg4j6BLCdZKleVtmNrfzJGI eGIOW1mGM/Dbw+t5bXnR8jdnsoY 2015-07-01 05:56:14 10.123.124.91 443 0 s Fast Running Stable Valid w Bandwidth=156 p reject 1-65535 We are looking for router digest 78620E5B598633F0DBC3EB796D79D1F23767B286: base64decode("eGIOW1mGM/Dbw+t5bXnR8jdnsoY") = 78620E5B598633F0DBC3EB796D79D1F23767B286 But there is no file bridge-descriptors-2015-07/server-descriptors/7/8/78620e5b598633f0dbc3eb796d79d1f23767b286. However, I did find it in the previous month's tarball, https://collector.torproject.org/archive/bridge-descriptors/bridge-descript… bridge-descriptors-2015-06/server-descriptors/8/3/835a43ff89db9c1be8ddf7536d759875878620e7 It seems rare that the bridge-server-descriptor is missing. In the 2015-07 tarball, it happened for 5891/477496 relays (1.2%). An additional 4/477496 (0.0%) had a bridge-server-descriptor but were missing bridge-extra-info. How do you handle cases like this? I had a browse through the Onionoo source code, but did not quickly understand it. Should I just always include the month preceding the earliest month I want to process?

3 4

Update: Proposal 237 - All relays are directory servers
by Matthew Finkel 09 Jul '15

09 Jul '15

Hi All, Here's an updated version of prop 237. I rewrote some parts for clarity and brought it inline with the current implementation for #12538. Updated branch on prop237_update in my torspec repo. Thanks! Matt

2 1

tor#16518: Read-Only Filesystem Error opening tor lockfile in 0.2.6.9 but not 0.2.5.12
by aexlfowley＠web.de 09 Jul '15

09 Jul '15

Hi devs, User here. teor send me over from IRC. It's regarding to this trac-ticket https://trac.torproject.org/projects/tor/ticket/16518 After upgrading from 0.2.5.12 (git-3731dd5c3071dcba) to 0.2.6.9 (git-145b2587d1269af4) an error occured. I'm on Debian Jessie (stable) on an AMD Athlon 64 X2. Tor won't start and these are the last lines in log: [warn] Couldn't open "/media/cRAID/Tor/lock" for locking: Read-only file system [err] set_options(): Bug: Acting on config options left us in a broken state. Dying. Here's the full log: http://www.file-upload.net/download-10748122/aexllog.gz.html Here's also a pastebin of some shell commands I did for the IRC chat: http://lpaste.net/136099 debug just also contains: [debug] tor_disable_debugger_attach(): Attemping to disable debugger attachment to Tor for unprivileged users. [debug] tor_disable_debugger_attach(): Debugger attachment disabled for unprivileged users. [info] tor_lockfile_lock(): Locking "/media/cRAID/Tor/lock" teor thinks that I "could be experiencing an issue with the tor sandbox and not getting the right paths or, tor is running with insufficient permissions" teor's help and mindfulness was much appreciated. It lasted about 7 days for me to realize that Tor wasn't running. I'm now operating 0.2.5.12 successfully, hoping that it's still secure. Kindest regards

7 12

Today's openssl vulnerability; preliminary analysis wrt Tor
by Nick Mathewson 09 Jul '15

09 Jul '15

tl;dr: CVE-2015-1793 does not appear to affect Tor. Update your OpenSSL anyway; other applications are certainly affected. Hi, all! Here's the announcement for today's major security issue in OpenSSL1.0.2c, 1.0.2b, 1.0.1n and 1.0.1o: https://www.openssl.org/news/secadv_20150709.txt So far, I have only looked at the announcement itself, and the commit messages--not yet the code. But from what I can tell, it should only affect programs that have trusted certificates in their store. Tor itself does not use trusted root certificates, so it is not affected. (Similarly, TorBrowser should not be affected: it uses NSS, not OpenSSL.) Still, you likely have lots of other programs that depend on OpenSSL and trusted certificates to build certificate chains, and those programs _will_ be affected. So, you should probably upgrade OpenSSL as soon as feasible. (I'll spend a little more time patches and reviewing Tor's code to confirm my analysis above, and I invite others to do so as well. I'm in recovery from my vacation today.) best wishes, -- Nick

1 0

3rd status report for OnioNS
by Jesse V 09 Jul '15

09 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello everyone, I have been making significant progress on the Onion Name System project, and it's really coming down the home stretch here. * I have been focusing on a four-party system: a hidden service operator, an authoritative server, a name server (mirror), and a client. Thus two servers are involved, so the responsibility is split. * The mirror can now subscribe to the authoritative server for events, thus it receives new Records (claims on domain names) and new signatures on the data structures. The design now prevents the mirror from lying to the client. * After some discussions with others at PETS, I have split my software into four separate repositories and four different packages: tor-onions-hs, tor-onions-client, tor-onions-server, and tor-onions-common. The first three now have a very focused responsibility and have dependencies on the -common package, a shared library that handles configuration parsing, talking to Tor, and other generic tasks. This also means that hidden service operators need only install the -hs package, which reduces the amount of code running on their protected server. * I have organized all the tickets on Github into their respective repository and assigned most of them to milestones to indicate how soon I will be tackling them. * I have also successfully integrated the software into the Tor Browser such that when the Tor Browser starts, Tor and the OnioNS software launch behind the scenes and in the correct order. My modification is a binary substitution rather than recompiling the Tor Launcher, but the result appears to be reliable and greatly simplifies the client end of things. * In the last status report, I discussed work on logging. I have made significant progress on this task, which is now a critical item because the software now runs in the background when launched from the Tor Browser, so normal stdout is not seen. The software logs to <Tor Browser dir>/Browser/TorBrowser/Data/OnioNS/ * I saw DonnchaC's alpha release. I am really working on making a prototype that everyone can beta test. The packaging and integration into the Tor Browser should make this very straightforward, but I need to have all the configurations in place and a few more bugs worked out before then. Jesse V. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBCAAGBQJVnnvTAAoJEK2XNk/CC+yATgQH/RSG1rNzUIF9pkyPQFl0YYHP fbor9zzdRXeHIADrnnn4trg4epGW43SCLJ2bpFiI7/keYTe6FFABE8j0W0mB8RbQ cIERbDmfGFBgrPrx0sPxMHj99637sF2j3igIxdln5rtqbvOuXLrgI3F+a6fWdCmr 6GFgTaurGexrCI5ybvqcaMz6p2/eRDuTl+k51gz6cWPzV7mzdCzzU9Gl1miqpS6M UB/RsjMwgGJDpcS/b0hlIXU6Zcsc1Kk3pP+czj99JsychHEUHZFSNB5qsVsDDCHq h/BS44O/v+x2aJg1ZwL6hn02ghn0HE9hapIglERQFLxZ/pQvTsVJzUL2gaL5L8I= =BC+c -----END PGP SIGNATURE-----

1 0

Summer of Privacy: Enhance GetTor
by ilv 07 Jul '15

07 Jul '15

Hello people, I just wanted to let you know that this summer I'll be working in improving the current GetTor. I've wrote about this before here [0] and I detailed my plan for the summer here [1]. Thanks to the nice will of the people running this program and my mentors, I'm starting with a different schedule ("Winter schedule" [2]), which started yesterday :) If you have comments or new ideas about GetTor, I'd love the hear it. Best regards, Israel Leiva. [0] https://lists.torproject.org/pipermail/tor-dev/2015-April/008718.html [1] https://people.torproject.org/~ilv/sop_proposal_2015.html [2] https://trac.torproject.org/projects/tor/wiki/doc/gsoc

1 0

Re: [tor-dev] onionoo: new field: measured flag (#16020)
by l.m 07 Jul '15

07 Jul '15

Hi nusenu, Since you posted to tor-dev I guess you're asking for community input too. About your use cases, Onionoo is for obtaining data about running relays, not tor network health, or BWAuth activity. You can answer this question by looking at the latest consensus data from CollecTor, counting relays, and unmeasured. So this use case is out of the scope of Onionoo use. Your second use case doesn't help the operator debug. What does not being measured mean in the context posed by the ticket? (whichever, boolean or int) a) the relay was not measured b) the relay was supposed to be measured but didn't respond c) the relay was supposed to be measured but the measurement was blocked due to network interference d) the relay was measured but the threshold number of measurements was not obtained ... and so on. Not being measured doesn't help with debugging. The only thing which matters is being `measured` in which case a relay has consensus weight. The only data which can be relied upon, and which is also a property of a running relay that has been measured is consensus weight. You can track this using the Weights document. I hope that helps --leeroy

1 0

onionoo: new field: measured flag (#16020)
by nusenu 06 Jul '15

06 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Karsten, could you give me a short opinion about #16020 https://trac.torproject.org/projects/tor/ticket/16020 Something like: - - no, never going to add measurement data to onionoo ever or - - yes, it makes sense and has a chance to be implemented+deployed within the next 12 months thanks! -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVmtOUAAoJEFv7XvVCELh0k3oQAJ6WXaQJmLGuMFEq2rV9Iz8D Rsv4HF+sDmaV6IiP6JilhuLQfvIuHKdW0Rq5qkD9P3Bg7RYg8Xo2gEWWBMv2H1ro /CzZ/XxAV4MzeExq2oYV/orjpEmLLKh8IB3CgJpRXr6aTuDg10g4WpONvzbO1D6x O6k2BJOe8h6t3t+8Eho5i4tyxJJzji/m5gwIO5ct+93gCuFmcr++tyum699KXRRT o1133JpKML+mwS5/uhzx54gMoHgzGgmo391gDYY/8yAf0WWeqeh0gmJqL/DTxa6o OLsstA4sEKuGX7TgkM326Rrp+4GPUSh548AMCovlJpvcIx63qzSN5iBPXmlHoOGo ZdwfLiTc07ELuKYRzXG+rrKvX12/ZlwK9AR/m6oe8BhN2wX5t/Ib0gAoiNXRzspi FHSdJfrS18FdxgiZI3QBlcrw0Tm4y4atwhsg4xMCb5HyUFTicnyqJflqlGrD7K5S KesB/299wMgzQ1kt8i46JmKqfvOM6+xHuV5aDSljFoG45A8vsIGFW0YX64RT6MTs MW/JaCVa2uaIJfR/bF1p83962uU8eXZIFUdaQpz3icshEE1QVsNRAcMp/wEA6RZ3 9URU6pS8O34QZY25nl0+glgYQzBKXzwK+megKi2rQetEHFaXOpxA9468NJGM7SdJ 1ZEejaB3ZkKuu4Fm7HlT =f7hD -----END PGP SIGNATURE-----

1 0

Summary of meek's costs, June 2015
by David Fifield 06 Jul '15

06 Jul '15

Here's the summary of meek's CDN fees for June 2015. App Engine + Amazon + Azure = total by month February 2014 $0.09 + -- + -- = $0.09 March 2014 $0.00 + -- + -- = $0.00 April 2014 $0.73 + -- + -- = $0.73 May 2014 $0.69 + -- + -- = $0.69 June 2014 $0.65 + -- + -- = $0.65 July 2014 $0.56 + $0.00 + -- = $0.56 August 2014 $1.56 + $3.10 + -- = $4.66 September 2014 $4.02 + $4.59 + $0.00 = $8.61 October 2014 $40.85 + $130.29 + $0.00 = $171.14 November 2014 $224.67 + $362.60 + $0.00 = $587.27 December 2014 $326.81 + $417.31 + $0.00 = $744.12 January 2015 $464.37 + $669.02 + $0.00 = $1133.39 February 2015 $650.53 + $604.83 + $0.00 = $1255.36 March 2015 $690.29 + $815.68 + $0.00 = $1505.97 April 2015 $886.43 + $785.37 + $0.00 = $1671.80 May 2015 $871.64 + $896.39 + $0.00 = $1768.03 June 2015 $601.83 + $820.00 + $0.00 = $1421.83 -- total by CDN $4765.72 + $5509.18 + $0.00 = $10274.90 grand total The rate limiting of meek-google and meek-amazon has been partially effective in bringing costs down. The bill this month is the lowest it has been since February. I aimed to get the meek-google bill down to $500, the amount of the monthly grant we're getting from Google, and it came in a little above that. meek-amazon, on the other hand, did not decrease much at all. I think it may be because of Amazon's per-request billing. The number of users was pretty stable, staying mostly in the range 3000–4500. https://metrics.torproject.org/userstats-bridge-transport.html?graph=userst… If you want to set up your own bridge and CDN instance without rate limiting, I can help you do it. Here are some docs to look at: https://trac.torproject.org/projects/tor/wiki/doc/meek#Howtorunameek-server… https://trac.torproject.org/projects/tor/wiki/doc/meek#GoogleAppEngine https://trac.torproject.org/projects/tor/wiki/doc/meek#AmazonCloudFront https://trac.torproject.org/projects/tor/wiki/doc/meek#MicrosoftAzure == App Engine a.k.a. meek-google == Rate limiting brought down bandwidth usage almost 50%. Despite that, instance hours went up. The rate limiting happens at the bridge, so it's possible that requests are taking longer to process at the App Engine level, requiring more instance hours. Here is how the Google costs broke down: 2807 GB $336.87 5299 instance hours $264.96 Compared to the previous month: 5359 GB $643.07 4571 instance hours $228.57 https://globe.torproject.org/#/bridge/88F745840F47CE0C6A4FE61D827950B06F9E4… == Amazon a.k.a. meek-amazon == Bandwidth use was down about 16% compared to the previous month. The number of requests stayed about the same. As you can see, the major contributor to the Amazon bill is not bandwidth but the number of requests. I hope this ticket will help reduce the number of requests we make: "Use streaming downloads" https://trac.torproject.org/projects/tor/ticket/12857 Asia Pacific (Singapore) 114M requests $136.84 831 GB $111.35 Asia Pacific (Sydney) 845K requests $1.06 2 GB $0.40 Asia Pacific (Tokyo) 57M requests $68.43 336 GB $43.11 EU (Ireland) 184M requests $221.41 1454 GB $108.52 South America (Sao Paulo) 8M requests $18.87 53 GB $12.54 US East (Northern Virginia) 64M requests $64.23 422 GB $33.25 -- total 429M requests $510.84 3101 GB $309.17 https://globe.torproject.org/#/bridge/3FD131B74D9A96190B1EE5D31E91757FADA1A… == Azure a.k.a. meek-azure == meek-azure bandwidth use continues to increase, up 17% compared to the previous month. Keep in mind that our grant expires in October, so you should not count it continuing to work after that. Speaking of the grant, they switched me over to a new kind of sponsored account that will enable me to see the exact bandwidth use and what its cost would be. That should be available starting next month. Until now, we have had to estimate bandwidth usage by measuring at the bridge. https://onionoo.torproject.org/bandwidth?fingerprint=AA033EEB61601B2B7312D8… Estimated bandwidth use this month: 2015-06 3038 GB Compared to last month: 2015-05 2592 GB https://globe.torproject.org/#/bridge/AA033EEB61601B2B7312D89B62AAA23DC3ED8… Earlier reports in this series: https://lists.torproject.org/pipermail/tor-dev/2014-August/007429.html https://lists.torproject.org/pipermail/tor-dev/2014-October/007576.html https://lists.torproject.org/pipermail/tor-dev/2014-November/007716.html https://lists.torproject.org/pipermail/tor-dev/2014-December/007916.html https://lists.torproject.org/pipermail/tor-dev/2015-January/008082.html https://lists.torproject.org/pipermail/tor-dev/2015-February/008235.html https://lists.torproject.org/pipermail/tor-dev/2015-March/008427.html https://lists.torproject.org/pipermail/tor-dev/2015-April/008596.html https://lists.torproject.org/pipermail/tor-dev/2015-May/008767.html https://lists.torproject.org/pipermail/tor-dev/2015-June/008932.html

1 0

Chutney (was Re: Performance testing using chutney)
by Cory Pruce 06 Jul '15

06 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > Well, you could test my latest branches for #14175: > > https://trac.torproject.org/projects/tor/ticket/14175#comment:8 > > There's a branch which modifies src/test/test-network.sh in tor, > and another with the performance measurement code in chutney. > > The command-line arguments/environmental variables are in the ticket, > and I've modified the chutney README to include performance testing. Cool! I'll check it out tonight and keep you posted :-) > > Maybe look at these for a start. It's something that's on my TODO > list, but I wouldn't complain if someone else happens to do it before I > do, and it would help HS scalability considerably[0]. > > https://trac.torproject.org/projects/tor/ticket/13737 > https://trac.torproject.org/projects/tor/ticket/13738 > > (If you happen to be more interested in making non-HS use cases faster, > then look elsewhere. :P) > > Regards, > Haha I'm interested in helping the performance/scalability of both HS and non-HS. I read the descriptions but they are pretty short and high level. Could you give any pointers to start? Does worker imply changes to cpuworker.*? - - Cory -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJVmqHwAAoJEB7DuCIJauCgdtMP/02Ztn3IoikLU1s+MdiJRyNY /X9pTU667Lu6VAd2yhbMpYSQlgoPkR6bFElGhOWucihFN84zTxSBbNxaPwW36mAE rBNHgw41cGEBSva6chHNLJJGx/wpThbLDGYTqPvoQrDGWqgKNB1hK4IRRRFaRiC0 H+nBKR0xiH/sBY4PzIIfiX3BBVrA0ogV+kOzEIwDSjmnw4x24OMjD/Fx2VOK2tai dwpHw0QTW6Ao+WEVtXWPvzhKZTnvKcPKDkvLwo7fzdSmVTS9IUqQJCh8KdYUnt/5 uO4HR0WEdPOT7anA1TvrKzhRTe/8KHzTcPrCHJo7Dbt/rvc7RW0v4GSClTEtBlIU EcHRCvmG/SaCaw+H2IcCK9k5T/rt7azsADcKIhqDWMvttJkVmsHo9FbVldpzNB/z hwHfVYKsuOPWanpPrucn4opf5Td8LcQ+v6IAtwqQbslpe6myf6aX2GrulLZLi74R PT/lo0wqGh4V59HeAS5rG6mUVeI7vADmqlrzzFnsDxNMYP2OLsLquKl2noAFSbFk kD2xcQmy3kygS6SiLrDX95c1CefB0W56CcJBdkHrvyuhSltFngpXpcDyVSvyZnN8 giXWxWuZC9unIAvgWvlKnEnUQgGxVWT3ItCzxohXUFM0qturow9ii3D9iWsO4zkL K5EB0TLP8Fc3OstFcpt1 =tGow -----END PGP SIGNATURE-----

1 0