- tor-dev - lists.torproject.org

3rd status report for OnioNS
by Jesse V 09 Jul '15

09 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello everyone, I have been making significant progress on the Onion Name System project, and it's really coming down the home stretch here. * I have been focusing on a four-party system: a hidden service operator, an authoritative server, a name server (mirror), and a client. Thus two servers are involved, so the responsibility is split. * The mirror can now subscribe to the authoritative server for events, thus it receives new Records (claims on domain names) and new signatures on the data structures. The design now prevents the mirror from lying to the client. * After some discussions with others at PETS, I have split my software into four separate repositories and four different packages: tor-onions-hs, tor-onions-client, tor-onions-server, and tor-onions-common. The first three now have a very focused responsibility and have dependencies on the -common package, a shared library that handles configuration parsing, talking to Tor, and other generic tasks. This also means that hidden service operators need only install the -hs package, which reduces the amount of code running on their protected server. * I have organized all the tickets on Github into their respective repository and assigned most of them to milestones to indicate how soon I will be tackling them. * I have also successfully integrated the software into the Tor Browser such that when the Tor Browser starts, Tor and the OnioNS software launch behind the scenes and in the correct order. My modification is a binary substitution rather than recompiling the Tor Launcher, but the result appears to be reliable and greatly simplifies the client end of things. * In the last status report, I discussed work on logging. I have made significant progress on this task, which is now a critical item because the software now runs in the background when launched from the Tor Browser, so normal stdout is not seen. The software logs to <Tor Browser dir>/Browser/TorBrowser/Data/OnioNS/ * I saw DonnchaC's alpha release. I am really working on making a prototype that everyone can beta test. The packaging and integration into the Tor Browser should make this very straightforward, but I need to have all the configurations in place and a few more bugs worked out before then. Jesse V. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBCAAGBQJVnnvTAAoJEK2XNk/CC+yATgQH/RSG1rNzUIF9pkyPQFl0YYHP fbor9zzdRXeHIADrnnn4trg4epGW43SCLJ2bpFiI7/keYTe6FFABE8j0W0mB8RbQ cIERbDmfGFBgrPrx0sPxMHj99637sF2j3igIxdln5rtqbvOuXLrgI3F+a6fWdCmr 6GFgTaurGexrCI5ybvqcaMz6p2/eRDuTl+k51gz6cWPzV7mzdCzzU9Gl1miqpS6M UB/RsjMwgGJDpcS/b0hlIXU6Zcsc1Kk3pP+czj99JsychHEUHZFSNB5qsVsDDCHq h/BS44O/v+x2aJg1ZwL6hn02ghn0HE9hapIglERQFLxZ/pQvTsVJzUL2gaL5L8I= =BC+c -----END PGP SIGNATURE-----

1 0

Summer of Privacy: Enhance GetTor
by ilv 08 Jul '15

08 Jul '15

Hello people, I just wanted to let you know that this summer I'll be working in improving the current GetTor. I've wrote about this before here [0] and I detailed my plan for the summer here [1]. Thanks to the nice will of the people running this program and my mentors, I'm starting with a different schedule ("Winter schedule" [2]), which started yesterday :) If you have comments or new ideas about GetTor, I'd love the hear it. Best regards, Israel Leiva. [0] https://lists.torproject.org/pipermail/tor-dev/2015-April/008718.html [1] https://people.torproject.org/~ilv/sop_proposal_2015.html [2] https://trac.torproject.org/projects/tor/wiki/doc/gsoc

1 0

Re: [tor-dev] onionoo: new field: measured flag (#16020)
by l.m 07 Jul '15

07 Jul '15

Hi nusenu, Since you posted to tor-dev I guess you're asking for community input too. About your use cases, Onionoo is for obtaining data about running relays, not tor network health, or BWAuth activity. You can answer this question by looking at the latest consensus data from CollecTor, counting relays, and unmeasured. So this use case is out of the scope of Onionoo use. Your second use case doesn't help the operator debug. What does not being measured mean in the context posed by the ticket? (whichever, boolean or int) a) the relay was not measured b) the relay was supposed to be measured but didn't respond c) the relay was supposed to be measured but the measurement was blocked due to network interference d) the relay was measured but the threshold number of measurements was not obtained ... and so on. Not being measured doesn't help with debugging. The only thing which matters is being `measured` in which case a relay has consensus weight. The only data which can be relied upon, and which is also a property of a running relay that has been measured is consensus weight. You can track this using the Weights document. I hope that helps --leeroy

1 0

onionoo: new field: measured flag (#16020)
by nusenu 07 Jul '15

07 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Karsten, could you give me a short opinion about #16020 https://trac.torproject.org/projects/tor/ticket/16020 Something like: - - no, never going to add measurement data to onionoo ever or - - yes, it makes sense and has a chance to be implemented+deployed within the next 12 months thanks! -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVmtOUAAoJEFv7XvVCELh0k3oQAJ6WXaQJmLGuMFEq2rV9Iz8D Rsv4HF+sDmaV6IiP6JilhuLQfvIuHKdW0Rq5qkD9P3Bg7RYg8Xo2gEWWBMv2H1ro /CzZ/XxAV4MzeExq2oYV/orjpEmLLKh8IB3CgJpRXr6aTuDg10g4WpONvzbO1D6x O6k2BJOe8h6t3t+8Eho5i4tyxJJzji/m5gwIO5ct+93gCuFmcr++tyum699KXRRT o1133JpKML+mwS5/uhzx54gMoHgzGgmo391gDYY/8yAf0WWeqeh0gmJqL/DTxa6o OLsstA4sEKuGX7TgkM326Rrp+4GPUSh548AMCovlJpvcIx63qzSN5iBPXmlHoOGo ZdwfLiTc07ELuKYRzXG+rrKvX12/ZlwK9AR/m6oe8BhN2wX5t/Ib0gAoiNXRzspi FHSdJfrS18FdxgiZI3QBlcrw0Tm4y4atwhsg4xMCb5HyUFTicnyqJflqlGrD7K5S KesB/299wMgzQ1kt8i46JmKqfvOM6+xHuV5aDSljFoG45A8vsIGFW0YX64RT6MTs MW/JaCVa2uaIJfR/bF1p83962uU8eXZIFUdaQpz3icshEE1QVsNRAcMp/wEA6RZ3 9URU6pS8O34QZY25nl0+glgYQzBKXzwK+megKi2rQetEHFaXOpxA9468NJGM7SdJ 1ZEejaB3ZkKuu4Fm7HlT =f7hD -----END PGP SIGNATURE-----

1 0

Summary of meek's costs, June 2015
by David Fifield 07 Jul '15

07 Jul '15

Here's the summary of meek's CDN fees for June 2015. App Engine + Amazon + Azure = total by month February 2014 $0.09 + -- + -- = $0.09 March 2014 $0.00 + -- + -- = $0.00 April 2014 $0.73 + -- + -- = $0.73 May 2014 $0.69 + -- + -- = $0.69 June 2014 $0.65 + -- + -- = $0.65 July 2014 $0.56 + $0.00 + -- = $0.56 August 2014 $1.56 + $3.10 + -- = $4.66 September 2014 $4.02 + $4.59 + $0.00 = $8.61 October 2014 $40.85 + $130.29 + $0.00 = $171.14 November 2014 $224.67 + $362.60 + $0.00 = $587.27 December 2014 $326.81 + $417.31 + $0.00 = $744.12 January 2015 $464.37 + $669.02 + $0.00 = $1133.39 February 2015 $650.53 + $604.83 + $0.00 = $1255.36 March 2015 $690.29 + $815.68 + $0.00 = $1505.97 April 2015 $886.43 + $785.37 + $0.00 = $1671.80 May 2015 $871.64 + $896.39 + $0.00 = $1768.03 June 2015 $601.83 + $820.00 + $0.00 = $1421.83 -- total by CDN $4765.72 + $5509.18 + $0.00 = $10274.90 grand total The rate limiting of meek-google and meek-amazon has been partially effective in bringing costs down. The bill this month is the lowest it has been since February. I aimed to get the meek-google bill down to $500, the amount of the monthly grant we're getting from Google, and it came in a little above that. meek-amazon, on the other hand, did not decrease much at all. I think it may be because of Amazon's per-request billing. The number of users was pretty stable, staying mostly in the range 3000–4500. https://metrics.torproject.org/userstats-bridge-transport.html?graph=userst… If you want to set up your own bridge and CDN instance without rate limiting, I can help you do it. Here are some docs to look at: https://trac.torproject.org/projects/tor/wiki/doc/meek#Howtorunameek-server… https://trac.torproject.org/projects/tor/wiki/doc/meek#GoogleAppEngine https://trac.torproject.org/projects/tor/wiki/doc/meek#AmazonCloudFront https://trac.torproject.org/projects/tor/wiki/doc/meek#MicrosoftAzure == App Engine a.k.a. meek-google == Rate limiting brought down bandwidth usage almost 50%. Despite that, instance hours went up. The rate limiting happens at the bridge, so it's possible that requests are taking longer to process at the App Engine level, requiring more instance hours. Here is how the Google costs broke down: 2807 GB $336.87 5299 instance hours $264.96 Compared to the previous month: 5359 GB $643.07 4571 instance hours $228.57 https://globe.torproject.org/#/bridge/88F745840F47CE0C6A4FE61D827950B06F9E4… == Amazon a.k.a. meek-amazon == Bandwidth use was down about 16% compared to the previous month. The number of requests stayed about the same. As you can see, the major contributor to the Amazon bill is not bandwidth but the number of requests. I hope this ticket will help reduce the number of requests we make: "Use streaming downloads" https://trac.torproject.org/projects/tor/ticket/12857 Asia Pacific (Singapore) 114M requests $136.84 831 GB $111.35 Asia Pacific (Sydney) 845K requests $1.06 2 GB $0.40 Asia Pacific (Tokyo) 57M requests $68.43 336 GB $43.11 EU (Ireland) 184M requests $221.41 1454 GB $108.52 South America (Sao Paulo) 8M requests $18.87 53 GB $12.54 US East (Northern Virginia) 64M requests $64.23 422 GB $33.25 -- total 429M requests $510.84 3101 GB $309.17 https://globe.torproject.org/#/bridge/3FD131B74D9A96190B1EE5D31E91757FADA1A… == Azure a.k.a. meek-azure == meek-azure bandwidth use continues to increase, up 17% compared to the previous month. Keep in mind that our grant expires in October, so you should not count it continuing to work after that. Speaking of the grant, they switched me over to a new kind of sponsored account that will enable me to see the exact bandwidth use and what its cost would be. That should be available starting next month. Until now, we have had to estimate bandwidth usage by measuring at the bridge. https://onionoo.torproject.org/bandwidth?fingerprint=AA033EEB61601B2B7312D8… Estimated bandwidth use this month: 2015-06 3038 GB Compared to last month: 2015-05 2592 GB https://globe.torproject.org/#/bridge/AA033EEB61601B2B7312D89B62AAA23DC3ED8… Earlier reports in this series: https://lists.torproject.org/pipermail/tor-dev/2014-August/007429.html https://lists.torproject.org/pipermail/tor-dev/2014-October/007576.html https://lists.torproject.org/pipermail/tor-dev/2014-November/007716.html https://lists.torproject.org/pipermail/tor-dev/2014-December/007916.html https://lists.torproject.org/pipermail/tor-dev/2015-January/008082.html https://lists.torproject.org/pipermail/tor-dev/2015-February/008235.html https://lists.torproject.org/pipermail/tor-dev/2015-March/008427.html https://lists.torproject.org/pipermail/tor-dev/2015-April/008596.html https://lists.torproject.org/pipermail/tor-dev/2015-May/008767.html https://lists.torproject.org/pipermail/tor-dev/2015-June/008932.html

1 0

Chutney (was Re: Performance testing using chutney)
by Cory Pruce 07 Jul '15

07 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > Well, you could test my latest branches for #14175: > > https://trac.torproject.org/projects/tor/ticket/14175#comment:8 > > There's a branch which modifies src/test/test-network.sh in tor, > and another with the performance measurement code in chutney. > > The command-line arguments/environmental variables are in the ticket, > and I've modified the chutney README to include performance testing. Cool! I'll check it out tonight and keep you posted :-) > > Maybe look at these for a start. It's something that's on my TODO > list, but I wouldn't complain if someone else happens to do it before I > do, and it would help HS scalability considerably[0]. > > https://trac.torproject.org/projects/tor/ticket/13737 > https://trac.torproject.org/projects/tor/ticket/13738 > > (If you happen to be more interested in making non-HS use cases faster, > then look elsewhere. :P) > > Regards, > Haha I'm interested in helping the performance/scalability of both HS and non-HS. I read the descriptions but they are pretty short and high level. Could you give any pointers to start? Does worker imply changes to cpuworker.*? - - Cory -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJVmqHwAAoJEB7DuCIJauCgdtMP/02Ztn3IoikLU1s+MdiJRyNY /X9pTU667Lu6VAd2yhbMpYSQlgoPkR6bFElGhOWucihFN84zTxSBbNxaPwW36mAE rBNHgw41cGEBSva6chHNLJJGx/wpThbLDGYTqPvoQrDGWqgKNB1hK4IRRRFaRiC0 H+nBKR0xiH/sBY4PzIIfiX3BBVrA0ogV+kOzEIwDSjmnw4x24OMjD/Fx2VOK2tai dwpHw0QTW6Ao+WEVtXWPvzhKZTnvKcPKDkvLwo7fzdSmVTS9IUqQJCh8KdYUnt/5 uO4HR0WEdPOT7anA1TvrKzhRTe/8KHzTcPrCHJo7Dbt/rvc7RW0v4GSClTEtBlIU EcHRCvmG/SaCaw+H2IcCK9k5T/rt7azsADcKIhqDWMvttJkVmsHo9FbVldpzNB/z hwHfVYKsuOPWanpPrucn4opf5Td8LcQ+v6IAtwqQbslpe6myf6aX2GrulLZLi74R PT/lo0wqGh4V59HeAS5rG6mUVeI7vADmqlrzzFnsDxNMYP2OLsLquKl2noAFSbFk kD2xcQmy3kygS6SiLrDX95c1CefB0W56CcJBdkHrvyuhSltFngpXpcDyVSvyZnN8 giXWxWuZC9unIAvgWvlKnEnUQgGxVWT3ItCzxohXUFM0qturow9ii3D9iWsO4zkL K5EB0TLP8Fc3OstFcpt1 =tGow -----END PGP SIGNATURE-----

1 0

Re: [tor-dev] Roster introduction (Philipp Winter) (Sean Saito)
by saitosean＠ymail.com 06 Jul '15

06 Jul '15

Hi Philipp, > - I searched for my relay family by fingerprint and the resulting page's > URL didn't contain my family, so I couldn't send the URL to somebody > else. Could you give me some more details about this? Currently the search bar supports all running relays. The URL will contain the fingerprint you put into the search bar. > - The colour coding for relay flags is hard to read because of the low > contrast. Perhaps use green for assigned flags and grey for missing > flags? > - I would put the exit policy in a fixed-size box. Otherwise, there's a > lot of scrolling for large exit policies. Certainly possible. Thanks a lot for the suggestions. Sean

2 1

Hi, all!
by Nick Mathewson 06 Jul '15

06 Jul '15

I'm about to go on vacation, but before I started, Isabela asked me for coverage information on Tor, to help people decide where more tests should be written. I've re-run coverage, and uploaded the gcov output here: https://people.torproject.org/~nickm/volatile/coverage-65a1e27.tar.bz2 That archive includes coverage on current master from * make check * make test-network (uses chutney) * make test-stem (uses stem) * all of the above, combined I'm also attaching a script that I wrote long long ago, and use to summarize coverage file-by-file. Example use: gcov-lines.py ./coverage-65a1e27/cov-all/* yrs, -- Nick

2 1

Re: [tor-dev] Roster introduction (nusenu) (Sean Saito)
by saitosean＠ymail.com 05 Jul '15

05 Jul '15

Hi nusenu, > A few comments: > > - - the page seems to be AS name centric > - first column of the table is AS name > - only clickable column in the table is AS name > > - - the AS column suggests that all relays in a family run in one AS only > > - - the family page of a specific family shows the AS name as title, > this is a bit confusing as one might think that the page shows all > families/relays in the selected AS after clicking an as name on the > start page Agreed, this was just a placeholder. Will put contact info instead for now. In the future I would like families to have ids/nicknames as well. > Family identifier > > - - by looking at your page I assume your family identifier is the > fingerprint of the oldest relay in the family, does that mean that an > operator has to start over at 0 with collecting "points" once he kills > his oldest relay? That is not the case. Thanks for the comments! Sean

1 0

Bi-directional families in Onionoo and consensus weight as measure of relayed bandwidth
by Karsten Loesing 03 Jul '15

03 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Moving this discussion here from another list with Virgil's permission. On 02/07/15 08:42, Virgil Griffith wrote: > Big issues right now are: * Bugs (?) in Onionoo --- Onionoo doesn't > sanitize its data. For example, there's a lack of bidirectionality > between relays of many families. > https://drive.google.com/file/d/0Bwjagz1RgJOnSkx0YTlhMHdfMFU/view?usp=shari… > > There are currently about 665 pairs of family relays without > bidirectionality. This is caused by the .torrc of some relays not > pointing to its family members. > > I am considering doing a service on top of Onionoo that sanitizes > the raw Tor consensus to ensure things like bidirectional families. > It's unclear how much other data needs sanitization. I'd rather want to fix/change Onionoo than have you write another service that processes Tor descriptors. There's even a ticket for this, we're just somewhat stuck by arguing about the best fix. Maybe I should just fix it somehow and, if necessary, fix it more later. https://trac.torproject.org/projects/tor/ticket/16276 Would that solve your problem? What other problems would there be with Onionoo's data? Can you make a wish list? > * A semi-reliable measure for the magnitude of traffic a relay has > routed. We have confirmed instances of relays forging their > observed bandwidth, ergo we can't use that. And thus far > Consensus Weight is the best we've found, but it's unclear whether > we can use that as a proxy of magnitude of relayed traffic. --- > https://docs.google.com/document/d/1v1rutylD6RkBei9rEmSvsgmvQXhrIHXOr85NL3I… > > Right now the lack of a reliable measure of how much bandwidth is > relayed is the largest sticking point. Actually, consensus weight (fraction) is a fine measure, and I like how you're calling it "bandwidth points" in your prototype which doesn't imply a bits per second or related unit. I'd say assign 10,000 bandwidth points to all relays per day, depending on what fraction of total consensus weight a relay had. To me, it's fine that this doesn't translate to bits or bytes. How does that sound? All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJVlPKJAAoJEJD5dJfVqbCrZWQH/0lHSdgy4PF7nQ8RMZryKpnf o3Fvw8VkcIwZgJgp0MOLIVu0fZhcD8hhvSWd9yYTSpQwGwBayUJuPE0ao4MbfZYf mwz5hkngzq1Z7654K65m/fYLu7EIbXI86vT4/Cwwh8cnGl/ezaliFVvVMOmKTyOb UtV7T+Lgk5IgsGJOxQbpNHCTxyAokbAygqZ9Eq/6ZWqjZFBZb1P2XjV+IaziGyJl yuxrD66cJe4ZmcpPe9g7mTa9JyQ5kmUOWogXhKTFWDFCcPslc0M49iiYohDmiNxC 5RGKp1dMuYkL6th9b3Uuc3W4TdCMaDHV96BDUD3qdlqCWBU0fD617f31+Hsb6Bg= =0KdX -----END PGP SIGNATURE-----

4 7