- tor-dev - lists.torproject.org

Summary of meek's costs, June 2015
by David Fifield 06 Jul '15

06 Jul '15

Here's the summary of meek's CDN fees for June 2015. App Engine + Amazon + Azure = total by month February 2014 $0.09 + -- + -- = $0.09 March 2014 $0.00 + -- + -- = $0.00 April 2014 $0.73 + -- + -- = $0.73 May 2014 $0.69 + -- + -- = $0.69 June 2014 $0.65 + -- + -- = $0.65 July 2014 $0.56 + $0.00 + -- = $0.56 August 2014 $1.56 + $3.10 + -- = $4.66 September 2014 $4.02 + $4.59 + $0.00 = $8.61 October 2014 $40.85 + $130.29 + $0.00 = $171.14 November 2014 $224.67 + $362.60 + $0.00 = $587.27 December 2014 $326.81 + $417.31 + $0.00 = $744.12 January 2015 $464.37 + $669.02 + $0.00 = $1133.39 February 2015 $650.53 + $604.83 + $0.00 = $1255.36 March 2015 $690.29 + $815.68 + $0.00 = $1505.97 April 2015 $886.43 + $785.37 + $0.00 = $1671.80 May 2015 $871.64 + $896.39 + $0.00 = $1768.03 June 2015 $601.83 + $820.00 + $0.00 = $1421.83 -- total by CDN $4765.72 + $5509.18 + $0.00 = $10274.90 grand total The rate limiting of meek-google and meek-amazon has been partially effective in bringing costs down. The bill this month is the lowest it has been since February. I aimed to get the meek-google bill down to $500, the amount of the monthly grant we're getting from Google, and it came in a little above that. meek-amazon, on the other hand, did not decrease much at all. I think it may be because of Amazon's per-request billing. The number of users was pretty stable, staying mostly in the range 3000–4500. https://metrics.torproject.org/userstats-bridge-transport.html?graph=userst… If you want to set up your own bridge and CDN instance without rate limiting, I can help you do it. Here are some docs to look at: https://trac.torproject.org/projects/tor/wiki/doc/meek#Howtorunameek-server… https://trac.torproject.org/projects/tor/wiki/doc/meek#GoogleAppEngine https://trac.torproject.org/projects/tor/wiki/doc/meek#AmazonCloudFront https://trac.torproject.org/projects/tor/wiki/doc/meek#MicrosoftAzure == App Engine a.k.a. meek-google == Rate limiting brought down bandwidth usage almost 50%. Despite that, instance hours went up. The rate limiting happens at the bridge, so it's possible that requests are taking longer to process at the App Engine level, requiring more instance hours. Here is how the Google costs broke down: 2807 GB $336.87 5299 instance hours $264.96 Compared to the previous month: 5359 GB $643.07 4571 instance hours $228.57 https://globe.torproject.org/#/bridge/88F745840F47CE0C6A4FE61D827950B06F9E4… == Amazon a.k.a. meek-amazon == Bandwidth use was down about 16% compared to the previous month. The number of requests stayed about the same. As you can see, the major contributor to the Amazon bill is not bandwidth but the number of requests. I hope this ticket will help reduce the number of requests we make: "Use streaming downloads" https://trac.torproject.org/projects/tor/ticket/12857 Asia Pacific (Singapore) 114M requests $136.84 831 GB $111.35 Asia Pacific (Sydney) 845K requests $1.06 2 GB $0.40 Asia Pacific (Tokyo) 57M requests $68.43 336 GB $43.11 EU (Ireland) 184M requests $221.41 1454 GB $108.52 South America (Sao Paulo) 8M requests $18.87 53 GB $12.54 US East (Northern Virginia) 64M requests $64.23 422 GB $33.25 -- total 429M requests $510.84 3101 GB $309.17 https://globe.torproject.org/#/bridge/3FD131B74D9A96190B1EE5D31E91757FADA1A… == Azure a.k.a. meek-azure == meek-azure bandwidth use continues to increase, up 17% compared to the previous month. Keep in mind that our grant expires in October, so you should not count it continuing to work after that. Speaking of the grant, they switched me over to a new kind of sponsored account that will enable me to see the exact bandwidth use and what its cost would be. That should be available starting next month. Until now, we have had to estimate bandwidth usage by measuring at the bridge. https://onionoo.torproject.org/bandwidth?fingerprint=AA033EEB61601B2B7312D8… Estimated bandwidth use this month: 2015-06 3038 GB Compared to last month: 2015-05 2592 GB https://globe.torproject.org/#/bridge/AA033EEB61601B2B7312D89B62AAA23DC3ED8… Earlier reports in this series: https://lists.torproject.org/pipermail/tor-dev/2014-August/007429.html https://lists.torproject.org/pipermail/tor-dev/2014-October/007576.html https://lists.torproject.org/pipermail/tor-dev/2014-November/007716.html https://lists.torproject.org/pipermail/tor-dev/2014-December/007916.html https://lists.torproject.org/pipermail/tor-dev/2015-January/008082.html https://lists.torproject.org/pipermail/tor-dev/2015-February/008235.html https://lists.torproject.org/pipermail/tor-dev/2015-March/008427.html https://lists.torproject.org/pipermail/tor-dev/2015-April/008596.html https://lists.torproject.org/pipermail/tor-dev/2015-May/008767.html https://lists.torproject.org/pipermail/tor-dev/2015-June/008932.html

1 0

Chutney (was Re: Performance testing using chutney)
by Cory Pruce 06 Jul '15

06 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > Well, you could test my latest branches for #14175: > > https://trac.torproject.org/projects/tor/ticket/14175#comment:8 > > There's a branch which modifies src/test/test-network.sh in tor, > and another with the performance measurement code in chutney. > > The command-line arguments/environmental variables are in the ticket, > and I've modified the chutney README to include performance testing. Cool! I'll check it out tonight and keep you posted :-) > > Maybe look at these for a start. It's something that's on my TODO > list, but I wouldn't complain if someone else happens to do it before I > do, and it would help HS scalability considerably[0]. > > https://trac.torproject.org/projects/tor/ticket/13737 > https://trac.torproject.org/projects/tor/ticket/13738 > > (If you happen to be more interested in making non-HS use cases faster, > then look elsewhere. :P) > > Regards, > Haha I'm interested in helping the performance/scalability of both HS and non-HS. I read the descriptions but they are pretty short and high level. Could you give any pointers to start? Does worker imply changes to cpuworker.*? - - Cory -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJVmqHwAAoJEB7DuCIJauCgdtMP/02Ztn3IoikLU1s+MdiJRyNY /X9pTU667Lu6VAd2yhbMpYSQlgoPkR6bFElGhOWucihFN84zTxSBbNxaPwW36mAE rBNHgw41cGEBSva6chHNLJJGx/wpThbLDGYTqPvoQrDGWqgKNB1hK4IRRRFaRiC0 H+nBKR0xiH/sBY4PzIIfiX3BBVrA0ogV+kOzEIwDSjmnw4x24OMjD/Fx2VOK2tai dwpHw0QTW6Ao+WEVtXWPvzhKZTnvKcPKDkvLwo7fzdSmVTS9IUqQJCh8KdYUnt/5 uO4HR0WEdPOT7anA1TvrKzhRTe/8KHzTcPrCHJo7Dbt/rvc7RW0v4GSClTEtBlIU EcHRCvmG/SaCaw+H2IcCK9k5T/rt7azsADcKIhqDWMvttJkVmsHo9FbVldpzNB/z hwHfVYKsuOPWanpPrucn4opf5Td8LcQ+v6IAtwqQbslpe6myf6aX2GrulLZLi74R PT/lo0wqGh4V59HeAS5rG6mUVeI7vADmqlrzzFnsDxNMYP2OLsLquKl2noAFSbFk kD2xcQmy3kygS6SiLrDX95c1CefB0W56CcJBdkHrvyuhSltFngpXpcDyVSvyZnN8 giXWxWuZC9unIAvgWvlKnEnUQgGxVWT3ItCzxohXUFM0qturow9ii3D9iWsO4zkL K5EB0TLP8Fc3OstFcpt1 =tGow -----END PGP SIGNATURE-----

1 0

Re: [tor-dev] Roster introduction (Philipp Winter) (Sean Saito)
by saitosean＠ymail.com 06 Jul '15

06 Jul '15

Hi Philipp, > - I searched for my relay family by fingerprint and the resulting page's > URL didn't contain my family, so I couldn't send the URL to somebody > else. Could you give me some more details about this? Currently the search bar supports all running relays. The URL will contain the fingerprint you put into the search bar. > - The colour coding for relay flags is hard to read because of the low > contrast. Perhaps use green for assigned flags and grey for missing > flags? > - I would put the exit policy in a fixed-size box. Otherwise, there's a > lot of scrolling for large exit policies. Certainly possible. Thanks a lot for the suggestions. Sean

2 1

Hi, all!
by Nick Mathewson 05 Jul '15

05 Jul '15

I'm about to go on vacation, but before I started, Isabela asked me for coverage information on Tor, to help people decide where more tests should be written. I've re-run coverage, and uploaded the gcov output here: https://people.torproject.org/~nickm/volatile/coverage-65a1e27.tar.bz2 That archive includes coverage on current master from * make check * make test-network (uses chutney) * make test-stem (uses stem) * all of the above, combined I'm also attaching a script that I wrote long long ago, and use to summarize coverage file-by-file. Example use: gcov-lines.py ./coverage-65a1e27/cov-all/* yrs, -- Nick

2 1

Re: [tor-dev] Roster introduction (nusenu) (Sean Saito)
by saitosean＠ymail.com 05 Jul '15

05 Jul '15

Hi nusenu, > A few comments: > > - - the page seems to be AS name centric > - first column of the table is AS name > - only clickable column in the table is AS name > > - - the AS column suggests that all relays in a family run in one AS only > > - - the family page of a specific family shows the AS name as title, > this is a bit confusing as one might think that the page shows all > families/relays in the selected AS after clicking an as name on the > start page Agreed, this was just a placeholder. Will put contact info instead for now. In the future I would like families to have ids/nicknames as well. > Family identifier > > - - by looking at your page I assume your family identifier is the > fingerprint of the oldest relay in the family, does that mean that an > operator has to start over at 0 with collecting "points" once he kills > his oldest relay? That is not the case. Thanks for the comments! Sean

1 0

Bi-directional families in Onionoo and consensus weight as measure of relayed bandwidth
by Karsten Loesing 02 Jul '15

02 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Moving this discussion here from another list with Virgil's permission. On 02/07/15 08:42, Virgil Griffith wrote: > Big issues right now are: * Bugs (?) in Onionoo --- Onionoo doesn't > sanitize its data. For example, there's a lack of bidirectionality > between relays of many families. > https://drive.google.com/file/d/0Bwjagz1RgJOnSkx0YTlhMHdfMFU/view?usp=shari… > > There are currently about 665 pairs of family relays without > bidirectionality. This is caused by the .torrc of some relays not > pointing to its family members. > > I am considering doing a service on top of Onionoo that sanitizes > the raw Tor consensus to ensure things like bidirectional families. > It's unclear how much other data needs sanitization. I'd rather want to fix/change Onionoo than have you write another service that processes Tor descriptors. There's even a ticket for this, we're just somewhat stuck by arguing about the best fix. Maybe I should just fix it somehow and, if necessary, fix it more later. https://trac.torproject.org/projects/tor/ticket/16276 Would that solve your problem? What other problems would there be with Onionoo's data? Can you make a wish list? > * A semi-reliable measure for the magnitude of traffic a relay has > routed. We have confirmed instances of relays forging their > observed bandwidth, ergo we can't use that. And thus far > Consensus Weight is the best we've found, but it's unclear whether > we can use that as a proxy of magnitude of relayed traffic. --- > https://docs.google.com/document/d/1v1rutylD6RkBei9rEmSvsgmvQXhrIHXOr85NL3I… > > Right now the lack of a reliable measure of how much bandwidth is > relayed is the largest sticking point. Actually, consensus weight (fraction) is a fine measure, and I like how you're calling it "bandwidth points" in your prototype which doesn't imply a bits per second or related unit. I'd say assign 10,000 bandwidth points to all relays per day, depending on what fraction of total consensus weight a relay had. To me, it's fine that this doesn't translate to bits or bytes. How does that sound? All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJVlPKJAAoJEJD5dJfVqbCrZWQH/0lHSdgy4PF7nQ8RMZryKpnf o3Fvw8VkcIwZgJgp0MOLIVu0fZhcD8hhvSWd9yYTSpQwGwBayUJuPE0ao4MbfZYf mwz5hkngzq1Z7654K65m/fYLu7EIbXI86vT4/Cwwh8cnGl/ezaliFVvVMOmKTyOb UtV7T+Lgk5IgsGJOxQbpNHCTxyAokbAygqZ9Eq/6ZWqjZFBZb1P2XjV+IaziGyJl yuxrD66cJe4ZmcpPe9g7mTa9JyQ5kmUOWogXhKTFWDFCcPslc0M49iiYohDmiNxC 5RGKp1dMuYkL6th9b3Uuc3W4TdCMaDHV96BDUD3qdlqCWBU0fD617f31+Hsb6Bg= =0KdX -----END PGP SIGNATURE-----

4 7

Tor Messenger Third Alpha
by Sukhbir Singh 01 Jul '15

01 Jul '15

Hi, Tor Messenger is an instant messaging client currently under development. It is designed to make connections using the Tor network and will therefore be a valuable piece in the privacy-enhancing software toolkit (web: Tor Browser, email: Thunderbird + TorBirdy, chat: Tor Messenger.) Based on the Instantbird IM client, Tor Messenger: - sends all traffic over Tor, - enforces Off-the-Record Messaging (OTR) for one-on-one conversations by default, - can be used with a wide variety of chat networks, - has an easy-to-use graphical user interface localized into multiple languages. * Current Status Today we are releasing the third alpha version with which we hope to gain both usability and security related feedback. There have been two previous alpha releases to the mailling lists that have already helped smooth out some of the rougher edges. * What's New in Tor Messenger 0.0.6 - Support for Windows and OS X builds (in addition to Linux) - Improved privacy and security with network and application-related enhancements - Improved usability of the Instantbird OTR extension - Private key and known fingerprint management - XMPP in-band account registration support - Build localization support (currently Arabic; Chinese, Spanish, Farsi, and other languages coming soon) * Bundles Linux (32-bit): https://people.torproject.org/~sukhbir/tor-messenger-0.0.6/tor-messenger-0.… Linux (64-bit): https://people.torproject.org/~sukhbir/tor-messenger-0.0.6/tor-messenger-0.… Windows: https://people.torproject.org/~sukhbir/tor-messenger-0.0.6/tor-messenger-0.… OS X: https://people.torproject.org/~sukhbir/tor-messenger-0.0.6/tor-messenger-0.… sha256sum: https://people.torproject.org/~sukhbir/tor-messenger-0.0.6/sha256sums.txt https://people.torproject.org/~sukhbir/tor-messenger-0.0.6/sha256sums.txt.a… The bundles are signed with the key 0xB297B391. * Instructions - On Linux, extract the bundle and then run: ./start-tor-messenger. - On OS X, copy the Tor Messenger application from the disk image to your local disk before running it. On all platforms, Tor Messenger sets the profile folder to the installation directory. * Code We are doing automated builds of Tor Messenger for all platforms using rbm (Reproducible Build Manager): https://gitweb.torproject.org/tor-messenger-build.git * Arabic Language Pack https://people.torproject.org/~sukhbir/tor-messenger-0.0.6/tor-messenger-ar… Install the language pack as an add-on and then set "general.useragent.locale" to "ar". (Thanks to Sherief Alaa for the translation.) * Feedback This is an early release and there may be serious privacy leaks and other issues -- please DO NOT recommend Tor Messenger to end users; this release is only for developers and advanced users who would like to help us with testing but understand the risks involved with using alpha software. With this release, we invite feedback not limited to anonymity leaks, usability issues, feature requests and suggestions for improvement. Please submit your feedback using the bug tracker (bugs.torproject.org; select the "Tor Messenger" component). You can also talk to us on this mailing list or on IRC. Thanks, Arlo Breault, Nicolas Vigier, Sukhbir Singh

3 4

Stormy update
by Griffin Boyce 01 Jul '15

01 Jul '15

Hey all, It seems like time to give the tor-dev list an update on Stormy's development. Right now, the scripts are undergoing third-party testing to identify any obvious bugs before sending them to security auditors. Testing should be finished imminently, any bugs found will be fixed this week, and then sent to auditors (along with the GUI). The security audit may take about two to three weeks [1]. Currently, Stormy's functions include installing typical onion service dependencies (webserver+tor), setting up a Ghost-based content management system, creating a personal cloud server to handle files/rss feeds/calendars/tasks, installing an XMPP/jabber server for private communications, and installing an IRC server for group communications. Shortcomings and future work: Like all software projects, Stormy has some shortcomings. Users can't configure multiple onion services on the same machine as Stormy doesn't account for virtual hosts. For users to run more than one onion service, they must be on separate physical or virtual machines. Stormy also doesn't detect the currently-running clearnet service, so users who seek to make their existing service also an onion service may need to adjust their configuration manually. Currently, Stormy lives on my github page [2], though once it passes a security audit, the goal is for it to live within Tor's git repositories. best, Griffin [1] Cupcake's audit took about two weeks, so that's really my only real data point for audit timing. [2] https://github.com/glamrock/stormy -- “Sometimes the questions are complicated and the answers are simple.” ― Dr. Seuss

2 1

Re: [tor-dev] Discussions
by Cory Pruce 01 Jul '15

01 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Yep, you got it! People find it easiest to read mailing list posts if the subjects match. And the threading works better that way, too. Any time you reply to and quote a previous email, cut out the extra stuff the mailing list adds in. And cutting out people's long signatures helps, too. Imagine you've googled a problem, and you're reading each email by itself - what do people need to know to follow the conversation? > I see now about top-posting though :D > Thanks, we all do that from time to time - most mail clients automatically top-post. > I kept the mailing list in the > loop for the readpassphrase.h but it's your call where you want to go > through the environment. I'm cool either way :-) > Stick as much as you like on the mailing list. People will tell you if it's too much. In fact, I'll stick my reply to your question about testing on tor-dev, as it's relevant. Tim Tim Wilson-Brown (teor) teor2345 at gmail dot com pgp 0xABFED1AC https://gist.github.com/teor2345/d033b8ce0a99adbc89c5 teor at blah dot im OTR D5BE4EC2 255D7585 F3874930 DB130265 7C9EBBC7 - --------------------------------------------------------------------------------------------- Awesome thanks Tim! I'm going to set up chutney later. Could you please send me the next steps you think would be good for a Tor testing environment? Thanks for your help! Cory -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJVksKGAAoJEB7DuCIJauCgM7MP/1ZbuSuo53Mz/8YBzA8FZPX1 yATAWJudqPaPH0m1nQS72x2Bzarz5UhCCGq2YpAhocoo1rEam+wIqcSJ7KUVso1H 1ViuKRjUWjgevAB33zdCWkZZtaKhg/r8zFpqMD8VL6N2ae3ypiY+nnw0vyEJZNZf YnUjoH+YNab/ftRv16/d72bmzHeylVRBcJX6OmSzSlCuCWROlptez3hG+xl52NHv q1obXuyWn36ba8agIeo3OFIX7sQ9Ok3Hvyr/Sve9QEt+FnB24yswS5X3izl8MtZZ Rs9Kz/RZsrH8AGAWmz+h6IIdGK/cmycsLk7bQyDzWHjzuDuIOtvXe5B4xizmoNEW in0Qxg5LfSklc8nN2531AbMExMoQsOW7ciNbQK7lQQlm8ZRb5U/ghWhjKc6PqZ8Z bWGv/vQQyzOQofsgaOm9Tul8vOPGk/e7a5mWRLj167ZuFRY/9d8I4K2mdjm45OQT Lz/gloDCV61IcMzEmmj3Ga8U2oWDicmKbJeNk7RnD9nPNr3JR9yKLEDRUD59Fp43 i4ZplSV0eIFDzozx0aZnVE/ctqyK9wDWWBVaTbsJfXcdV5h+DfGbtzPIdq2CA/cl 1Id94kqqGL00MlYuzw0eHcH5ssUIpVRGg96njolcCFgE/ipfWrFCTW+dxpuXD+p2 0unsHjaY0Am+T+NMPmCr =v2h0 -----END PGP SIGNATURE-----

2 2

Re: [tor-dev] Discussions
by teor 30 Jun '15

30 Jun '15

> On 1 Jul 2015, at 01:29 , Cory Pruce <corypruce(a)gmail.com> wrote: > > I've been able to do a > make install and am ready to continue setting up the test environment. I > see chutney on your github. I'll get that set up when I get home. Do you > have any advice for testing Tor? Also, feel free to move this to the > mailing list whenever you feel it's appropriate. > Hi Cory, I'm moving this to tor-dev as it's relevant to other developers. Chutney Please use the official chutney repository at git.torproject.org <http://git.torproject.org/> My chutney repository a little messy at the moment, and I've merged draft throughput-testing code into my master branch. You're welcome to test the draft code by merging https://github.com/teor2345/chutney/tree/bug14175-bandwidth <https://github.com/teor2345/chutney/tree/bug14175-bandwidth> The throughput-testing functionality works, but it needs to be tidied up, and have the hard-coded constants turned into command-line options. So it's not ready for release, but it is ready for an early review to check for functionality gaps and bugs. You might find throughput metrics useful to confirm that any multithreading changes are actually improving performance. Testing Once tor is built, you can test it using: # basic tests make test # extended tests make check # benchmarks /src/test/bench # verify tor connectivity using a local test network - requires chutney make test-network # verify that the core functionality of tor works, requires chutney and IPv6 on localhost src/test/test-network.sh --flavour bridges+ipv6+hs # verify that the core functionality of tor works, using chutney, but without IPv6 src/test/test-network.sh --flavour bridges src/test/test-network.sh --flavour hs There's also various other testing tools: shadow Tor network simulator static analysers, such as coverity and clang-scan dynamic sanitizers: Undefined Behaviour (UBSan), Address (ASAN), … fuzzing (I'm working on some tor-specific harnesses for fuzzing, but they're not ready) and I'm sure there are others which I've missed. Generally, I'm happy with code once I know it: * compiles with no warnings, * passes the extended unit tests (make check), including any tests written for new functionality, and * passes all the connectivity tests in a comprehensive test network (bridges+ipv6+hs). For bonus points, you can compile tor using UBSan and ASAN. If you're using clang, you'll find that tor/contrib/clang helps with setting this up. It lists known undefined behaviour in the Tor codebase, so it may be useful if you want to do something similar with gcc. (However, getting sanitizers to work can be incredibly fiddly and a total time-suck, too. So if it's just not working, skip it.) I'll also occasionally run code through a static analyser, to find subtle bugs which haven't been uncovered using UBSan/ASAN. Again, this isn't something everyone needs to do. Others may have advice on testing multithreading code in particular. Tim Tim Wilson-Brown (teor) teor2345 at gmail dot com pgp 0xABFED1AC https://gist.github.com/teor2345/d033b8ce0a99adbc89c5 teor at blah dot im OTR D5BE4EC2 255D7585 F3874930 DB130265 7C9EBBC7

1 0