August 2015 - tor-dev - lists.torproject.org

First release of OnioNS for beta testing
by Jesse V 08 Aug '15

08 Aug '15

Happy Saturday everyone, At long last, 310 commits later, I am pleased to present a release of the Onion Name System (OnioNS), a DNS for Tor hidden services. This release is a usability test; it offers reliable behind-the-scenes integration with the Tor Browser, a friendly command-line dialog for claiming domain names and subdomains, and many options for hosting and configuring a server. The system utilizes two servers: a single Quorum node which hidden services upload their claims to and another server which clients query against. I am looking for feedback as to how usable the system is and areas where it could be improved. Most of the changes going forward will be behind-the-scenes. The software is divided into three primary pieces, OnioNS-client, OnioNS-HS, and OnioNS-server. These all have OnioNS-common (a shared library) as a dependency. You can install whichever one you'd like, or all of them. This software is currently Linux-only, and Debian 7 and 8, Ubuntu 14.04 - 15.10, Mint 17 - 17.2, and Fedora 21 - 23 are supported. I provide packages for Debian 7 and a software repository for currently-supported versions of Ubuntu and Mint on 32-bit, 62-bit, and ARM systems. If possible, please use the repository. Please see the READMEs in the following repositories for more information, including installation, initialization, and configuration procedures. Manpages are also included for your convenience. https://github.com/Jesse-V/OnioNS-common https://github.com/Jesse-V/OnioNS-client https://github.com/Jesse-V/OnioNS-HS https://github.com/Jesse-V/OnioNS-server Please star the repository if it works well for you. I have claimed "example.tor" for my project's HS and claimed the "arma.example.tor" subdomain that points to Roger's site, so you can test this from the client. Please open a ticket if you find a new bug, or contact me if you don't have a Github account. A brief FAQ: Q: How does one pronounce "OnioNS"? A: As one would pronounce the lowercase form: "onions", the plural of "onion". Q: It only takes a couple of minutes to claim a domain name, isn't that too easy? A: In this release, I have set a very small difficulty level. It will certainly be harder in the future and more counter-measures are being considered. Also, since the claims are not yet saved to disk, I offer no guarantee of their long-term survival. Q: Should I use this on production hidden services? A: No, this software is not ready. This release introduces _features_, not security. Tor circuits are used on both the client and HS side, but I can't guarantee that my SOCKS use is leak-proof, for example. I'm asking for help with finding bugs that may compromise anonymity. Q: I'm running Windows/OSX/BSD/Arch/Gentoo/LFS/etc, can I test OnioNS? A: Yes, but I'm not currently supporting that environment, so you're mostly on your own. However, if you can give me compilation instructions, I may be able to. I am also looking to coordinate with anyone familiar with RPM or Windows development. Q: Is there security on your network communications? A: Client and HS communication occurs over Tor circuits, and there are some integrity checks, but simply getting everything to work is the priority here. Most of the infrastructure is set up so adding signatures and such will be easy, but that is next on the list. Once that occurs, the name server (Mirror) the client uses can be malicious with no significant impact. Q: Where can I learn more about this project? A: The normal project page, onions55e7yam27n.onion, is currently being rewritten. A simple page is in its place, so "example.tor" is still there. Literature on this project may be found at https://github.com/Jesse-V/OnioNS-literature. Please see the PDFs under the respective folders. Note that the distributed design will be changing to use George's commit-and-reveal scheme. Q: Are the servers reliable enough to run under Comcast? A: I have not tested them in production or otherwise under https://github.com/tylertreat/comcast, but I may in the future. I welcome help in this area. Enjoy, Jesse V.

2 1

simple onionoo patch
by firebrand＠ruggedinbox.com 07 Aug '15

07 Aug '15

Hi, I have submitted a small patch some time ago to fix some issues in onionoo. Could someone have a look? https://trac.torproject.org/projects/tor/ticket/16626 ** firebrand

2 1

Relay Dashboard Code Review
by Damian Johnson 05 Aug '15

05 Aug '15

Hi Cristobal, again apologies for the delay in getting you feedback. Just gave RWSD a whirl and worked great - nice work! >From a UI perspective only feedback on what we have so far is... * Both graphs just have the title of 'Graph component'. Better would be to say if it's inboud or outbound. * Graph's y-axis should indicate its units. * Personally I kinda like Vidalia's line graph better than bar graphs for this, but obviously just something to experiment with. Lots of options. https://upload.wikimedia.org/wikipedia/commons/e/ef/Tor-non-exit-relay-band… * Only having fifteen seconds of information is pretty short. Actually, if you raise it to something high (like several minutes of data) then you might want to consider prepopulating bandwidth information from before RWSD started. Here's how Nyx does it... https://gitweb.torproject.org/nyx.git/tree/nyx/graph_panel.py#n244 Unfortunately I'm still a bit of an Angular noob so I can't comment much on those bits. Arturo: do you have any thoughts? Otherwise, here's some feedback on the code! ============================================================ run_client.py ============================================================ 23 settings = dict( 24 template_path=os.path.join(os.path.dirname(__file__), "client/templates"), 25 static_path=os.path.join(os.path.dirname(__file__), "client/static"), 26 autoescape=None, 27 ) 28 handlers = [ 29 (r"/", MainHandler), 30 ] 31 cyclone.web.Application.__init__(self, handlers, **settings) Very minor thing, but little odd using dict() that way. This could also be... handlers = [ (r"/", MainHandler), ] cyclone.web.Application.__init__(self, handlers, { template_path: os.path.join(os.path.dirname(__file__), "client/templates"), static_path: os.path.join(os.path.dirname(__file__), "client/static"), autoescape: None, }) Again, minor and up to you. run_server.py has the same. This is similar to what I do all throughout my codebases... https://gitweb.torproject.org/nyx.git/tree/nyx/graph_panel.py#n71 ------------------------------------------------------------ 37 reactor.listenTCP(int(config.get('client.port')), Application()) Actually, you don't need the int() if you provide a default. If you call... config.get('client.port', 9051) ... then the config will give you an int, and emit a warning if the config has something else... https://stem.torproject.org/api/util/conf.html#stem.util.conf.Config.get Same in run_server.py. ============================================================ run_server.py ============================================================ 35 if(dual_mode()): Good idea on this config option. In python you don't use parentheses for conditionals, unless you're doing some grouping. Speaking of which, might I suggest baking pyflakes and pep8 checks into your project? Even if you don't have unit tests yet these can speed development by catching minor issues, and help the codebase have a more uniform style. Stem provides a couple utilities that should make this easy. Here's an example of using them... https://gitweb.torproject.org/nyx.git/tree/run_tests.py Oh, and it'll catch the trailing whitespaces that's slipped in. ;) % grep -R ' $' rwsd/* | grep -v 'Binary file' | wc -l 17 ------------------------------------------------------------ 45 @uses_settings 46 def main(config, dual = True): The 'dual = True' argument is unused. You use the dual_mode() util function instead. ------------------------------------------------------------ 87 except: 88 log.msg("unable to attach listeners, controller off") 89 except: 90 log.msg("unable to start tor controller from rwsd") _conn_listener() is a neat approach. I like it. Generally it's good idea to both only catch the exception types you need, and include the exception in the message so there's some hint for troubleshooting. The second, for instance, is in effect just calling stem.connection.connect(), which only raises ValueErrors (for invalid arguments)... https://stem.torproject.org/api/connection.html#stem.connection.connect Seeing how you use it here you probably want Controller.from_port() instead. As we discussed earlier connect() prints errors to stdout and returns None if it can't connect. However, you're not checking if it returns None so the second try/catch will hit an AttributeError. I suspect there *might* be a bug around double attaching listeners when tor is connected then reconnected, but not sure. ============================================================ util/__init__.py ============================================================ 49 def msg(message, config, **attr): Unused, but feel free to leave it alone if you plan on using it later. ============================================================ server/handlers/main.py ============================================================ Your run_client.py defines a MainHandler but then it seems you have an unused copy of it here too. Since run_server.py uses it too maybe drop it from run_client.py and use this instead? ============================================================ server/handlers/status.py ============================================================ 14 def _handle_status_event(controller, state, timestamp): 15 if state == State.CLOSED: 16 output = {'status': 'off'} 17 else: 18 output = {'status': 'on'} 19 20 send_data(WebSocketType.STATUS, output) Certainly fine, but can condense this if you'd like. def _handle_status_event(controller, state, timestamp): send_data(WebSocketType.STATUS, {'status': 'off' if state == State.CLOSED else 'on'}) ============================================================ server/handlers/graph.py ============================================================ 24 if read_total and write_total: 25 output.update({'read_total': read_total}) 26 output.update({'write_total': write_total}) No need to call update() for single values. This is the same as... if read_total and write_total: output.update['read_total'] = read_total output.update['write_total'] = write_total Same for more calls below. ============================================================ server/handlers/base.py ============================================================ 33 def init_websockets(): 34 global WEBSOCKETS 35 WEBSOCKETS = {} 36 for ws_type in WebSocketType: 37 WEBSOCKETS.update({ws_type: []}) Could also be... def init_websockets(): global WEBSOCKETS WEBSOCKETS = dict([(ws_type, []) for ws_type in WebSocketType]) ------------------------------------------------------------ 40 def get_websockets(ws_type = None): 41 global WEBSOCKETS 42 if ws_type: 43 if WEBSOCKETS[ws_type]: 44 return WEBSOCKETS[ws_type] 45 else: 46 return None 47 else: 48 return WEBSOCKETS The inner conditional made me pause for a second until I realized it's to convert empty lists to None. Are you familiar with dictionary's get() method? It allows you to specify a default, and might allow this to be improved a bit. def get_websockets(ws_type = None): websocket = WEBSOCKETS.get(ws_type, []) return websocket if websocket else None Also, you only need the 'global' keyword if you're doing assignment. Your init_websockets() is the only function that needs it. ============================================================ client/static/css/* client/static/font/* ============================================================ Perfectly fine, but our readme should say we're bundling a copy of angular, bootstrap, and awesome. Also, please note their licenses. I assume bundling is the usual approach for this? ============================================================ client/static/img/body-bg.png ============================================================ Is this just a solid color? If so, any reason it's an image rather than a css background color property? ============================================================ client/static/js/* ============================================================ Please separate our custom javascript from bundled modules. This way it's clear what is ours verses a copy. ============================================================ client/static/js/bandwidthGraph.controller.js ============================================================ 9 function bandwidthGraph($scope, bandwidthWebsocket) { 10 activate(); 11 $scope.readData.data = bandwidthWebsocket.read_data; 12 $scope.writtenData.data = bandwidthWebsocket.written_data; Ahh, so we're only using read_data and written_data. Our bandwidth handler is doing a bit of extra work each second to fetch additional data from tor. Fine for now, but we should later drop those GETINFO calls if they remain unused. Is the activate() getting called on every bandwidth event? On first glance looks like $scope.readData and $scope.writtenData are getting reset each time. ============================================================ client/templates/index.html ============================================================ 10 <link href="http://fonts.googleapis.com/css?family=Open+Sans:400italic,600italic,400,600" rel="stylesheet"> ... 15  16  Fine for now, but cross domain fetches to google (googleapis.com and googlecode.com) are likely a no-go for our final version. Also, it might also be unwise to link to Google Code since it's being phased out... https://code.google.com/p/support/wiki/ReadOnlyTransition Doesn't matter in the short term (being read-only is, of course, fine) but sounds like that may go away eventually. ------------------------------------------------------------ 19 <style> 20 .chart { 21 width: 100%; 22 height: 300px; 23 } 24 </style> Should we start our own application.css in static for things like this? No doubt this'll just be the first of many. ------------------------------------------------------------ 68 <div class="main"> 69 <div class="main-inner"> 70 <div class="container"> 71 <div class="row" ng-controller="bandwidthGraph"> ... Angular supports templates which store html snippets. Essentially partials. Maybe that would help break this up? It would be nice if our base index.html was dead simple, and pulled in modular components which do the actual work.

4 6

Summary of meek's costs, July 2015
by David Fifield 05 Aug '15

05 Aug '15

Here's the summary of meek's CDN fees for July 2015. App Engine + Amazon + Azure = total by month February 2014 $0.09 + -- + -- = $0.09 March 2014 $0.00 + -- + -- = $0.00 April 2014 $0.73 + -- + -- = $0.73 May 2014 $0.69 + -- + -- = $0.69 June 2014 $0.65 + -- + -- = $0.65 July 2014 $0.56 + $0.00 + -- = $0.56 August 2014 $1.56 + $3.10 + -- = $4.66 September 2014 $4.02 + $4.59 + $0.00 = $8.61 October 2014 $40.85 + $130.29 + $0.00 = $171.14 November 2014 $224.67 + $362.60 + $0.00 = $587.27 December 2014 $326.81 + $417.31 + $0.00 = $744.12 January 2015 $464.37 + $669.02 + $0.00 = $1133.39 February 2015 $650.53 + $604.83 + $0.00 = $1255.36 March 2015 $690.29 + $815.68 + $0.00 = $1505.97 April 2015 $886.43 + $785.37 + $0.00 = $1671.80 May 2015 $871.64 + $896.39 + $0.00 = $1768.03 June 2015 $601.83 + $820.00 + $0.00 = $1421.83 July 2015 $732.01 + $837.08 + $0.00 = $1569.09 -- total by CDN $5497.73 + $6346.26 + $0.00 = $11843.99 grand total The big event last month was an outage of meek-azure that was caused by a mistake during an account migration: https://lists.torproject.org/pipermail/tor-talk/2015-July/038487.html The outage is unfortunately still ongoing, although there's been progress in getting it fixed (I opened a ticket with Microsoft support and they have to migrate things manually). In the meantime, you can use Tor Browser 5.0a4, which uses a newly configured CDN endpoint. The number of users dropped below 3000, perhaps because of the meek-azure outage. https://metrics.torproject.org/userstats-bridge-transport.html?graph=userst… If you want to set up your own bridge and CDN instance without rate limiting, I can help you do it. Here are some docs to look at: https://trac.torproject.org/projects/tor/wiki/doc/meek#Howtorunameek-server… https://trac.torproject.org/projects/tor/wiki/doc/meek#GoogleAppEngine https://trac.torproject.org/projects/tor/wiki/doc/meek#AmazonCloudFront https://trac.torproject.org/projects/tor/wiki/doc/meek#MicrosoftAzure == App Engine a.k.a. meek-google == Instance hours were up sharply, over 40%, I don't know why. Here is how the Google costs broke down: 2934 GB $352.11 7598 instance hours $379.90 Compared to the previous month: 2807 GB $336.87 5299 instance hours $264.96 https://globe.torproject.org/#/bridge/88F745840F47CE0C6A4FE61D827950B06F9E4… == Amazon a.k.a. meek-amazon == Number of requests was up 8% while bandwidth was down 11%. Asia Pacific (Singapore) 155M requests $187.04 1034 GB $137.11 Asia Pacific (Sydney) 1M requests $2.46 5 GB $0.66 Asia Pacific (Tokyo) 28M requests $34.08 157 GB $20.10 EU (Ireland) 198M requests $238.14 1124 GB $86.91 South America (Sao Paulo) 7M requests $16.02 50 GB $11.94 US East (Northern Virginia) 71M requests $71.71 395 GB $30.90 -- total 463M requests $549.45 2768 GB $287.62 https://globe.torproject.org/#/bridge/3FD131B74D9A96190B1EE5D31E91757FADA1A… == Azure a.k.a. meek-azure == The meek-azure outage started on July 20. You can see it clearly in the attached graph, where bandwidth dropped almost to zero. If you are affected by this outage, you should try Tor Browser 5.0a4, which uses a working CDN endpoint for meek-azure. You can also manually change the configuration in other versions of Tor Browser: https://lists.torproject.org/pipermail/tor-talk/2015-July/038496.html https://trac.torproject.org/projects/tor/ticket/16634 https://onionoo.torproject.org/bandwidth?fingerprint=AA033EEB61601B2B7312D8… Estimated bandwidth use this month: 2015-07 1974 GB Compared to last month: 2015-06 3038 GB https://globe.torproject.org/#/bridge/AA033EEB61601B2B7312D89B62AAA23DC3ED8… Earlier reports in this series: https://lists.torproject.org/pipermail/tor-dev/2014-August/007429.html https://lists.torproject.org/pipermail/tor-dev/2014-October/007576.html https://lists.torproject.org/pipermail/tor-dev/2014-November/007716.html https://lists.torproject.org/pipermail/tor-dev/2014-December/007916.html https://lists.torproject.org/pipermail/tor-dev/2015-January/008082.html https://lists.torproject.org/pipermail/tor-dev/2015-February/008235.html https://lists.torproject.org/pipermail/tor-dev/2015-March/008427.html https://lists.torproject.org/pipermail/tor-dev/2015-April/008596.html https://lists.torproject.org/pipermail/tor-dev/2015-May/008767.html https://lists.torproject.org/pipermail/tor-dev/2015-June/008932.html https://lists.torproject.org/pipermail/tor-dev/2015-July/009030.html

1 0

Is anyone using tor-fw-helper? (Was Re: BOINC-based Tor wrapper)
by Nick Mathewson 05 Aug '15

05 Aug '15

Yawning's mail below reminds me: I am considering removing the C implementation of tor-fw-helper from the tor distribution, and recommending Yawning's pure-Go implementation instead. But before I do this, I'd like to get some sense of whether folks are shipping tor-fw-helper today, or using it in practice. On Jul 21, 2015 11:26 AM, "Yawning Angel" <yawning(a)schwanenlied.me> wrote: > > On Wed, 22 Jul 2015 01:06:41 +1000 > teor <teor2345(a)gmail.com> wrote: > > > > > > On 20 Jul 2015, at 11:11 , Serg <std.serg(a)gmail.com> wrote: > > > > > >> How do you plan to map ports on NAT devices? > > > > > > If it can't be done automatically using UPnP, This must be done > > > manually. No alternative cases. > > > > Our experience is that most routers' UPnP / NAT-PMP implementations > > don't work well with (our) automated tools. So this would have to be > > done manually, significantly reducing the pool of available > > volunteers. > > Just chiming in here. This may well work for a good number of users, > but the support overhead for when it fails is utterly gigantic because > certain brands of consumer routers have extremely poor UPnP/NAT-PMP > implementations. > > The usual symptom of a poor implementation is "the router crashes" but > certain other behaviors have been documented in the past by people > trying to use UPnP in ways that are spec compliant such as "the router > crashes and requires a NVRAM reset", "random port mappings get > obliterated", "the UPnP/NAT-PMP stack on the router crashes" etc. I wonder how commercial software handles these cruddy routers. Do they restrict themselves to a tiny part of the spec? Do they probe for problematic firmware, and maintain a list of unreliable versions? [I am very glad it is not our job to maintain such a list.] -- Nick

8 20

curve25519_donna vs. crypto_scalarmult_curve25519?
by Steve Snyder 05 Aug '15

05 Aug '15

Given a contemporary release of Tor with a contemporary version of OpenSSL, under what circumstances is the intrinsic curve25519_donna() preferred over the libsodium/NaCl crypto_scalarmult_curve25519(), or vice versa? Does it come down to 32-bit vs. 64-bit? Or CPU instruction sets detected at build time? Or has libsodium/NaCl been deprecated altogether and just not removed from the configure script? I'm trying to determine if installation of libsodium in a Tor build environment will yield a "better" Tor binary. Thanks.

2 1

5th status report for OnioNS
by Jesse V 04 Aug '15

04 Aug '15

Hi everyone, As I stated in my last status report, I promised to work on server-to-server communication in preparation for a beta test. Quite a bit of work was done in this area, with the result that the server-server protocols are now working as intended and fairly reliably. The changes include: * Ability to bind to a specific IPv4 address. This is useful not only in servers with multiple interfaces, but allows two instances of OnioNS-server to run simultaniously on localhost, which is great for testing and debugging. * Improved server-side logging and diagnostic information. Each network session has it's own identification number, which makes it easy to keep track of events and conversations. * Rewrote the networking code responsible for retransmitting information such as new Records. The servers now correctly sends Records to subscribers. * A Mirror server now reestablishes the network connection with a Quorum server if the Quorum server drops from the network. * Many bug fixes, addressing numerous issues with crashes, networking issues, and serialization faults. With these changes, the OnioNS project is in a pretty good state. Here's an overview of the system's order of operations: 1) A single Quorum server starts up and listens on a specific IP and port. It's a TCP server and waits for JSON-encoded messages. 2) A Mirror server starts up, connects with and subscribes to the Quorum server. 3) A hidden service operator generates a claim on a domain name (say "example.tor") and optionally some subdomains. This claim is represented in the Record data structure. 4) The operator sends the Record over a Tor circuit to the Quorum server, who inspects it, caches it, and sends it to all subscribers, which is the single Mirror node. 5) The Mirror inspects the Record, caches it, and sends it to all subscribers, but here it doesn't have any. 6) A client starts up the Tor Browser with the OnioNS software installed. This means that the starting of the Tor Browser also triggers the launching of the OnioNS software: a binary and a Python script. 7) The user types in "example.tor". Their Python script detects the pseudo-TLD and performs a query lookup to the Mirror node, who replies with the Record. 8) The OnioNS-client software inspects the Record, extracts the hidden service address, and passes the address back to the Stem script. 9) The Stem script rewrites the original stream request from step 7 to this HS address and tells Tor to attach the stream to some circuit. 10) Tor attaches the stream with the effect that the hidden service loads in the Tor Browser. All of these pieces are now in place and generally work as intended without requiring any user intervention. The step of separating the responsibility into two different servers is now behind me, which when coupled with the working subscriber-based server-server communication, gives me many options to move forward towards a distributed system. I have a couple of annoyance-level bugs to fix but a prototype is very near. I will be shortly resume building packages and will currently be supporting Ubuntu, Mint, Debian, and Fedora on i386, amd64, and armhf. As a final note, a quick thanks to George for the well-written commit-reveal-consensus proposal. As he stated in his introduction, I certainly need that in order to integrate OnioNS with the Tor network. Jesse V.

1 0

CompCert C compiler
by Mansour Moufid 04 Aug '15

04 Aug '15

Hi everyone, This is just to let you know that Tor compiles without modification using the CompCert C compiler. This week I'll run it in the "C interpreter" to see what happens. http://compcert.inria.fr/man/manual004.html#cinterp Mansour

1 0

[Measurement Team] Next IRC meeting happens on Wednesday, Aug 5, 14:00 UTC in #tor-project
by Karsten Loesing 03 Aug '15

03 Aug '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello list, This is an announcement/reminder that there will be an IRC meeting of the Measurement Team on Wednesday, August 5, 2015, 14:00 UTC in #tor-project https://www.timeanddate.com/worldclock/fixedtime.html?iso=20150805T14 Last week's meeting only had two participants, and while I had a really good time chatting with Philipp, I think it's safe to suggest the same agenda for this week: - Introductions, problems, products, priorities: any feedback, remarks, comments on what we did in the previous four weeks - 1-1-1 task exchange: you get 1 minute to describe a task that would take somebody else roughly 1 hour and that they will do for you within 1 week (review a document, write some analysis code, fix a small bug, etc.; better come prepared to get the most out of this; give 1, take 1) - Misc/discussion: whatever comes up during the meeting If you think this can be fun, please join us at the meeting! Talk to you on Wednesday! All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJVv8ZVAAoJEJD5dJfVqbCrJuAH/2wSlJn8v3m7SDy4kzBEyA5Y lSItMFAn8Q+p6zd4aCkeBxMFUaEJZ8qpNPuq7m8Nv+L5lDAMISpjn8D9z7cH4qLb HGM+Y6oHZwX+e/o/Xj6iVEG8MvJVXB2l1QhXynohGDpJoMX8Lj/lff1bDdt19n2f 5cAULUH5xtFKf4l9XZ7gRgRW+iIZZ3NdI08A/8uZT6PU/rvoKHN9j/52gUIAM+mB j0Vr6xIW38fC3O3N8CnlVmoXglTiuLXJ0HC56VPbGlKRU/Al1pM2M1vk+6ddsi7q 9zcB3qqd485Oj8GFeOUS238GEBawrQk61prKh1xS58Vdji/XfPB/i7vuDgs0Pms= =L7j1 -----END PGP SIGNATURE-----

1 0