- tor-dev - lists.torproject.org

rendezvous on non-OR circuit with purpose Acting as rendevous
by Ian Goldberg 23 Nov '17

23 Nov '17

I just restarted my node, and saw this in the log: Nov 23 14:07:21.000 [warn] Tried to establish rendezvous on non-OR circuit with purpose Acting as rendevous (pending) What does this mean? I'm a little worried someone out there is playing games with the protocol...

2 1

Proposal: Move IPv6 ORPorts to the Microdesc Consensus
by teor 21 Nov '17

21 Nov '17

Hi all, We would like to move IPv6 ORPorts from microdescriptors to the microdesc consensus. This makes it easier for IPv6 clients to bootstrap and choose reachable guards. The proposal is inlined below, it is also available with the corresponding dir-spec updates in my torspec branch bug23826-23828 on GitHub: https://github.com/teor2345/torspec.git The tor code that implements these new consensus methods is in my tor branch on bug23826-23828 on GitHub: https://github.com/teor2345/tor.git The parent ticket for these related changes is #20916. The code changes are being tracked in #23826 and #23828, and the spec changes and proposal in #23898: https://trac.torproject.org/projects/tor/ticket/20916 If we've spoken about this, and I've left you out as an author, please let me know! Here is the proposal text: Filename: xxx-ipv6-in-micro-consensus.txt Title: Move IPv6 ORPorts from microdescriptors to the microdesc consensus Author: Tim Wilson-Brown (teor) Created: 18-Oct-2017 Status: Open Target: 0.3.3.x 1. Summary Moving IPv6 ORPorts from microdescs to the microdesc consensus will make it easier for IPv6 clients to bootstrap and select reachable guards. Since consensus method 14, authorities have voted for IPv6 address/port pairs (ORPorts) in "a" lines. Unreachable IPv6 ORPorts are dropped from the full consensus. But for clients that use microdescriptors (the default), IPv6 ORPorts are placed in microdescriptors. So these clients can only tell if an IPv6 ORPort is unreachable when a majority of voting authorities mark the relay as not Running. This proposal puts reachable relay IPv6 ORPorts in an "a" line in the microdesc consensus. This allows clients to discover unreachable IPv6 ORPorts, even if a minority of voting authorities set AuthDirHasIPv6Connectivity 1. 2. Proposal We add two new consensus methods, here represented as M and N (M < N), to be allocated when this proposal's implementation is merged. These consensus methods move IPv6 ORPorts from microdescs to the microdesc consensus. We use two different methods because this allows us to modify client code based on each method. Also, if a bug is discovered in one of the methods, authorities can be patched to stop voting for it, and then we can implement a fix in a later method. 2.1. Add Reachable IPv6 ORPorts to the Microdesc Consensus We specify that microdescriptor consensuses created with methods M or later contain reachable IPv6 ORPorts. 2.2. Remove IPv6 ORPorts from Microdescriptors We specify that microdescriptors created with methods N or later do not contain any IPv6 ORPorts. 3. Retaining Existing Behaviour The following existing behaviour will be retained: 3.1. Authority IPv6 Reachability Only authorities configured with AuthDirHasIPv6Connectivity 1 will test IPv6 ORPort reachability, and vote for IPv6 ORPorts. This means that: * if no voting authorities set AuthDirHasIPv6Connectivity 1, there will be no IPv6 ORPorts in the consensus, * if a minority of voting authorities set AuthDirHasIPv6Connectivity 1, unreachable IPv6 ORPort lines will be dropped from the consensus, but the relay will still be listed as Running, * if a majority of voting authorities set AuthDirHasIPv6Connectivity 1, relays with unreachable IPv6 ORPorts will be dropped from the consensus. We will document this behaviour in the tor manual page, see #23870. 3.2. Full Consensus IPv6 ORPorts The full consensus will continue to contain reachable IPv6 ORPorts. 3.3. Clients that use Full Descriptors Tor clients that use full descriptors already ignore unreachable IPv6 ORPorts, and have done so since at least 0.2.8.x. 4. Impact and Related Changes 4.1. Directory Authority Configuration We will work to get a super-majority (75%) of authorities checking relay IPv6 reachability, to avoid Running-flag flapping. To do this, authorities need to get IPv6 connectivity, and set AuthDirHasIPv6Connectivity 1. 4.2. Relays and Bridges Tor relays and bridges do not currently use IPv6 ORPorts from the consensus. We expect that 2/3 of authorities will be voting for consensus method N before future Tor relay or bridge versions use IPv6 ORPorts from the consensus. 4.3. Clients 4.3.1. Legacy Clients 4.3.1.1. IPv6 ORPort Circuits Tor clients on versions 0.2.8.x to 0.3.2.x check directory documents for ORPorts in the following order: * descriptors (routerinfo, available if using bridges or full descriptors) * consensus (routerstatus) * microdescriptors (IPv6 ORPorts only) Their behaviour will be identical to the current behaviour for consensus methods M and earlier. When consensus method N is used, they will ignore unreachable IPv6 ORPorts without any code changes. 4.3.1.2. IPv6 ORPort Bootstrap Tor clients on versions 0.2.8.x and 0.2.9.x are currently unable to bootstrap over IPv6 only connections when using microdescriptors. This happens because the microdesc consensus does not contain IPv6 ORPorts. When consensus method M is used, they will be able to bootstrap over IPv6 only connections using microdescriptors, without any code changes. 4.3.2. Future Clients 4.3.2.1. Ignoring IPv6 ORPorts in Microdescs Tor clients on versions 0.3.3.x and later will ignore unreachable IPv6 ORPorts once consensus method M or later is in use. (See #23827.) 4.3.2.2. IPv6 ORPort Bootstrap If a bootstrapping IPv6-only client has a consensus made with method M or later, it should download microdescriptors from one of the IPv6 ORPorts in that consensus. Previously, IPv6-only clients would use fallback directory mirrors to download microdescs, because there were no IPv6 ORPorts in the microdesc consensus. (See #23827.) 4.3.2.3. Ignoring Addresses in Unused Directory Documents If a client doesn't use a particular directory document type for a node, it should ignore any addresses in that document type. (See #23975.) 5. Data Size This change removes 2-50 bytes from the microdescriptors of relays that have an IPv6 ORPort, and adds them to reachable IPv6 relays' microdesc consensus entries. As of October 2017, 600 relays (9%) have IPv6 ORPorts in the full consensus. Their "a" lines take up 19 KB, or 33 bytes each on average. The microdesc consensus is 1981 KB, so this represents about 1% of its uncompressed size. Most tor clients are already running 0.3.1.7, which implements consensus diffs. We expect that most directory mirrors will also implement consensus diffs by the time 2/3 of authorities are voting for consensus method M. So we expect that this change will have a minimal impact, which is made even smaller by compression and consensus diffs. 6. External Impacts We don't expect this change to impact Onionoo and similar projects, because they typically use the full consensus. Metrics doesn't currently graph IPv6 usage in Tor, but would like to in future. -- Tim / teor PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------

3 4

Re: [tor-dev] GAEuploader now supports Windows
by Katherine Li 20 Nov '17

20 Nov '17

Hello Tor-Dev Community, GAE(Google App Engine) uploader now supports Windows, in addition to Mac and Linux. GAEuploader automates the process of uploading the meek code to App Engine. If you already have a Google account, it's pretty easy: the program will open a browser so you can authenticate and accept the App Engine terms of service, then upload the code under a domain name you choose. At the end, GAEuploader prints out a bridge line to paste into Tor Browser. The difference between this and the old meek-google is that the App Engine code you upload is just for you, not shared with anyone else. I would really appreciate user testing on GAEuploader. You can download it at: https://github.com/katherinelitor/GAEuploader/releases README: https://github.com/katherinelitor/GAEuploader Tor wiki page, containing step-by-step screenshots of how to use: https://trac.torproject.org/projects/tor/wiki/doc/GAEuploader If you have any questions, feedback or bug reports, please email me at katherineli.tor(a)gmail.com. I will respond promptly. Best, Katherine Li On Sun, Jan 22, 2017 at 3:53 PM, Katherine Li <katherineli.tor(a)gmail.com> wrote: > Hello Tor-Dev Community, > > GAE(Google App Engine) uploader has just released v1.0-beta. GAEuploader > enables clients to set up their own private meek-google in one easy step: > just run it and paste the bridge line into Tor Browser. > > Background information: Meek > <https://trac.torproject.org/projects/tor/wiki/doc/meek#Overview> is > pluggable transport that uses domain fronting. Meek-google got suspended > <https://lists.torproject.org/pipermail/tor-talk/2016-June/041699.html>, > but clients can set up their own Google App Engine app and configure their > Tor Browser accordingly. However, the process of creating and deploying the > Google App Engine can be confusing. GAEuploader significantly simplifies > this process for clients. > > I would really appreciate user testing on GAEuploader. You can download it > at: https://github.com/katherinelitor/GAEuploader/releases > README: https://github.com/katherinelitor/GAEuploader > Tor wiki page, containing step-by-step screenshots of how to use: > https://trac.torproject.org/projects/tor/wiki/doc/GAEuploader > > *GAEuploader runs on Mac or Linux, but currently does not support Windows. > *GAEuploader uses the Google Cloud SDK > <https://cloud.google.com/sdk/docs/#linux>. If you are interested in how > GAEuploader works, the README contains the list of commands that were run. > > If you have any questions, feedback or bug reports, please email me at > katherineli.tor(a)gmail.com. I will respond promptly. > > Best, > Katherine Li >

1 0

[release] [protocol] Onionoo 4.3-1.7.1
by iwakeh 20 Nov '17

20 Nov '17

Hi there! Onionoo's protocol was extended and has a minor version jump to 4.3. In addition, this release *prepares* a major *upcoming version* jump 5.0. Take note of the field "next_major_version_scheduled":"2017-12-17" in documents. Download available at: https://dist.torproject.org/onionoo/4.3-1.7.1/ Protocol changes (also summarized in [0]): Added support for a new "host_name" parameter to filter by host name and a new "unreachable_or_addresses" field with declared but unreachable OR addresses [1]. The changes are already deployed on all onionoo.torproject.org instances. Please direct comments and questions to the metrics-team mailing list [2]. Cheers, iwakeh [0] https://metrics.torproject.org/onionoo.html#versions_4_3 [1] https://metrics.torproject.org/onionoo.html#details_relay_unreachable_or_ad… [2] https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team

1 0

Tor Metrics Roadmap 2017/18
by Karsten Loesing 20 Nov '17

20 Nov '17

Hello everyone, we, the Tor Metrics Team, have finished writing our roadmap for the 12 months between October 2017 and September 2018: https://trac.torproject.org/projects/tor/raw-attachment/wiki/org/teams/Metr… https://trac.torproject.org/projects/tor/wiki/org/teams/MetricsTeam#Roadmap… In the process of writing this roadmap we incorporated feedback from various people in the Tor community, including suggestions made on this list. And if something didn't make it on this year's roadmap, we made a note to reconsider in about a year from now. In any case, thanks a lot for contributing to this process! Stay tuned for more Tor Metrics! All the best, Karsten

2 3

Re: [tor-dev] UX improvement proposal: Onion auto-redirects using Alt-Svc HTTP header
by Alec Muffett 18 Nov '17

18 Nov '17

Apologies, I am waiting for a train and don't have much bandwidth, so I will be brief: 1) There is no point in issuing <any header of any kind> to anyone unless they are accessing <website> via an exit node. 2) It's inefficient to issue the header upon every web access by every person in the world; when the header is only relevant to 1-in-a-few-thousand users, you will be imposing extra bandwidth cost upon the remaining 99.99...% -- which is unfair to them 3) Therefore: the header should only be issued to people arriving via an exit node. The means of achieving this are a) Location b) Bending Alt-Svc to fit and breaking web standards c) Creating an entirely new header 4) Location already works and does the right thing. Privacy International already use this and issue it to people who connect to their .ORG site from an Exit Node. 5) Bending Alt-Svc to fit, is pointless, because Location already works 6) Creating a new header? Given (4) and (5) above, the only potential material benefit of it that I can see would be to "promote Tor branding" - and (subjective opinion) this would actually harm the cause of Tor at all because it is *special*. 6 Rationale) The majority the "Dark Net" shade which had been thrown at Tor over the past 10 years has pivoted upon "needing special software to access", and creating (pardon me) a "special" header to onionify a fetch seems to be promoting the weirdness of Tor, again. The required goal of redirection to the corresponding Onion site does not require anything more than a redirect, and - pardon me - but there are already 4x different kinds of redirects that are supported by the Location header (301, 302, 307, 308) with useful semantics. Why reinvent 4 wheels specially for Tor? 7) Story: when I was implementing the Facebook onion, I built infra to support such (eventual) redirection and/or exit-node-usage tracking. Hit " facebook.com/si/proxy/" from Tor/NonTor to see it in action. The most challenging thing for me was to get a reliable and cheap way to look-up, locally, quickly, cheaply and reliably, whether a given IP address corresponded to an exit node. The closest that I could get to that idea was to scrape Onionoo every so often and to cache the results into a distributed, almost-memcache-like table for the entire site. ie: squillions of machines. This mechanism suffers from all the obvious flaws, notably Onionoo crashes and/or "lag" behind the state of the consensus. 8) So, to pass concrete advice on the basis of experience: rather than pursue novel headers and reinvent a bunch of established, widely-understood web redirection technologies, I would ask that Tor focus its efforts instead upon providing a service - perhaps a listener service embedded in little-t tor as an enable-able service akin to SOCKSListener - which can accept a request from <cidr-subnetmask>, receive an newline-terminated IP address, and return a set of flags associated with that IP (exit node, relay, whatever) - or "none" where the IP is not part of the tor network. Riff on this protocol as you see fit. This would mean more people running more tor daemons in more datacentres (and possibly configuring some of them as relays) - using this lookup service to establish quickly whether $CLIENT_ADDR is an exit node or not, and whether it should be issued "308 Permanent Redirect With Same Method" I think this is a better goal for Tor to be pursuing. What do you think? - alec

7 9

Connection, Channel and Scheduler - An Intense Trek
by David Goulet 16 Nov '17

16 Nov '17

Hello everyone! DISCLAIMER: The following is enormous and tries to describe in some level of details the situation in tor with connection<->channel<->scheduler. This comes after we've merged the KIST scheduler, we've realized many things we'ren't what they were suppose to be or meant for. In the end, I'm asking questions so we can move forward with development and fixing things. Last thing before you start your journey in the depth of Tor, the 3 subsystems I'm going to talk about and how they interact are kind of very complicated so it is very possible that I might have gotten things wrong or miss some details. Please, point them out so we can better document, better be informed and make good decisions. I plan to document as much as I can from this process for a new file in torguts.git repository. This is part of the work from ticket #23993. Enjoy! == Part One - The Distant Past == Once upon a time, Tor had 0.2.3 years old. It was before the "big change" and it was using connections and circuits in a simplistic but yet clever way. I'll briefly describe here how it worked out and why we wanted to change it. Roughly, once a cell comes in, it is put in the inbound buffer called "inbuf" of a connection_t and it is processed. Then, if it needs to be relayed that is sent along the circuit it came on, it is put in that circuit queue. Each connection_t had a priority queue of "active circuits" using the EWMA policy to prioritize. The first circuit was to be flushed of a certain amount of cells onto the connection_t outbound buffer or "outbuf" where libevent main loop takes over to actually write the bytes on the network from the outbuf. Then, the circuit pqueue is updated and we would move on. The following is a high level diagram of a relayed cell: +--------------+ +-----------+ +----------------+ |connection_t A| | circuit_t | | connection_t B| | (inbuf) |--->| (pqueue) |--->| (outbuf) |---> Network +--------------+ +-----------+ +----------------+ That had at least one big problem. Not considering the entire set of circuits could make one connection (with some circuits on it) bloat the network link of the relay. It would actually be worst than that if it had bandwidth limitation, you want to pick the connection that has the highest priority for the circuit on it to flush All in all, tor needed to move to a logic where _all_ circuits are considered using the EWMA policy, flushing the cell(s) of that picked circuit on the connection outbuf so it is prioritized on the network. Then in 0.2.4, everything changed! == Part Two - The Big Change == Two new abstractions were added to Tor 0.2.4 (#6465), channel and circuitmux (short for circuit multiplexer or multiplexing, history is unclear ;). Up until this day, this is what tor is using. The channel_t object was intended to provide a "transport" abstraction layer for relay to relay connection which is reprensented by a connection_t object. Each channel has a circuitmux_t object that encapsulates the queuing logic and selection policy of circuits of a channel. In other words, the circuits that goes through that connection are listed in that object so we can do cell prioritization. To summarize, this roughly looks like this now: +------------+ |connection_t| +------------+ | v 1 +---------+ |channel_t| +---------+ | v 0..n +---------+ |circuit_t| +---------+ Channels are suppose to work as a protocol abstraction on that connection. The original intent of this was to be able to implement more easily different protocl like for instance helping researchers to experiment with other things more easily. Decoupling TLS from connection_t and putting them in a subclass of channel_t and thus was born channel_tls_t. It implements the channel_t interface and is in theory designed to handle TLS layer of relay to relay connection (handshake, cells, etc). So far, tor only has that channel class. Now that we had this layer and a way to multiplex circuits on a single connection through the use of channel, we needed one last piece, a way to prioritize cells on the network considering all circuits. Along came the "cell scheduler" (scheduler.c) in Tor 0.2.6 and now called the Vanilla scheduler. In a nutshell, libevent main loop calls the scheduler at a constant rate and the subsystem will consider all channels between each other comparing circuits priority using the EWMA policy (an interface was created to implement more policies but I won't get into that in this email). So now, tor finally had a good infrastructure to prioritize cells on the network by looking at all circuits at once. TCP connections would be established between relays, the channel layer would be handling the TLS business and the scheduler would be flushing cells on the network by considering all circuits on all channels. It is a really nice modularized design and would make tor improve in performance. But, that's not entirely the reality of things right now. == Part Three - The Thruth == In 2014, a paper came out titled: Never Been KIST: Tor’s Congestion Management Blossoms with Kernel-Informed Socket Transport (http://www.robgjansen.com/publications/kist-sec2014.pdf) An improved way for tor to scheduled data transmission properly by asking the kernel information about the state of the TCP buffers so tor could flush data on the network without bloating the local link. Then in 2016 up to now, Matthew Traudt implemented KIST in tor which required an important refactoring of the scheduler subsystem in order to make it work with different multiple scheduling type. It is today the Scheduler option in torrc. And all 0.3.2 tor uses KIST by default. Many things were discovered during that piece of work by Matt. For instance, this very important #20459 bug for which we were badly comparing cmux policy thus ultimately prioritizing wrongly our circuits. But fundemental issues in the design of channels and the scheduler were found for which I'll list some here and will ultimately lead me to Part Four of this email about our next steps moving forward. 1. Channels implemented cell queues that weren't really queues. If you look at the channel_t object, you'll notice "incoming_queue" and "outgoing_queue" which basically allow the transition of a cell from a connection_t inbuf to the circuit queue and then to the connection_t oubuf. It looks like this for a relayed cell: conn inbuf -> chan in_queue -> circ queue -> chan out_queue -> conn outbuf The idea behind this is to quickly empty the conncetion inbuf to an intermediary object (channel) leaving the kernel inbound TCP queue as much room as we can. In other words, offloading the kernel buffer to user space since we need to run some processing on that data and there was no need to leave that data on the kernel buffer during that time. Then from the channel, a cell would be put on the circuit queue and in turn would be process by the scheduler. The scheduler would select a circuit (EWMA policy), flush cells from its queue onto the channel outgoing queue and then it would be flushed to the connection outbuf which is taken care of by libevent to write on the socket (to the network). One could argue here that in terms of design we have too many queues but that would be true if the channel queues were actually acting like queues. It turns out that the incoming and outgoing queues of a channel are always empty (#23709) and this is due to the fact that the code queues a cell then processes it immediately meaning either put it on the circuit queue or connection outbuf. Each cell is memcpy() over the a channel queue and then processed right away just to be copied again into the circuit queue or outbuf. This has a performance cost over time as a relay sees thousands of cells a second. But also it makes the channel subsystem much more complicated in terms of code with trying to handle those queues every part of the way in and out of the channel subsystem. But the worst part is that the scheduler subsystem assumed some decisions based on the outgoing queue size. And when it is always 0, issue arise like #23676. 2. DESTROY cells handling Within a circuitmux object, there is a "destroy cell queue" on which a DESTROY cell is put in for one of the circuit on the cmux. An important thing for tor is that when it needs to send a DESTROY, it needs to _stop_ sending any queued cell on that circuit, dump them and only send the DESTROY cell. Many things are problematic currently. For starter, sending a DESTROY cell bypasses the scheduler (#23712). Tor will immediately try to flush the cell on the network *if* the channel doesn't have queued *writes* which means if the connection outbuf is empty. If not, once it is sufficiently drained, the channel will be scheduled in and the DESTROY cell finally sent. I've observed this process going from 0.5 seconds up to 10 seconds on a normal relay if the outbuf has data on it. Second, because of this concept of different queue for the DESTROY cell, tor will back and forth between that queue and the normal queue of a circuit. Remember, that the destroy queue is *not* per circuit but per circuitmux. This is used by the "cmux->last_cell_was_destroy" which is set to 1 if the last cell the cmux handled was a DESTROY or 0 if not. This has a side effect. Before sending a DESTROY cell, the circuit queue is cleared out without changing the circuit EWMA priority, in order to make sure we don't send cells from that point on. However, because we back and forth, the EWMA policy will pick the right channel based on the overall state of the circuits (see scheduler_compare_channels()) on it but will not prioritize the circuit with the DESTROY cell on using that policy. Furthermore, the destroy cell queue is a FIFO so if 10 DESTROY cells were queued bu th 10th one is actually on the circuit with the highest priority, we won't process it until those 9 previous who came in before are flushed. 3. Connection and Channel decoupling is kind of a lie The connection_t object is bound to be a TCP connection except for DNSPort which will be a UDP. That object also has a "TLS connection state" (tor_tls_t) and a channel_tls_t object (chan). First, it turns out that the connection subsystem will establish the TLS session and not the channel tls layer. When opening a circuit, it looks like this: channel_connect_for_circuit() |- channel_connect() |- channel_tls_connect() |- connection_or_connect() |- connection_tls_start_handshake() Second, when processing a cell from the connection inbuf in connection_or_process_cells_from_inbuf(), we'll directly call channel_tls_handle_cell(). The channel_tls_t layer handles PADDING, VERSIONS, CERTS, AUTH_CHALLENGE, AUTHENTICATE and NETINFO cells. The other cell types will be queued on the corresponding circuit queue. Third, once the TLS session has been established, a VERSIONS cell can be sent which is done by the connection subsystem with connection_or_send_versions(). But it is bypassing the channel_tls_t layer that processes them and should write them using channel_tls_write_cell_method(). Futhermore, the channel_tls_t layer is using the connection layer to send those cells using connection_or_send_certs_cell() and cie. My point here is that the channel abstraction is kind of violated in many ways by either directly calling the TLS channel layer or not using it to send specific cells that are for the TLS handshake. And also, the TLS state is baked in the connection subsystem. This means that anyone wanting to implement a new channel will need to do a major refactoring first which unfortunately leaves this abstraction kind of pointless. And do not make the mistake to label the channel subsystem as a "transport layer", it is at best a protocol layer, connection_t handles transport, not channels. 4. Vanilla scheduler is not really a scheduler. It turns out that the Vanilla scheduler is not really scheduling things but rather shoving as much as it could onto the connection outbuf. This ain't ideal because tor is selecting circuit based on a policy (EWMA) and when the circuit is selected, we would push on the connection outbuf by either writing as much as we can on the outbuf or stop at 1000 cells (see MAX_FLUSH_CELLS) which is 512 * 1000 bytes == 0.5MB. One can argue, is it too little to only write 0.5MB per scheduler tick or actually too much? So imagine tor flushed 1000 cells everytime it picks a circuit and then moves on to the next circuit. I'm not sure this is playing well with a relay with bandwidth limitation for instance. If you have 100 circuits, and 100KB to spend, you might not want to spend it all on one single circuit? Also, because this scheduler runs as often as possible, only one channel was considered everytime. Matt's experiment showed that in practice which means that a scheduler that is only considering one channel doesn't make good priority decision because there is none in the first place. Finally, when trying to bloat the outbuf as much as possible, and for that I'm not sure how libevent operates, but it leaves the decision of when and what to flush on the network to the process that handles outbuf. In other words, the scheduler doesn't actually control when the data is written to the socket so the scheduling decision aren't necessarly followed through on the transport layer. This is something the scheduler and connection subsystem should be thightly connected. == Part Four - The Conclusion == Through this epic journey, we've discovered some issues as well as design problems. Now the question is what should and can do about it? In a nutshell, there are a couple of questions we should ask our selfves and try to answer so we can move forward: * I believe now that we should seriously discuss the relevance of channels. Originally, the idea was good that is providing an abstraction layer for the relay to relay handshake and send/process cells related to the protocol. But, as of now, they are half doing it. There is an important cost in code and maintanance of something that is not properly implemented/finished (channel abstraction) and also something that is unused. An abstraction implemented only for one thing is not really useful except maybe to offer an example for others? But we aren't providing a good example right now imo... That being said, we can spend time fixing the channel subsystem, trying to turn it in a nicer interface, fixing all the issues I've described above (and I suspect there might be more) so the cell scheduler can play nicely with channels. Or, we could rip them off eliminating lots of code and reducing our technical debt. I would like us to think about what we want seriously because that channel subsystem is _complicated_ and very few of us fully understands it afaict. Which would bring us back to (which is btw basically what we have now considering the channel queues are useless): conn inbuf -> circ queue -> conn outbuf If we don't want to get rid of channel, the fixes are non trivial. For starter, we have to decide if we want to keep the channel queue or not and if yes, we need to almost start from square 1 in terms of testing because we would basically introduce a new layer of queuing cells. * Dealing with the DESTROY cell design issue will require a bit more tricky work I think. We must not starve circuit with a DESTROY cell pending to be sent else the other side keeps sending data. But we should also not starve all the circuits because if we ever need to send a gazillion DESTROY cell in priority, we'll make the relay useless (DoS vector). The question is, do we trust our EWMA policy to be wise enough to pick the circuit in a reasonable amount of time so we can flush the DESTROY cell from the circuit queue? Or we really need to bypass or prioritize somehow that cell in order to send them asap in order to avoid load on the network because the other side of the circuit is still sending? * In the short term, we should get rid of Vanilla scheduler because it complefixies a lot the scheduler code by adding uneeded things to channel_t but also bloated the scheduler interface with pointless function pointers for instance. And in my opinion, it is not helping performance the way it is done right now. Cheers! David -- tU3F3BDoF3rwk9y3O+UAixfuPnil7VNkwl0LGe2cdpA=

7 12

UX improvement proposal #2: Encrypted bookmarks for onions
by George Kadianakis 16 Nov '17

16 Nov '17

Hello, here is another onion-related UX improvement proposal. We still don't have a plan for how to concretely fix the onion naming issue, and we recently released next gen onions so names just got bigger. Ideally we should start experimenting with solutions sooner than later (also see https://blog.torproject.org/cooking-onions-names-your-onions) I think a local-solution akin to bookmarks makes sense to start with, so I opened a trac ticket today about encrypted bookmarks on Tor Browser: https://trac.torproject.org/projects/tor/ticket/24310 Please let me know if you are aware of firefox addon projects that do encrypted bookmarks that we could use or start basing our work on. Cheers!

2 1

Understanding the guard/md issue (#21969)
by George Kadianakis 15 Nov '17

15 Nov '17

Hey Tim, just wanted to ask a clarifying question wrt #21969. First of all there are various forms of #21969 (aka the "missing descriptors for some of our primary entry guards" issue). Sometimes it occurs for 10 mins and then goes away, whereas for other people it disables their service permanently (until restart). I call this the hardcore case of #21969. It has happened to me and disabled my service for days, and I've also seen it happen to other people (e.g. dgoulet). So. We have found various md-related bugs and put them as children of #21969. Do you think we have found the bugs that can cause the hardcore case of #21969? That is, is any of these bugs (or a bug combo) capable of permanently disabling an onion service? It seems to me that all the bugs identified so far can only cause #21969 to occur for a few hours before it self-heals itself. IIUC, even the most fundamental bugs like #23862 and #23863 are only temporarily, since eventually one of the dirguards will fetch the missing mds and give them to the client. Do you think that's the case? I'm asking you because I plan to spend some serious time next week on #21969-related issues, and I'd like to prioritize between bug hunting and bug fixing. That is, if the root cause of the hardcore case of #21969 is still out there, I'd like to continue bug hunting until I find it. Let me know what you think! Perhaps you have other ideas here of how we should approach this issue. Cheers!! :) PS: Sending this as an email since our timezones are making it kind hard to synch up on IRC.

2 5

Question on Tor Design (current and maybe past and future)
by ng0 11 Nov '17

11 Nov '17

Hi, can one of the core developers tell me if there exists a current summary of the design of Tor? I've started looking into OR for GNUnet and as far as I know from the papers in the anonbib and one 2013 presentation slides I've read so far, Tor went through many changes over the years. I'm interested in what you use today, where you are heading (or what you are considering for the future) and maybe reasons why past designs were changed. Best regards, ng0 -- GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588 GnuPG: https://dist.ng0.infotropique.org/dist/keys/ WWW: https://ng0.infotropique.org

4 5

Proposal 284: Hidden Service v3 Control Port
by David Goulet 09 Nov '17

09 Nov '17

Hi everyone, Attached is the proposal draft for the hidden service v3 contro port specification. The idea with this proposal is to _only_ extend the current commands and events to v3. Nothing new is added. We can think of more things to add after but for now, I wanted a baseline to start with that is only extending what exists. Any kind of feedbacks is welcome! :) Cheers! David -- Zu3IyL4LcdnKNkQIZqEqaTNUapUEJFdEcN02dPwo5FQ=

7 20

A ContactInfo specification
by nusenu 09 Nov '17

09 Nov '17

Hello, an earlier version of this has been send to tor-relays [1] and I got some feedback from irl and relay operators via ML and github issues. Now I'd like to hear from the tor developer community. Please send your feedback until 2017-10-27. example ContactInfo: https://atlas.torproject.org/#details/1EBFC733CCD952F7F7616EBBA7BC6B8E6F2F0… Overview --------- Tor's ContactInfo descriptor field was primarily intended to contain an email address and PGP key fingerprint but since this field accepts an arbitrary string it has been used for multiple other other purposes (website urls, donation information, bitcoin addresses, ...). Making use of provided information in an automated way is hard since there is no specification on how this string should look like. This is an specification to formalize the ContactInfo string. Motivation (excerpt) ----------- - collect additional (self-reported) relay metrics (for things like atlas and https://nusenu.github.io/OrNetStats ) - exmples: How many use tor's Sandbox/OfflineMasterKey/KIST feature? - This data could provide tor developers with information on how well tested/how much used new features (like Sandboxes) are before changing defaults. The entire document can be found here: https://github.com/nusenu/ContactInfo-Information-Shareing-Specification regards, nusenu [1] https://lists.torproject.org/pipermail/tor-relays/2017-October/013274.html -- https://mastodon.social/@nusenu twitter: @nusenu_

2 6

Re: [tor-dev] tor-dev Digest, Vol 82, Issue 8
by Nathan Kettle 07 Nov '17

07 Nov '17

unsubscribe On Tue, Nov 7, 2017 at 11:12 AM, <tor-dev-request(a)lists.torproject.org> wrote: > Send tor-dev mailing list submissions to > tor-dev(a)lists.torproject.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev > or, via email, send a message with subject or body 'help' to > tor-dev-request(a)lists.torproject.org > > You can reach the person managing the list at > tor-dev-owner(a)lists.torproject.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of tor-dev digest..." > > > Today's Topics: > > 1. Proposal 284: Hidden Service v3 Control Port (David Goulet) > 2. Re: Proposal 284: Hidden Service v3 Control Port (AntiTree) > 3. Re: Proposal 284: Hidden Service v3 Control Port (Damian Johnson) > 4. Re: Proposal 284: Hidden Service v3 Control Port (meejah) > 5. Fwd: Nyx 2.0 Release (Damian Johnson) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 6 Nov 2017 09:59:07 -0500 > From: David Goulet <dgoulet(a)ev0ke.net> > To: tor-dev <tor-dev(a)lists.torproject.org> > Subject: [tor-dev] Proposal 284: Hidden Service v3 Control Port > Message-ID: <20171106145907.p7dm7dmqpnzlqsmj@raoul> > Content-Type: text/plain; charset="utf-8" > > Hi everyone, > > Attached is the proposal draft for the hidden service v3 contro port > specification. > > The idea with this proposal is to _only_ extend the current commands and > events to v3. Nothing new is added. We can think of more things to add > after > but for now, I wanted a baseline to start with that is only extending what > exists. > > Any kind of feedbacks is welcome! :) > > Cheers! > David > > -- > Zu3IyL4LcdnKNkQIZqEqaTNUapUEJFdEcN02dPwo5FQ= >

1 0

Fwd: Nyx 2.0 Release
by Damian Johnson 07 Nov '17

07 Nov '17

Hi tor-dev. Sorry for the cross post but while tor-relays@ is the perfect spot to announce Nyx, tor-dev@ is the traditional place to unveil Stem. I'm pleased to announce Stem 1.6, the accumulation of a full year of improvements for our controller library... https://stem.torproject.org/change_log.html#version-1-6 Besides features such as descriptor creation and ed25519 support the main highlight for this release is performance tuning. Descriptor parsing is ~25% faster and low-level control socket handling got some special attention. Cheers! -Damian ---------- Forwarded message ---------- From: Damian Johnson <atagar(a)torproject.org> Date: Mon, Nov 6, 2017 at 3:41 PM Subject: Nyx 2.0 Release To: tor-relays(a)lists.torproject.org Hi all, after years of being in the works I'm pleased to announce Nyx! A long overdue modernization of arm. http://blog.atagar.com/nyx-release-2-0/ https://nyx.torproject.org/ Even more important for our controller space at large, Nyx is coming hand-in-hand with Stem 1.6. A full year of improvements that include descriptor creation support, ed25519 certificates, performance tuning, and much, much more... https://stem.torproject.org/change_log.html#version-1-6 Cheers! -Damian

1 0

anonbib
by ng0 06 Nov '17

06 Nov '17

Hi, our plan with the bibliography collection of GNUnet is to implement something similar to your/freehaven's anonbib. While running the build and cache update of it from current git HEAD on the anonbib.cfg I noticed a number of outdated and broken links. I'm currently playing with 2 options: re-use anonbib as it is and change the style + some of its content (for us at GNUnet) or write something similar to it. From my perspective option 1 would be the best as we could work on fixing links together, keep the content up-to-date and at the same time keep the duplicate efforts and work down to a minimum. What do you think? -- ng0 GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588 GnuPG: https://dist.ng0.infotropique.org/dist/keys/ https://www.infotropique.org https://ng0.infotropique.org

6 12

Namecoin resolution for Tor Prop279
by Jeremy Rand 05 Nov '17

05 Nov '17

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Tor devs, I've written up some documentation on using Namecoin with Prop279. This can be used for human-meaningful onion service names, as well as standard Namecoin A/AAAA/CNAME records. In theory, this can also be used with arbitrary non-Namecoin naming systems that speak the DNS protocol, although I haven't attempted to do so. Documentation is here: https://www.namecoin.org/docs/tor-resolution/ You can test it with this URL: http://federalistpapers.bit/ Curious what you all think of it -- please feel free to give feedback. Cheers, - -- - -Jeremy Rand Lead Application Engineer at Namecoin Mobile email: jeremyrandmobile(a)airmail.cc Mobile OpenPGP: 2158 0643 C13B B40F B0FD 5854 B007 A32D AB44 3D9C Send non-security-critical things to my Mobile with OpenPGP. Please don't send me unencrypted messages. My business email jeremy(a)veclabs.net is having technical issues at the moment. -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJZ/5k5AAoJELPy0WV4bWVwlSEQAIeF5fiZUBsaiCGyxaDAz97b l4KVyEKAcPOKCvjryKMqGjxHJPfw9uJJY4x0hD4taXYHszfAMBayOs8EkWHd/eFq 3FDAGW2VaS29k9+y+URiIa9gI3H2et+HgwTOxwCk1O/B2l/2TmWWdGnyh8UCr1a1 yibzBRcamaaMyWwDg7gK0ZvN0WPGwCXiFxVC3fT37dKW2Ve9PwvqZEfsZ/i7eVN1 1Vb+o1Q11qe70yx7XQv3KUFpccU1UsIlR/C0hp0ISxAjV5CpMnN8Lu3/E8s0ViFL Z3PKL/cpbRltaPPdv8U0mMLy3KY+/u4yRtl2lmHejsNnCwA9gBrZ3XY9A81ZX/wD QKVBvHf7tO1FlbwKnz0Vb4aY2Ls09vRlXKAQ3zEWtXAT9pofH/dadfI3LYKykeAd +vuDjDOfsGrIVarn0vWqoVwf43P9iDuaDmk0hNDkbBieEcjIOYbuy3xXiMFxMjpF Na0StOMYsI1xSlyUqq6+MT5MRrGmh7zk5d8/PYAEdQM8mPJ6wOzgN98sEoGfZLE+ s1rbeZdMiJqne18G9ulUCh4DPOTyZ4K36oJoUAtr/JSt5IO9vGIrTElbw3BjYm6T 7bTeawoWoX+Lx+BJW+dDq57J2r91wwFZdHAqlbQhWvUi8IynVOVqsZUJbw+g9ioP BAzC5zPauEzD2/awtx/5 =0kqw -----END PGP SIGNATURE-----

1 0

v3 hidden services: inconsistencies between spec and implementation
by inkylatenoth 03 Nov '17

03 Nov '17

Whilst implementing v3 hidden services myself I found some inconsistencies between the specs and the current implementation. I wanted to share these in case someone from the Tor organization wants to update the specs and/or the implementation. # rend-spec-v3.txt ## 2.4 * after decrypting the `superencrypted' object from a descriptor, the resulting document does not end with the NL character. This means that it does not strictly conform to the document meta-format described in section 1.2 of dir-spec.txt. ## A.2 * the blinded key param is defined as H(BLIND_STRING | A | s | B | N). In practice I found that I had to add a null byte after BLIND_STRING in order to reach the same value as the C implementation: param = H(BLIND_STRING | INT_1(\x00) | A | s | B | N) In all other cases where a string constant is used like this (e.g. computing the nonce N above), I found that the trailing null byte is not required. * when clamping the blinding factor, the second bitwise operation is `param[31] &= 127' in the spec but `param[31] &= 63' in the C implementation. These are equivalent in practice when followed by the third operation (`param[31] |= 64'), but it might be nice to use a consistent representation for the benefit of human readers. # 220-ecc-ids-keys.txt # 2.1 * 'The signature is formed by signing the first N-64 bytes of the certificate prefixed with the string "Tor node signing key certificate v1".' I found this to be false; the signatures only validate without the string prefix. ## A.1 * I realized that the certificate types here are outdated. The signing-key extension is listed as type [04], when in rend-spec-v3.txt and the C implementation it is type [08].

3 3

Reporting Relay Bandwidth Less Often
by teor 01 Nov '17

01 Nov '17

Hi all, We would like to make Tor relays report their bandwidth statistics every 24 hours, rather than every 4 hours. We believe that this is a safer interval for clients. It makes it harder to discover the guards of clients that use a lot of bandwidth, particularly onion services. Here's how this kind of guard discovery can happen: * a client repeatedly downloads a large file, or an onion service becomes very popular, or is repeatedly asked for a large file * the traffic statistics for the client's guard increase dramatically in the next 4 hours * an adversary watches the traffic statistics across the whole network, and finds the ones with dramatic increases Increasing the bandwidth statistics interval slows down this attack: * it requires more bandwidth to produce a 24 hour spike * each statistics interval is longer, so it takes more time to be sure of the guard One of the impacts of this change is that relay bandwidth graphs are less detailed. We will encourage relay operators to view detailed graphs using local tools like Nyx or Munin or similar, because this is safer for clients. We are tracking this work in this trac ticket: https://trac.torproject.org/projects/tor/ticket/23856 Tim -- Tim / teor PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------

2 3

Fwd: Fennec now builds with clang instead of gcc
by Nathan Freitas 29 Oct '17

29 Oct '17

For those working on Tor Browser for Android... ----- Original message ----- From: Nathan Froyd <nfroyd(a)mozilla.com> To: "dev-platform" <dev-platform(a)lists.mozilla.org>, mobile-firefox-dev(a)mozilla.org Subject: Fennec now builds with clang instead of gcc Date: Sun, 29 Oct 2017 19:15:50 -0400 Hi all, Bug 1163171 has been merged to mozilla-central, moving our Android builds over to using clang instead of GCC. Google has indicated that the next major NDK release will render GCC unsupported (no bugfixes will be provided), and that it will be removed entirely in the near future. Switching to clang now makes future NDK upgrades easier, provides for better integration with the Android development tools, and brings improvements in performance/code size/standards support. For non-Android platforms, the good news here is that compiling Fennec with clang was the last major blocker for turning on C++14 support. Using clang on Android also opens up the possibility of running our static analyses on Android. If you run into issues, please file bugs blocking bug 1163171. Thanks, -Nathan _______________________________________________ mobile-firefox-dev mailing list mobile-firefox-dev(a)mozilla.org https://mail.mozilla.org/listinfo/mobile-firefox-dev

1 0

[release] CollecTor 1.4.1, Onionoo 4.2-1.6.1
by iwakeh 27 Oct '17

27 Oct '17

Hi there! three new releases are available: https://dist.torproject.org/collector/1.4.1/ https://dist.torproject.org/onionoo/4.2-1.6.1/ These releases enable CollecTor and Onionoo to handle unusually structured (but valid) descriptors (cf. change-logs [0], [1]). For Onionoo this prevented updates, which are running again now as all instances use the new version now. So, all clients like Atlas receive fresh data again. The CollecTor fix was deployed first on all tp.o instances and no data was lost. Thanks to all who waited patiently for Atlas to receive new data! And, to all who kindly reported the problems they noticed! Please direct comments and questions to the metrics-team mailing list [3]. Cheers, iwakeh [0] https://gitweb.torproject.org/collector.git/tree/CHANGELOG.md?h=collector-1… [1] https://gitweb.torproject.org/onionoo.git/tree/CHANGELOG.md?h=onionoo-4.2-1… [2] https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team

1 0

onionoo.tpo stuck at 2017-06-12 12:00?
by nusenu 26 Oct '17

26 Oct '17

Hi, just wanted to let you know that the delta between relays_published and current time is unusually high. https://onionoo.torproject.org/details?limit=0 {"version":"4.0", "relays_published":"2017-06-12 12:00:00", -- https://mastodon.social/@nusenu https://twitter.com/nusenu_

2 4

Re: [tor-dev] https://gitlab.com/torproject
by Hans-Christoph Steiner 25 Oct '17

25 Oct '17

Hans-Christoph Steiner: > > > Nick Mathewson: >> On Wed, Aug 10, 2016 at 5:01 AM, Hans-Christoph Steiner >> <hans(a)guardianproject.info> wrote: >>> >>> FYI I registered https://gitlab.com/torproject to claim the name in the >>> gitlab namespace. I'm happy to hand it over to a more official Tor >>> person if anyone wants. Or just let me know if anyone ever wants to >>> actually use it. >>> >>> I've been using gitlab for a while now with F-Droid, and its quite nice. >>> Guardian Project is going to move our stuff there too. >>> >>> .hc >> >> Sure, I'd be happy to take it. >> >> We've been using gitlab's code review feature a fair bit, and though >> we don't plan to stop self-hosting our repositories, we do rather like >> having a better way to review big branches. > > Hey Nick, > > Sorry I was on vacation when I received this, and it got lost in the > fray. Send me your gitlab username, and I'll turn this over to you. > > .hc > I granted nickm_tor full ownership to https://gitlab.com/torproject I'm happy to answer questions about gitlab, especially gitlab-ci. I use it a lot with https://gitlab.com/fdroid. gitlab-ci is really great. .hc -- PGP fingerprint: EE66 20C7 136B 0D2C 456C 0A4D E9E2 8DEA 00AA 5556 https://pgp.mit.edu/pks/lookup?op=vindex&search=0xE9E28DEA00AA5556

1 0

Request for resource requirement
by Ankit Rathore 24 Oct '17

24 Oct '17

Hello, I am currently working on a project where I require the resource which was originally present at this link https://trac.torproject.org/projects/tor/wiki/sponsors/SponsorD/June2011 The contents of this page are no longer available. It would be really helpful if someone could help me get the original contents of this page Thank you

2 1

Do Tor relays rely on ICMP type 11 (time exceeded / timeout in transit)?
by Igor Mitrofanov 23 Oct '17

23 Oct '17

Hi, On my relays I am dropping any traffic that Tor itself does not rely on. I wonder if I should allow or block incoming and/outgoing ICMP type 11 (time exceeded / timeout in transit)? My host does receive some ICMP type 11 packets, and does seem to send some out, but I am not sure if Tor is the source or destination. Do Tor relays use some 'traceroute'-like mechanism to detect unreachable relays? "netstat -s: ... ICMP input histogram: ... timeout in transit: 1923 ... ICMP output histogram: ... timeout in transit: 1277 " I remember seeing outgoing TCP packets with TTL set to 1 - those were the ones triggering incoming ICMP type 11 packets. Thanks, - Igor

2 2

metrics: collecting circuit build failures from relays to detect network reachability issues and broken relays
by nusenu 19 Oct '17

19 Oct '17

Hello, In [1] David describes his preliminary results from scanning a portion of the tor network to detect connectivity problems (partitions) in the presumed tor mesh network (where every relay is expected to be able to reach every other relay). I believe this information (How far off are we from a complete mesh network?) is crucial for the anonymity properties of the tor network and collecting the data to answer that question should be an integral and continuous part of tor. Actively scanning the tor network for connectivity issues is resource intensive. What if we could collect this data without actively scanning for it? This could be achieved by collecting reachability information passively by relays themselves and uploaded via extra-info descriptors. This data could help reduce the overall scanning effort. So instead of scanning _all_ relays we can reduce the scanning to relays where some threshold of relays consistently said that they are hardly reachable. (to confirm their measurements) Due to the mutual nature of the information collection, single or a minor number of lying relays would not be a big issue. relays could: - aggregate reachability issues over the past 24 hours (or week?) per outbound destination relay (which relay failed what percentage of circuit build attempts) - if a relay failed more than - some threshold value - check if the relay in question: - had his uptime reset during that timeframe (reachability problem expected) - dropped out of consensus during that timeframe - only include it in uploaded data if it didn't drop out of consensus and did not reset his uptime (using a week instead of a day would help with reducing the amount of data that we need to process after collecting it) In addition to that passive collection, relays - when idle/not overloaded - could actively attempt to create circuits to measure their reachability to relays for which they did not collect any any passive data. (maybe limit it to fast and stable destination relays) To reduce the load on the tor network we could limit the active outbound connection tests based on relay flags: - Guard-only relays are more likely to establish outbound connections to non-exit relays - middle-only relays (no guard and exit flag) are more likely to establish connections to exits than to guards - exits are more likely to get inbound connections from non-guards (maybe skip active probes from exits since they are a bottleneck already?) Reachability issues could also be displayed to the relay operator warning them about the potential problem via log entries. So they could actively work on debugging problems themselfes. There is no doubt that this information is also valuable for (powerful) adversaries (it could help them reduce their effort when they know weak spots in the network). So you would have to decide what data you collect an what you would publish (collector.tpo) - even if that means I might never get to see that interesting data ;) This is a medium/long term goal, with the usual steps: - proposal - implementation (- deployment - data collection - the amount of data could be huge - data analysis) If there is a consensus that this makes sense and if someone would actually implement it I would be happy to work/help on a proposal. This entire idea would be an opt-in torrc setting at the beginning and a opt-out feature once we are more confident about its implications and safety. Please let me know what you think about this idea. regards, nusenu [1] https://lists.torproject.org/pipermail/tor-project/2017-October/001492.html related trac tickets: https://trac.torproject.org/projects/tor/ticket/12131 https://trac.torproject.org/projects/tor/ticket/19068 -- https://mastodon.social/@nusenu twitter: @nusenu_

3 2