- tor-dev - lists.torproject.org

How to integrate an external name resolver into Tor
by Nick Mathewson 08 Aug '16

08 Aug '16

Hi, all! I've seen a couple of emails from people looking into new ways to do naming for onion services. That's great! Before anybody gets too far, I'd like to send this quick note to let you know that integrating stuff like this into Tor is actually easier than you think. Here's how you do it, using a Tor controller. (See control-spec.txt for protocol documentation. Also see one of the several friendly libraries, like steam, that exist to interface with Tor over this protocol.) First, you set the Tor option "__LeaveStreamsUnattached". This tells Tor that it shouldn't try to handle new client requests immediately, but it should instead let the controller take responsibility. In the controller, you make sure that you are watching STREAM events so that you find out about new streams. Whenever there's a new stream, you check its address. If the address is one that you don't want to rewrite, you just call ATTACHSTREAM on it, with a circuit ID of 0. (The 0 means "Tor, you figure out how to attach this one.". Otherwise, you do whatever magic dance you do in order to find out the real address for the stream. If the lookup operation is successful, you say "REDIRECTSTREAM (stream ID) (new address". And then you ATTACHSTREAM as above. If the lookup operation fails, you call "CLOSESTREAM (stream ID) 2". (The 2 means "resolve failed". And that's it for the Tor integration! All you need to do now is figure out how the name lookup works. cheers, -- Nick

5 7

Re: [tor-dev] Onioncat and Prop224
by str4d 08 Aug '16

08 Aug '16

On 08/08/16 01:22, Jeremy Rand wrote: >> And most certainly any external layer must be capable of having >> nodes binding natively in the Tor / I2P / etc networks, and >> preferably being strictly private entirely within them (like how >> private Tor / I2P / Bitcoin nets can be deployed by generating >> new authorities, keys, genesis, etc). Outproxy is not an option. > > Bitcoin and Namecoin nodes can be exposed as Tor hidden services (I > believe there's even code in there now for automatically configuring > such a hidden service via the Tor control port). I don't think there's > any built-in support for I2P (unless something was added since I last > looked), Not yet, but there's an open issue for adding I2P support (alongside support for Tor's HS2.0) [0]. See also an old Bitcoin fork with I2P support [1], and other relevant links in Zcash's I2P issue [2]. Any native I2P support would automatically set up the I2P Destinations, because that's how the SAM API operates [3]. > but if I2P can expose a local TCP service as an I2P service, Yes. > and if I2P allows such TCP services to be connected to by exposing a > SOCKS interface, Yes. > I'm guessing that it should mostly work (famous last > words). The only thing that would break that way is peer exchange: > Bitcoin and Namecoin nodes normally share peer addresses, and although > this works just fine with Tor hidden services, I don't think it would > work out of the box with I2P (unless someone's added that since I last > looked). The issue here is that I2P addresses (even B32s) are longer than the internal message fields for sharing addresses. Tor's HS2.0 will suffer from the same issue, because their proposed addresses are just as long as I2P's B32s. > > Although it is possible to deploy private Bitcoin and Namecoin networks > (I did it at a workshop I gave at DWS so that I could mine blocks on > demand), usually people do this for testing or demonstration purposes, > not for real-world applications. The reason is that blockchains' > security assumptions include the assumption that they have a reasonably > high hashrate from a reasonably diverse set of miners. An > Onioncat-specific Namecoin network would probably have a very low > hashrate compared to the main Namecoin network, which means that it > would be much easier to perform a 51% attack. Merged mining could, > hypothetically, raise the hashrate, except that to do merged mining, the > miners would have to be connected to the public Bitcoin network. In > addition, it's usually very unprofitable to mine Bitcoin or Namecoin > over Tor or I2P, because the additional latency causes much higher > orphan rates. (Most Bitcoin and Namecoin miners spend considerable > effort obtaining low-latency connections for relaying blocks to and from > other miners.) > > Might I ask what the motivation is for having a completely separate > Namecoin network inside Tor/I2P, that can't be satisfied by binding > participants' nodes to Tor/I2P hidden services while allowing some users > who don't want or need anonymity to relay transactions and blocks > between clearnet and Tor/I2P? One potential issue is that if the user's endpoint isn't configured correctly, Namecoin names that resolve to clearnet addresses could deanonymise the user (checks at the browser or app level wouldn't help here because all it would see is an IPv6-style address IIUC). It's not unavoidable, but a point to be wary of when considering the overall integration strategy. Using the global Namecoin blockchain also puts the bridging nodes into a position of importance and trust, that can only be lessened by greatly increasing the number of bridging nodes. This is more of a concern for I2P than Tor, as I2P-only peers can only reach the bridging nodes, whereas Tor-only peers can connect to clearnet Namecoin nodes too (although as with Bitcoin there is the issue of exit node traffic manipulation). str4d [0] https://github.com/bitcoin/bitcoin/issues/2091 [1] https://github.com/VirtualDestructor/bitcoin-qt-i2p [2] https://github.com/zcash/zcash/issues/1111 [3] https://geti2p.net/en/docs/api/samv3

1 0

CollecTor will soon serve descriptors that don't pass metrics-lib's parser
by Karsten Loesing 05 Aug '16

05 Aug '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello CollecTor data consumers, note: if you don't know what CollecTor is or if you have never used CollecTor data before, you can safely stop reading now and enjoy your Friday. But in case you're now curious what CollecTor is, feel free to take a look: https://collector.torproject.org/ So, we're about to remove a sanity check in CollecTor where we'd feed relay descriptors into metrics-lib and only serve them if they pass the parsing step. This step has so far eliminated descriptors with non-ASCII characters in unexpected places, like: dirreq-v3-ips us=192,ru=144,de=136,??=112,fr=64,gb=56,br=40,it=40,es=24,in=24,jp=24,pl=24,ar=16,at=16,au=16,ca=16,ch=16,cn=16,cz=16,ir=16,kr=16,mx=16,nl=16,ua=16,za=16,^Wg=8,ae=8,ao=8,be=8,bg=8,bj=8,bo=8,bs=8,bw=8,by=8,cd=8,cl=8,cm=8,co=8,cr=8,cu=8,dd=8,dk=8,du=8,dz=8,ec=8,eg=8,fi=8,gf=8,gh=8,gr=8,gt=8,hk=8,hn=8,hr=8,hu=8,id=8,ie=8,il=8,i�=8,ke=8,kz=8,lk=8,lt=8,lv=8,ly=8,ma=8,md=8,mk=8,mu=8,my=8,nc=8,ng=8,ni=8,no=8,nz=8,pa=8,pe=8,ph=8,pk=8,pr=8,pt=8,ro=8,rs=8,se=8,sg=8,sk=8,sn=8,sy=8,th=8,tn=8,tr=8,tw=8,ug=8,ve=8,vn=8,�p=8 Very soon (next week?), CollecTor will serve those descriptors, just like the Tor directory authorities serve them. If you're obtaining and processing descriptors from CollecTor, you'll somehow have to handle them. Of course, if you're using metrics-lib, nothing changes, and I hear Stem discards descriptors with non-ASCII characters in the line above, too. If you want, you can find more details on the ticket: https://trac.torproject.org/projects/tor/ticket/19170 All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJXpGOhAAoJEC3ESO/4X7XBVmoIALbd2AmUEMX6DrzPsEvPLItR YP3ySblh9knkc5NvrGxO/hcYABjX4j5k/smrAotvcT+5qUVyQiXbmD4kO0Rkq5i9 nC4+MkBiuigj4eeXi+Y57oBVuXidJNqFoMFmNpeXxU/csiwsVXsWX/UTnoiox6Cg 5KKnGpr9lr4IQt74Jw1gIYAnhXE6ga8JyWv8dcv0dtOdChl1zsjjCOgFuFpPlFJl ycFAhX0jM+iOpVV8m1WTA0r1PAAAndtKBotCuhGizdfNSRcPiA3hhUFewNxbgY7N b/fvIRQRumNCv2uAYve25iIuL+0vE197rUO9vY8IlRB/po/qDQu8AVp6HjrlZCU= =nzBR -----END PGP SIGNATURE-----

1 0

Tor and Namecoin
by Jeremy Rand 04 Aug '16

04 Aug '16

Hello Tor devs, Namecoin is interested in collaboration with Tor in relation to human-readable .onion names; I'm reaching out to see how open the Tor community would be to this, and to get feedback on how exactly the integration might work. The new hidden service spec is going to substantially increase the length of .onion names, which presents usability concerns. Namecoin provides a way to resolve a human-readable .bit name to a .onion name. Another benefit of Namecoin is that it provides a way to lookup TLS fingerprints for clearnet .bit sites, which reduces the risk of MITM attacks on clearnet websites from malicious or compromised CA's. I had the pleasure of meeting Mike Perry at the Decentralized Web Summit at the Internet Archive in June; I talked to him about Namecoin's rough plans and he suggested I post here. I understand that Riccardo Spagni from Monero discussed this topic as well with Roger Dingledine at the Security in Times of Surveillance conference at Ei_PSI. The two most major concerns that I expect would be brought up involve anonymity and blockchain size. Here's how we plan to deal with these issues: Namecoin already provides location-anonymity for name registrations assuming that it's routed via Tor. It's also necessary to broadcast transactions for different names to different peers, which isn't coded yet, but this is just coding work rather than an engineering challenge -- a usable workaround today is running multiple Namecoin wallets. The more interesting challenge is blockchain anonymity for registrations, due to the linkability required for blockchain validation. An important point here is that transactions for a given name are inherently linkable to each other, and that this isn't problematic. The problem would come when multiple names are linked together, or when a name is linked with currency transactions. The solution I've come up with is to use atomic cross-chain trades, which let a user buy namecoins using a cryptocurrency that is designed to provide anonymity (such as Monero or Zcash, both of which have cryptographic proofs of anonymity, given a certain anonymity set and security assumptions). The user would use an anonymous cryptocurrency to buy a small amount of namecoins (enough to register a single name and keep it renewed for a while). If the user wanted to register another name, she would perform another atomic cross-chain trade, receiving namecoins that are not linked to the namecoins obtained for the first name. As long as those namecoins are not mixed by the wallet software, the names remain unlinked. Many users won't want to download the full Namecoin blockchain (around 3 GB at the moment). I have a proof-of-concept SPV-based Namecoin name lookup client working as of early June. I just got a large part of that code upstreamed into libdohj, and I'm working on getting the rest upstreamed and released. It's in Java (based on BitcoinJ), so it's not subject to the memory safety concerns that C/C++ code are. The SPV name lookups are implemented in 3 ways, depending on the user's needs: Option A: 1. Block headers are synced over the Namecoin P2P network. (Over clearnet this takes about 5 minutes the first time it runs.) 2. An index mapping unexpired block heights to block hashes is constructed, so that lookups can be done quickly. (This occurs when the SPV client starts, after syncup has completed; it's fast enough that I haven't found a need to benchmark it.) 3. When a name lookup request is received, the client asks a remote API server for the height of the last update of the name. 4. The client looks up the block hash of that height from its index, and requests that block over the P2P network. 5. The client verifies that the received block matches the correct hash and that the block follows Namecoin rules (e.g. verifying the merkle root). 6. The client looks through the transactions in the block until it finds the one that updates the name. 7. The client retrieves the value of the name from that transaction, and returns it to the user. Option B: 1 through 3. Same as Option A. 4. The API server also provides the full content of the transaction, as well as a merkle proof of inclusion in the block. 5. The client verifies that the merkle proof links the hash of the provided transaction to the merkle root of the block header with the given height. 6. The client retrieves the value of the name from the provided transaction, and returns it to the user. Option C: 1. Block headers are synced over the Namecoin P2P network, as well as full blocks for the past year (meaning that all full blocks that contain unexpired name data will be synced). (Over clearnet this takes about 10 minutes the first time it runs.) 2. An index mapping names to transactions is constructed as the full blocks are downloaded. (This uses LevelDB.) 3. When a name lookup request is received, the client looks up the transaction in the LevelDB index. 4. The client retrieves the value of the name from that transaction, and returns it to the user. For Options A and B, if the API server is malicious, it can do any of the following: 1. Falsely claim that the name doesn't exist. 2. Provide outdated name data that is less than 36000 blocks old (the expiration period for Namecoin). (Option C is not vulnerable to either of those attacks.) If multiple API servers are consulted, and they return different results, it is easy to tell which is lying (although I haven't implemented any such logic yet). The API server cannot do any of the following: 1. Provide name data that isn't from the blockchain with the most work. 2. Provide name data that is more than 36000 blocks old (the expiration period for Namecoin). The reason an API server is used in Options A and B instead of the P2P network, is that the P2P network is unauthenticated and easy to Sybil. The P2P network is great for getting data that is independently verifiable (e.g. block headers and contents of blocks), but it's unwise to rely on the P2P network to get unverifiable data such as a block height of a name. An API server is authenticated (currently via CA-based TLS, but a cert pin or PGP signing is certainly doable), which reduces the possible points of attack. This is analogous to why Tor uses centralized directory authorities -- authenticated trust points are harder to Sybil. (We do have longer term plans to introduce a way for SPV clients to get the latest transaction associated with a name, without using an API server or needing to download any full blocks, but that's out of scope of this email.) Options A and B do reveal to the API server which name is being looked up. If mode A is used, it also reveals to a P2P peer which block height is being looked up (which narrows the set of names by a factor of ~36000). Therefore, Tor stream isolation should be used in such cases. (That's not implemented yet.) Option C doesn't generate any network traffic on lookups, so it doesn't reveal anything. In my testing, an SPV-based name lookup using Option A takes around 650 milliseconds (over clearnet). The vast majority of this is latency to the API server (the server I'm testing with is on a low-budget hosting plan). The portion consisting of a block retrieval over P2P takes around 98 milliseconds (although it varies by block size). Option C takes around 4 milliseconds. The storage overhead of Option C's LevelDB database is around 400 MB right now, although I believe it's feasible to reduce this significantly. There are a few options I can think of for integrating this with Tor for .onion naming. One would be to modify OnioNS to call the Namecoin SPV client. This would concern me because OnioNS is in C++, which introduces the risk of memory safety vulnerabilities. Another would be to use an intermediate proxy like Yawning's or-ctl-filter. A third option would be to try to get external name resolution implemented in Tor itself, which I believe Jeff Burdges has suggested in the past. If Option A or B is used, any solution would need to pass the stream isolation info to the SPV client. Integrating this with Tor Browser for TLS certificate validation might involve a Firefox patch. There are tricks that can be done with the CertDB and SiteSecurityService XPCOM interfaces that will do the job without Firefox patches, but XPCOM is being phased out by Mozilla in favor of WebExtensions, and I'm unaware of any equivalent features in WebExtensions. (Also, it's unclear to me whether CertDB and SiteSecurityService would introduce isolation issues -- I can't think of any obvious attacks, but I haven't thought very hard about it.) I'm trying to engage with Mozilla to see if we can work out a WebExtensions feature for this, but nothing conclusive has happened on that front yet. On the subject of reproducible builds, I've never tried to build Java code in Gitian, so I'm not certain how difficult it's going to be. Since Android uses Java, maybe the Guardian Project devs would have some insight into the best way to do it. One of the Namecoin developers (Joseph Bisch) is really good with reproducible builds (you probably know him since he's the author of the Debian guest support in Gitian), so I'm reasonably confident that a way to do it can be found. I'd love to hear feedback on all of this. Cheers, -Jeremy Rand Lead Application Engineer of Namecoin

4 7

Re: [tor-dev] [rfc] Usability Improvements for Atlas
by tordev123＠Safe-mail.net 03 Aug '16

03 Aug '16

IMO, your improvements make the user experience much worse. The current version is far more space efficient, your new versions wastes giant amounts of screen space. > * There is no super-wide list of relays that doesn't fit even into Tor > Browser window (panels instead with upstatus indication) The important stuff does fit, usually no scrolling is required. > * Updated navigation bar with new version of Bootstrap Takes up more space with no usability improvement. > * Stretching of the graphs They disappear with TBB set to high security. > P.S. Some of functions like list sorting, display range, detailed search > and maybe more are broken/gone now. I'm wondering if these functions are > useful? Yes, they are useful.

1 0

Usability Improvements for Atlas (was Re: Globe is now retired)
by Iain R. Learmonth 02 Aug '16

02 Aug '16

Hi All, On 29/06/16 09:56, Karsten Loesing wrote: > tl;dr: Globe is now retired in favor of an improved Atlas. Atlas is improved, but I'd like to improve it further. I've been tackling #6787 looking to improve the homepage and make Atlas easier to use for someone that isn't already a Tor expert. The changes I'm proposing in my fix for #6787 [1] do make some noticeable UI changes and I just want to gather some opinions on whether or not Atlas users feel that this is taking it in the right direction. The "Top 10 Relays" page was added to Atlas in order to make sure that functionality was not lost when shutting down globe. (See #5430 [2]) In my proposed patch, I'm also adding an "Authorities" page and adding text to both of these pages to explain what you're seeing. These are linked from the homepage, and from the top navbar. I'm also moving the introduction and explanation of the Atlas features from the About page to the homepage and introducing a new Help page that explains the details that are found in the relay details pages. You can find a live example of this patch at: https://irl.github.io/atlas/#/ and a full list of changes are detailed in #6787. Please let me know if you can see any reason to not merge this, or if you have any suggestions for small changes before merging. Thanks, Iain. [1]: https://trac.torproject.org/projects/tor/ticket/5430 [2]: https://trac.torproject.org/projects/tor/ticket/6787

5 5

Tor Browser and Privoxy
by Paulo Roberto 02 Aug '16

02 Aug '16

Hello, By default the Tor Browser Bundle connects through SOCKS5 to the TOR daemon. Is there any advantage today to configure Privoxy to use TOR and configure the Tor Browser in the Tor Network Settings to use proxy access type HTTP/HTTPS pointing to Privoxy? Thanks in advance. Paulo Roberto.

3 3

prop224: zero bits in addresses
by teor 02 Aug '16

02 Aug '16

Hi all, At the Montreal hidden service hackfest, we discussed another scheme shortening prop224 .onion addresses. The only drawback is that it's exponentially difficult, so it can only chop off a few characters. I'm describing it here because it has other useful properties, and can be used as: 1) an address versioning scheme, and/or 2) a configurable proof-of-work threshold for generating valid addresses. The basic scheme works like this: (those of you familiar with bitcoin might recognise it) A) Valid .onion address hashes must start with a certain, hard-coded number of 0 bits, followed by a 1 bit. (We include the 1 bit, so that addresses with additional 0 bits can be used by a later version of the protocol.) B) When generating a private key, the hidden service must discard any keys that yield .onion addresses that don't fit this bit pattern. C) When creating a textual .onion address, Tor removes the hard-coded bits from the address. D) When parsing a textual .onion address, Tor adds the hard-coded bits from the address. Otherwise, the protocol proceeds as it would in prop224. This kind of scheme is mentioned as a TODO in the existing proposal: [TODO: Alternatively, we could require that the first bit of the master key always be zero, and use a 51-byte encoding. Or we could require that the first two bits be zero, and use a 51-byte encoding and reserve the first bit. Or we could require that the first nine bits, or ten bits be zero, etc.] In my very basic tests using Yawning's horse25519 on a laptop, it seems that 15 bits takes less than a second, but 20 bits can take from a few seconds to around a minute. Prop 224 .onion addresses already have 4 unused bits, so we could easily chop off 3-4 characters using this scheme, with only a small amount of computation the first time the hidden service runs. Without a scheme like this, it's possible to generate hundreds of thousands of valid .onion addresses per minute on most recent processors. I'm not sure if this is a problem or not. So we have to decide whether the added complexity is worth having addresses that are 3-4 characters shorter. I'm not sure that it is, unless we like the idea of making it harder to generate valid .onion addresses, or unless we want to allow for extensible future versioning of .onion addresses, beyond the existing 4 unused bits. Tim Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmmp: teor at torproject dot org

2 1

metrics-lib/DescripTor 1.3.1 is released
by Karsten Loesing 01 Aug '16

01 Aug '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello devs, we just released metrics-lib/DescripTor 1.3.1: https://dist.torproject.org/descriptor/1.3.1/ - From the change log: # Changes in version 1.3.1 - 2016-08-01 * Medium changes - Adapt to CollecTor's new date format to make DescriptorCollector work again. Happy descriptor collecting! All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJXn0XrAAoJEC3ESO/4X7XB+q8IAMtu1b/mhvUvS7nlETa2ejCF BDhHNMK2v23pH0usku2AaZuzc0AkG4r0d3Nyb0jDRH5J/oNu2kDCUXIbDqJxwfX7 y5UlNaZgLTcYV7eJjWAqbvAmCBQMpEYJLhhtPqO81DgZpNbMP08doAgis30TPfiE KRUrKTIJxcDVQLfvsSFOOZj3PDjkQScWtW+icBF2PdD1u62/Dhv1QEJWzBTKH2xq etmc5VsNAir/mV2/le68CsIlSNuwGH9yU4xRlZ8QEb+MOnnS76p3onnsdqmHhytJ V0hXlViWNRjFd0EzcuCp4tvkWK3//xBC19AjQkwuRCOCbbeuOpQLZI00BJV9NCg= =0qss -----END PGP SIGNATURE-----

1 0

Re: [tor-dev] prop224: Ditching key blinding for shorter onion addresses
by Nick Mathewson 30 Jul '16

30 Jul '16

whoops, didn't reply-all. On Sat, Jul 30, 2016 at 1:00 PM, Nick Mathewson <nickm(a)alum.mit.edu> wrote: > On Sat, Jul 30, 2016 at 9:32 AM, George Kadianakis <desnacked(a)riseup.net> wrote: > [...] >> >> Please let me know in what way this scheme is completely broken! > > I think it has the same problem as the last one: there does not appear > to be a way to keep an attacker from putting their own record in place > using your query_key, and thereby performing a DoS on your hidden > service. > > > -- > Nick

1 0