- tor-dev - lists.torproject.org

CollecTor will soon serve descriptors that don't pass metrics-lib's parser
by Karsten Loesing 05 Aug '16

05 Aug '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello CollecTor data consumers, note: if you don't know what CollecTor is or if you have never used CollecTor data before, you can safely stop reading now and enjoy your Friday. But in case you're now curious what CollecTor is, feel free to take a look: https://collector.torproject.org/ So, we're about to remove a sanity check in CollecTor where we'd feed relay descriptors into metrics-lib and only serve them if they pass the parsing step. This step has so far eliminated descriptors with non-ASCII characters in unexpected places, like: dirreq-v3-ips us=192,ru=144,de=136,??=112,fr=64,gb=56,br=40,it=40,es=24,in=24,jp=24,pl=24,ar=16,at=16,au=16,ca=16,ch=16,cn=16,cz=16,ir=16,kr=16,mx=16,nl=16,ua=16,za=16,^Wg=8,ae=8,ao=8,be=8,bg=8,bj=8,bo=8,bs=8,bw=8,by=8,cd=8,cl=8,cm=8,co=8,cr=8,cu=8,dd=8,dk=8,du=8,dz=8,ec=8,eg=8,fi=8,gf=8,gh=8,gr=8,gt=8,hk=8,hn=8,hr=8,hu=8,id=8,ie=8,il=8,i�=8,ke=8,kz=8,lk=8,lt=8,lv=8,ly=8,ma=8,md=8,mk=8,mu=8,my=8,nc=8,ng=8,ni=8,no=8,nz=8,pa=8,pe=8,ph=8,pk=8,pr=8,pt=8,ro=8,rs=8,se=8,sg=8,sk=8,sn=8,sy=8,th=8,tn=8,tr=8,tw=8,ug=8,ve=8,vn=8,�p=8 Very soon (next week?), CollecTor will serve those descriptors, just like the Tor directory authorities serve them. If you're obtaining and processing descriptors from CollecTor, you'll somehow have to handle them. Of course, if you're using metrics-lib, nothing changes, and I hear Stem discards descriptors with non-ASCII characters in the line above, too. If you want, you can find more details on the ticket: https://trac.torproject.org/projects/tor/ticket/19170 All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJXpGOhAAoJEC3ESO/4X7XBVmoIALbd2AmUEMX6DrzPsEvPLItR YP3ySblh9knkc5NvrGxO/hcYABjX4j5k/smrAotvcT+5qUVyQiXbmD4kO0Rkq5i9 nC4+MkBiuigj4eeXi+Y57oBVuXidJNqFoMFmNpeXxU/csiwsVXsWX/UTnoiox6Cg 5KKnGpr9lr4IQt74Jw1gIYAnhXE6ga8JyWv8dcv0dtOdChl1zsjjCOgFuFpPlFJl ycFAhX0jM+iOpVV8m1WTA0r1PAAAndtKBotCuhGizdfNSRcPiA3hhUFewNxbgY7N b/fvIRQRumNCv2uAYve25iIuL+0vE197rUO9vY8IlRB/po/qDQu8AVp6HjrlZCU= =nzBR -----END PGP SIGNATURE-----

1 0

Tor and Namecoin
by Jeremy Rand 05 Aug '16

05 Aug '16

Hello Tor devs, Namecoin is interested in collaboration with Tor in relation to human-readable .onion names; I'm reaching out to see how open the Tor community would be to this, and to get feedback on how exactly the integration might work. The new hidden service spec is going to substantially increase the length of .onion names, which presents usability concerns. Namecoin provides a way to resolve a human-readable .bit name to a .onion name. Another benefit of Namecoin is that it provides a way to lookup TLS fingerprints for clearnet .bit sites, which reduces the risk of MITM attacks on clearnet websites from malicious or compromised CA's. I had the pleasure of meeting Mike Perry at the Decentralized Web Summit at the Internet Archive in June; I talked to him about Namecoin's rough plans and he suggested I post here. I understand that Riccardo Spagni from Monero discussed this topic as well with Roger Dingledine at the Security in Times of Surveillance conference at Ei_PSI. The two most major concerns that I expect would be brought up involve anonymity and blockchain size. Here's how we plan to deal with these issues: Namecoin already provides location-anonymity for name registrations assuming that it's routed via Tor. It's also necessary to broadcast transactions for different names to different peers, which isn't coded yet, but this is just coding work rather than an engineering challenge -- a usable workaround today is running multiple Namecoin wallets. The more interesting challenge is blockchain anonymity for registrations, due to the linkability required for blockchain validation. An important point here is that transactions for a given name are inherently linkable to each other, and that this isn't problematic. The problem would come when multiple names are linked together, or when a name is linked with currency transactions. The solution I've come up with is to use atomic cross-chain trades, which let a user buy namecoins using a cryptocurrency that is designed to provide anonymity (such as Monero or Zcash, both of which have cryptographic proofs of anonymity, given a certain anonymity set and security assumptions). The user would use an anonymous cryptocurrency to buy a small amount of namecoins (enough to register a single name and keep it renewed for a while). If the user wanted to register another name, she would perform another atomic cross-chain trade, receiving namecoins that are not linked to the namecoins obtained for the first name. As long as those namecoins are not mixed by the wallet software, the names remain unlinked. Many users won't want to download the full Namecoin blockchain (around 3 GB at the moment). I have a proof-of-concept SPV-based Namecoin name lookup client working as of early June. I just got a large part of that code upstreamed into libdohj, and I'm working on getting the rest upstreamed and released. It's in Java (based on BitcoinJ), so it's not subject to the memory safety concerns that C/C++ code are. The SPV name lookups are implemented in 3 ways, depending on the user's needs: Option A: 1. Block headers are synced over the Namecoin P2P network. (Over clearnet this takes about 5 minutes the first time it runs.) 2. An index mapping unexpired block heights to block hashes is constructed, so that lookups can be done quickly. (This occurs when the SPV client starts, after syncup has completed; it's fast enough that I haven't found a need to benchmark it.) 3. When a name lookup request is received, the client asks a remote API server for the height of the last update of the name. 4. The client looks up the block hash of that height from its index, and requests that block over the P2P network. 5. The client verifies that the received block matches the correct hash and that the block follows Namecoin rules (e.g. verifying the merkle root). 6. The client looks through the transactions in the block until it finds the one that updates the name. 7. The client retrieves the value of the name from that transaction, and returns it to the user. Option B: 1 through 3. Same as Option A. 4. The API server also provides the full content of the transaction, as well as a merkle proof of inclusion in the block. 5. The client verifies that the merkle proof links the hash of the provided transaction to the merkle root of the block header with the given height. 6. The client retrieves the value of the name from the provided transaction, and returns it to the user. Option C: 1. Block headers are synced over the Namecoin P2P network, as well as full blocks for the past year (meaning that all full blocks that contain unexpired name data will be synced). (Over clearnet this takes about 10 minutes the first time it runs.) 2. An index mapping names to transactions is constructed as the full blocks are downloaded. (This uses LevelDB.) 3. When a name lookup request is received, the client looks up the transaction in the LevelDB index. 4. The client retrieves the value of the name from that transaction, and returns it to the user. For Options A and B, if the API server is malicious, it can do any of the following: 1. Falsely claim that the name doesn't exist. 2. Provide outdated name data that is less than 36000 blocks old (the expiration period for Namecoin). (Option C is not vulnerable to either of those attacks.) If multiple API servers are consulted, and they return different results, it is easy to tell which is lying (although I haven't implemented any such logic yet). The API server cannot do any of the following: 1. Provide name data that isn't from the blockchain with the most work. 2. Provide name data that is more than 36000 blocks old (the expiration period for Namecoin). The reason an API server is used in Options A and B instead of the P2P network, is that the P2P network is unauthenticated and easy to Sybil. The P2P network is great for getting data that is independently verifiable (e.g. block headers and contents of blocks), but it's unwise to rely on the P2P network to get unverifiable data such as a block height of a name. An API server is authenticated (currently via CA-based TLS, but a cert pin or PGP signing is certainly doable), which reduces the possible points of attack. This is analogous to why Tor uses centralized directory authorities -- authenticated trust points are harder to Sybil. (We do have longer term plans to introduce a way for SPV clients to get the latest transaction associated with a name, without using an API server or needing to download any full blocks, but that's out of scope of this email.) Options A and B do reveal to the API server which name is being looked up. If mode A is used, it also reveals to a P2P peer which block height is being looked up (which narrows the set of names by a factor of ~36000). Therefore, Tor stream isolation should be used in such cases. (That's not implemented yet.) Option C doesn't generate any network traffic on lookups, so it doesn't reveal anything. In my testing, an SPV-based name lookup using Option A takes around 650 milliseconds (over clearnet). The vast majority of this is latency to the API server (the server I'm testing with is on a low-budget hosting plan). The portion consisting of a block retrieval over P2P takes around 98 milliseconds (although it varies by block size). Option C takes around 4 milliseconds. The storage overhead of Option C's LevelDB database is around 400 MB right now, although I believe it's feasible to reduce this significantly. There are a few options I can think of for integrating this with Tor for .onion naming. One would be to modify OnioNS to call the Namecoin SPV client. This would concern me because OnioNS is in C++, which introduces the risk of memory safety vulnerabilities. Another would be to use an intermediate proxy like Yawning's or-ctl-filter. A third option would be to try to get external name resolution implemented in Tor itself, which I believe Jeff Burdges has suggested in the past. If Option A or B is used, any solution would need to pass the stream isolation info to the SPV client. Integrating this with Tor Browser for TLS certificate validation might involve a Firefox patch. There are tricks that can be done with the CertDB and SiteSecurityService XPCOM interfaces that will do the job without Firefox patches, but XPCOM is being phased out by Mozilla in favor of WebExtensions, and I'm unaware of any equivalent features in WebExtensions. (Also, it's unclear to me whether CertDB and SiteSecurityService would introduce isolation issues -- I can't think of any obvious attacks, but I haven't thought very hard about it.) I'm trying to engage with Mozilla to see if we can work out a WebExtensions feature for this, but nothing conclusive has happened on that front yet. On the subject of reproducible builds, I've never tried to build Java code in Gitian, so I'm not certain how difficult it's going to be. Since Android uses Java, maybe the Guardian Project devs would have some insight into the best way to do it. One of the Namecoin developers (Joseph Bisch) is really good with reproducible builds (you probably know him since he's the author of the Debian guest support in Gitian), so I'm reasonably confident that a way to do it can be found. I'd love to hear feedback on all of this. Cheers, -Jeremy Rand Lead Application Engineer of Namecoin

4 7

Re: [tor-dev] [rfc] Usability Improvements for Atlas
by tordev123＠Safe-mail.net 04 Aug '16

04 Aug '16

IMO, your improvements make the user experience much worse. The current version is far more space efficient, your new versions wastes giant amounts of screen space. > * There is no super-wide list of relays that doesn't fit even into Tor > Browser window (panels instead with upstatus indication) The important stuff does fit, usually no scrolling is required. > * Updated navigation bar with new version of Bootstrap Takes up more space with no usability improvement. > * Stretching of the graphs They disappear with TBB set to high security. > P.S. Some of functions like list sorting, display range, detailed search > and maybe more are broken/gone now. I'm wondering if these functions are > useful? Yes, they are useful.

1 0

Usability Improvements for Atlas (was Re: Globe is now retired)
by Iain R. Learmonth 03 Aug '16

03 Aug '16

Hi All, On 29/06/16 09:56, Karsten Loesing wrote: > tl;dr: Globe is now retired in favor of an improved Atlas. Atlas is improved, but I'd like to improve it further. I've been tackling #6787 looking to improve the homepage and make Atlas easier to use for someone that isn't already a Tor expert. The changes I'm proposing in my fix for #6787 [1] do make some noticeable UI changes and I just want to gather some opinions on whether or not Atlas users feel that this is taking it in the right direction. The "Top 10 Relays" page was added to Atlas in order to make sure that functionality was not lost when shutting down globe. (See #5430 [2]) In my proposed patch, I'm also adding an "Authorities" page and adding text to both of these pages to explain what you're seeing. These are linked from the homepage, and from the top navbar. I'm also moving the introduction and explanation of the Atlas features from the About page to the homepage and introducing a new Help page that explains the details that are found in the relay details pages. You can find a live example of this patch at: https://irl.github.io/atlas/#/ and a full list of changes are detailed in #6787. Please let me know if you can see any reason to not merge this, or if you have any suggestions for small changes before merging. Thanks, Iain. [1]: https://trac.torproject.org/projects/tor/ticket/5430 [2]: https://trac.torproject.org/projects/tor/ticket/6787

5 5

Tor Browser and Privoxy
by Paulo Roberto 03 Aug '16

03 Aug '16

Hello, By default the Tor Browser Bundle connects through SOCKS5 to the TOR daemon. Is there any advantage today to configure Privoxy to use TOR and configure the Tor Browser in the Tor Network Settings to use proxy access type HTTP/HTTPS pointing to Privoxy? Thanks in advance. Paulo Roberto.

3 3

prop224: zero bits in addresses
by teor 03 Aug '16

03 Aug '16

Hi all, At the Montreal hidden service hackfest, we discussed another scheme shortening prop224 .onion addresses. The only drawback is that it's exponentially difficult, so it can only chop off a few characters. I'm describing it here because it has other useful properties, and can be used as: 1) an address versioning scheme, and/or 2) a configurable proof-of-work threshold for generating valid addresses. The basic scheme works like this: (those of you familiar with bitcoin might recognise it) A) Valid .onion address hashes must start with a certain, hard-coded number of 0 bits, followed by a 1 bit. (We include the 1 bit, so that addresses with additional 0 bits can be used by a later version of the protocol.) B) When generating a private key, the hidden service must discard any keys that yield .onion addresses that don't fit this bit pattern. C) When creating a textual .onion address, Tor removes the hard-coded bits from the address. D) When parsing a textual .onion address, Tor adds the hard-coded bits from the address. Otherwise, the protocol proceeds as it would in prop224. This kind of scheme is mentioned as a TODO in the existing proposal: [TODO: Alternatively, we could require that the first bit of the master key always be zero, and use a 51-byte encoding. Or we could require that the first two bits be zero, and use a 51-byte encoding and reserve the first bit. Or we could require that the first nine bits, or ten bits be zero, etc.] In my very basic tests using Yawning's horse25519 on a laptop, it seems that 15 bits takes less than a second, but 20 bits can take from a few seconds to around a minute. Prop 224 .onion addresses already have 4 unused bits, so we could easily chop off 3-4 characters using this scheme, with only a small amount of computation the first time the hidden service runs. Without a scheme like this, it's possible to generate hundreds of thousands of valid .onion addresses per minute on most recent processors. I'm not sure if this is a problem or not. So we have to decide whether the added complexity is worth having addresses that are 3-4 characters shorter. I'm not sure that it is, unless we like the idea of making it harder to generate valid .onion addresses, or unless we want to allow for extensible future versioning of .onion addresses, beyond the existing 4 unused bits. Tim Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmmp: teor at torproject dot org

2 1

metrics-lib/DescripTor 1.3.1 is released
by Karsten Loesing 01 Aug '16

01 Aug '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello devs, we just released metrics-lib/DescripTor 1.3.1: https://dist.torproject.org/descriptor/1.3.1/ - From the change log: # Changes in version 1.3.1 - 2016-08-01 * Medium changes - Adapt to CollecTor's new date format to make DescriptorCollector work again. Happy descriptor collecting! All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJXn0XrAAoJEC3ESO/4X7XB+q8IAMtu1b/mhvUvS7nlETa2ejCF BDhHNMK2v23pH0usku2AaZuzc0AkG4r0d3Nyb0jDRH5J/oNu2kDCUXIbDqJxwfX7 y5UlNaZgLTcYV7eJjWAqbvAmCBQMpEYJLhhtPqO81DgZpNbMP08doAgis30TPfiE KRUrKTIJxcDVQLfvsSFOOZj3PDjkQScWtW+icBF2PdD1u62/Dhv1QEJWzBTKH2xq etmc5VsNAir/mV2/le68CsIlSNuwGH9yU4xRlZ8QEb+MOnnS76p3onnsdqmHhytJ V0hXlViWNRjFd0EzcuCp4tvkWK3//xBC19AjQkwuRCOCbbeuOpQLZI00BJV9NCg= =0qss -----END PGP SIGNATURE-----

1 0

Re: [tor-dev] prop224: Ditching key blinding for shorter onion addresses
by Nick Mathewson 31 Jul '16

31 Jul '16

whoops, didn't reply-all. On Sat, Jul 30, 2016 at 1:00 PM, Nick Mathewson <nickm(a)alum.mit.edu> wrote: > On Sat, Jul 30, 2016 at 9:32 AM, George Kadianakis <desnacked(a)riseup.net> wrote: > [...] >> >> Please let me know in what way this scheme is completely broken! > > I think it has the same problem as the last one: there does not appear > to be a way to keep an attacker from putting their own record in place > using your query_key, and thereby performing a DoS on your hidden > service. > > > -- > Nick

1 0

[GSoC] CONIKS for Tor Messenger - status report #5
by Huy Vu Quoc 30 Jul '16

30 Jul '16

Hi everyone, These are things that I have been doing in the last two weeks: - I and my mentors have had a discussions for next steps for the project. We decided to reorganize our project’s repo [1]. All repos has been merged into one [1]. This helps us maintain and keep tracking the development easier. - A lot of issues and PRs have been resolved/merged. The merkletree library is almost done for now. - Thanks for Ismail and Arlo’s suggestions, the client library for crypto verifications (including the VRF verification, Ed25519 signature verification, auth path verification and so on) is implemented in Go and compiled with `c-shared` build mode [2]. This library then will be wrapped with js-ctypes to use in the add-on. I have successfully loaded the library into the add-on in Linux machines. There’s still some problem since Tor Messenger for OS X is compiled from a Linux machine, while the library is compiled from macOS, then when loading the library into Tor Messenger for OS X, Tor Messenger may crash. We will probably dig deeper on this issue. Windows’ DLL version of the library also should be taken care. Next steps are mostly to focus on the client-side, along with finishing the server reference implementation. Here’s thing I expect in the next weeks: - Merge the server-registration and registration-bots PR. - Merge the coniks-preference PR. - Merge the PR of client library. - Submit PR for client registration and lookup. I also expect to have a working version of project before August 15, including the functions which I described in my proposal. Best, Huy — [1] https://github.com/coniks-sys/coniks-go [2] https://github.com/coniks-sys/coniks-go/tree/client-cgo

1 0

[GSOC 16] Ahmia status update #5
by Ismael R 30 Jul '16

30 Jul '16

Hi everyone, I'm working on ahmia.fi, the hidden service search engine and you're reading status update #5. During the last two weeks I worked on improving search. It required working on a new index format and updating the crawler to fill this new index. Search now has these new features: - Token search is done in addition to pure full-text search. For instance, "foxes" and "fox" are different words but are indexed as the same token. Same for "running" and "run", "tor's and tor". - Associated words are also stored. For instance, "I’m not happy I’m working" have the same words than "I’m happy I’m not working", but the meaning is different. - Meaningful words have way more impact than frequent words. - The text used with a link (called anchor text) is now stored with the page which is the target of the link. Let's say a site A is linking a site B like that: <a href="B">B is a great search engine</a>. Searching "great search engine" is going to return B instead of A. These anchors could also replace a title in the results list if the site has no title (.txt files are like that). - A word present in the title, description, anchors has way more strength than a word present in the content of a page. - An authority score is computed for each page thanks to the pagerank algorithm. Important pages (like homepages) tends to appear first in the result list. It also solves problems we have with duplicated content. - The result list now returns pages instead of sites/domains. Before updating ahmia.fi, I need to finish porting the rest of the django site (API and statistics) to the new structure. I hope to finish this next week so you can try it. After this, here are the features I plan to implement in the remaining time: - Add a language to the search engine (site:msydqstlz2kzerdg.onion to specify a domain, -term to exclude a term, filetype:txt to specify a filetype, lang:fr to specify a language, etc). - Use If-Modified-Since HTTP header to avoid crawling unmodified pages. - Improve documentation - Add unit tests Also, I have some ideas I won't have the time to implement during GSoC. Hopefully, I will do it later because I plan to continue working with the Tor community :) A small list: - Improve search for languages other than english by making one index per language - Make the crawler better understand query parameters. Some are useful (?page=3), some are useless (?printable=1). - Better tokenize the url to extract words from it. - Add highlights to each result (especially for those without meta- description) to show the context in which query terms appears. - Make a script to make sure every indexed url is still online. - Make a true RESTful API. I see you in two weeks for the next status report and maybe sooner if I push something online. Cheers! Ismael

1 0