- tor-dev - lists.torproject.org

[GSOC 16] Ahmia status update #6
by Ismael R 12 Aug '16

12 Aug '16

Hi everyone, I'm working on ahmia.fi, the hidden service search engine and you're reading status update #6. During the last two weeks, I finished porting the django app to the new structure. I'm also working on last minute things before shipping the new site online. I will continue updating documentation and add some unit tests to the project. The code is not merged yet but you're welcome to check it on my forks. [1] [2] Since this status report is short, here is a list of goals I had in my initial project proposition and what work has been done on each. Review code and infrastructure: - Split the project in several repositories - Improve documentation - Automate testing (Travis.CI) - Track code quality (Landscape.IO) - Track requirements (Requires.IO) - Refactor each subproject Improve search results: - Better use of elasticsearch (use of stemmers, shingles, term-centric search) - Search results are now pages instead of domains. Improve UI/UX: Not much work has been done for this goal. The website has been in the process of porting old pages to a new design. All pages are now using the new design. Gather more statistics: - Pagerank is now used to compute an authority score for each page - I suggested that we could use a self hosted statistics framework like piwik [3] but no decision has been made. Use stats to better rank search results: - Results are ranked by authority score. Make sense of the indexed info to understand a search meaning: - Shingles enable us to differenciate these two queries: "i'm not happy i'm working" and "i'm happy i'm not working". - Synonyms could be used by the search algorithm if we provided a synonym dictionnary. No work has been done at making this work. Make a google trend-like interface to visualize searches over time: No work has been done to reach this optional goal. Even some stats fonctionnalities were dropped in the new site because they were "domain- centric" when a search engine needs to be "page-centric". We could probably index searches in elasticsearch and use Date Histogram Aggregation [4] to display trends. Make stats available with the API: No work has been done to reach this optional goal. Some API endpoints were also dropped because they were domain-centric. It would be great to have an API with a coherent url scheme. I think Django Rest Framework can help design that API while keeping the code simple. That's it for this week, Have a nice weekend. Ismael R. [1] https://github.com/iriahi/ahmia-site [2] https://github.com/iriahi/ahmia-crawler [3] https://piwik.org/ [4] https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggr… [5] http://www.django-rest-framework.org/

2 1

[GSOC16] Fingerprint Central - Status report n°6
by Pierre Laperdrix 12 Aug '16

12 Aug '16

Hi everyone, Here is my sixth status report for my GSOC project. Repository: https://github.com/plaperdr/fp-central Beta: https://fpcentral.irisa.fr 1- Development of the "Acceptable" feature When a user sends his fingerprint to the server, he can now see for each attribute if the collected value is "acceptable" or not (i.e., if it is an expected one). If the server returns that a value is not an acceptable one, this means that it is something to investigate. Does it come from a simple misconfiguration of the browser or could it be a genuine difference between TBB browsers that should be fixed? Right now, it is a simple Yes/No answer but I'll add more feedback so that a user understands the information that is presented. 2- i18n support with documentation I have finished the FR version of the website. Not everything is localized since some libraries I use for the front-end are only in English but the majority of FP Central is now translatable. I have written a small guide if anyone wants to contribute (https://github.com/plaperdr/fp-central/tree/master/translations) The process is pretty simple: you generate a file that references all the strings that need to be translated and you complete it with your translations. 3- Multiple bug fixes and improvements I fixed multiple bugs in the customStats page that were caused by the way I structured data in the page and it should be much smoother now. I'm sure other bugs are lurking in the code so do not hesitate to visit the beta website and report to me any bugs you may find. In terms of features, I have normally developed everything I wrote in my proposal. So my focus for the final stretch of the coding period is to clean the code, fix bugs, improve the website and write even more documentation. I'll also start planning the post-GSOC period to see what interesting features could be added in the future and to make sure that FP Central is ready for prime time for users and developers. Have a great week-end, Pierre

1 0

First CollecTor Release 1.0.0
by iwakeh 11 Aug '16

11 Aug '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi there! A while ago I wrote about the CollecTor improvement project [0]. Since July we intensified work there to prepare the very first release of CollecTor, which is now available here: https://dist.torproject.org/collector/1.0.0/ CollecTor itself is much older, the git repository dates back at least six years and there is even data from the year 2004 available [1]. The three most significant changes for release 1.0.0: * CollecTor consists of a single executable Java jar now. * CollecTor schedules all modules without the help of any crontab like service. * Configuration is done via one file and all important settings were made configurable. CollecTor is only a download away now :-) Cheers, iwakeh [0] https://lists.torproject.org/pipermail/tor-dev/2016-April/010742.html [1] https://collector.torproject.org/archive/relay-descriptors/tor/?C=N;O=A -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iF4EAREIAAYFAles67AACgkQsANAxTOj/ab/GQEAl/KUq2by8ooi2Nftw9RFh85L lLHf6uwzmHXD294lptQA/jHxOSCDUDUM4uEVV4HxvaUVbuToNNZ0mqbY45YbJz+r =rLeM -----END PGP SIGNATURE-----

1 0

https://gitlab.com/torproject
by Hans-Christoph Steiner 10 Aug '16

10 Aug '16

FYI I registered https://gitlab.com/torproject to claim the name in the gitlab namespace. I'm happy to hand it over to a more official Tor person if anyone wants. Or just let me know if anyone ever wants to actually use it. I've been using gitlab for a while now with F-Droid, and its quite nice. Guardian Project is going to move our stuff there too. .hc -- PGP fingerprint: EE66 20C7 136B 0D2C 456C 0A4D E9E2 8DEA 00AA 5556 https://pgp.mit.edu/pks/lookup?op=vindex&search=0xE9E28DEA00AA5556

3 2

How to integrate an external name resolver into Tor
by Nick Mathewson 08 Aug '16

08 Aug '16

Hi, all! I've seen a couple of emails from people looking into new ways to do naming for onion services. That's great! Before anybody gets too far, I'd like to send this quick note to let you know that integrating stuff like this into Tor is actually easier than you think. Here's how you do it, using a Tor controller. (See control-spec.txt for protocol documentation. Also see one of the several friendly libraries, like steam, that exist to interface with Tor over this protocol.) First, you set the Tor option "__LeaveStreamsUnattached". This tells Tor that it shouldn't try to handle new client requests immediately, but it should instead let the controller take responsibility. In the controller, you make sure that you are watching STREAM events so that you find out about new streams. Whenever there's a new stream, you check its address. If the address is one that you don't want to rewrite, you just call ATTACHSTREAM on it, with a circuit ID of 0. (The 0 means "Tor, you figure out how to attach this one.". Otherwise, you do whatever magic dance you do in order to find out the real address for the stream. If the lookup operation is successful, you say "REDIRECTSTREAM (stream ID) (new address". And then you ATTACHSTREAM as above. If the lookup operation fails, you call "CLOSESTREAM (stream ID) 2". (The 2 means "resolve failed". And that's it for the Tor integration! All you need to do now is figure out how the name lookup works. cheers, -- Nick

5 7

Re: [tor-dev] Onioncat and Prop224
by str4d 07 Aug '16

07 Aug '16

On 08/08/16 01:22, Jeremy Rand wrote: >> And most certainly any external layer must be capable of having >> nodes binding natively in the Tor / I2P / etc networks, and >> preferably being strictly private entirely within them (like how >> private Tor / I2P / Bitcoin nets can be deployed by generating >> new authorities, keys, genesis, etc). Outproxy is not an option. > > Bitcoin and Namecoin nodes can be exposed as Tor hidden services (I > believe there's even code in there now for automatically configuring > such a hidden service via the Tor control port). I don't think there's > any built-in support for I2P (unless something was added since I last > looked), Not yet, but there's an open issue for adding I2P support (alongside support for Tor's HS2.0) [0]. See also an old Bitcoin fork with I2P support [1], and other relevant links in Zcash's I2P issue [2]. Any native I2P support would automatically set up the I2P Destinations, because that's how the SAM API operates [3]. > but if I2P can expose a local TCP service as an I2P service, Yes. > and if I2P allows such TCP services to be connected to by exposing a > SOCKS interface, Yes. > I'm guessing that it should mostly work (famous last > words). The only thing that would break that way is peer exchange: > Bitcoin and Namecoin nodes normally share peer addresses, and although > this works just fine with Tor hidden services, I don't think it would > work out of the box with I2P (unless someone's added that since I last > looked). The issue here is that I2P addresses (even B32s) are longer than the internal message fields for sharing addresses. Tor's HS2.0 will suffer from the same issue, because their proposed addresses are just as long as I2P's B32s. > > Although it is possible to deploy private Bitcoin and Namecoin networks > (I did it at a workshop I gave at DWS so that I could mine blocks on > demand), usually people do this for testing or demonstration purposes, > not for real-world applications. The reason is that blockchains' > security assumptions include the assumption that they have a reasonably > high hashrate from a reasonably diverse set of miners. An > Onioncat-specific Namecoin network would probably have a very low > hashrate compared to the main Namecoin network, which means that it > would be much easier to perform a 51% attack. Merged mining could, > hypothetically, raise the hashrate, except that to do merged mining, the > miners would have to be connected to the public Bitcoin network. In > addition, it's usually very unprofitable to mine Bitcoin or Namecoin > over Tor or I2P, because the additional latency causes much higher > orphan rates. (Most Bitcoin and Namecoin miners spend considerable > effort obtaining low-latency connections for relaying blocks to and from > other miners.) > > Might I ask what the motivation is for having a completely separate > Namecoin network inside Tor/I2P, that can't be satisfied by binding > participants' nodes to Tor/I2P hidden services while allowing some users > who don't want or need anonymity to relay transactions and blocks > between clearnet and Tor/I2P? One potential issue is that if the user's endpoint isn't configured correctly, Namecoin names that resolve to clearnet addresses could deanonymise the user (checks at the browser or app level wouldn't help here because all it would see is an IPv6-style address IIUC). It's not unavoidable, but a point to be wary of when considering the overall integration strategy. Using the global Namecoin blockchain also puts the bridging nodes into a position of importance and trust, that can only be lessened by greatly increasing the number of bridging nodes. This is more of a concern for I2P than Tor, as I2P-only peers can only reach the bridging nodes, whereas Tor-only peers can connect to clearnet Namecoin nodes too (although as with Bitcoin there is the issue of exit node traffic manipulation). str4d [0] https://github.com/bitcoin/bitcoin/issues/2091 [1] https://github.com/VirtualDestructor/bitcoin-qt-i2p [2] https://github.com/zcash/zcash/issues/1111 [3] https://geti2p.net/en/docs/api/samv3

1 0

CollecTor will soon serve descriptors that don't pass metrics-lib's parser
by Karsten Loesing 05 Aug '16

05 Aug '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello CollecTor data consumers, note: if you don't know what CollecTor is or if you have never used CollecTor data before, you can safely stop reading now and enjoy your Friday. But in case you're now curious what CollecTor is, feel free to take a look: https://collector.torproject.org/ So, we're about to remove a sanity check in CollecTor where we'd feed relay descriptors into metrics-lib and only serve them if they pass the parsing step. This step has so far eliminated descriptors with non-ASCII characters in unexpected places, like: dirreq-v3-ips us=192,ru=144,de=136,??=112,fr=64,gb=56,br=40,it=40,es=24,in=24,jp=24,pl=24,ar=16,at=16,au=16,ca=16,ch=16,cn=16,cz=16,ir=16,kr=16,mx=16,nl=16,ua=16,za=16,^Wg=8,ae=8,ao=8,be=8,bg=8,bj=8,bo=8,bs=8,bw=8,by=8,cd=8,cl=8,cm=8,co=8,cr=8,cu=8,dd=8,dk=8,du=8,dz=8,ec=8,eg=8,fi=8,gf=8,gh=8,gr=8,gt=8,hk=8,hn=8,hr=8,hu=8,id=8,ie=8,il=8,i�=8,ke=8,kz=8,lk=8,lt=8,lv=8,ly=8,ma=8,md=8,mk=8,mu=8,my=8,nc=8,ng=8,ni=8,no=8,nz=8,pa=8,pe=8,ph=8,pk=8,pr=8,pt=8,ro=8,rs=8,se=8,sg=8,sk=8,sn=8,sy=8,th=8,tn=8,tr=8,tw=8,ug=8,ve=8,vn=8,�p=8 Very soon (next week?), CollecTor will serve those descriptors, just like the Tor directory authorities serve them. If you're obtaining and processing descriptors from CollecTor, you'll somehow have to handle them. Of course, if you're using metrics-lib, nothing changes, and I hear Stem discards descriptors with non-ASCII characters in the line above, too. If you want, you can find more details on the ticket: https://trac.torproject.org/projects/tor/ticket/19170 All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJXpGOhAAoJEC3ESO/4X7XBVmoIALbd2AmUEMX6DrzPsEvPLItR YP3ySblh9knkc5NvrGxO/hcYABjX4j5k/smrAotvcT+5qUVyQiXbmD4kO0Rkq5i9 nC4+MkBiuigj4eeXi+Y57oBVuXidJNqFoMFmNpeXxU/csiwsVXsWX/UTnoiox6Cg 5KKnGpr9lr4IQt74Jw1gIYAnhXE6ga8JyWv8dcv0dtOdChl1zsjjCOgFuFpPlFJl ycFAhX0jM+iOpVV8m1WTA0r1PAAAndtKBotCuhGizdfNSRcPiA3hhUFewNxbgY7N b/fvIRQRumNCv2uAYve25iIuL+0vE197rUO9vY8IlRB/po/qDQu8AVp6HjrlZCU= =nzBR -----END PGP SIGNATURE-----

1 0

Tor and Namecoin
by Jeremy Rand 04 Aug '16

04 Aug '16

Hello Tor devs, Namecoin is interested in collaboration with Tor in relation to human-readable .onion names; I'm reaching out to see how open the Tor community would be to this, and to get feedback on how exactly the integration might work. The new hidden service spec is going to substantially increase the length of .onion names, which presents usability concerns. Namecoin provides a way to resolve a human-readable .bit name to a .onion name. Another benefit of Namecoin is that it provides a way to lookup TLS fingerprints for clearnet .bit sites, which reduces the risk of MITM attacks on clearnet websites from malicious or compromised CA's. I had the pleasure of meeting Mike Perry at the Decentralized Web Summit at the Internet Archive in June; I talked to him about Namecoin's rough plans and he suggested I post here. I understand that Riccardo Spagni from Monero discussed this topic as well with Roger Dingledine at the Security in Times of Surveillance conference at Ei_PSI. The two most major concerns that I expect would be brought up involve anonymity and blockchain size. Here's how we plan to deal with these issues: Namecoin already provides location-anonymity for name registrations assuming that it's routed via Tor. It's also necessary to broadcast transactions for different names to different peers, which isn't coded yet, but this is just coding work rather than an engineering challenge -- a usable workaround today is running multiple Namecoin wallets. The more interesting challenge is blockchain anonymity for registrations, due to the linkability required for blockchain validation. An important point here is that transactions for a given name are inherently linkable to each other, and that this isn't problematic. The problem would come when multiple names are linked together, or when a name is linked with currency transactions. The solution I've come up with is to use atomic cross-chain trades, which let a user buy namecoins using a cryptocurrency that is designed to provide anonymity (such as Monero or Zcash, both of which have cryptographic proofs of anonymity, given a certain anonymity set and security assumptions). The user would use an anonymous cryptocurrency to buy a small amount of namecoins (enough to register a single name and keep it renewed for a while). If the user wanted to register another name, she would perform another atomic cross-chain trade, receiving namecoins that are not linked to the namecoins obtained for the first name. As long as those namecoins are not mixed by the wallet software, the names remain unlinked. Many users won't want to download the full Namecoin blockchain (around 3 GB at the moment). I have a proof-of-concept SPV-based Namecoin name lookup client working as of early June. I just got a large part of that code upstreamed into libdohj, and I'm working on getting the rest upstreamed and released. It's in Java (based on BitcoinJ), so it's not subject to the memory safety concerns that C/C++ code are. The SPV name lookups are implemented in 3 ways, depending on the user's needs: Option A: 1. Block headers are synced over the Namecoin P2P network. (Over clearnet this takes about 5 minutes the first time it runs.) 2. An index mapping unexpired block heights to block hashes is constructed, so that lookups can be done quickly. (This occurs when the SPV client starts, after syncup has completed; it's fast enough that I haven't found a need to benchmark it.) 3. When a name lookup request is received, the client asks a remote API server for the height of the last update of the name. 4. The client looks up the block hash of that height from its index, and requests that block over the P2P network. 5. The client verifies that the received block matches the correct hash and that the block follows Namecoin rules (e.g. verifying the merkle root). 6. The client looks through the transactions in the block until it finds the one that updates the name. 7. The client retrieves the value of the name from that transaction, and returns it to the user. Option B: 1 through 3. Same as Option A. 4. The API server also provides the full content of the transaction, as well as a merkle proof of inclusion in the block. 5. The client verifies that the merkle proof links the hash of the provided transaction to the merkle root of the block header with the given height. 6. The client retrieves the value of the name from the provided transaction, and returns it to the user. Option C: 1. Block headers are synced over the Namecoin P2P network, as well as full blocks for the past year (meaning that all full blocks that contain unexpired name data will be synced). (Over clearnet this takes about 10 minutes the first time it runs.) 2. An index mapping names to transactions is constructed as the full blocks are downloaded. (This uses LevelDB.) 3. When a name lookup request is received, the client looks up the transaction in the LevelDB index. 4. The client retrieves the value of the name from that transaction, and returns it to the user. For Options A and B, if the API server is malicious, it can do any of the following: 1. Falsely claim that the name doesn't exist. 2. Provide outdated name data that is less than 36000 blocks old (the expiration period for Namecoin). (Option C is not vulnerable to either of those attacks.) If multiple API servers are consulted, and they return different results, it is easy to tell which is lying (although I haven't implemented any such logic yet). The API server cannot do any of the following: 1. Provide name data that isn't from the blockchain with the most work. 2. Provide name data that is more than 36000 blocks old (the expiration period for Namecoin). The reason an API server is used in Options A and B instead of the P2P network, is that the P2P network is unauthenticated and easy to Sybil. The P2P network is great for getting data that is independently verifiable (e.g. block headers and contents of blocks), but it's unwise to rely on the P2P network to get unverifiable data such as a block height of a name. An API server is authenticated (currently via CA-based TLS, but a cert pin or PGP signing is certainly doable), which reduces the possible points of attack. This is analogous to why Tor uses centralized directory authorities -- authenticated trust points are harder to Sybil. (We do have longer term plans to introduce a way for SPV clients to get the latest transaction associated with a name, without using an API server or needing to download any full blocks, but that's out of scope of this email.) Options A and B do reveal to the API server which name is being looked up. If mode A is used, it also reveals to a P2P peer which block height is being looked up (which narrows the set of names by a factor of ~36000). Therefore, Tor stream isolation should be used in such cases. (That's not implemented yet.) Option C doesn't generate any network traffic on lookups, so it doesn't reveal anything. In my testing, an SPV-based name lookup using Option A takes around 650 milliseconds (over clearnet). The vast majority of this is latency to the API server (the server I'm testing with is on a low-budget hosting plan). The portion consisting of a block retrieval over P2P takes around 98 milliseconds (although it varies by block size). Option C takes around 4 milliseconds. The storage overhead of Option C's LevelDB database is around 400 MB right now, although I believe it's feasible to reduce this significantly. There are a few options I can think of for integrating this with Tor for .onion naming. One would be to modify OnioNS to call the Namecoin SPV client. This would concern me because OnioNS is in C++, which introduces the risk of memory safety vulnerabilities. Another would be to use an intermediate proxy like Yawning's or-ctl-filter. A third option would be to try to get external name resolution implemented in Tor itself, which I believe Jeff Burdges has suggested in the past. If Option A or B is used, any solution would need to pass the stream isolation info to the SPV client. Integrating this with Tor Browser for TLS certificate validation might involve a Firefox patch. There are tricks that can be done with the CertDB and SiteSecurityService XPCOM interfaces that will do the job without Firefox patches, but XPCOM is being phased out by Mozilla in favor of WebExtensions, and I'm unaware of any equivalent features in WebExtensions. (Also, it's unclear to me whether CertDB and SiteSecurityService would introduce isolation issues -- I can't think of any obvious attacks, but I haven't thought very hard about it.) I'm trying to engage with Mozilla to see if we can work out a WebExtensions feature for this, but nothing conclusive has happened on that front yet. On the subject of reproducible builds, I've never tried to build Java code in Gitian, so I'm not certain how difficult it's going to be. Since Android uses Java, maybe the Guardian Project devs would have some insight into the best way to do it. One of the Namecoin developers (Joseph Bisch) is really good with reproducible builds (you probably know him since he's the author of the Debian guest support in Gitian), so I'm reasonably confident that a way to do it can be found. I'd love to hear feedback on all of this. Cheers, -Jeremy Rand Lead Application Engineer of Namecoin

4 7

Re: [tor-dev] [rfc] Usability Improvements for Atlas
by tordev123＠Safe-mail.net 03 Aug '16

03 Aug '16

IMO, your improvements make the user experience much worse. The current version is far more space efficient, your new versions wastes giant amounts of screen space. > * There is no super-wide list of relays that doesn't fit even into Tor > Browser window (panels instead with upstatus indication) The important stuff does fit, usually no scrolling is required. > * Updated navigation bar with new version of Bootstrap Takes up more space with no usability improvement. > * Stretching of the graphs They disappear with TBB set to high security. > P.S. Some of functions like list sorting, display range, detailed search > and maybe more are broken/gone now. I'm wondering if these functions are > useful? Yes, they are useful.

1 0

Usability Improvements for Atlas (was Re: Globe is now retired)
by Iain R. Learmonth 02 Aug '16

02 Aug '16

Hi All, On 29/06/16 09:56, Karsten Loesing wrote: > tl;dr: Globe is now retired in favor of an improved Atlas. Atlas is improved, but I'd like to improve it further. I've been tackling #6787 looking to improve the homepage and make Atlas easier to use for someone that isn't already a Tor expert. The changes I'm proposing in my fix for #6787 [1] do make some noticeable UI changes and I just want to gather some opinions on whether or not Atlas users feel that this is taking it in the right direction. The "Top 10 Relays" page was added to Atlas in order to make sure that functionality was not lost when shutting down globe. (See #5430 [2]) In my proposed patch, I'm also adding an "Authorities" page and adding text to both of these pages to explain what you're seeing. These are linked from the homepage, and from the top navbar. I'm also moving the introduction and explanation of the Atlas features from the About page to the homepage and introducing a new Help page that explains the details that are found in the relay details pages. You can find a live example of this patch at: https://irl.github.io/atlas/#/ and a full list of changes are detailed in #6787. Please let me know if you can see any reason to not merge this, or if you have any suggestions for small changes before merging. Thanks, Iain. [1]: https://trac.torproject.org/projects/tor/ticket/5430 [2]: https://trac.torproject.org/projects/tor/ticket/6787

5 5