- tor-dev - lists.torproject.org

[GSoC 2013] Status report - Searchable metrics archive
by Kostas Jakeliunas 23 Aug '13

23 Aug '13

Hello! Updating on my Searchable Tor metrics archive project. (As is very evident) I'm very open for naming suggestions. :) To the best of my understanding and current satisfaction, I solved the database bottlenecks, or at least I am, as of now, satisfied with the current output from my benchmarking utility. Things may change, but I am confident (and have support to argue) that the whole thing runs swell at least on amazon m2.2xlarge instances. For fun and profit, a part of the database (which, has, for now, status entries (only) in the range [2010-01-01 00:00:00, 2013-05-31 23:00:00]), namely, what is currently used by the Onionoo-like API is now available online (not on EC2, though) - will now write a separate email so that everyone can inspect it. I should now move on with implementing / extending the Onionoo API, in particular, working on date range queries, and refining/rewriting the "list status entries" API point (see below). Need to carefully plan some things, and always keep an updated API document. (Also need to update and publish a separate, more detailed specification document.) More concrete report points: - re-examined my benchmarking approach, and wrote a rather simple but effective set of benchmarking tools (more like a simple script) [1] that can be hopefully used outside this project as well; at the very least, together with the profiling and the query_info tools, it is powerful (but also simple) enough to be used to test all kinds of bottlenecks in ORMs and elsewhere. - used this tool to generate benchmark reports on EC2 and on the (less powerful) dev server, and with different schema settings (usually rather minor schema changes that do not require re-importing all the data) - came up with a triple table schema that proves to render our queries quickly: we first do a search (using whatever criteria (e.g. nickname, fingerprint, address, running), if any) on a table which has a column with unique fingerprints; extract the relevant fingerprints; JOIN with the main status entry table, which is much larger; and get the final results. Benchmarked using this schema. <details> If we are only extracting a list of the latest status entries (with distinct on fingerprint), we can do LIMITs and OFFSETs already on the fingerprint table, before the JOIN. This helps us quite a bit. On the other hand, nickname searches etc. are also efficient. As of now, I have re-enabled nickname+address+fingerprint substring search (not from the middle (LIKE %substring%), but from the beginning of a substring (LIKE substring%), which is still nice), and all is well. Updated the higher-level ORM to reflect this new table [2] (I've yet to change some column names, though - but these are cosmetics.) </details> - found a way to generate the SQL queries that I need to generate using the higher-level SQLAlchemy SQL API using various SQLAlchemy-provided primitives, and always observing the resulting query statements. This is good, because everything becomes more modular: much easier to shape the query depending on the query parameters received, etc. (while still retaining it in sane order.) - hence (re)wrote a part of the Onionoo-like API that uses the new schema and the SQLAlchemy primitives. Extended the API a bit. [3] - wrote a very hacky API point for getting a list of status entries for a given fingerprint. I simply wanted a way (for myself and people) to query this kind of a relation easily and externally. It now works as part of the API. This part will probably need some discussion. - wrote a (kind of a stub) document explaining the current Onionoo-like API, what can be queried, what can be returned, what kinds of parameters work. [4] Will extend this later on. while writing the doc and rewriting part the API, stumbled upon a few things that make clear that I've made some shortcuts that may hurt later on. Will be happy to elaborate on them later on / separately. I need to carefully plan a few things, and then try rewriting the Onionoo API yet again, this time including more parameters and fields returned. TL;DR yay, a working database backend! I might give *one* more update detailing things I might have forgotten about soon re: this report - I don't want to make a habit of delaying reports (which I have consistently done), so reporting what I have now. [1]: https://github.com/wfn/torsearch/blob/master/torsearch/benchmark.py [2]: https://github.com/wfn/torsearch/blob/master/torsearch/models.py [3]: https://github.com/wfn/torsearch/blob/master/torsearch/onionoo_api.py [4]: https://github.com/wfn/torsearch/blob/master/docs/onionoo_api.md -- Kostas (wfn on OFTC) 0x0e5dce45 @ pgp.mit.edu

1 0

Proposal 222: Stop sending client timestamps
by Nick Mathewson 22 Aug '13

22 Aug '13

Filename: 222-remove-client-timestamps.txt Title: Stop sending client timestamps Authors: Nick Mathewson Created: 22 August 2013 Target: 0.2.5.x Status: Open 0. Summary There are a few places in Tor where clients and servers send timestamps. I list them and discuss how to eliminate them. 1. Introduction Despite this late date, many hosts aren't running NTP and don't have very well synchronized clocks. Even more hosts aren't running a secure NTP; it's probably easy to desynchronize target hosts. Given all of this, it's probably a fingerprinting opportunity whenever clients send their view of the current time. Let's try to avoid that. I'm also going to list the places where servers send their view of the current time, and propose that we eliminate some of those. Scope: This proposal is about eliminating passive timestamp exposure, not about tricky active detection mechanisms where you do something like offering a client a large number of about-to-expire/just-expired certificates to see which ones they accept. 2. The Tor link protocol 2.1. NETINFO (client and server) NETINFO cells specify that both parties include a 4-byte timestamp. Instead, let's say that clients should set this timestamp to 0. Nothing currently looks at a client's setting for this field, so this change should be safe. 2.2. AUTHENTICATE (server) The AUTHENTICATE cell is not ordinarily sent by clients. It contains an 8-byte timestamp and a 16-byte random value. Instead, let's replace both with a 24-byte (truncated) HMAC of the current time, using a random key. This will achieve the goal of including a timestamp in the cell (preventing replays even in the presence of bad entropy), while at the same time not including the time here. 2.3. TLS 2.3.1. ClientRandom in the TLS handshake See TLS proposal in appendix A. This presents a TLS fingerprinting/censorship opportunity. I propose that we investigate whether "random " or "zero" is more common on the wire, choose that, and lobby for changes to TLS implementations. 2.3.2. Certificate validity intervals Servers use the current time in setting certificate validity for their initial certificates. They randomize this value somewhat. I propose that we don't change this, since it's a server-only issue, and already somewhat mitigated. 3. Directory protocol 3.1. Published This field in descriptors is generated by servers only; I propose no change. 3.2. The Date header This HTTP header is sent by directory servers only; I propose no change. 4. The hidden service protocol 4.1. Descriptor publication time Hidden service descriptors include a publication time. I propose that we round this time down to the nearest N minutes, perhaps for N=30. 4.2. INTRODUCE2 cell timestamp INTRODUCE2 cells once limited the duration of their replay caches by including a timestamp in the INTRODUCE2 cells. Since 0.2.3.9-alpha, this timestamp is ignored, and key lifetime is used instead. When we determine that no hidden services are running on 0.2.2.x (and really, no hidden services should be running on 0.2.2.x!), we can simply send 0 instead. (See ticket #7803). This might be a good place to use a consensus parameter, so that a large number of clients switch at the same time. I claim this would be suitable for backport to 0.2.4. 5. The application layer The application layer is mostly out of scope for this proposal, except: TorBrowser already (I hear) drops the timestamp from the ClientRandom field in TLS. We should encourage other TLS applications to do so. (See Appendix A.) ================================================================= APPENDIX A: "Let's replace gmt_unix_time in TLS" PROBLEM: The gmt_unix_time field in the Random field in the TLS handshake provides a way for an observer to fingerprint clients. Despite the late date, much of the world is still not synchronized to the second via an ntp-like service. This means that different clients have different views of the current time, which provides a fingerprint that helps to track and distinguish them. This fingerprint is useful for tracking clients as they move around. It can also distinguish clients using a single VPN, NAT, or privacy network. (Tor's modified firefox avoids this by not sending the time.) Worse, some implementations don't send the current time, but the process time, or the computer's uptime, both of which are far more distinguishing than the current time() value. The information fingerprint here is strong enough to uniquely identify some TLS users (the ones whose clocks are hours off). Even for the ones whose clocks are mostly right (within a second or two), the field leaks a bit of information, and it only takes so many bits to make a user unique. WHY gmt_unix_time IN THE FIRST PLACE? According to third-hand reports -- (and correct me if I'm wrong!) it was introduced in SSL 3.0 to prevent complete failure in cases where the PRNG was completely broken, by making a part of the Random field that would definitely vary between TLS handshakes. I doubt that this goal is really achieved: on modern desktop environments, it's not really so strange to start two TLS connections within the same second. WHY ELSE IS gmt_unix_time USED? The consensus among implementors seems to be that it's unwise to depend on any particular value or interpretation for the field. The TLS 1.2 standard, RFC 5246, says that "Clocks are not required to be set correctly by the basic TLS protocol; higher-level or application protocols may define additional requirements." Some implementations set the entire field randomly; this appears not to have broken TLS on the internet. At least one tool (tlsdate) uses the server-side value of the field as an authenticated view of the current time. PROPOSAL 1: Declare that implementations MAY replace gmt_unix_time either with four more random bytes, or four bytes of zeroes. Make your implementation just do that. (Rationale: some implementations (like TorBrowser) are already doing this in practice. It's sensible and simple. You're unlikely to mess it up, or cause trouble.) PROPOSAL 2: Okay, if you really want to preserve the security allegedly provided by gmt_unix_time, allow the following approach instead: Set the Random field, not to 32 bytes from your PRNG, but to the HMAC-SHA256 of any high resolution timer that you have, using 32 bytes from your PRNG as a key. In other words, replace this: Random.gmt_unix_time = time(); Random.random_bytes = get_random_bytes(28) with this: now = hires_time(); // clock_gettime(), or concatenate time() // with a CPU timer, or process // uptime, or whatever. key = get_random_bytes(32); Random = hmac_sha256(key, now); This approach is better than the status quo on the following counts: * It doesn't leak your view of the current time, assuming that your PRNG isn't busted. * It actually fixes the problem that gmt_unix_time purported to fix, by using a high-resolution time that's much less likely to be used twice. Even if the PRNG is broken, the value is still nonrepeating. It is not worse than the status quo: * It is unpredictable from an attacker's POV, assuming that the PRNG works. (Because an HMAC, even of known data, with an unknown random key is supposed to look random). CONSIDERATIONS: I'd personally suggest proposal 1 (just set the field at random) for most users. Yes, it makes things a little worse if your PRNG can generate repeat values... but nearly everything in cryptography fails if your PRNG is broken. You might want to apply this fix on clients only. With a few exceptions (like hidden services) the server's view of the current time is not sensitive. Implementors might want to make this feature optional and on-by-default, just in case some higher-level application protocol really does depend on it. ==================================================================

4 3

Segfault trying to start tor in 0.2.4.16-rc with bufferevents
by Ian Goldberg 22 Aug '13

22 Aug '13

I just tried to upgrade my tor exit node to 0.2.4.16-rc, and got this: Aug 22 09:55:09.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Aug 22 09:55:10.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Aug 22 09:55:12.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Aug 22 09:55:12.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Aug 22 09:55:12.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Aug 22 09:55:16.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Aug 22 09:55:17.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Segmentation fault The configure line was: ./configure --with-openssl-dir=../openssl-1.0.1e --enable-bufferevents Running under gdb, I see: Program received signal SIGSEGV, Segmentation fault. buf_datalen (buf=0x0) at src/or/buffers.c:523 523 return buf->datalen; (gdb) bt #0 buf_datalen (buf=0x0) at src/or/buffers.c:523 #1 0x0000555555611b89 in connection_write_to_buf_impl_ ( string=0x7fffffffdd20 "\026\003\001", len=168, conn=0x55555682d070, zlib=0) at src/or/connection.c:3598 #2 0x0000555555587779 in connection_write_to_buf (conn=0x55555682d070, len=<optimized out>, string=<optimized out>) at src/or/connection.h:136 #3 connection_edge_process_relay_cell (cell=0x7fffffffdd10, circ=0x555557282610, conn=0x55555682d070, layer_hint=<optimized out>) at src/or/relay.c:1340 #4 0x0000555555589391 in circuit_receive_relay_cell (cell=0x7fffffffdd10, circ=0x555557282610, cell_direction=CELL_DIRECTION_OUT) at src/or/relay.c:212 #5 0x00005555555fa57c in command_process_relay_cell (chan=0x5555575b9f40, cell=0x7fffffffdd10) at src/or/command.c:462 #6 command_process_cell (chan=0x5555575b9f40, cell=0x7fffffffdd10) at src/or/command.c:148 #7 0x00005555555db873 in channel_tls_handle_cell (cell=0x7fffffffdd10, conn=0x555557ccc140) at src/or/channeltls.c:923 #8 0x000055555561c3f9 in connection_or_process_cells_from_inbuf ( conn=0x555557ccc140) at src/or/connection_or.c:1972 #9 0x000055555560f965 in connection_handle_read_cb (bufev=<optimized out>, arg=0x555557ccc140) at src/or/connection.c:3114 #10 0x00007ffff749457e in ?? () from /usr/lib/libevent-2.0.so.5 ---Type <return> to continue, or q <return> to quit--- #11 0x00007ffff748ae47 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #12 0x000055555556b1fe in do_main_loop () at src/or/main.c:1987 #13 0x000055555556c9cf in tor_main (argc=<optimized out>, argv=0x7fffffffe3c8) at src/or/main.c:2703 #14 0x00007ffff689c76d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6 #15 0x0000555555566cdd in _start () I rebuilt without bufferevents, and it hasn't crashed yet. (I also don't see the "Something tried to close an or_connection_t" warnings.) - Ian

2 2

Using Stem's descriptor fetching module to replace the Java consensus-health checker
by Karsten Loesing 20 Aug '13

20 Aug '13

Hi Damian, we briefly discussed Stem's new descriptor fetching module and how we could extend the existing simple monitors [0] towards a replacement of the Java consensus-health checker [1]. Moving this discussion to this list with your permission. So, you asked what exactly the consensus-health checker a.k.a. DocTor looks for. Let me try to give you a quick overview of the different parts [2]. Going through the Java source files in an order that hopefully explains best how everything works together: - Warning.java is an enum of all different warnings that DocTor can emit. Each warning contains a little documentation string saying what it means. If these are ambiguous, let me know, and I can probably explain them better. - Checker.java contains the various checks that are performed on previously downloaded consensuses and votes. For example, checkMissingConsensuses goes through the (hard-coded) list of known directory authorities and emits a ConsensusDownloadTimeout warning if we couldn't download the consensus from at least one of them. As you see, there are plenty more check* methods. - StatusFileReport.java uses the results from Checker by putting all warnings in two output files, one of them containing all warnings, the other only containing new warnings. Each warning has a severity, which can be ERROR, WARNING, or NOTICE. Also, each warning defines a time after which we consider the exact same warning string new even though the warning hasn't changed. The latter is useful to rate-limit warnings. For example, the fact that a certificate is going to expire in two months from now doesn't have to be repeated every hour. - MetricsWebsiteReport.java is the second output of DocTor. It's the website available at [3]. The idea is that the website gives more information about warnings received on IRC or via email. It's actually a hack that this website is presented on metrics. In a rewrite, PyDoctor would have its own little webserver to present consensus-health details. Once it's in place and we shut down DocTor, I'm going to replace the website on metrics with a static page linking to PyDoctor. - DownloadStatistics.java keeps statistics about consensus download times which are displayed on the website. - Downloader.java is a wrapper for metrics-lib's descriptor downloader. - Main.java puts everything together. It first downloads everything, then writes the status files containing warnings, and then generates the website output. So, that's what DocTor does right now. Here are two more things that would be great to have in DocTor or PyDoctor: - Warn if directory authorities assign flags to unusually few or many relays [4]. This enhancement has the potential of generating lots of warnings, because the directory authorities currently vote *very* differently on certain flags. The result will be a lot of directory authority operator nagging. Just saying, you should be prepared for that when deploying this! - Ignore certain known warnings [5]. This will reduce a lot of noise on the consensus-health mailing list. The fewer noise there is the more people will pay attention to actually valid warnings. In theory. Hope that makes sense. Happy to provide more input or review code. Just let me know! All the best, Karsten [0] https://lists.torproject.org/pipermail/tor-dev/2013-July/005209.html [1] https://www.torproject.org/getinvolved/volunteer#metrics-pyDoctor [2] https://gitweb.torproject.org/doctor.git/tree/HEAD:/src/org/torproject/doct… [3] https://metrics.torproject.org/consensus-health.html [4] https://trac.torproject.org/projects/tor/ticket/9103 [5] https://trac.torproject.org/projects/tor/ticket/8797

2 13

[GSoC 2013] Status report - Searchable metrics archive
by Kostas Jakeliunas 19 Aug '13

19 Aug '13

Hello, another busy benchmarking + profiling period for database querying, but this time more rigorous and awesome. * wrote a generic query analyzer which logs query statements, EXPLAIN, ANALYZE, spots and informs of particular queries that yield inefficient query plans; * wrote a very simple but rather exhaustive profiler (using python's cProfile) which logs query times, function calls, etc.; output is used to see which parts of the e.g. backend are slow during API calls; output can be easily used to construct a general query 'profile' for a particular database, etc.; [1] * benchmarked lots of different queries using these tools, recorded query times, was able to observe deviations/discrepancies; * uploaded the whole database and benchmarked briefly on an amazon EC2 m2.2xlarge instance; * concluded that, provided there is enough memory to cache *and hold* the indexes in cache, query times are good; * in particular, tested the following query scheme extensively: [2] (see comments there as well if curious); concluded that it runs well; * opted for testing raw SQL queries (from within Flask/python) - so far, translating them into ORM queries (while being careful) resulted in degraded performance; if we have to end up using raw SQL, I will create a way to encapsulate them nicely; * made sure data importing is not slowed and remains a quick-enough procedure; * researched PostgreSQL stuff, especially its two-layer caching; I now have an understanding of the way pgsql caches things in memory, how statistics on index usage are gathered and used for maintaining buffer_cache, etc. The searchable metrics archive would work best when all of its indexes are kept in memory. * to this end, looked into buffer cache hibernation [3], etc.; I think pg_prewarm [4, 5] would serve our purpose well. (Apparently many business/etc. solutions do find cache prewarming relevant - pity it's not supported in stock PostgreSQL.) The latter means that * I don't think we can avoid using certain postgresql extensions (if only one) - which means that deploying will always take more than apt-get && pip install, but I believe it is needed; * next on my agenda is testing pg_prewarm on EC2 and, hopefully, putting our beloved database bottleneck problem to rest. I planned to expose the EC2 for public tor-dev inquiry (and ended up delaying status report yet again), but I'll have to do this separately. This is possible, however. Sorry for the delayed report. ## More generally, I'm happy with my queer queries [2] now; the two constraints/goals of * being able to run Onionoo-like queries on the whole descriptor / status entry database * being able to get a list of status entries for a particular relay will hopefully be put to rest very soon. The former is done, provided I have no trouble setting up a database index precaching system (which will ensure that all queries of the same syntax/scheme run quick enough.) Overall, I'm spending a bit too much time on a specific problem, but at least I have a more intimate lower-level knowledge of PostgreSQL, which turns out to be very relevant to this project. I hope to be able to soon move to extending Onionoo support and providing a clean API for getting lists of consensuses in which a particular relay was present. And maybe start with the frontend. :) Kostas. [1]: https://github.com/wfn/torsearch/commit/8e6f16a07c40f7806e98e9c71c1ce0f8e38… [2]: https://github.com/wfn/torsearch/blob/master/misc/nested_join.sql [3]: http://postgresql.1045698.n5.nabble.com/patch-for-new-feature-Buffer-Cache-… [4]: http://www.postgresql.org/message-id/CA+TgmobRrRxCO+t6gcQrw_dJw+Uf9ZEdwf9be… [5]: http://raghavt.blogspot.com/2012/04/caching-in-postgresql.html

2 8

Draft of proposal "Stop HS address enumeration by HSDirs"
by George Kadianakis 18 Aug '13

18 Aug '13

Greetz, I'm posting the draft of a proposal that specifies how to hide HS descriptors and addresses from the hidden service directories. This proposal is supposed to go along with the proposal that specifies how to upgrade HS identity keys to use Ed25519. I posted that proposal some seconds ago. This proposal (like the other one) is incredibly drafty in the sense that I might have forgotten to specify things that need to be specified. On the other hand, "release early; release often" they say, so here it goes. Inlining: Filename: xxx-onion-antienumeration-hsdirs.txt Title: Stop HS address enumeration by HSDirs Author: George Kadianakis Created: 16 August 2013 Target: 0.2.5.x Status: Draft [More draft than Guiness.] 0. Proposal overview and motivation: Currently, it is the case that, HSDirs can read the descriptors of the Hidden Services they host. Even if the descriptor was encrypted, HSDirs could still learn the address of the HSes by logging the client directory requests (which contain the hidden service address). This proposal: a) Defines cryptographic procedures to be followed by Tor clients and hidden services to allow HS address antienumeration. b) Defines a new descriptor format which hides the descriptor content from entities who don't know the public key of the HS. c) Changes the size (and semantics) of Hidden Service addresses. 1. Acknowledgments The cryptography behind this proposal was originally proposed by Robert Ransom in a private email threadi and subsequently posted in tor-dev [0]. Discussion was continued in trac ticket #8106 [1]. During the past 6 months many bright people have looked at the cryptography behind this scheme. The list of people includes Nadia Heninger, Leonid Reyzin, Nick Hopper, Aggelos Kiayias, Tanja Lange, Dan J. Bernstein and probably others that I can't recall at this point. Thanks! 2. Parameters TIME_PERIOD is XXX 3. Scheme overview Currently, Hidden Services upload their unencrypted descriptor to hidden service directories (HSDirs). HSDirs store the unencrypted descriptor in an internal map of: <hs address> -> <hs descriptor> When a client wants the descriptor of an HS, it asks an HSDir for the descriptor that corresponds to <hs address>. If the HSDir has such an index in its map, it returns the <hs descriptor> to the client. This proposal asks Hidden Services, to periodically derive a new ephemeral keypair from its long-term identity key; the new keypair is a function of the identity key and a time-dependent nonce. The derivation should be one-way; if you know the identity key you should be able to derive the ephemeral key, but not the other way around. Finally, a client should be able to derive the ephemeral HS public key from the long-term HS public key without knowing the long-term HS private key (#KEYPAIRDERIVATION) Hidden Services then encrypt their descriptor with a symmetric key (derived from the ephemeral public key) and sign the ciphertext and the ephemeral public key with their ephemeral private key. Then they place the ephemeral public key, the encrypted descriptor and the signature in a "metadescriptor" document (#METADESC) and send it to the HSDir (#DESCPUBLISH). The HSDir validates the signature of the "metadescriptor", and if it's legit it stores the metadescriptor in an internal map of: <ephemeral public key> -> <metadescriptor> . (#DESCPUBLISH) Now, out of band, the HS gives to its clients a <z>.onion address. <z> in this case is the long-term public key of the HS (this is different from the current situation where <z> is the hash of the long-term public key). A client that knows the <z>.onion address and wants to acquire the HS descriptor, derives the <ephemeral public key> of the HS by using <z> and the same key derivation procedure that the HS uses. She also derives the symmetric key that decrypts the encrypted HS descriptor (#DESCFETCH). The client then contacts the appropriate HSDir and inquires for the descriptor with index <ephemeral public key>. If the HSDir has such an index in its internal map, it passes the <metadescriptor> to the client (#DESCFETCH). The client then verifies the signature of the metadescriptor, and if it's legit she decrypts the encrypted descriptor with the symmetric key. The client now has the Hidden Servide descriptor she was so looking for (#DESCDECRYPTION). 4. Security proof XXX A security proof of the above scheme is under development. We are not going to implement or deploy anything before we have a solid proof of this. 5. Changes to the current HS protocol 5.0. Related proposals This proposal is supposed to be applied on top of the "Migrate HS identity keys to Ed25519" proposal. 5.1. Ephemeral keypair derivation (#KEYPAIRDERIVATION) XXX Leaving this unpsecified for now till the security proof comes along. For now, let's assume that after this paragraph each HS has a per-TIME_PERIOD ephemeral keypair. It also has a symmetric key derived from the ephemeral public key (maybe even the hash of the ephemeral public key) to encrypt its descriptor. Let's also assume that each client can generate the ephemeral public key of a HS given only its long-term public key (and knowledge of the current time-dependent nonce). 5.2. HS descriptor publishing 5.2.1. Metadescriptor format (#METADESC) The format of the metadescriptor that is uploaded to the HSDir is: "ephemeral-public-key" NL public-key [At start, exactly once] The ephemeral public key for this time period in base64 encoding. "encrypted-descriptor" NL encrypted-descriptor [Exactly once] An encrypted v3 hidden service descriptor (as specified in xxx-hs-ecc-id-keys.txt). It's encrypted using the ephemeral symmetric key of the HS, encoded in base64 and surrounded with "-----BEGIN MESSAGE-----" and "-----END MESSAGE-----". "signature" NL signature [At end, exactly once] A signature of all fields above with the ephemeral private key of the hidden service. 5.2.2. Metadescriptor publishing (#DESCPUBLISH) The metadescriptor specified above is published to the appropriate HSDir by sending an HTTP 'POST' request to "/tor/rendezvous3/publish" as specified in the "Migrate HS identity keys to Ed25519" proposal. # XXX should this get its own URI, even though we assume that these two proposals will be implemented and deployed simultaneously? Upon receiving a descriptor, the directory server checks the signature, and discards the descriptor if the signature does not match the enclosed public key. If the signature matches, the directory server saves the descriptor in a map of: <ephemeral-public-key> -> <metadescriptor>. 5.3. Metadescriptor fetching (#DESCFETCH) A client that knows the long-term public key (onion address) of a hidden service can derive the ephemeral public key and the ephemeral symmetric key as specified in section 5.1. A client that wants to fetch the metadescriptor of an HS, does an HTTP 'GET' request to the appropriate directory server asking for "/tor/rendezvous3/<z>" where <z> is the ephemeral public key of the HS. # XXX should this get its own URI, even though we assume that these two proposals will be implemented and deployed simultaneously? 5.4. Metadescriptor decryption (#DESCDECRYPTION) The client then verifies the signature of the metadescriptor, and discards it if it doesn't match the ephemeral public key that was previously derived. If the signature matches, the client uses the derived ephemeral symmetric key to decrypt the 'encrypted-descriptor' part of the metadescriptor. 6. Discussion [Points here might deserve their own sections] Do we need the HSDir hash ring, even though the HS address and the descriptor are now hidden from HSDirs? An ed25519 public key is 32 bytes. 32 bytes in base32 encoding is 56 characters (or 52 with the '=' padding removed). Do we want a different URL encoding or are we happy with addresses like: mfrggzdfmztwq2lknnwg23tpobyxe43uov3ho6dzpjaueq2eivda.onion ? Refs: [0]: https://lists.torproject.org/pipermail/tor-dev/2012-September/004026.html . [1]: https://trac.torproject.org/projects/tor/ticket/8106

5 7

Globe without search term
by Karsten Loesing 15 Aug '13

15 Aug '13

21:43 #tor-dev: < rndm> karsten, looks like this now if it gets the maximum number of results http://globe.rndm.de/#/search/query= You're not online now, so replying via email: Looks really good! Minor tweaks to the phrasing: "To avoid too many requests," -> the better motivation is that a) we don't want to overload the poor Onionoo server and b) reduce delays and bandwidth requirements for the client. I think users will understand b) better and not care much about a). How about "To reduce waiting time and limit bandwidth usage," ? "try to use a search word" -> this message also shows up if the user tried a search term that is too general. How about "try to refine your search" ? Thanks! Nice work! Best, Karsten

2 5

[draft] Proposal 220: Migrate server identity keys to Ed25519
by Nick Mathewson 14 Aug '13

14 Aug '13

Here's a proposal I wrote about node key migration. I hope it meshes well with the authority identity migration proposal that Jake and Linus are doing. There are probably holes and mistakes here: let's fix them. Filename: 220-ecc-id-keys.txt Title: Migrate server identity keys to Ed25519 Authors: Nick Mathewson Created: 12 August 2013 Target: 0.2.5.x Status: Draft [Note: This is a draft proposal; I've probably made some important mistakes, and there are parts that need more thinking. I'm publishing it now so that we can do the thinking together.] 0. Introduction In current Tor designs, identity keys are limited to 1024-bit RSA keys. Clearly, that should change, because RSA doesn't represent a good performance-security tradeoff nowadays, and because 1024-bit RSA is just plain too short. We've already got an improved circuit extension handshake protocol that uses curve25519 in place of RSA1024, and we're using (where supported) P256 ECDHE in our TLS handshakes, but there are more uses of RSA1024 to replace, including: * Router identity keys * TLS link keys * Hidden service keys This proposal describes how we'll migrate away from using 1024-bit RSA in the first two, since they're tightly coupled. Hidden service crypto changes will be complex, and will merit their own proposal. In this proposal, we'll also (incidentally) be extirpating a number of SHA1 usages. 1. Overview When this proposal is implemented, every router will have an Ed25519 identity key in addition to its current RSA1024 public key. Ed25519 (specifically, Ed25519-SHA-512 as described and specified at http://ed25519.cr.yp.to/) is a desirable choice here: it's secure, fast, has small keys and small signatures, is bulletproof in several important ways, and supports fast batch verification. (It isn't quite as fast as RSA1024 when it comes to public key operations, since RSA gets to take advantage of small exponents when generating public keys.) (For reference: In Ed25519 public keys are 32 bytes long, private keys are 64 bytes long, and signatures are 64 bytes long.) To mirror the way that authority identity keys work, we'll fully support keeping Ed25519 identity keys offline; they'll be used to sign long-ish term signing keys, which in turn will do all of the heavy lifting. A signing key will get used to sign the things that RSA1024 identity keys currently sign. 1.1. 'Personalized' signatures Each of the keys introduced here is used to sign more than one kind of document. While these documents should be unambiguous, I'd going to forward-proof the signatures by specifying each signature to be generated, not on the document itself, but on the document prefixed with some distinguishing string. 2. Certificates and Router descriptors. 2.1. Certificates When generating a signing key, we also generate a certificate for it. Unlike the certificates for authorities' signing keys, these certificates need to be sent around frequently, in significant numbers. So we'll choose a compact representation. VERSION [1 Byte] TYPE [1 Byte] CERTIFIED_KEY [32 Bytes] EXPIRATION_DATE [4 Bytes] EXTRA_STUFF [variable length] SIGNATURE [64 Bytes] The "VERSION" field holds the value [01]. The "TYPE" field holds the value [01]. The CERTIFIED_KEY field is an Ed25519 public key. The expiration date is a day, given in DAYS since the epoch, after which this certificate isn't valid. The EXTRA_STUFF field is left for a future version of this format. [XXXX Is "EXTRA_STUFF" a good idea? -NM] Before processing any certificate, parties MUST know which identity key it is supposed to be signed by, and then check the signature. The signature is formed by signing the first N-64 bytes of the certificate prefixed with the string "Tor node signing key certificate v1". We also specify a revocation document for revoking a signing key or an identity key. Its format is: FIXED_PREFIX [8 Bytes] VERSION [1 Byte] KEYTYPE [1 Byte] IDENTITY_KEY [32 Bytes] REVOKED_KEY [32 Bytes] PUBLISHED [8 Bytes] EXTRA_STUFF [variable length] SIGNATURE [64 Bytes] FIXED_PREFIX is "REVOKEID" or "REVOKESK". VERSION is [01]. KEYTYPE is [01] for revoking a signing key or [02] or revoking an identity key. REVOKED_KEY is the key being revoked; IDENTITY_KEY is the node's Ed25519 identity key. PUBLISHED is the time that the document was generated, in seconds since the epoch. EXTRA_STUFF is left for a future version of this document. The SIGNATURE is generated with the same key as in IDENTITY_KEY, and covers the entire revocation, prefixed with "Tor key revocation v1". Using these revocation documents is unspecified. 2.2. Managing keys By default, we can keep the easy-to-setup key management properties that Tor has now, so that node operators aren't required to have offline public keys: * When a Tor node starts up with no Ed25519 identity keys, it generates a new identity keypair. * When a Tor node has an Ed25519 identity keypair, and it has no signing key, or its signing key is going to expire within the next 48 hours, it generates a new signing key to last 30 days. But we also support offline identity keys: * When a Tor node starts with an Ed25519 public identity key but no private identity key, it checks wither it has a currently valid certified signing keypair. If it does, it starts. Otherwise, it refuses to start. * If a Tor node's signing key is going to expire soon, it starts warning the user. If it is expired, then the node shuts down. 2.3. Router descriptors We specify the following element that may appear at most once in each router descriptor: "identity-ed25519" SP identity-key SP certification NL The identity-key and certification are base64 encoded with terminating =s removed. When this element is present, it MUST appear as the first or second element in the router descriptor. [XXX The rationale here is to allow extracting the identity key and signing key and checking the signature before fully parsing the rest of the document. -NM] When an identity-ed25519 element is present, there must also be an "router-signature-ed25519" element. It MUST be the next-to-last element in the descriptor, appearing immediately before RSA signature. It MUST contain an ed25519 signature of the entire document, from the first character up to but not including the "router-signature-ed25519" element, prefixed with the string "Tor router descriptor signature v1". Its format is: "router-signature-ed25519" SP signature NL Were 'signature' is encoded in base64 with terminating =s removed. The identity key in the identity-ed25519 key MUST be the one used to sign the certification, and the signing key in the certification MUST be the one used to sign the document. Note that these keys cross-certify as follows: the ed25519 identity key signs the ed25519 signing key in the certificate. The ed25519 signing key signs itself and the ed25519 identity key and the RSA identity key as part of signing the descriptor. And the RSA identity key also signs all three keys as part of signing the descriptor. 2.3.1. Checking descriptor signatures. Current versions of Tor will handle these new formats by ignoring the new fields, and not checking any ed25519 information. New version of Tor will have a flag that tells them whether to check ed25519 information. When it is set, they must check: * All RSA information and signatures that Tor implementations currently check. * If the identity-ed25519 line is present, it must be well-formed, and the certificate must be well-formed and correctly signed, and there must be a valid. * If we require an ed25519 key for this node (see 3.1 below), the ed25519 key must be present. Authorities and directory caches will have this flag always-on. For clients, it will be controlled by a torrc option and consensus option, to be set to "always-on" in the future once enough clients support it. 2.3.2. Extra-info documents Extrainfo documents now include "identity-ed25519" and "router-signature-ed25519" fields in the same positions in which they appear in router descriptors. Additionally, we add the base64-encoded, =-stripped SHA256 digest of a node's extra-info document field to the extra-info-digest line. (All versions of Tor that recognize this line allow an extra field there.) 2.3.3. A note on signature verification Here and elsewhere, we're receiving a certificate and a document signed with the key certified by that certificate in the same step. This is a fine time to use the batch signature checking capability of Ed25519, so that we can check both signatures at once without (much) additional overhead over checking a single signature. 3. Consensus documents and authority operation 3.1. Handling router identity at the authority When receiving router descriptors, authorities must track mappings between RSA and Ed25519 keys. Rule 1: Once an authority has seen an Ed25519 identity key and an RSA identity key together on the same (valid) descriptor, it should no longer accept any descriptor signed by that RSA key with a different Ed25519 key, or that Ed25519 key with a different RSA key. Rule 2: Once an authority has seen an Ed25519 identity key and an RSA identity key on the same descriptor, it should no longer accept any descriptor signed by that RSA key unless it also has that Ed25519 key. These rules together should enforce the property that, even if an attacker manages to steal or factor a node's RSA identity key, the attacker can't impersonate that node to the authorities, even when that node is identified by its RSA key. Enforcement of Rule 1 should be advisory-only for a little while (a release or two) while node operators get experience having Ed25519 keys, in case there are any bugs that cause or force identity key replacement. Enforcement of Rule 2 should be advisory-only for little while, so that node operators can try 0.2.5 but downgrade to 0.2.4 without being de-listed from the consensus. [XXX I could specify a way to do a signed "I'm downgrading for a while!" statement, and kludge some code back into 0.2.4.x to better support that?] 3.2. Formats Vote and consensus documents now contain an optional "id" field for each routerstatus section. Its format is: "id" SP ed25519-identity NL where ed25519-identity is base-64 encoded, with trailing = characters omitted. In vote documents, it may be replaced by the format: "id" SP "none" NL which indicates that the node does not have an ed25519 identity. (In the consensus, a lack of "id" line means that the node has no ed25519 identity.) [XXXX Should the id entries in consensuses go into microdescriptors instead? I think perhaps so. -NM] A vote or consensus document is ill-formed if it includes the same ed25519 identity key twice. 3.3. Generating votes An authority should pick which descriptor to choose for a node as before, and include the ed25519 identity key for the descriptor if it's present. As a transition, before Rule 1 and Rule 2 in 3.1 are fully enforced, authorities need a way to deal with the possibility that there might be two nodes with the same ed25519 key but different RSA keys. In that case, it votes for the one with the most recent publication date. (The existing rules already prevent an authority from voting for two servers with the same RSA identity key.) 3.4. Generating a consensus from votes This proposal requires a new consensus vote method. When we deploy it, we'll pick the next available vote method in sequence to use for this. When the new consensus method is in use, we must choose nodes first by ECC key, then by RSA key. First, for every {ECC identity key, RSA identity key} pair listed by over half of the voting authorities, list it, unless some other RSA identity key digest is listed more popularly for the ECC key. Break ties in favor of low RSA digests. Treat all routerstatus entries that mention this <ECC,RSA> pair as being for the same router, and all routerstatus entries that mention the same RSA key with an unspecified ECC key as being for the same router. Then, for every node that has previously not been listed, perform the current routerstatus algorithm: listing a node if it has been listed by at least N/2 voting authorities, and treating all routerstatuses containing the same identity as the same router. In other words: Let Entries = [] for each ECC ID listed by any voter: Find the RSA key associated with that ECC ID by the most voters, breaking ties in favor of low RSA keys. If that ECC ID and RSA key ID are listed by > N/2 voting authorities: Add the consensus of the routerstatus entries for those voters, along with the routerstatus entry for every voter that included that RSA key with no ECC key, to Entries. Include the ECC ID in the consensus. For each RSA key listed by any voter: If that RSA key is already in Entries, skip it. If the RSA key is listed by > N/2 voting authorities: Add the consensus of the routerstatus entries for those voters to Entries. Do not include an ECC key in the consensus. [XXX Think about this even more.] 4. The link protocol [XXX This section won't make much sense unless you grok the v3 connection protocol as described in tor-spec.txt, first proposed in proposal 195.] We add four new CertType values for use in CERTS cells: 4: Ed25519 signing key 5: Link key certificate certified by Ed25519 signing key 6: TLS authentication key certified by Ed25519 signing key 7: RSA cross-certification for Ed25519 identity key The content of certificate type 4 is: Ed25519 identity key [32 byets] Signing key certificate as in 2.1 above [variable length] The content of certificate type 5 is: CERT_DIGEST [32 bytes] EXPIRATION_DATE [4 bytes] EXTRA_STUFF [variable] SIGNATURE [64 bytes] where CERT_DIGEST is a SHA256 digest of the X.509 certificate used for the TLS link, EXPIRATION_DATE is a date in *days* since the epoch starting on which the certificate is invalid, and SIGNATURE is a signature using the signing key of the above two fields, prefixed with "Tor TLS link certificate check v1". The content of certificate type 6 is the same as the signing key in 2.1 above, except that the TYPE field is 02 and the fixed string used in signing is "". The Ed25519 key certified by this key can be used in AUTHENTICATE cells. The content of certificate type 7 is: ED25519_KEY [32 bytes] EXPIRATION_DATE [4 bytes] SIGNATURE [128 bytes] Here, the Ed25519 identity key is signed with router's identity key, to indicate that authenticating with a key certified by the Ed25519 key counts as certifying with RSA identity key. (The signature is computed on the SHA256 hash of the non-signature parts of the certificate, prefixed with the string "Tor TLS RSA/Ed25519 cross-certification".) (There's no reason to have a corresponding Ed25519-signed-RSA-key certificate here, since we do not treat authenticating with an RSA key as proving ownership of the Ed25519 identity.) Relays with Ed25519 keys should always send these certificate types in addition to their other certificate types. Non-bridge relays with Ed25519 keys should generate TLS link keys of appropriate strength, so that the certificate chain from the Ed25519 key to the link key is strong enough. We add a new authentication type for AUTHENTICATE cells: "Ed25519-TLSSecret", with AuthType value 2. Its format is the same as "RSA-SHA256-TLSSecret", except that the CID and SID fields support more key types; some strings are different, and the signature is performed with Ed25519 using the authentication key from a type-6 cert. TYPE: The characters "AUTH0002" [8 octets] CID: A SHA256 hash of the initiator's RSA1024 identity key [32 octets] SID: A SHA256 hash of the responder's RSA1024 identity key [32 octets] CID_ED: The initiator's Ed25519 identity key [32 octets] SID_ED: The responder's Ed25519 identity key, or all-zero. [32 octets] SLOG: A SHA256 hash of all bytes sent from the responder to the initiator as part of the negotiation up to and including the AUTH_CHALLENGE cell; that is, the VERSIONS cell, the CERTS cell, the AUTH_CHALLENGE cell, and any padding cells. [32 octets] CLOG: A SHA256 hash of all bytes sent from the initiator to the responder as part of the negotiation so far; that is, the VERSIONS cell and the CERTS cell and any padding cells. [32 octets] SCERT: A SHA256 hash of the responder's TLS link certificate. [32 octets] TLSSECRETS: A SHA256 HMAC, using the TLS master secret as the secret key, of the following: - client_random, as sent in the TLS Client Hello - server_random, as sent in the TLS Server Hello - the NUL terminated ASCII string: "Tor V3 handshake TLS cross-certification with Ed25519" [32 octets] TIME: The time of day in seconds since the POSIX epoch. [8 octets] RAND: A 16 byte value, randomly chosen by the initiator. [16 octets] SIG: A signature of all previous fields using the initiator's Ed25519 authentication flags. [variable length] If you've got a consensus that lists an ECC key for a node, but the node doesn't give you an ECC key, then refuse this connection. 5. The extend protocol We add a new NSPEC node specifier for use in EXTEND2 cells, with LSTYPE value [03]. Its length must be 32 bytes; its content is the Ed25519 identity key of the target node. Clients should use this type only when: * They know an Ed25519 identity key for the destination node. * The source node supports EXTEND2 cells * A torrc option is set, _or_ a consensus value is set. We'll leave the consensus value off for a while until more clients support this, and then turn it on. When picking a channel for a circuit, if this NSPEC value is provided, then the RSA identity *and* the Ed25519 identity must match. If we have a channel with a given Ed25519 ID and RSA identity, and we have a request for that Ed25519 ID and a different RSA identity, we do not attempt to make another connection: we just fail and DESTROY the circuit. If we receive an EXTEND or EXTEND2 request for a node listed in the consensus, but that EXTEND/EXTEND2 request does not include an Ed25519 identity key, the node SHOULD treat the connection as failed if the Ed25519 identity key it receives does not match the one in the consensus. 6. Naming nodes in the interface Anywhere in the interface that takes an $identity should be able to take an ECC identity too. ECC identities are case-sensitive base64 encodings of Ed25519 identity keys. You can use $ to indicate them as well; we distinguish RSA identity digests length. When we need to indicate an Ed25519 identity key in an hostname format (as in a .exit address), we use the lowercased version of the name, and perform a case-insensitive match. (This loses us one bit per byte of name, Nodes must not list Ed25519 identities in their family lines; clients and authorities must not honor them there. Clients shouldn't accept .exit addresses with Ed25519 names on SOCKS or DNS ports by default, even when AllowDotExit is set. We need an identity-to-node map for ECC identity and for RSA identity. The controller interface will need to accept and report Ed25519 identity keys as well as (or instead of) RSA identity keys. That's a separate proposal, though. 7. Hidden service changes out of scope Hidden services need to be able identity nodes by ECC keys, just as they will need to include ntor keys as well as TAP keys. Not just yet though. This needs to be part of a bigger hidden service revamping strategy. 8. Proposed migration steps Once a few versions have shipped with Ed25519 key support, turn on "Rule 1" on the authorities. (Don't allow an Ed25519<->RSA pairing to change.) Once 0.2.5.x is in beta or rc, turn on the consensus option for everyone who receives descriptors with Ed25519 identity keys to check them. Once 0.2.5.x is in beta or rc, turn on the consensus option for clients to generate EXTEND2 requests with Ed25519 identity keys. Once 0.2.5.x has been stable for a month or two, turn on "Rule 2" on authorities. (Don't allow nodes that have advertised an Ed25519 key to stop.) 9. Future proposals * Ed25519 identity support on the controller interface * Supporting nodes without RSA keys * Remove support for nodes without Ed25519 keys * Ed25519 support for hidden services * Bridge identity support. * Ed25519-aware family support

5 8

[draft] Proposal 219: Support for full DNS and DNSSEC resolution in Tor
by Nick Mathewson 13 Aug '13

13 Aug '13

This is a proposal of Ondrej's from last year that I've edited a little, with permission. I'm assigning it a number so that we can keep it on the radar. Discussion invited! Let's try to answer the open questions too. Filename: 219-expanded-dns.txt Title: Support for full DNS and DNSSEC resolution in Tor Authors: Ondrej Mikle Created: 4 February 2012 Modified: 2 August 2013 Target: 0.2.5.x Status: Draft 0. Overview Adding support for any DNS query type to Tor. 0.1. Motivation Many applications running over Tor need more than just resolving FQDN to IPv4 and vice versa. Sometimes to prevent DNS leaks the applications have to be hacked around to be supplied necessary data by hand (e.g. SRV records in XMPP). TLS connections will benefit from planned TLSA record that provides certificate pinning to avoid another Diginotar-like fiasco. 0.2. What about DNSSEC? Routine DNSSEC resolution is not practical with this proposal alone, because of round-trip issues: a single name lookup can require dozens of round trips across a circuit, rendering it very slow. (We don't want to add minutes to every webpage load time!) For records like TLSA that need extra signing, this might not be an unacceptable amount of overhead, but routine hostname lookup, it's probably overkill. [Further, thanks to the changes of proposal 205, DNSSEC for routine hostname lookup is less useful in Tor than it might have been back when we cached IPv4 and IPv6 addresses and used them across multiple circuits and exit nodes.] See section 8 below for more discussion of DNSSEC issues. 1. Design 1.1 New cells There will be two new cells, RELAY_DNS_BEGIN and RELAY_DNS_RESPONSE (we'll use DNS_BEGIN and DNS_RESPONSE for short below). 1.1.1. DNS_BEGIN DNS_BEGIN payload: FLAGS [2 octets] DNS packet data (variable length, up to length of relay cell.) The DNS packet must be generated internally by Tor to avoid fingerprinting users by differences in client resolvers' behavior. [XXXX We need to specify the exact behavior here: saying "Just do what Libunbound does!" would make it impossible to implement a Tor-compatible client without reverse-engineering libunbound. - NM] The FLAGS field is reserved, and should be set to 0 by all clients. Because of the maximum length of the RELAY cell, the DNS packet may not be longer than 496 bytes. [XXXX Is this enough? -NM] Some fields in the query must be omitted or set to zero: see section 3 below. 1.1.2. DNS_RESPONSE DNS_RESPONSE payload: STATUS [1 octet] CONTENT [variable, up to length of relay cell] If the low bit of STATUS is set, this is the last DNS_RESPONSE that the server will send in response to the given DNS_BEGIN. Otherwise, there will be more DNS_RESPONSE packets. The other bits are reserved, and should be set to zero for now. The CONTENT fields of the DNS_RESPONSE cells contain a DNS record, split across multiple cells as needed, encoded as: total length (2 octets) data (variable) So for example, if the DNS record R1 is only 300 bytes long, then it is sent in a single DNS_RESPONSE cell with payload [01 01 2C] R1. But if the DNS record R2 is 1024 bytes long, it's sent in 3 DNS_RESPONSE cells, with contents: [00 04 00] R2[0:495], [00] R2[495:992], and [01] R2[992:1024] respectively. [NOTE: I'm using the length field and the is-this-the-last-cell field to allow multi-packet responses in the future. -NM] AXFR and IXRF are not supported in this cell by design (see specialized tool below in section 5). 1.1.3. Matching queries to responses. DNS_BEGIN must use a non-zero, distinct StreamID. The client MUST NOT re-use the same stream ID until it has received a complete response from the server or a RELAY_END cell. The client may cancel a DNS_BEGIN request by sending a RELAY_END cell. The server may refused to answer, or abort answering, a DNS_BEGIN cell by sending a RELAY_END cell. 2. Interfaces to applications DNSPort evdns - existing implementation will be updated to use DNS_BEGIN. [XXXX we should add a dig-like tool that can work over the socksport via some extension, as tor-resolve does now. -NM] 3. Limitations on DNS query Clients must only set query class to IN (INTERNET), since the only other useful class CHAOS is practical for directly querying authoritative servers (OR in this case acts as a recursive resolver). Servers MUST return REFUSED for any for class other than IN. Multiple questions in a single packet are not supported and OR will respond with REFUSED as the DNS error code. All query RR types are allowed. [XXXX I originally thought about some exit policy like "basic RR types" and "all RRs", but managing such list in deployed nodes with extra directory flags outweighs the benefit. Maybe disallow ANY RR type? -OM] Client as well as OR MUST block attempts to resolve local RFC 1918, 4193, or 4291 adresses (PTR). REFUSED will be returned as DNS error code from OR. [XXXX Must they also refuse to report addresses that resolve to these? -NM] [XXX I don't think so. People often use public DNS records that map to private adresses. We can't effectively separate "truly public" records from the ones client's dnsmasq or similar DNS resolver returns. - OM] [XXX Then do you mean "must be returned as the DNS error from the OP"?] Request for special names (.onion, .exit, .noconnect) must never be sent, and will return REFUSED. The DNS transaction ID field MUST be set to zero in all requests and replies; the stream ID field plays the same function in Tor. 4. Implementation notes Client will periodically purge incomplete DNS replies. Any unexpected DNS_RESPONSE will be dropped. AD flag must be zeroed out on client unless validation is performed. [XXXX libunbound lowlevel API, Tor+libunbound libevent loop libunbound doesn't publicly expose all the necessary parts of low-level API. It can return the received DNS packet, but not let you construct a packet and get it in wire-format, for example. Options I see: a) patch libunbound to be able feed wire-format DNS packets and add API to obtain constructed packets instead of sending over network b) replace bufferevents for sockets in unbound with something like libevent's paired bufferevents. This means that data extracted from DNS_RESPONSE/DNS_BEGIN cells would be fed directly to some evbuffers that would be picked up by libunbound. It could possibly result in avoiding background thread of libunbound's ub_resolve_async running separate libevent loop. c) bind to some arbitrary local address like 127.1.2.3:53 and use it as forwarder for libunbound. The code there would pack/unpack the DNS packets from/to libunbound into DNS_BEGIN/DNS_RESPONSE cells. It wouldn't require modification of libunbound code, but it's not pretty either. Also the bind port must be 53 which usually requires superuser privileges. Code of libunbound is fairly complex for me to see outright what would the best approach be. ] 5. Separate tool for AXFR The AXFR tool will have similar interface like tor-resolve, but will return raw DNS data. Parameters are: query domain, server IP of authoritative DNS. The tool will transfer the data through "ordinary" tunnel using RELAY_BEGIN and related cells. This design decision serves two goals: - DNS_BEGIN and DNS_RESPONSE will be simpler to implement (lower chance of bugs) - in practice it's often useful do AXFR queries on secondary authoritative DNS servers IXFR will not be supported (infrequent corner case, can be done by manual tunnel creation over Tor if truly necessary). 6. Security implications As proposal 171 mentions, we need mitigate circuit correlation. One solution would be keeping multiple streams to multiple exit nodes and picking one at random for DNS resolution. Other would be keeping DNS-resolving circuit open only for a short time (e.g. 1-2 minutes). Randomly changing the circuits however means that it would probably incur additional latency since there would likely be a few cache misses on the newly selected exits. [This needs more analysis; We need to consider the possible attacks here. It would be good to have a way to tie requests to SocksPorts, perhaps? -NM] 7. TTL normalization idea A bit complex on implementation, because it requires parsing DNS packets at exit node. TTL in reply DNS packet MUST be normalized at exit node so that client won't learn what other clients queried. The normalization is done in following way: - for a RR, the original TTL value received from authoritative DNS server should be used when sending DNS_RESPONSE, trimming the values to interval [5, 600] - does not pose "ghost-cache-attack", since once RR is flushed from libunbound's cache, it must be fetched anew 8. DNSSEC notes 8.1. Where to do the resolution? DNSSEC is part of the DNS protocol and the most appropriate place for DNSSEC API would be probably in OS libraries (e.g. libc). However that will probably take time until it becomes widespread. On the Tor's side (as opposed to application's side), DNSSEC will provide protection against DNS cache-poisoning attacks (provided that exit is not malicious itself, but still reduces attack surface). 8.2. Round trips and serialization Following are two examples of resolving two A records. The one for addons.mozila.org is an example of a "common" RR without CNAME/DNAME, the other for www.gov.cn an extreme example chained through 5 CNAMEs and 3 TLDs. The examples below are shown for resolving that started with an empty DNS cache. Note that multiple queries are made by libunbound as it tries to adjust for the latency of network. "Standard query response" below that does not list RR type is a negative NOERROR reply with NSEC/NSEC3 (usually reply to DS query). The effect of DNS cache plays a great role - once DS/DNSKEY for root and a TLD is cached, at most 3 records usually need to be fetched for a record that does not utilize CNAME/DNAME (3 roundtrips for DS, DNSKEY and the record itself if there are no zone cuts below). Query for addons.mozilla.org, 6 roundtrips (not counting retries): Standard query A addons.mozilla.org Standard query A addons.mozilla.org Standard query A addons.mozilla.org Standard query A addons.mozilla.org Standard query A addons.mozilla.org Standard query response A 63.245.217.112 RRSIG Standard query response A 63.245.217.112 RRSIG Standard query response A 63.245.217.112 RRSIG Standard query A addons.mozilla.org Standard query response A 63.245.217.112 RRSIG Standard query response A 63.245.217.112 RRSIG Standard query A addons.mozilla.org Standard query response A 63.245.217.112 RRSIG Standard query response A 63.245.217.112 RRSIG Standard query DNSKEY <Root> Standard query DNSKEY <Root> Standard query response DNSKEY DNSKEY RRSIG Standard query response DNSKEY DNSKEY RRSIG Standard query DS org Standard query response DS DS RRSIG Standard query DNSKEY org Standard query response DNSKEY DNSKEY DNSKEY DNSKEY RRSIG RRSIG Standard query DS mozilla.org Standard query response DS RRSIG Standard query DNSKEY mozilla.org Standard query response DNSKEY DNSKEY DNSKEY RRSIG RRSIG Query for www.gov.cn, 16 roundtrips (not counting retries): Standard query A www.gov.cn Standard query A www.gov.cn Standard query A www.gov.cn Standard query A www.gov.cn Standard query A www.gov.cn Standard query response CNAME www.gov.chinacache.net CNAME www.gov.cncssr.chinacache.net CNAME www.gov.foreign.ccgslb.com CNAME wac.0b51.edgecastcdn.net CNAME gp1.wac.v2cdn.net A 68.232.35.119 Standard query response CNAME www.gov.chinacache.net CNAME www.gov.cncssr.chinacache.net CNAME www.gov.foreign.ccgslb.com CNAME wac.0b51.edgecastcdn.net CNAME gp1.wac.v2cdn.net A 68.232.35.119 Standard query A www.gov.cn Standard query response CNAME www.gov.chinacache.net CNAME www.gov.cncssr.chinacache.net CNAME www.gov.foreign.ccgslb.com CNAME wac.0b51.edgecastcdn.net CNAME gp1.wac.v2cdn.net A 68.232.35.119 Standard query response CNAME www.gov.chinacache.net CNAME www.gov.cncssr.chinacache.net CNAME www.gov.foreign.ccgslb.com CNAME wac.0b51.edgecastcdn.net CNAME gp1.wac.v2cdn.net A 68.232.35.119 Standard query response CNAME www.gov.chinacache.net CNAME www.gov.cncssr.chinacache.net CNAME www.gov.foreign.ccgslb.com CNAME wac.0b51.edgecastcdn.net CNAME gp1.wac.v2cdn.net A 68.232.35.119 Standard query A www.gov.cn Standard query response CNAME www.gov.chinacache.net CNAME www.gov.cncssr.chinacache.net CNAME www.gov.foreign.ccgslb.com CNAME wac.0b51.edgecastcdn.net CNAME gp1.wac.v2cdn.net A 68.232.35.119 Standard query response CNAME www.gov.chinacache.net CNAME www.gov.cncssr.chinacache.net CNAME www.gov.foreign.ccgslb.com CNAME wac.0b51.edgecastcdn.net CNAME gp1.wac.v2cdn.net A 68.232.35.119 Standard query A www.gov.chinacache.net Standard query response CNAME www.gov.cncssr.chinacache.net CNAME www.gov.foreign.ccgslb.com CNAME wac.0b51.edgecastcdn.net CNAME gp1.wac.v2cdn.net A 68.232.35.119 Standard query A www.gov.cncssr.chinacache.net Standard query response CNAME www.gov.foreign.ccgslb.com CNAME wac.0b51.edgecastcdn.net CNAME gp1.wac.v2cdn.net A 68.232.35.119 Standard query A www.gov.foreign.ccgslb.com Standard query response CNAME wac.0b51.edgecastcdn.net CNAME gp1.wac.v2cdn.net A 68.232.35.119 Standard query A wac.0b51.edgecastcdn.net Standard query response CNAME gp1.wac.v2cdn.net A 68.232.35.119 Standard query A gp1.wac.v2cdn.net Standard query response A 68.232.35.119 Standard query DNSKEY <Root> Standard query response DNSKEY DNSKEY RRSIG Standard query DS cn Standard query response Standard query DS net Standard query response DS RRSIG Standard query DNSKEY net Standard query response DNSKEY DNSKEY RRSIG Standard query DS chinacache.net Standard query response Standard query DS com Standard query response DS RRSIG Standard query DNSKEY com Standard query response DNSKEY DNSKEY RRSIG Standard query DS ccgslb.com Standard query response Standard query DS edgecastcdn.net Standard query response Standard query DS v2cdn.net Standard query response An obvious idea to avoid so many roundtrips is to serialize them together. There has been an attempt to standardize such "DNSSEC stapling" [1], however it's incomplete for the general case, mainly due to various intricacies - proofs of non-existence, NSEC3 opt-out zones, TTL handling (see RFC 4035 section 5). References: [1] https://www.ietf.org/mail-archive/web/dane/current/msg02823.html

2 1

[GSOC 2013] Steganography Browser Extension Status Report
by Hareesan 10 Aug '13

10 Aug '13

Hi, In my plan for this week, I supposed to work on image steganography libraries. But due to some latency in decision making about the steganography libraries, I continued working on adding preferences and creating data saving mechanism for keys and filters using sqlite databases. In my next week plan, I will be working on image steganography libraries part of the extension to add image steganography to the extension. -- Hareesan

1 0