- tor-dev - lists.torproject.org

Status report - Stream-RTT
by ra 26 Aug '13

26 Aug '13

Hello! Here goes my fourth status report on reducing the RTT of Tor circuits: I continued to evaluate lots of RTT-data of Tor circuits and drew some conclusions from that: *) The RTT measurements of circuits are subject to multiple influences and hence very diverse. *) There is no single distribution that fits them for all circuits. I am currently in the process of doing a demonstration of the improvement the active RTT probing yields. I delayed this report to provide some actual results but the demo is still half-baked.. Best, Robert

1 0

[GSOC 2013] Steganography Browser Extension Status Report
by Hareesan 25 Aug '13

25 Aug '13

Hi, In my previous mail, I was planning to add steganography libraries to the extension. But with the meeting with mentor, we planned to complete the code with proper documentation and to add public key cryptography features. But in my plan, I was able to complete all the UIs and back end framework for all the parts of the extension other than few updates remaining in options page. In cumming weeks, I will be working on completion of code and will add public key cryptography libraries which need some modification from existing libraries. -- Hareesan

1 0

Haskell packages?
by Nikita Karetnikov 23 Aug '13

23 Aug '13

I'd like to improve my Haskell skills. Are there any opportunities? I've been told there is at least one project that uses Haskell, which is not maintained. (For example, this page [1] mentions TorDNSEL, which was replaced by TorBEL.) [1] https://www.torproject.org/getinvolved/volunteer.html.en

5 21

GSOC Status Report - EvilGenius
by Johannes Fürmann 23 Aug '13

23 Aug '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello *, During the last two weeks I was primarily busy packing literally everything I own into boxes and unpacking it again, as well as other logistics that have to be done when moving. My equipment as well as my brain and the rest of me should be fully operational again on monday. In spite of the loads of stuff I had to do the last weeks, I still made a few improvement on my projects: * Probe running and logging functionality is now in place * A few improvements on networking: Addresses are now actually set and the corresponding links are pulled up. * Traceroute instrument for debugging purposes. You all have a wonderful weekend ~johannes -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJSF64/AAoJEAgie5o3Q/JpVN4P/00Zbt3l5H0ASF5KBeUE1/7F jTfyDZzYBKr1dV48V4dExkIdud6dtS6gTeaU9gjlrkH/jr6n81Dg4cxJVEH17n2M wrFSN7QiwVOrD/iPfApJhERIT+h3S503ra2Jejg2nnQTwF86Ifcgvr8gMvKus1b5 ddP8nZQSZqY5e+oZBkVUnjvXW4elc7f/XjRmfs6DmwyvTUGYC8Q+B1+bTMEauQU3 9VlY96HzjTzAKqST/7VihVFwToN0T7KK8pIZeH6O6JN7xcrbX2PfjqvRbPrMgrCE PV82yGVLnzX2jJkIbCFjkZZBTZJKd3X8hZ0A6lftko7SKHJBriSl9lEmxRHtq1pP FQVGAB1x/LEDgcaImupXT9IcnY6BJm2mdZ1Le+CPwY3RJq/CdChdxftGDCPqxl62 6yKOYMIUPxE+I9Cw+LZviFOcRCyH9TxOe918fLJeZ/buPViq+Xf1tw9duBHqq/xr OWTZL2qb5fjcJTG5uomGvwi7fOYq3gTHI/1JFlq9abVpUICrns1nEtevwP7RSb7D LyxIm1zsyIzt+8Qgyxh9uVnOrQY6nZiFmAGy8V3KTxIboAr+Z2booF+EgHwroR/g Sn8bcNng6Ln/cvWsU24OoG53/a4stLo5yNMzFj+p/hU5uvYF9R/4/IDJ6SqqqBzR cEOFb9tzJhCZVD5W9rLb =gX0j -----END PGP SIGNATURE-----

1 0

Global semi-passive adversary: suggestion of using expanders
by Paul-Olivier Dehaye 23 Aug '13

23 Aug '13

Hello, Thank you for working on Tor. I have a suggestion and would appreciate input. Please bear with me as I have a limited understanding of the design of Tor and all the different threats that it is meant to mitigate. Below, a (?) indicates a place where I need some confirmation that my understanding is correct, and N indicates either the number of Tor nodes, the number of end-users, or the amount of traffic (I assume these are all linearly related). As far as I can tell, the main threat by a global passive adversary comes from traffic analysis (?). This attack should become easier as the number of Tor nodes increases (?): Tor uses a clique topology, so the number of edges potentially carrying traffic grows like N^2. A dual way to see this is that not enough mixing can happen around a node for incoming/outgoing edge pairs, bar injecting a huge amount of fake traffic. To compensate, it seems natural to look for a sparse yet highly mixing network topology. Mathematically, those are called expanders [1]. A typical example of a family of expanders would be the Erdos-Renyi model [2], and indeed I have found in the literature suggestions for basing anonymizing protocols on such a model. The analysis in the presence of an active adversary becomes very difficult though. Alternatively, one could use a different method for constructing that expander topology, working "all at once". This comes from recent mathematics research (<= 5 years, certainly not my own, see [3]). The graph is then a Cailey graph [4] in a matrix group (the group is fixed and determined by an approximation to the number of Tor nodes, such as nearest third power of a prime number). In some sense this construction interpolates between mixing chains and Tor, and can be seen as a lot of mixing chains interwoven. In the setting of Tor, constructing the Cailey graph would require making two distributed randomize choices: - a matching of elements of the group to Tor nodes (possibly 2:1 for some Tor nodes) - a small subset of generators for the Cailey graph >From my understanding of security protocols, it should be easy to do these two choices safely and fast, as it amounts to choosing a random element in S_N and filling lots of matrix entries with random elements between 1 and a prime p, with some rejection. Once that is done, the network topology is fully determined, and with very high probability gives an expander. This means that traffic gets mixed up in very few hops. The number of hops needed grows as log N, with a constant that can be mitigated by chosing a large generating set above. This is the only downside I see (apart from difficulty to explain the math behind this): the latency would increase, from 3 in the current protocol to maybe 10 or so. I don't know the details of the behaviour of the constants in the last paragraph, and would appreciate feedback from the list before looking too much into this. Paul Dehaye [1] http://en.wikipedia.org/wiki/Expander_graph [2] http://en.wikipedia.org/wiki/Erd%C5%91s%E2%80%93R%C3%A9nyi_model [3] http://terrytao.wordpress.com/2011/12/02/245b-notes-1-basic-theory-of-expan… 15 and remark below [4] http://en.wikipedia.org/wiki/Cayley_graph

4 4

[GSoC 2013] Status report - Searchable metrics archive
by Kostas Jakeliunas 23 Aug '13

23 Aug '13

Hello! Updating on my Searchable Tor metrics archive project. (As is very evident) I'm very open for naming suggestions. :) To the best of my understanding and current satisfaction, I solved the database bottlenecks, or at least I am, as of now, satisfied with the current output from my benchmarking utility. Things may change, but I am confident (and have support to argue) that the whole thing runs swell at least on amazon m2.2xlarge instances. For fun and profit, a part of the database (which, has, for now, status entries (only) in the range [2010-01-01 00:00:00, 2013-05-31 23:00:00]), namely, what is currently used by the Onionoo-like API is now available online (not on EC2, though) - will now write a separate email so that everyone can inspect it. I should now move on with implementing / extending the Onionoo API, in particular, working on date range queries, and refining/rewriting the "list status entries" API point (see below). Need to carefully plan some things, and always keep an updated API document. (Also need to update and publish a separate, more detailed specification document.) More concrete report points: - re-examined my benchmarking approach, and wrote a rather simple but effective set of benchmarking tools (more like a simple script) [1] that can be hopefully used outside this project as well; at the very least, together with the profiling and the query_info tools, it is powerful (but also simple) enough to be used to test all kinds of bottlenecks in ORMs and elsewhere. - used this tool to generate benchmark reports on EC2 and on the (less powerful) dev server, and with different schema settings (usually rather minor schema changes that do not require re-importing all the data) - came up with a triple table schema that proves to render our queries quickly: we first do a search (using whatever criteria (e.g. nickname, fingerprint, address, running), if any) on a table which has a column with unique fingerprints; extract the relevant fingerprints; JOIN with the main status entry table, which is much larger; and get the final results. Benchmarked using this schema. <details> If we are only extracting a list of the latest status entries (with distinct on fingerprint), we can do LIMITs and OFFSETs already on the fingerprint table, before the JOIN. This helps us quite a bit. On the other hand, nickname searches etc. are also efficient. As of now, I have re-enabled nickname+address+fingerprint substring search (not from the middle (LIKE %substring%), but from the beginning of a substring (LIKE substring%), which is still nice), and all is well. Updated the higher-level ORM to reflect this new table [2] (I've yet to change some column names, though - but these are cosmetics.) </details> - found a way to generate the SQL queries that I need to generate using the higher-level SQLAlchemy SQL API using various SQLAlchemy-provided primitives, and always observing the resulting query statements. This is good, because everything becomes more modular: much easier to shape the query depending on the query parameters received, etc. (while still retaining it in sane order.) - hence (re)wrote a part of the Onionoo-like API that uses the new schema and the SQLAlchemy primitives. Extended the API a bit. [3] - wrote a very hacky API point for getting a list of status entries for a given fingerprint. I simply wanted a way (for myself and people) to query this kind of a relation easily and externally. It now works as part of the API. This part will probably need some discussion. - wrote a (kind of a stub) document explaining the current Onionoo-like API, what can be queried, what can be returned, what kinds of parameters work. [4] Will extend this later on. while writing the doc and rewriting part the API, stumbled upon a few things that make clear that I've made some shortcuts that may hurt later on. Will be happy to elaborate on them later on / separately. I need to carefully plan a few things, and then try rewriting the Onionoo API yet again, this time including more parameters and fields returned. TL;DR yay, a working database backend! I might give *one* more update detailing things I might have forgotten about soon re: this report - I don't want to make a habit of delaying reports (which I have consistently done), so reporting what I have now. [1]: https://github.com/wfn/torsearch/blob/master/torsearch/benchmark.py [2]: https://github.com/wfn/torsearch/blob/master/torsearch/models.py [3]: https://github.com/wfn/torsearch/blob/master/torsearch/onionoo_api.py [4]: https://github.com/wfn/torsearch/blob/master/docs/onionoo_api.md -- Kostas (wfn on OFTC) 0x0e5dce45 @ pgp.mit.edu

1 0

Proposal 222: Stop sending client timestamps
by Nick Mathewson 22 Aug '13

22 Aug '13

Filename: 222-remove-client-timestamps.txt Title: Stop sending client timestamps Authors: Nick Mathewson Created: 22 August 2013 Target: 0.2.5.x Status: Open 0. Summary There are a few places in Tor where clients and servers send timestamps. I list them and discuss how to eliminate them. 1. Introduction Despite this late date, many hosts aren't running NTP and don't have very well synchronized clocks. Even more hosts aren't running a secure NTP; it's probably easy to desynchronize target hosts. Given all of this, it's probably a fingerprinting opportunity whenever clients send their view of the current time. Let's try to avoid that. I'm also going to list the places where servers send their view of the current time, and propose that we eliminate some of those. Scope: This proposal is about eliminating passive timestamp exposure, not about tricky active detection mechanisms where you do something like offering a client a large number of about-to-expire/just-expired certificates to see which ones they accept. 2. The Tor link protocol 2.1. NETINFO (client and server) NETINFO cells specify that both parties include a 4-byte timestamp. Instead, let's say that clients should set this timestamp to 0. Nothing currently looks at a client's setting for this field, so this change should be safe. 2.2. AUTHENTICATE (server) The AUTHENTICATE cell is not ordinarily sent by clients. It contains an 8-byte timestamp and a 16-byte random value. Instead, let's replace both with a 24-byte (truncated) HMAC of the current time, using a random key. This will achieve the goal of including a timestamp in the cell (preventing replays even in the presence of bad entropy), while at the same time not including the time here. 2.3. TLS 2.3.1. ClientRandom in the TLS handshake See TLS proposal in appendix A. This presents a TLS fingerprinting/censorship opportunity. I propose that we investigate whether "random " or "zero" is more common on the wire, choose that, and lobby for changes to TLS implementations. 2.3.2. Certificate validity intervals Servers use the current time in setting certificate validity for their initial certificates. They randomize this value somewhat. I propose that we don't change this, since it's a server-only issue, and already somewhat mitigated. 3. Directory protocol 3.1. Published This field in descriptors is generated by servers only; I propose no change. 3.2. The Date header This HTTP header is sent by directory servers only; I propose no change. 4. The hidden service protocol 4.1. Descriptor publication time Hidden service descriptors include a publication time. I propose that we round this time down to the nearest N minutes, perhaps for N=30. 4.2. INTRODUCE2 cell timestamp INTRODUCE2 cells once limited the duration of their replay caches by including a timestamp in the INTRODUCE2 cells. Since 0.2.3.9-alpha, this timestamp is ignored, and key lifetime is used instead. When we determine that no hidden services are running on 0.2.2.x (and really, no hidden services should be running on 0.2.2.x!), we can simply send 0 instead. (See ticket #7803). This might be a good place to use a consensus parameter, so that a large number of clients switch at the same time. I claim this would be suitable for backport to 0.2.4. 5. The application layer The application layer is mostly out of scope for this proposal, except: TorBrowser already (I hear) drops the timestamp from the ClientRandom field in TLS. We should encourage other TLS applications to do so. (See Appendix A.) ================================================================= APPENDIX A: "Let's replace gmt_unix_time in TLS" PROBLEM: The gmt_unix_time field in the Random field in the TLS handshake provides a way for an observer to fingerprint clients. Despite the late date, much of the world is still not synchronized to the second via an ntp-like service. This means that different clients have different views of the current time, which provides a fingerprint that helps to track and distinguish them. This fingerprint is useful for tracking clients as they move around. It can also distinguish clients using a single VPN, NAT, or privacy network. (Tor's modified firefox avoids this by not sending the time.) Worse, some implementations don't send the current time, but the process time, or the computer's uptime, both of which are far more distinguishing than the current time() value. The information fingerprint here is strong enough to uniquely identify some TLS users (the ones whose clocks are hours off). Even for the ones whose clocks are mostly right (within a second or two), the field leaks a bit of information, and it only takes so many bits to make a user unique. WHY gmt_unix_time IN THE FIRST PLACE? According to third-hand reports -- (and correct me if I'm wrong!) it was introduced in SSL 3.0 to prevent complete failure in cases where the PRNG was completely broken, by making a part of the Random field that would definitely vary between TLS handshakes. I doubt that this goal is really achieved: on modern desktop environments, it's not really so strange to start two TLS connections within the same second. WHY ELSE IS gmt_unix_time USED? The consensus among implementors seems to be that it's unwise to depend on any particular value or interpretation for the field. The TLS 1.2 standard, RFC 5246, says that "Clocks are not required to be set correctly by the basic TLS protocol; higher-level or application protocols may define additional requirements." Some implementations set the entire field randomly; this appears not to have broken TLS on the internet. At least one tool (tlsdate) uses the server-side value of the field as an authenticated view of the current time. PROPOSAL 1: Declare that implementations MAY replace gmt_unix_time either with four more random bytes, or four bytes of zeroes. Make your implementation just do that. (Rationale: some implementations (like TorBrowser) are already doing this in practice. It's sensible and simple. You're unlikely to mess it up, or cause trouble.) PROPOSAL 2: Okay, if you really want to preserve the security allegedly provided by gmt_unix_time, allow the following approach instead: Set the Random field, not to 32 bytes from your PRNG, but to the HMAC-SHA256 of any high resolution timer that you have, using 32 bytes from your PRNG as a key. In other words, replace this: Random.gmt_unix_time = time(); Random.random_bytes = get_random_bytes(28) with this: now = hires_time(); // clock_gettime(), or concatenate time() // with a CPU timer, or process // uptime, or whatever. key = get_random_bytes(32); Random = hmac_sha256(key, now); This approach is better than the status quo on the following counts: * It doesn't leak your view of the current time, assuming that your PRNG isn't busted. * It actually fixes the problem that gmt_unix_time purported to fix, by using a high-resolution time that's much less likely to be used twice. Even if the PRNG is broken, the value is still nonrepeating. It is not worse than the status quo: * It is unpredictable from an attacker's POV, assuming that the PRNG works. (Because an HMAC, even of known data, with an unknown random key is supposed to look random). CONSIDERATIONS: I'd personally suggest proposal 1 (just set the field at random) for most users. Yes, it makes things a little worse if your PRNG can generate repeat values... but nearly everything in cryptography fails if your PRNG is broken. You might want to apply this fix on clients only. With a few exceptions (like hidden services) the server's view of the current time is not sensitive. Implementors might want to make this feature optional and on-by-default, just in case some higher-level application protocol really does depend on it. ==================================================================

4 3

Segfault trying to start tor in 0.2.4.16-rc with bufferevents
by Ian Goldberg 22 Aug '13

22 Aug '13

I just tried to upgrade my tor exit node to 0.2.4.16-rc, and got this: Aug 22 09:55:09.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Aug 22 09:55:10.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Aug 22 09:55:12.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Aug 22 09:55:12.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Aug 22 09:55:12.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Aug 22 09:55:16.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Aug 22 09:55:17.000 [warn] Something tried to close an or_connection_t without going through channels at src/or/connection.c:3185 Segmentation fault The configure line was: ./configure --with-openssl-dir=../openssl-1.0.1e --enable-bufferevents Running under gdb, I see: Program received signal SIGSEGV, Segmentation fault. buf_datalen (buf=0x0) at src/or/buffers.c:523 523 return buf->datalen; (gdb) bt #0 buf_datalen (buf=0x0) at src/or/buffers.c:523 #1 0x0000555555611b89 in connection_write_to_buf_impl_ ( string=0x7fffffffdd20 "\026\003\001", len=168, conn=0x55555682d070, zlib=0) at src/or/connection.c:3598 #2 0x0000555555587779 in connection_write_to_buf (conn=0x55555682d070, len=<optimized out>, string=<optimized out>) at src/or/connection.h:136 #3 connection_edge_process_relay_cell (cell=0x7fffffffdd10, circ=0x555557282610, conn=0x55555682d070, layer_hint=<optimized out>) at src/or/relay.c:1340 #4 0x0000555555589391 in circuit_receive_relay_cell (cell=0x7fffffffdd10, circ=0x555557282610, cell_direction=CELL_DIRECTION_OUT) at src/or/relay.c:212 #5 0x00005555555fa57c in command_process_relay_cell (chan=0x5555575b9f40, cell=0x7fffffffdd10) at src/or/command.c:462 #6 command_process_cell (chan=0x5555575b9f40, cell=0x7fffffffdd10) at src/or/command.c:148 #7 0x00005555555db873 in channel_tls_handle_cell (cell=0x7fffffffdd10, conn=0x555557ccc140) at src/or/channeltls.c:923 #8 0x000055555561c3f9 in connection_or_process_cells_from_inbuf ( conn=0x555557ccc140) at src/or/connection_or.c:1972 #9 0x000055555560f965 in connection_handle_read_cb (bufev=<optimized out>, arg=0x555557ccc140) at src/or/connection.c:3114 #10 0x00007ffff749457e in ?? () from /usr/lib/libevent-2.0.so.5 ---Type <return> to continue, or q <return> to quit--- #11 0x00007ffff748ae47 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #12 0x000055555556b1fe in do_main_loop () at src/or/main.c:1987 #13 0x000055555556c9cf in tor_main (argc=<optimized out>, argv=0x7fffffffe3c8) at src/or/main.c:2703 #14 0x00007ffff689c76d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6 #15 0x0000555555566cdd in _start () I rebuilt without bufferevents, and it hasn't crashed yet. (I also don't see the "Something tried to close an or_connection_t" warnings.) - Ian

2 2

Using Stem's descriptor fetching module to replace the Java consensus-health checker
by Karsten Loesing 20 Aug '13

20 Aug '13

Hi Damian, we briefly discussed Stem's new descriptor fetching module and how we could extend the existing simple monitors [0] towards a replacement of the Java consensus-health checker [1]. Moving this discussion to this list with your permission. So, you asked what exactly the consensus-health checker a.k.a. DocTor looks for. Let me try to give you a quick overview of the different parts [2]. Going through the Java source files in an order that hopefully explains best how everything works together: - Warning.java is an enum of all different warnings that DocTor can emit. Each warning contains a little documentation string saying what it means. If these are ambiguous, let me know, and I can probably explain them better. - Checker.java contains the various checks that are performed on previously downloaded consensuses and votes. For example, checkMissingConsensuses goes through the (hard-coded) list of known directory authorities and emits a ConsensusDownloadTimeout warning if we couldn't download the consensus from at least one of them. As you see, there are plenty more check* methods. - StatusFileReport.java uses the results from Checker by putting all warnings in two output files, one of them containing all warnings, the other only containing new warnings. Each warning has a severity, which can be ERROR, WARNING, or NOTICE. Also, each warning defines a time after which we consider the exact same warning string new even though the warning hasn't changed. The latter is useful to rate-limit warnings. For example, the fact that a certificate is going to expire in two months from now doesn't have to be repeated every hour. - MetricsWebsiteReport.java is the second output of DocTor. It's the website available at [3]. The idea is that the website gives more information about warnings received on IRC or via email. It's actually a hack that this website is presented on metrics. In a rewrite, PyDoctor would have its own little webserver to present consensus-health details. Once it's in place and we shut down DocTor, I'm going to replace the website on metrics with a static page linking to PyDoctor. - DownloadStatistics.java keeps statistics about consensus download times which are displayed on the website. - Downloader.java is a wrapper for metrics-lib's descriptor downloader. - Main.java puts everything together. It first downloads everything, then writes the status files containing warnings, and then generates the website output. So, that's what DocTor does right now. Here are two more things that would be great to have in DocTor or PyDoctor: - Warn if directory authorities assign flags to unusually few or many relays [4]. This enhancement has the potential of generating lots of warnings, because the directory authorities currently vote *very* differently on certain flags. The result will be a lot of directory authority operator nagging. Just saying, you should be prepared for that when deploying this! - Ignore certain known warnings [5]. This will reduce a lot of noise on the consensus-health mailing list. The fewer noise there is the more people will pay attention to actually valid warnings. In theory. Hope that makes sense. Happy to provide more input or review code. Just let me know! All the best, Karsten [0] https://lists.torproject.org/pipermail/tor-dev/2013-July/005209.html [1] https://www.torproject.org/getinvolved/volunteer#metrics-pyDoctor [2] https://gitweb.torproject.org/doctor.git/tree/HEAD:/src/org/torproject/doct… [3] https://metrics.torproject.org/consensus-health.html [4] https://trac.torproject.org/projects/tor/ticket/9103 [5] https://trac.torproject.org/projects/tor/ticket/8797

2 13

[GSoC 2013] Status report - Searchable metrics archive
by Kostas Jakeliunas 19 Aug '13

19 Aug '13

Hello, another busy benchmarking + profiling period for database querying, but this time more rigorous and awesome. * wrote a generic query analyzer which logs query statements, EXPLAIN, ANALYZE, spots and informs of particular queries that yield inefficient query plans; * wrote a very simple but rather exhaustive profiler (using python's cProfile) which logs query times, function calls, etc.; output is used to see which parts of the e.g. backend are slow during API calls; output can be easily used to construct a general query 'profile' for a particular database, etc.; [1] * benchmarked lots of different queries using these tools, recorded query times, was able to observe deviations/discrepancies; * uploaded the whole database and benchmarked briefly on an amazon EC2 m2.2xlarge instance; * concluded that, provided there is enough memory to cache *and hold* the indexes in cache, query times are good; * in particular, tested the following query scheme extensively: [2] (see comments there as well if curious); concluded that it runs well; * opted for testing raw SQL queries (from within Flask/python) - so far, translating them into ORM queries (while being careful) resulted in degraded performance; if we have to end up using raw SQL, I will create a way to encapsulate them nicely; * made sure data importing is not slowed and remains a quick-enough procedure; * researched PostgreSQL stuff, especially its two-layer caching; I now have an understanding of the way pgsql caches things in memory, how statistics on index usage are gathered and used for maintaining buffer_cache, etc. The searchable metrics archive would work best when all of its indexes are kept in memory. * to this end, looked into buffer cache hibernation [3], etc.; I think pg_prewarm [4, 5] would serve our purpose well. (Apparently many business/etc. solutions do find cache prewarming relevant - pity it's not supported in stock PostgreSQL.) The latter means that * I don't think we can avoid using certain postgresql extensions (if only one) - which means that deploying will always take more than apt-get && pip install, but I believe it is needed; * next on my agenda is testing pg_prewarm on EC2 and, hopefully, putting our beloved database bottleneck problem to rest. I planned to expose the EC2 for public tor-dev inquiry (and ended up delaying status report yet again), but I'll have to do this separately. This is possible, however. Sorry for the delayed report. ## More generally, I'm happy with my queer queries [2] now; the two constraints/goals of * being able to run Onionoo-like queries on the whole descriptor / status entry database * being able to get a list of status entries for a particular relay will hopefully be put to rest very soon. The former is done, provided I have no trouble setting up a database index precaching system (which will ensure that all queries of the same syntax/scheme run quick enough.) Overall, I'm spending a bit too much time on a specific problem, but at least I have a more intimate lower-level knowledge of PostgreSQL, which turns out to be very relevant to this project. I hope to be able to soon move to extending Onionoo support and providing a clean API for getting lists of consensuses in which a particular relay was present. And maybe start with the frontend. :) Kostas. [1]: https://github.com/wfn/torsearch/commit/8e6f16a07c40f7806e98e9c71c1ce0f8e38… [2]: https://github.com/wfn/torsearch/blob/master/misc/nested_join.sql [3]: http://postgresql.1045698.n5.nabble.com/patch-for-new-feature-Buffer-Cache-… [4]: http://www.postgresql.org/message-id/CA+TgmobRrRxCO+t6gcQrw_dJw+Uf9ZEdwf9be… [5]: http://raghavt.blogspot.com/2012/04/caching-in-postgresql.html

2 8