- tor-dev - lists.torproject.org

Limiting Tor RAM usage
by Gordon Morehouse 23 Sep '13

23 Sep '13

Hi all, I brought this up in tor-relay, where I'm keeping most of my relatively easy/low key stuff on getting Tor to work real solidly on the Raspberry Pi. However, the biggest problem I'm having right now (with the latest version) is occasional visits from the OOM killer, usually right after my relay gets a Stable flag and traffic increases. :( I really need a way to hard-limit the RSS (physical memory usage) of the Tor process. Would setrlimit(2)[1] work on the Tor process? This is Linux. For those that don't want to cruise the man page, here's the relevant discouraging text which may or may not apple to Tor, from a blog post on limiting memory and CPU time usage on Linux[2]: > Ulimit (the console tool, not the system call) is the first thing a Linux user would think about. You launch it in the shell, set "soft" and "hard" limits for each resource, and children, which are invoked in this shell, are limited in memory and time. > > Its limits are imposed like this: > [snip] > > Memory limit is set via the same system call. Memory limit is enforced by returning ENOMEM from a violating malloc() syscall. This usually leads to an unhandled exception of sorts, and the program (or its interpreter) is forced to terminate. If that won't work, how can I achieve this with a high degree of precision? MaxAdvertisedBandwidth *is not* going to cut it for systems that are low on RAM, but have enough CPU horsepower to happily forward a few Mbps. Thanks much, -Gordon M.

1 0

[GSOC] Status report - Tor capabilities
by Cristian-Matei Toader 23 Sep '13

23 Sep '13

Hello tor-dev, For the past two couple of weeks, stage 2 has ended [1], and stage 3 has begun [2]. Stage 2 didn't change by much, apart from a few change requests from nickm. As part of stage 3, some important changes were made: - the structures involving how a sandbox configuration is being managed has changed, in order to allow multiple configurations; the sandbox type now consists of the syscall array of no-parameter filters, and a list of parameter syscall configurations; now each parameter syscall is handled as a function pointer and a list of parameters which are used with the function pointer in order to initialise that particular filter; this all may sound over-complicated, but it's actually a quite intuitive non-hacky version of what was going on before. - a filter was developed (it's actually unexpectedly short) for worker threads only, and it is currently working just as it should in the stage 3 public branch - currently I have updated the general filter to include the prctl filter required to allow loading other seccomp filters, which is necessary for the purpose of stage 3; there is also an option for a general filter which disallows any further filters to be loaded, and I am currently working at loading this updated filter at an optimum location. I am planning to continue working on stage 3 past the hard deadline for the GSOC project, but when university starts I will probably prioritise studies over this project. It has been a pleasure working for the tor project, I will keep contributing (either sandboxing related or otherwise), and will make sure to keep in touch on IRC. [1] https://trac.torproject.org/projects/tor/ticket/9249 [2] https://trac.torproject.org/projects/tor/ticket/9730

1 0

[GSoC '13] Status report - Searchable metrics archive
by Kostas Jakeliunas 23 Sep '13

23 Sep '13

Hi all, TL;DR of TL;DR: all good as far as GSoC scope goes; work will continue; the live torsearch backend has been updated; will update API doc today. TL;DR: - updated database / imported data up until current date - hourly cronjob continuously rsync's latest archival data, imports to DB (+ tested and deployed): * had some trouble ensuring no duplication but also no excessive DB querying (the intermediary 'fingerprint' table (that basically solves the DB bottleneck problem) was previously updated in semi-manual-mode); solved with a kind of an 'upsert'; works well; * made switching from archival data import process to the rsync'ed ::metrics-recent import seamless (data can overlap, importer will take care of it; import speed is good) - Onionoo API improvements: * date range queries for all three document types currently provided * the network status entry document was expanded/improved, as per previous discussion and Karsten's feedback * yet to finish off IP address and nickname summarization - for now, providing two document versions - original and 'condensed' (valid-after ranges) * updating Onionoo API doc soon (sorry - had meant to update until now.) - caching: * tried some things, settled for a semi-vanilla python-level-caching solution for the time being, some testing was done * have yet to still work out some snags, not deploying for now * more (pure json response dumping to disk, etc.) can be done later on - other things: * wrote a working stub / minimal version for a DB-maintenance-from-within-torsearch component (e.g. for a full VACUUM with no DB access) * other small things I may have forgotten to mention General picture: the metrics archive / torsearch is in decent shape and an appropriate (to gsoc) stage of development. Development will continue as we're not done here yet, but the archive is in a functioning state, delivering the minimum functionality required. I will finish cleaning up before the hard pencils down date; and then continue development (at whatever pace). :) The things that were blocking before are no longer blocking, so ironically, I think I'll now be able to write more code i.r.l. to torsearch than over the summer. Anyway - I've been working on things falling within the GSoC scope, finishing the most urgent matters that were left re: gsoc milestone - quoting from my previous status report re: what to do next (see the bullet point list above for a concise no-ramble-mode version of all this): > update the database to the latest consensuses and descriptors Done. However, > import 2008 year data, etc. I had parts of 2009 data in previously while working on DB bottlenecks, but presently, the statuses still start at 2010; this got pushed down the priority lane and takes a lot of (passive) time. Will be able to batch-import now and confirm. > turn on the cronjob for rsync and import of newest archives Done. [0, 1] What kept me a bit busy was integrating the third helper/mediating 'fingerprint' table (we query this one before extracting things from the massive 'statusentry' table (which contains all network status entries)) into the batch_import_consensuses process used for import. <details> I had some problems making sure data in the fingerprint table is not duplicated and contains latest info (last valid-after etc), while minimizing the amount of work and queries needed to test for row/entry existence, etc. Solved with an actually simple hybrid 'upsert' approach [2], all is well (did some testing, OK, but not extensive.) Improved the archival importer for it to be able to process duplicate consensus documents from different directories (hence they're bypassing Stem's persistence file check - so need to take care of duplicated processing ourselves) - this might happen in production when, after massively importing data from the downloaded archives, we switch to rsync's ::metrics-recent/relay-descriptors folder for import. Consensuses may overlap (they overlapped when I did the switch.) When passed down a consensus document by Stem (so after checking for duplication in Stem's persistence file), we simply check whether we already store this consensus, using a separate 'consensus' table, which only stores consensus document validity ranges (this is fast and works nicely). [2] </details> > [from an older report] Onionoo date range queries Done. [4] Works for all three document types (details, summary, statuses); can be used together with offset+limit, etc. Smart/decent datetime parsing - can pass '2011-07-28 12:00:00', or '2011-07', or '2011', etc. Example: http://ts.mkj.lt:5555/details?search=200.&from=2010&to=2012 > caching Tried all kinds of options, and since I wanted to have something decent and stable working for now, opted for a very simplistic stock python-level caching approach (no onionoo-like 'JSON documents to disk' thing yet.) But I've yet to work out some things, delayed deploying code and live version. > hopefully later integration with the bandwidth and weights documents in Onionoo proper Out of current scope; hopefully will be able to work on this next / soon. > [from an older report] Documentation, specification/implementation document Not finished yet - not very good. I should try more of this "publish early and often" thing. > expand the list of fields contained in the three [...] documents Tried things, but e.g. providing assigned flag data was pushed to later. Current live backend not changed in regards to this. > [from an older report] rewrite/improve the network status entry document / improve the 'statuses' API point Done [7] (deployed changes, will update API doc - this is not a nice thing to do (i.e. doc being not updated), but since it's not production yet, deployed early) For now, we are providing two network status document versions - the original one (+ 'relays_published'), and a condensed one (?condensed=true). The latter basically zips up all the valid-after values into ranges, as per Karsten's suggestion, basically telling the client where any gaps in consensuses were present (which may turn out to be a rather useful thing, by the way.) This works well together with from..to date ranges, offset, and limit. I'm yet to finish off IP address and nickname summarization - for now, when in 'condensed' presentation mode, each range contains its last addresses and nickname. Example: http://ts.mkj.lt:5555/statuses?lookup=F397038ADC51336135E7B80BD99CA38443602… Also, as torsearch might one day be deployed on a machine such that the maintainer won't have direct access to the DB server, we'll need to do VACUUM FULL et al. from within torsearch. It is advisable to do a full VACUUM or even a REINDEX now and then, especially after a ton of entries is added (say, after a massive/bulk data batch-import.) Wrote a working stub for a 'torsearch/DB maintenance' component (for vacuum full, for now.) [8] Other maintenance things may be added to this separate file/component later on. I think that's all for now. Hopefully I'm past the 'experiment with all the things!' stage (I've still been trying and tinkering with stuff / different approaches to problems; but for what it's worth, the current codebase and the live backend are in decent shape.) Comments / suggestions / barbaric yawps? [0]: https://github.com/wfn/torsearch/commit/b9a31fdabc52bca1baa00064359c6f691af… [1]: see e.g. http://ts.mkj.lt:5555/details [2]: https://github.com/wfn/torsearch/commit/74cb2b19e1a5b47fa95b62df8d89e0c9fb2… # there is no 3. It's.. gone! [4]: https://github.com/wfn/torsearch/commit/5f862930b81ad34e04d6315c6c57bfd3cb4… # more numbers have mysteriously disappeared! [API doc commit would go here] [7]: https://github.com/wfn/torsearch/commit/fa2183bb9fb58bff5bc1469fa947367ecd0… [8]: https://github.com/wfn/torsearch/commit/0f24fee4fb73a12fd088f0d1e271dbdb056… -- Kostas (wfn on #tor-dev) 0x0e5dce45 @ pgp.mit.edu

1 1

Tor on the Beaglebone Black
by Joshua Datko 22 Sep '13

22 Sep '13

I've been running a relay on a Beaglebone black for about two weeks now and I'm wondering where to take this thing. The Beaglebone black is a 1GHz armhf System on a Chip, with 512 MB RAM. At the moment, it's a dedicated relay running Ubuntu (raring) but Debian is equally as easy to setup. I have a post with very basic installations: http://datko.net/2013/08/24/beagleboneblacktor/ So, what next? I looked at the torrouter project and I think I would have to probably have to redo the roadmap to suit the Beaglebone. Also, there is only one onboard NIC, so I'm not sure if that's a show stopper. I was going to do a fresh install of Debian, install Tor and then build an image so that it could be booting (or flashed to the eMMC) from a SD card, but wether that's better than installing tor oneself, I don't know. Anyway, just throwing an idea out there. Thanks everybody for the work on tor and I'm happy to be running a relay now. Josh

2 1

GSoC '13 Status Report - EvilGenius
by Johannes Fürmann 20 Sep '13

20 Sep '13

Hello tor-dev, Since the last report on EvilGenius, I was able to make a few last changes to EvilGenius which is now more or less feature-complete: * Internal routing issues are now solved packets are flowing happily. * Integrated OONI backend and probe * Wrote some user documentation I guess what's left to do for me is to fix some bugs, clean up the code a bit and get the developer documentation up to speed. Additionally, I will proceed to implement some actual censorship providers based on the ones I already had implemented in earlier versions. Have a nice weekend! Johannes

1 0

Pluggable Transport *Biweekly* Meeting, hence no meeting today
by vmonmoonshine＠gmail.com 20 Sep '13

20 Sep '13

Thanks for everybody who participated last week. The summary of the meeting can be found here: https://trac.torproject.org/projects/tor/wiki/doc/PluggableTransports/meeti… We decided that we are going to have these meetings biweekly till we see if we need more or less frequent meetings. So the next one will be next week. See you all there. Best, Vmon

1 0

entry guards and linkability
by Leif Ryge 19 Sep '13

19 Sep '13

Is the following statement correct? When a user connects to Tor from multiple locations where the network is monitored by the same adversary, their persistent use of the same set of entry guards uniquely identifies them and reveals their location to the adversary. Assuming this is an accurate assessment, wouldn't it make sense to maintain separate sets of entry guards for each network that the user connects from? ~leif

2 3

What to read to analyze Tor's use of NTor?
by Roger Dingledine 19 Sep '13

19 Sep '13

Hi Nick, Ian, I've been pointing people to "Section 6 of http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.228.6223" when they ask what NTor is. But then I realized that that's not the best (single) place to send cryptographers when I ask them to analyze whether we've designed or built it right. Then I found https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/216-ntor-han… which looks great but then I also found https://gitweb.torproject.org/torspec.git/blob/HEAD:/tor-spec.txt#l852 which looks crisper. So the questions: A) Which combination and order of these three resources should I point people at? That is, does the tor-spec stanza replace proposal 216 completely, or is there still some use to looking at the proposal too, or is the proposal wrong now because you fixed stuff since then but didn't change the proposal, etc? Did I miss any good resources? B) What are the sketchiest parts -- the parts of the design or the implementation that you most want review on, or that you think would be most fruitful for finding issues? C) What else should I be asking you, in terms of how to get this thing reviewed the mostest and the bestest? We rolled out NTor quicker than we rolled out TAP, relatively speaking, and now it would count as breaking a widely deployed system so I bet we can get some more people evaluating it. Thanks! --Roger

2 1

Using TorPS to estimate chances of all-TAP paths
by Roger Dingledine 19 Sep '13

19 Sep '13

Hi Aaron, Sathya, I have another use for your fine path simulator. :) See https://trac.torproject.org/projects/tor/ticket/9777 The first question is: for a recent consensus, what's the chance that your middle hop and your exit hop will both be running 0.2.4.7-alpha or earlier? The extra credit for question one is to generate the answers to question one for each of the past n consensuses (where n is, say, the past month) so somebody can graph them. The second question is: if we change path selection to try again if we choose such a path, what does that do to our anonymity, for various definitions of anonymity like the ones in the upcoming CCS paper? The extra credit for question two, if you haven't guessed it already, is to get some data points for question two from the past as well. I'm guessing getting 24*30 of them will be no fun, so whatever isn't no fun is fine. What can I do to help make it happen? If the answers are "actually not all that high" and "oh hey those curves aren't all that different" respectively, I want to put #9777 into 0.2.4 and enable it. If those aren't the answers, we'll need to sort out a better plan. Thanks! --Roger

1 0

OnionMail First Test
by Liste 19 Sep '13

19 Sep '13

Work complete for the version 0.0.0Beta http://anopticon.ws/13c03c09/OnionMail-0.0.0.B.zip to create user: telnet 127.0.0.1 9100 C = client S = Server S: +OK ControlPort 1.0 sfl‚4£$£"f C: server alice hackme <- default password S: +OK KESIA [example] `example` C: addusr test <-- local part S: +OK New user created S: Mail=test(a)your.onion S: User=test S: SMTPPasswd=lunc6){><uFXSpaoWnJHl1px S: POP3Passwd=kfzb};VW,NKwLz"JM2Hmrwot S: Server=your.onion S: MsgSize=2000000 S: MaxMsg=128 S: MsgOld=40 S: Inet= S: QFDN=your.onion S: . C: quit S: +OK Closing smtp server (tls if available) 127.0.0.1 25 user test etc...etc... pop3 port 110 (tls if available) 127.0.0.1

5 6