- tor-dev - lists.torproject.org

tor-bundle:How to extend the firefox context menu?
by mabra＠manfbraun.de 27 Sep '13

27 Sep '13

Hello ! I am doing really very boosted webserach in the last month. I am using the tor-bundle, if I am searching in several different and sensitiv areas, like wiki-leaks etc. >From my long term experience with all other search engines, beside google, they really dont allow you to continue, if the browser crashes. But this happens in minimum 10 times a day :-( So I want to use google to search, without allowing them to track me. This is why I use the tor-bundle. Using tor for simple webpages is usually slow. What would beome best for me, is, to use the tor-bundle together with the google search engine, but then, have a context menue (right click) to open another browser for a given search result item. The other browser then, is not configured to use the tor network. Does anyone has an information, how I, as an end-user, can exceed the context menue in the tor-bundles browser? Thanks anyway and best regards, ++mabra

1 0

Using the consensus-health web page to debug the Tor network
by Karsten Loesing 26 Sep '13

26 Sep '13

Hi Damian, Roger and I discussed the consensus-health web page that the Java version of DocTor produces [0] but your Python version does not. Roger says he uses that page to debug the Tor network. In particular, he says he scans the huge relay flags table to spot differences between relay flags assigned by the directory authorities. I'm pasting the IRC log below, but we should move this discussion to email, because it seems unlikely that we're all present in #tor-dev at the same time to discuss this. I see four ways how we can make most people mostly happy: 1.) We add votes documents to Onionoo (#9778) and include relay flags contained in votes in relay details pages of Atlas/Globe, so that people can find out what the different directory authorities think about their relay. We write a new Onionoo client that fetches this information for all running relays and produces a long table of relay flags. Before we do all this we make sure that there's nothing else on the consensus-health web page that Roger secretly hoped to keep. Then we shut down the Java DocTor. 2.) We rewrite the website-generating part of DocTor in a language we're happier to maintain long-term. This could be part of your DocTor, or of a new tool. Then we shut down the Java DocTor. 3.) We keep the Java DocTor running to generate the website, but disable the part that sends notifications to the mailing list and to the IRC bot. Or rather, I'll want to move the page away from metrics.tpo to, say, consensus-health.tpo or consensus-explorer.tpo, because it has always been a hack to serve it on metrics.tpo. Then we hope the Java DocTor never breaks. 4.) We do something much smarter which I didn't think of yet. What do you think? All the best, Karsten [0] https://metrics.torproject.org/consensus-health.html 08:04:49 < karsten> armadev: about removing per-relay votes from https://metrics.torproject.org/consensus-health.html. that was intentional, because the page took so long to load and it seemed nobody cared. 08:05:14 < karsten> armadev: but it seems the next step will be to kill that web page entirely.. 08:05:34 < karsten> the reason is that atagar's consensus-health checker will take over very soon. 08:05:56 < karsten> so, what information from the consensus-health page do you still need, either the current one or even the one with per-relay votes? 08:06:10 < karsten> we should try to rescue that information then. 08:07:16 < armadev> karsten: is atagar's consensus health checker up yet? 08:07:22 < karsten> yes, it is. 08:07:26 < armadev> url? 08:07:33 < karsten> it's sending mail to tor-consensus-health@ and to #tor-bots. 08:07:36 < karsten> it has no website 08:07:48 < karsten> it's sending out warnings if something goes wrong. 08:07:56 < karsten> when* :) 08:08:54 < armadev> karsten: in this case, what i wanted from the consensus health page was the set of votes per authority for 000000000000myTOR2 08:08:56 < karsten> right now, both consensus-health things send mail to that list. the ones at :03 are from mine, the other ones from atagar's. 08:09:11 < armadev> i pieced it together manually from moria1's v3-status-votes, but i'm special so i can do that 08:09:16 < karsten> right 08:09:25 < karsten> I thought about this a while ago. we could include votes in onionoo. 08:09:34 < karsten> not sure how yatei will like that though. 08:09:50 < karsten> but it's the votes you care about, ok. anything else from the consensus-health page? 08:10:16 < karsten> wfn: ^^ we should talk about this. 08:11:05 < armadev> s/votes/votes about flags/ here 08:11:23 < karsten> right. 08:37:19 < armadev> it seems i used consensus-health as consensus-explorer 10:11:35 < armadev> karsten: can you fwd me a sample mail from the health checker mailing list? 10:12:07 < karsten> armadev: they're archived: https://lists.torproject.org/pipermail/tor-consensus-health/2013-September/… 10:13:43 < armadev> oh good 10:13:56 < armadev> that's a very short mail. 10:14:02 < armadev> it has, like, nothing from the consensus health checker 10:14:20 < karsten> that's the things that need fixing to make it quiet again. 10:14:42 < armadev> why is "it takes a long time to load" a bug in consensus-health.html ? 10:15:37 < karsten> well, the page is huge, and firefox can't handle that very well. that's not a bug, but an inconvenience. 10:16:09 < armadev> sure 10:16:25 < karsten> I could have split the page into two. 10:16:34 < karsten> but anyway, the entire page is going away soon. 10:16:39 < armadev> if you were planning to drop it entirely, you could just leave it alone? 10:17:12 < karsten> as unmaintained service? 10:17:36 < karsten> I'd rather take the pieces we like and integrate them in maintained services. 10:17:37 < armadev> i guess. or write a howto for how to take votes in and generate the html, and i'll run it on moria 10:17:53 < armadev> though i think it needs more than votes as input 10:18:11 < armadev> i've used most parts of this page over the past few months 10:18:13 < karsten> it downloads votes and consensuses. 10:18:19 < karsten> I see. 10:18:42 < armadev> i have votes and a consensus on moria 10:18:49 < armadev> if that's the input, i can just cron it to run on moria 10:18:58 < armadev> based on the votes i see and the consensus i see 10:19:09 < armadev> unless i guess it requires a massive tomcat engine too :) 10:19:15 < karsten> it doesn't. 10:19:17 < karsten> it requires java. 10:19:29 < karsten> and produces a static html that you can service with whatever. 10:19:55 < karsten> well, if you need it, it's easier to keep it running on yatei. 10:19:57 < armadev> i don't have any java. i guess i could add it. or we could stick it on people.tp.o or something 10:19:59 < armadev> right 10:20:09 < karsten> and just turn off the email notifications, because atagar's thing will send those. 10:20:11 < armadev> i use it for debugging the network 10:20:32 < karsten> ok 10:20:58 < karsten> would #9778 solve the problem with the missing relay flags table? 10:20:59 -zwiebelbot:#tor-dev- [tor#9778: Adding votes documents to Onionoo] 10:21:08 < karsten> assuming atlas or globe would show those? 10:21:32 < armadev> do i have to load each relay one page at a time? 10:22:05 < karsten> probably. 10:22:31 < karsten> so, you're looking through the table to spot problems? 10:22:41 < karsten> or, have been looking, when the table was there. 10:22:41 < armadev> yes. and to get a sense of trends. 10:22:44 < armadev> right 10:22:57 < armadev> i guess in this most recent case i just wanted one relay 10:23:11 < armadev> but after that, i realized that i wanted to know how many relays had a Running flag from those two authorities and no Running flag from any others 10:23:33 < karsten> ok 10:23:42 < armadev> (i still want to know :) 10:24:06 < karsten> let me re-enable the table and think about a better long-term solution.. :) 10:24:07 < armadev> (related to https://trac.torproject.org/projects/tor/ticket/9775 ) 10:24:08 -zwiebelbot:#tor-dev- [tor#9775: Authorities should report when they don't vote Running but some addresses are still reachable] 10:24:48 < armadev> we are dropping an unknown number of relays from the consensus because they enable ipv6 but screw up the reachability 10:25:16 < armadev> hopefully every relay operator who knows what ipv6 is also knows how to check relay-search for themselves 10:28:21 < karsten> the next consensus-health.html will contain the relay flags table again. in roughly 35 mins. 10:29:51 < armadev> thanks. sorry for the troubles. and i understand the drive to get rid of extra things. i wish we had more things that were actually extra. 10:32:10 < karsten> yeah, we have a lot of stuff. and some of it is really old and hasn't seen love for too long.

2 6

Kickoff Meeting Times for Project Otter [all next week]
by Tom Lowenthal 26 Sep '13

26 Sep '13

There seem to be relatively good times for all three Project Otter kickoff meetings next week. There's no need to RSVP, but if you can't make it and have something to contribute please try to provide that input as soon as possible in the form of a "comments" section on the relevant page on the wiki. Otter index page: <https://trac.torproject.org/projects/tor/wiki/org/sponsors/Otter> All the meetings will be held in #tor-dev on `irc.oftc.net`. Here are the times: * Attentive Otters [IM bundle]: Weds 2013-10-02 1100-1200 Pacific * Boisterous Otters [Help/outreach/l10n]: Weds 2013-10-02 0900-1000 Pacific * Cute Otters [Hidden services & publishing]: Weds 2013-10-02 1600-1700 Pacific If you have other questions or there are other problems, please reply here. -Tom

1 0

IRC meeting to discuss sponsor F progress on Mon September 30, 18:00 to 19:00 in #tor-dev
by Karsten Loesing 24 Sep '13

24 Sep '13

Hi all, I'd like to schedule an IRC meeting to discuss what progress we made on sponsor F deliverables in September. Time and place are: Mon September 30, 18:00 to 19:00 UTC in #tor-dev That's in roughly 6 days 3 hours from now. The time in other timezones is: 11:00 in San Francisco 14:00 in Boston 20:00 in Berlin 21:00 in Athens 23:30 in New Delhi People who should attend are: Andrea, Andrew, Arlo, Colin, David, George, Linus, Matt, Nathan, Nick, Roger, Tom, Ximin, and anyone else who wants to attend. If your name is on that list and you're working on sponsor F stuff, but the date or time doesn't work for you, please email me your progress and I'll report it for you at the meeting. Thanks, Karsten

1 0

Re: [tor-dev] File verification GUI tool
by Matt Pagan 24 Sep '13

24 Sep '13

On Mon, 23 Sep 2013 17:59:23 -0400 Roger Dingledine <arma(a)mit.edu> wrote: > On Mon, Sep 23, 2013 at 05:45:36PM -0400, Sukhbir Singh wrote: > > > I am starting to work on a small GUI tool for file verification > > > because I find guiding users through the verification process on > > > Windows/Mac through the command line painful. > > > > Just a thought: have you considered doing this through a Firefox > > extension instead? > > It does seem like a Firefox extension is a better alternative. The > "here's an executable you just fetched from the Internet, please run > it" model is likely bad news, and also isn't the right habit to be > teaching users. > If you go the browser extension route, there is a now defunct Firefox add-on called FireGPG that implemented GnuPG in the web browser. This add-on was discontinued in 2010 and it's primary focus was encrypting and decrypting gmail messages. However, the software was open source and the code is still available. If you decide to implement this project as a browser extension, you might want to see if there are any usable parts you can salvage from that rubble (there might not be). http://getfiregpg.org/s/home

3 2

Onion IPv6 transport needed [was: ed25519 HS identity keys]
by grarpamp 24 Sep '13

24 Sep '13

On 9/24/13, Bernhard R. Fischer <bf(a)abenteuerland.at> wrote: > On Thursday 19 September 2013 10:53:13 grarpamp wrote: >> On 9/13/13, George Kadianakis <desnacked(a)riseup.net> wrote: >> > This proposal is in serious need for comments. >> > >> > 1.2. From the PoV of the HS client: >> > Tor clients can distinguish new-style HS addresses from old ones by >> > their length. Legacy addresses are 16 base32 characters, while new >> > ones are 56 (XXX) base32 characters. >> > >> > If Tor is asked to connect to a legacy address it SHOULD throw a >> > warning and advocate the use of new-style addresses (it should still >> > connect to the HS however). In the future, when old-style HS >> > addresses are close to depletion, we can introduce a torrc parameter >> > that blocks client connections to old-style HS addresses. >> >> As commented earlier on this list... >> https://lists.torproject.org/pipermail/tor-dev/2013-August/005286.html >> >> This addressing change will break onioncat (IPv6) within Tor. >> https://www.onioncat.org/ >> >> I suggest a solution to transporting IPv6 within Tor be >> maintained/deployed concurrently with any change in >> current onion addressing and or transport mechanics. > > > 56 base32 characters means 280 bits. > And yes, the new addresses will break Onioncat addressing mechanism. > > Actually this is the same problem as with I2P which uses 256 bit addresses > for > the destinations. > Then, as a workaround I implemented an /etc/hosts file-based lookup. This is > > not a perfect solution but it was better than nothing ;) > > I'll think about a new method of addressing, which allows forward > compatibility (DNS comes into my mind -- I know...) I know, yet I don't mind the central nature of DNS for naming. It works on clearnet, you can subscribe to multiple roots to avoid censure and under an anonymous system someone will always be willing to carry the sensational records. So you can resolve just fine. The problem is the transport. Being able to resolve is pointless if you can't transport. And if you can transport you can resolve on top of that any which way you want, including DNS. I'm sure this is not really a naming issue but an addressing one. If people expect Tor, I2P etc to ever evolve to support for real p2p/social apps and services and build communities beyond what crippling unidirectional TCP is capable of (which I suggest we should), they need to transport IPv6. And to do that you need to have nodes fall within private RFC/48 addressing bound to an interface... whether natively as Tor currently does by byproduct with onioncat providing the layer, or via some future address mapping layer [1]. The 'DNS' you speak of could map 80bit to n-greater-than-80bit, (256/280/512 bit) within an internal backend of a new ocat that then transports to the higher bits. But if actual DNS was used for that it would invoke the root/censor/uptime/trust issues of DNS for *addressing* not naming, and we don't really want that type of breakdown there. Ocat currently does not have those problems. And if someone comes up with the likes of [1], a central trust system like DNS won't be needed. To the extent this is a wider summary, if there's an IPv6 transport (mod hidden addressing) ticket where reference to this should be lodged, please do. [1] Search for IPv6 on zzz.i2p, there were some comments found there of possible map considerations like signed announcements into DHT. Maybe also namecoin-like addressing.

1 0

Re: [tor-dev] Limiting Tor RAM usage
by Gordon Morehouse 24 Sep '13

24 Sep '13

References for prior message: 1. http://linux.die.net/man/2/setrlimit 2. http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/40 Best, -Gordon M.

1 0

Limiting Tor RAM usage
by Gordon Morehouse 24 Sep '13

24 Sep '13

Hi all, I brought this up in tor-relay, where I'm keeping most of my relatively easy/low key stuff on getting Tor to work real solidly on the Raspberry Pi. However, the biggest problem I'm having right now (with the latest version) is occasional visits from the OOM killer, usually right after my relay gets a Stable flag and traffic increases. :( I really need a way to hard-limit the RSS (physical memory usage) of the Tor process. Would setrlimit(2)[1] work on the Tor process? This is Linux. For those that don't want to cruise the man page, here's the relevant discouraging text which may or may not apple to Tor, from a blog post on limiting memory and CPU time usage on Linux[2]: > Ulimit (the console tool, not the system call) is the first thing a Linux user would think about. You launch it in the shell, set "soft" and "hard" limits for each resource, and children, which are invoked in this shell, are limited in memory and time. > > Its limits are imposed like this: > [snip] > > Memory limit is set via the same system call. Memory limit is enforced by returning ENOMEM from a violating malloc() syscall. This usually leads to an unhandled exception of sorts, and the program (or its interpreter) is forced to terminate. If that won't work, how can I achieve this with a high degree of precision? MaxAdvertisedBandwidth *is not* going to cut it for systems that are low on RAM, but have enough CPU horsepower to happily forward a few Mbps. Thanks much, -Gordon M.

1 0

[GSOC] Status report - Tor capabilities
by Cristian-Matei Toader 23 Sep '13

23 Sep '13

Hello tor-dev, For the past two couple of weeks, stage 2 has ended [1], and stage 3 has begun [2]. Stage 2 didn't change by much, apart from a few change requests from nickm. As part of stage 3, some important changes were made: - the structures involving how a sandbox configuration is being managed has changed, in order to allow multiple configurations; the sandbox type now consists of the syscall array of no-parameter filters, and a list of parameter syscall configurations; now each parameter syscall is handled as a function pointer and a list of parameters which are used with the function pointer in order to initialise that particular filter; this all may sound over-complicated, but it's actually a quite intuitive non-hacky version of what was going on before. - a filter was developed (it's actually unexpectedly short) for worker threads only, and it is currently working just as it should in the stage 3 public branch - currently I have updated the general filter to include the prctl filter required to allow loading other seccomp filters, which is necessary for the purpose of stage 3; there is also an option for a general filter which disallows any further filters to be loaded, and I am currently working at loading this updated filter at an optimum location. I am planning to continue working on stage 3 past the hard deadline for the GSOC project, but when university starts I will probably prioritise studies over this project. It has been a pleasure working for the tor project, I will keep contributing (either sandboxing related or otherwise), and will make sure to keep in touch on IRC. [1] https://trac.torproject.org/projects/tor/ticket/9249 [2] https://trac.torproject.org/projects/tor/ticket/9730

1 0

[GSoC '13] Status report - Searchable metrics archive
by Kostas Jakeliunas 23 Sep '13

23 Sep '13

Hi all, TL;DR of TL;DR: all good as far as GSoC scope goes; work will continue; the live torsearch backend has been updated; will update API doc today. TL;DR: - updated database / imported data up until current date - hourly cronjob continuously rsync's latest archival data, imports to DB (+ tested and deployed): * had some trouble ensuring no duplication but also no excessive DB querying (the intermediary 'fingerprint' table (that basically solves the DB bottleneck problem) was previously updated in semi-manual-mode); solved with a kind of an 'upsert'; works well; * made switching from archival data import process to the rsync'ed ::metrics-recent import seamless (data can overlap, importer will take care of it; import speed is good) - Onionoo API improvements: * date range queries for all three document types currently provided * the network status entry document was expanded/improved, as per previous discussion and Karsten's feedback * yet to finish off IP address and nickname summarization - for now, providing two document versions - original and 'condensed' (valid-after ranges) * updating Onionoo API doc soon (sorry - had meant to update until now.) - caching: * tried some things, settled for a semi-vanilla python-level-caching solution for the time being, some testing was done * have yet to still work out some snags, not deploying for now * more (pure json response dumping to disk, etc.) can be done later on - other things: * wrote a working stub / minimal version for a DB-maintenance-from-within-torsearch component (e.g. for a full VACUUM with no DB access) * other small things I may have forgotten to mention General picture: the metrics archive / torsearch is in decent shape and an appropriate (to gsoc) stage of development. Development will continue as we're not done here yet, but the archive is in a functioning state, delivering the minimum functionality required. I will finish cleaning up before the hard pencils down date; and then continue development (at whatever pace). :) The things that were blocking before are no longer blocking, so ironically, I think I'll now be able to write more code i.r.l. to torsearch than over the summer. Anyway - I've been working on things falling within the GSoC scope, finishing the most urgent matters that were left re: gsoc milestone - quoting from my previous status report re: what to do next (see the bullet point list above for a concise no-ramble-mode version of all this): > update the database to the latest consensuses and descriptors Done. However, > import 2008 year data, etc. I had parts of 2009 data in previously while working on DB bottlenecks, but presently, the statuses still start at 2010; this got pushed down the priority lane and takes a lot of (passive) time. Will be able to batch-import now and confirm. > turn on the cronjob for rsync and import of newest archives Done. [0, 1] What kept me a bit busy was integrating the third helper/mediating 'fingerprint' table (we query this one before extracting things from the massive 'statusentry' table (which contains all network status entries)) into the batch_import_consensuses process used for import. <details> I had some problems making sure data in the fingerprint table is not duplicated and contains latest info (last valid-after etc), while minimizing the amount of work and queries needed to test for row/entry existence, etc. Solved with an actually simple hybrid 'upsert' approach [2], all is well (did some testing, OK, but not extensive.) Improved the archival importer for it to be able to process duplicate consensus documents from different directories (hence they're bypassing Stem's persistence file check - so need to take care of duplicated processing ourselves) - this might happen in production when, after massively importing data from the downloaded archives, we switch to rsync's ::metrics-recent/relay-descriptors folder for import. Consensuses may overlap (they overlapped when I did the switch.) When passed down a consensus document by Stem (so after checking for duplication in Stem's persistence file), we simply check whether we already store this consensus, using a separate 'consensus' table, which only stores consensus document validity ranges (this is fast and works nicely). [2] </details> > [from an older report] Onionoo date range queries Done. [4] Works for all three document types (details, summary, statuses); can be used together with offset+limit, etc. Smart/decent datetime parsing - can pass '2011-07-28 12:00:00', or '2011-07', or '2011', etc. Example: http://ts.mkj.lt:5555/details?search=200.&from=2010&to=2012 > caching Tried all kinds of options, and since I wanted to have something decent and stable working for now, opted for a very simplistic stock python-level caching approach (no onionoo-like 'JSON documents to disk' thing yet.) But I've yet to work out some things, delayed deploying code and live version. > hopefully later integration with the bandwidth and weights documents in Onionoo proper Out of current scope; hopefully will be able to work on this next / soon. > [from an older report] Documentation, specification/implementation document Not finished yet - not very good. I should try more of this "publish early and often" thing. > expand the list of fields contained in the three [...] documents Tried things, but e.g. providing assigned flag data was pushed to later. Current live backend not changed in regards to this. > [from an older report] rewrite/improve the network status entry document / improve the 'statuses' API point Done [7] (deployed changes, will update API doc - this is not a nice thing to do (i.e. doc being not updated), but since it's not production yet, deployed early) For now, we are providing two network status document versions - the original one (+ 'relays_published'), and a condensed one (?condensed=true). The latter basically zips up all the valid-after values into ranges, as per Karsten's suggestion, basically telling the client where any gaps in consensuses were present (which may turn out to be a rather useful thing, by the way.) This works well together with from..to date ranges, offset, and limit. I'm yet to finish off IP address and nickname summarization - for now, when in 'condensed' presentation mode, each range contains its last addresses and nickname. Example: http://ts.mkj.lt:5555/statuses?lookup=F397038ADC51336135E7B80BD99CA38443602… Also, as torsearch might one day be deployed on a machine such that the maintainer won't have direct access to the DB server, we'll need to do VACUUM FULL et al. from within torsearch. It is advisable to do a full VACUUM or even a REINDEX now and then, especially after a ton of entries is added (say, after a massive/bulk data batch-import.) Wrote a working stub for a 'torsearch/DB maintenance' component (for vacuum full, for now.) [8] Other maintenance things may be added to this separate file/component later on. I think that's all for now. Hopefully I'm past the 'experiment with all the things!' stage (I've still been trying and tinkering with stuff / different approaches to problems; but for what it's worth, the current codebase and the live backend are in decent shape.) Comments / suggestions / barbaric yawps? [0]: https://github.com/wfn/torsearch/commit/b9a31fdabc52bca1baa00064359c6f691af… [1]: see e.g. http://ts.mkj.lt:5555/details [2]: https://github.com/wfn/torsearch/commit/74cb2b19e1a5b47fa95b62df8d89e0c9fb2… # there is no 3. It's.. gone! [4]: https://github.com/wfn/torsearch/commit/5f862930b81ad34e04d6315c6c57bfd3cb4… # more numbers have mysteriously disappeared! [API doc commit would go here] [7]: https://github.com/wfn/torsearch/commit/fa2183bb9fb58bff5bc1469fa947367ecd0… [8]: https://github.com/wfn/torsearch/commit/0f24fee4fb73a12fd088f0d1e271dbdb056… -- Kostas (wfn on #tor-dev) 0x0e5dce45 @ pgp.mit.edu

1 1