February 2016 - tor-dev - lists.torproject.org

Re: [tor-dev] Hashring understanding
by George Kadianakis 04 Feb '16

04 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hi, > > Sorry for this - had some questions about the hashring component of > #259 that I haven't been able to figure out myself. I'm sure it's just > me being unused to the Tor code base and how you write your proposals, > but it would be super helpful if I can get a quick answer to a few > questions. > > First, my simplified understanding of the data structures is this: > GUARDS are all the guards in the consensus > > UTOPIC_GUARDS is all utopic guards from GUARDS > DYSTOPIC_GUARDS is all dystopic guards from GUARDS > > UTOPIC_GUARDLIST is a subset of UTOPIC_GUARDS > DYSTOPIC_GUARDLIST is a subset of DYSTOPIC_GUARDS > > We will only ever choose guards to use from UTOPIC/DYSTOPIC_GUARDLIST > > The idea of the hashring is to be a structure to make it possible for us to > go from UTOPIC/DYSTOPIC_GUARDS to UTOPIC/DYSTOPIC_GUARDLIST. > The reason we want to avoid shifting is because we want the hashring to be a > long lived data structure that can be cheaply updated over p> several consensuses. > > Here are my questions: > - Is the above understanding correct? Yes, that's also my understanding. I think what prop259 tries to do with that hashring construction is to provide a deterministic way to sample guards, with minimum shifting as time passes. The aim here is to make it harder to track and fingerprint clients based on their guard lists. The idea is that when you are in WiFi "home" you always have one set of guards, and when you move to the WiFi "cafe" you have a different set of guards. See #10969 for some background and check http://meetbot.debian.net/tor-dev/2016/tor-dev.2016-01-19-16.03.log.html starting from 16:21:58 for some recent discussion. Personally, I think that the hashring is kind of a red herring in the sense that it makes prop259 more complex than it needs to be, and it also tries to tackle a problem that we don't really know how to solve. I mean even if we have the hashring construction, how do we use it? Do we just blend in the MAC of the network gateway when we sample guards? What happens if an evil gateway changes its MAC constantly, till we sample the guards it wants? I think solving the guard fingerprinting issue should be the subject of a separate proposal. Isis suggested this during the meeting but I was too dump at that point. Until we have a proper solution here I'd suggest we stick to the KISS thing, which probably is to sample guards from a simple list weighted by their bandwidth like Tor is currently doing (see node_sl_choose_by_bandwidth()). Of course, if you guys are interested in this and want to simulate the hashring idea (or any othe similar idea) that's fine as well. That's the good part of a simulation; that we don't need to commit to any particular construction. > - At what point do we add/update/change guards in UTOPIC/DYSTOPIC_GUARDLIST? I think prop259 suggests that we populate the guardlists the first time Tor attempts to connect to a network. This is different to how Tor currently does it, where it starts with a minimal guardlist (one guard), and adds new guards to it on demand (when the first guard fails). Both approaches have their merits. I'm not sure which one is better. We add more guards to guardlists if all the previous guards don't work anymore. This can be seen in step 4 of §2. We also need to mark (update) guards as unreachable when they get removed from the consensus, or when we fail to connect to them. Indeed the proposal should be updated to better specify when these actions happen and how they work. > - Since each key in the hashring structure depends acutely on bw_weight_total > and the BW of the guard, how will it be possible to update it > efficiently? My assumption is that the bw_weight_total will change on every > consensus, and at that point all the old keys are invalid?

1 0

(no subject)
by Christian Sturm 04 Feb '16

04 Feb '16

Hello, a while ago I set up a BSD buildbot for tor[1]. It's actually not just for BSDs, but also for other systems, such as Solaris. I wanted to make it easier to notice when something breaks on non-popular systems, not actively used by the developers. There also is reporting through an IRC bot and I wanted to ask if it would be interesting to have it report there. Currently it just lurks in there and lacks the permission to report build failures. So it would be great if the user 'bsdbuildbot' could get those permissions. I was wondering whether this is something that developers are interested in and if so what kind of reporting they'd like to have. Currently (if it would be allowed to) it would report individual build failures and success, but of course if that's too noisy could also change to just notify when this changes. Thanks to weasel there also is a FreeBSD builder for Jenkins. However that's only FreeBSD for now. The buildbot already has some other operating systems as build slaves and it's rather simple to set it up. Chris [1] http://buildbot.pixelminers.net/

1 1

Next (and subsequent) proposal discussion meetings
by Nick Mathewson 04 Feb '16

04 Feb '16

Hi, friends! This is a followup to the previous thread on the subject of proposal reviews. If you didn't read it, you can find it here: https://lists.torproject.org/pipermail/tor-dev/2016-January/010219.html [If you don't know how proposals work, see Lunar's explanation here: https://lists.torproject.org/pipermail/tor-dev/2016-January/010246.html ] First, the next meeting is scheduled on #tor-dev this Thursday, at 9:00 am EST (=1400 UTC). We'll be discussing these proposals: * Proposal 263: Postquantum circuit extension * Proposal 249: Large create cells Second, I'm trying to use the wiki to plan and schedule these meetings going forward. I've asked people who need or want to attend them to comment on a set of proposed times, and tried to list the people who really need to be at each one. The wiki page is at: https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/Meeting… I'll try to start outlining meeting times roughly a month in advance there, and finalizing them at least a week in advance. (No promises; it will take a while to ramp up.) best wishes, -- Nick

5 7

Re: [tor-dev] Getting people started on guard selection algorithms [prop259]
by George Kadianakis 03 Feb '16

03 Feb '16

> On Tue, Feb 02, 2016 at 02:29:22PM -0500, Ola Bini wrote: >> Hi, >> >> We have now started looking at the proposal and the existing code. Our >> current plan is to first of all code more of the simulations referred >> to in stuff-to-test.txt. We also noticed that the hashring >> implementation for choosing the [DYSTOPIC/UTOPIC]_GUARDLIST hasn't >> been implemented in the simulation code - in fact, in the code it >> seems the [DYSTOPIC/UTOPIC]_GUARDS is used as the guardlist. We are >> planning on implementing the two different varieties of the guardlist >> selection algorithm as well. >> Sounds good. Another thing to note is that during our last meeting, we decided to not have the utopic/dystopic guardlists be disjoint (mainly for load balancing reasons): https://lists.torproject.org/pipermail/tor-dev/2016-January/010265.html Check the meeting logs if you care for more info. Unfortunately, I think prop259 has not been updated to specify this new behavior. This might be too researchy for what you are trying to do, but finding a nice behavior here would be very helpful. Here are is an example idea of doing the 80/443 fascist firewall detection heuristic without two disjoint guard pools: Alice initializes a single guard list with all the guard nodes. Then she does steps 1 to 4 from §2 of prop259. Then in step 5, if Alice has tried more than GUARDLIST_FAILOVER_THRESHOLD guards from her guard list, she goes into "dystopic firewall" mode. During this mode, Alice only picks 80/443 nodes as guards (maybe from a separate dystopic guardlist). If those don't work either (she tries GUARDLIST_FAILOVER_THRESHOLD of them), then Alice "should make a note to herself that the network has potentially gone down" as suggested by step 5.b. So in the above idea, there are two guardlists. Guardlist GUARDS has all the guards in the network, and then there is a DYSTOPIC_GUARDS guard list (which is a subset of GUARDS) which is only used during "dystopic firewall" mode. I think this has better load balancing and anonymity properties. But there might be even better behaviors. Feel free to come up with your own and test them! >> After that, we are going to start looking at where it fits in the main >> Tor code base. >> Sounds good. You might enjoy choose_random_entry_impl() as a starting point. Feel free to ask us any questions you have about the Tor code base. Either here or on IRC. >> The only thing I'm a bit unclear about from the specification is the >> idea of primary guards, and what the procedure is when no primary >> guards are possible - 259.§2.3 talks about "all available and fitting >> entry guards" - is this from the list of primary guards or the >> guardlist? >> IIUC it's from the whole guardlist, but this should only happen if the primary guards are unreachable. The idea with primary guards is that in an ideal world, a Tor client would always only connect to the top guard of its guardlist. To expose itself minimally to the network. Unfortunately, the network is fiddly so this is not possible because the top guard will eventually go down. The concept of primary guards tries to compensate for that, by going to extra lengths to ensure that at least you always connect to one of your N=3 top guards in your guard list. It does this by periodically checking the reachability of those top N=3 guards, and marking them online if they are (see step 2 of §2). So, even if you or your guards have reachability issues and you drift on your 12th guard or something, you will eventually come back to one of your primary guards when they are found online again. Also, for the above heuristic to work, nodes that are not listed in the latest consensus should not be considered primary guards. >> 259.§2.4 says "adds a new entry guard" - is that adding it to the list >> of primary guards or something else? >> Good question. I'm not sure what the proposal means there. Maybe isis can clarify this further? Here is an attempt to help. Hope I don't confuse you further. The currently implemented Tor guard algorithm keeps a list USED_GUARDS of the guards it has already connected to (it's also saved on disk and is the guard list you see in your Tor state file). Everytime Tor tries a new guard node, it adds it to USED_GUARDS. The top N guards of USED_GUARDS are the primary guards. If you have exhausted all of the guards in USED_GUARDS and you still can't connect, then you need to add a new node to USED_GUARDS and attempt to connect to it. I imagine that the "list of all available and fitting entry guards" referenced in step 3 is something like USED_GUARDS. (It's actually not called USED_GUARDS in the code. I just named it like this for this email.)

1 0

Git users, enable fsck by default!
by Nick Mathewson 02 Feb '16

02 Feb '16

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813157 for more information why. The tl;dr here is: * By default Git doesn't verify the sha1 checksums it receives by default. * It doesn't look like we've got any inconsistencies in our repositories I use, though. That's good! * To turn on verification, I think you run: git config --add transfer.fsckobjects true git config --add fetch.fsckobjects true git config --add receive.fsckobjects true peace, -- Nick

4 3

Re: [tor-dev] Monitoring Service
by Karsten Loesing 02 Feb '16

02 Feb '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Tom, On 27/01/16 20:02, Tom Ritter wrote: > [feel free to reply adding tor-project or whomever] Sure, let me copy tor-dev@. > Remember a while ago I lamented that I wished there was some > monitoring service that could tell me when my metrics service; > relay; or bwauth went down? I finally built one. I'm still kicking > the tires on it, and I intend to improve it more over the next week > or two - but I think it's here to stay. > > https://github.com/tomrittervg/checker > > Right now I have it running several monitoring jobs, with a second > instance running with no jobs but serving as a peer. I have it > checking a number of TCP ports (to see if my relays are still up), > and I have custom jobs for metrics and the bwauth. They're in the > samplejobs folder. They're very simplistic and bare-bones. My > hope is that they can be fleshed out over time to account for more > imaginative ways things could fail. > > I'm already discovering that my bwauth file sometimes gets more > than two hours behind.... > > But I think the most useful thing here is that now I have a > minimal framework for writing simplistic python jobs and having it > monitor things for me. Maybe it would be useful for more people? Yes! Well, I can't speak for other people, but having a monitoring system for Metrics-related services would be very useful for me. In fact, it's been on my list for a long time now. This seems like a great opportunity to spend more thoughts on it. I'm not sure if I mentioned this before, but we're using Nagios to monitor Onionoo. The Nagios script we're using makes a tiny request to Onionoo to see whether the contained timestamps are still recent. That's an indirect way to notice problems with the data back-end, and it has helped with detecting numerous problems in the past. More details here: https://gitweb.torproject.org/admin/tor-nagios.git/tree/tor-nagios-checks/c… So, I don't know Nagios enough to say how it compares to your system. But I could imagine that we write a similar check for CollecTor that runs on your system and notifies us of problems. And maybe it's possible to write that script in a way that it can also be deployed on Nagios. Here's what I could imagine that the script would do: every, say, 10 minutes it would fetch CollecTor's index.json which is specified here: https://collector.torproject.org/#index-json The script would then run a series of checks and report one of statuses OK, WARNING, CRITICAL, or UNKNOWN: - host is unreachable or index.json cannot be found for at least 30 minutes (CRITICAL) - index.json contains invalid JSON for all checks in the last 30 minutes (CRITICAL) - the contained "index_created" timestamp is older than 30 minutes (WARNING) or older than 3 hours (CRITICAL) - when concatenating "path" fields of nested objects, most recent "last_modified" timestamp by path prefix is more than X behind "index_created" (CRITICAL): - /archive/bridge-descriptors/: 5 days - /archive/exit-lists/: 5 days - /archive/relay-descriptors/certs.tar.xz: 5 days - /archive/relay-descriptors/consensuses/: 5 days - /archive/relay-descriptors/extra-infos/: 5 days - /archive/relay-descriptors/microdescs/: 5 days - /archive/relay-descriptors/server-descriptors/: 5 days - /archive/relay-descriptors/votes/: 5 days - /archive/torperf/: 5 days - /recent/torperf/: 12 hours - /recent/bridge-descriptors/extra-infos/: 3 hours - /recent/bridge-descriptors/server-descriptors/: 3 hours - /recent/bridge-descriptors/statuses/: 3 hours - /recent/exit-lists/: 3 hours - /recent/relay-descriptors/consensuses/: 1.5 hours - /recent/relay-descriptors/extra-infos/: 1.5 hours - /recent/relay-descriptors/microdescs/consensus-microdesc/: 1.5 hours - /recent/relay-descriptors/microdescs/micro/: 1.5 hours - /recent/relay-descriptors/server-descriptors/: 1.5 hours - /recent/relay-descriptors/votes/: 1.5 hours In the detailed checks above, the script would not warn if index.json does not contain any files with a given prefix (so that you can run the script on your CollecTor instance that doesn't collect all the things). And ideally, the script would include all warnings in its output, not just the first. That's one check, and it would probably catch most problems where things get stale. I would like to add more checks, but those would need more access to the CollecTor host than its index.json which is publicly available. (That could mean that we export more status in a debug.json or in another public place.) Some examples as follows, and I'm only listing them for later: - Is the host soon going to run out of disk space or inodes? (This can easily be done with Nagios, I think.) - Did the relay-descriptor part of CollecTor fail to parse a descriptor using metrics-lib and hence did not store it to disk? (I'm receiving hourly cron mails in this case, but I'd prefer something better.) - Are we missing more than a certain threshold of relay descriptors referenced from other relay descriptors? For example, are we missing more than 0.5% of server descriptors referenced from the consensus? (I'm also receiving hourly cron mails, but the checks there are somewhat broken and can create a lot of noise.) - Same as above but for sanitized bridge descriptors. (I don't have such checks in place right now.) - Does the bridge sanitizer have trouble sanitizing bridge IP addresses in particular storing secret keys for the keyed hash function? (Cron would send mail, but that never happened so far.) - Did the bridge sanitizer run into an unknown line and hence refrain from storing the sanitized version of a bridge descriptor? (Cron sends mail which happens every now and then.) - Do the exit lists downloaded from TorDNSEL contain recent publication times but no recent scan times? - Does one or more of the Torperf hosts not report recent measurements? There, that's a long list. And it's just the list for CollecTor. Next would be more detailed checks for Onionoo that look into its internal operation or into problems with its web-facing part; Metrics including the various data-aggregating scripts running in the background; and ExoneraTor with its database importer to make sure it always has fresh and complete descriptors. Hope this was not too overwhelming. If you're still reading this and still want to help, let's talk more about writing that first script for CollecTor. Thanks! All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWsIYbAAoJEJD5dJfVqbCrY3EH/00mw1INpPniTRXnd981nJEl EAZdySgKNcuA6NIVrGMrRdRgH3p3W2UhVJYlyHpHquZOx0nLmEDH8XDyBcxvICEQ UviwPGZGiQCw+TYqFYlUPvM2d0DwZmjyQbMeoANo4r8BczqFKmwi9GC1xI/zc3/3 UT91OXyZeKIbhopaeyIWW868p/u8Cs9sGDSJuWZHmDqQjPL2sE0lM8bQtdK1wJWj UNTCYPT35FfSHYeuWXASDgqt0e8qrppWKhEzQgmWggAU6sPx1MMSz323fXymdPCX OyHbXYjzFsABE6+CAO5iR1c6AI3TFHxOtVutA8gfXC52GjgDApVETd+kywrNSZM= =kRdX -----END PGP SIGNATURE-----

1 0

Want to help in community team
by Carolin Zöbelein 02 Feb '16

02 Feb '16

--------------------------------------- Want to help in community team => Question: Is this the right mailing list for this? --------------------------------------- Hello, for several weeks I have already watched the goings-on at Tor (irc, blog, wiki, ...). Now I decided to say hello. I'm a very creative and autistic artist, satirist (writing), photographer, physicist, ... . Caused by some talent I work as mathematical adviser and now I'm also studying math to pretend more professionalism. Also I do what I call "creative (crazy) Nerd-consultant" after I didn't found an already existing job title which describes what I do. People say: "You are extremely strange, crazy and creative" and I thought I should use this. Sadly I'm not the best programmer although I earned my money with it in the past. Or better I'm able to code but I prefer to spend my time with math or other creative work. It just makes more fun. I think I'm best suited into the community team with my skills. But I'm also open for other tasks of course ;-). > > Monday, 18.01.2016, Georg Koppen <gk(a)torproject.org>: > Hi, > > I realized recently that there seem to be no community team meetings > anymore (is that the case or am I just missing something?). At least not > in #tor-outreach. Nor are there status reports sent to tor-reports. > > Is there any other way to get to know what the community team is > currently discussing/planning/concerned with? > > Georg > > > _______________________________________________ > tor-project mailing list > tor-project(a)lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-project > That's a good question. In particular for new people like me. What's the typical way to get involved in the community team? I hope I'm not too crazy for Tor people and don't know if and how I can help. So that's enough for a public mail. Bye, Carolin -- ________________________________________________________________________ URL: http://www.carolin-zoebelein.de E-mail: contact(a)carolin-zoebelein.de PGP: D4A7 35E8 D47F 801F 2CF6 2BA7 927A FD3C DE47 E13B

2 1

Applications Team Meeting Next Tues (Feb 9) at 19:00 UTC
by Mike Perry 01 Feb '16

01 Feb '16

Due to conflicts with other meetings, Georg and I both will be busy tomorrow during the regularly scheduled applications meeting (first Tuesday of each month at 19:00 UTC). So instead, we will have this month's Applications Team meeting next Tuesday, Feb 9, at 19:00 UTC. As usual, we will discuss usability, configuration, and any recent user study progress. Anyone interested in these topics for any user-facing Tor application is welcome and encouraged to participate! -- Mike Perry

1 0