July 2015 - tor-dev - lists.torproject.org

Relay Database: Existing Schemas?
by nusenu 27 Feb '16

27 Feb '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, I'm planing to store relay data in a database for analysis. I assume others have done so as well, so before going ahead and designing a db schema I'd like to make sure I didn't miss pre-existing db schemas one could build on. Data to be stored: - - (most) descriptor fields - - everything that onionoo provides in a details record (geoip, asn, rdns, tordnsel, cw, ...) - - historic records I didn't find something matching so far, so I'll go ahead, but if you know of other existing relay db schemas I'd like to hear about it. thanks, nusenu "GSoC2013: Searchable Tor descriptor archive" (Kostas Jakeliunas) https://www.google-melange.com/gsoc/project/details/google/gsoc2013/wfn/ 5866452879933440 https://lists.torproject.org/pipermail/tor-dev/2013-May/004923.html https://lists.torproject.org/pipermail/tor-dev/2013-September/005357.htm l https://github.com/wfn/torsearch (btw, someone knows the license of this?) > This is true: the summary/details documents (just like in Onionoo > proper) deal with the *last* known info about relays. ernie https://gitweb.torproject.org/metrics-db.git/plain/doc/manual.pdf (didn't find db/tordir.sql mentioned in the pdf) "Instructions for setting up relay descriptor database" https://lists.torproject.org/pipermail/tor-dev/2010-March/001783.html "Set up descriptor database for other researchers" https://trac.torproject.org/projects/tor/ticket/1643 -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVLrmVAAoJEFv7XvVCELh0+ycP/0VkRWMGAhx7YwWhixtOPB0q ouitosse3SdLCMd1W7cps7nnQhcGv8E1xeUPYeizNQ+RIK9ldvnwgxhVt9ETOI3e oL6bcx3B/NSbTcGfZhIU6XCCX7+39CXz3/+/gXg4c1a5Cd9rYKuwTn7ZApPUNAK2 9yfckCwHaqh67ZDHB1A7nrHX1BJAjwhri5JRGomHSUXPwNZOyMzpgPkB+I92esqc Z4ajQo2D+Vyk42qF7apqp5ApVBLdt5ycUtW1j4s4KvyJH2BjeP4tCXvvYLKv9IvN TGHMJ111NQXbuk6H1OdaDmMZhhW6utPk8YAbNc7RZHqCdH7No40zFVlikLDrV2Xl uLt8FMlck7hGQv/4udA59Dw3PGHfrSiCbvJXb0S2jarnWOH2O+zSSuMs8jQM/x7k 6tIOlwtRDVGw3VuvNbb4MJnLzdHRxq3qy6ueDYuhmhHslxjD1Cbr3eE1RA7Da9aX kNDgjfXtdah52HWUZbIIxHQYzyCg5G3M5yy51GhUlA2Fe1sbPpNcEBeyEDK/jH3Y +7G+tgR+CdMvSFraCiYKHa1drVPJHeqNvZAqPNHKxpKwOwOoJEYrsW4wb/mJ7ybe o8xNYK691hcaMvHqwI47tw4hZfrI0f0+gQKKzENQQk8/8yUZ/0jgMRlW/Cz0yYIa FZEg+OJ/yEs8vKfYerIN =Rfn4 -----END PGP SIGNATURE-----

3 2

Proposal: Merging Hidden Service Directories and Introduction Points
by John Brooks 29 Sep '15

29 Sep '15

Hello! George and I, along with the other participants of this hidden services meeting, have written a proposal for the idea to merge hidden service directories and introduction points into the same entity along with proposal 224. Comments are encouraged, especially if there are downsides or side effects that we haven’t written about yet, or that you have a different opinion on. The intent is that we can decide to do this before implementing proposal 224, so they can be implemented together. The proposal is attached, and also available from: https://raw.githubusercontent.com/special/torspec/224-no-hsdir/proposals/id… Thanks! - John

11 19

Get Stem and zoossh to talk to each other
by Philipp Winter 17 Aug '15

17 Aug '15

Hi Damian, I'm interested in building a lightweight, internal domain-specific language to explore archived Tor data. The goal is to make it easy to answer questions like the one that recently came up on tor-relays, "how many guards shift location significantly across the Internet, and how often?" Combining Stem and zoossh seems like a good solution. Ideally, zoossh should do the heavy lifting as it's implemented in a compiled language. For data exploration, however, having a Stem-enabled Python shell with a set of analysis methods sounds better. Now the question is how to pass potentially large amounts of readily-parsed consensuses and descriptors from zoossh to Stem? In a perfect world, we would have bindings to use zoossh in Python. The gopy [0] folks are working on that, but it's a young project; interfaces are not yet supported. Two workarounds come to mind until gopy catches up, both requiring some glue code: 1. Let zoossh do the data filtering and then return a list of files that are then parsed again by Stem. That's easy to implement, but can be quite inefficient if the filtering step still returns plenty of data. 2. Have some IPC mechanism that passes objects from zoossh to Stem. Objects could be serialised in some way to minimise unnecessary parsing. While that might be the most efficient option for now, it probably requires too much work. 3. ...something else I didn't consider? Please let me know if you have any thoughts. [0] <https://github.com/go-python/gopy> Cheers, Philipp

3 10

Is anyone using tor-fw-helper? (Was Re: BOINC-based Tor wrapper)
by Nick Mathewson 05 Aug '15

05 Aug '15

Yawning's mail below reminds me: I am considering removing the C implementation of tor-fw-helper from the tor distribution, and recommending Yawning's pure-Go implementation instead. But before I do this, I'd like to get some sense of whether folks are shipping tor-fw-helper today, or using it in practice. On Jul 21, 2015 11:26 AM, "Yawning Angel" <yawning(a)schwanenlied.me> wrote: > > On Wed, 22 Jul 2015 01:06:41 +1000 > teor <teor2345(a)gmail.com> wrote: > > > > > > On 20 Jul 2015, at 11:11 , Serg <std.serg(a)gmail.com> wrote: > > > > > >> How do you plan to map ports on NAT devices? > > > > > > If it can't be done automatically using UPnP, This must be done > > > manually. No alternative cases. > > > > Our experience is that most routers' UPnP / NAT-PMP implementations > > don't work well with (our) automated tools. So this would have to be > > done manually, significantly reducing the pool of available > > volunteers. > > Just chiming in here. This may well work for a good number of users, > but the support overhead for when it fails is utterly gigantic because > certain brands of consumer routers have extremely poor UPnP/NAT-PMP > implementations. > > The usual symptom of a poor implementation is "the router crashes" but > certain other behaviors have been documented in the past by people > trying to use UPnP in ways that are spec compliant such as "the router > crashes and requires a NVRAM reset", "random port mappings get > obliterated", "the UPnP/NAT-PMP stack on the router crashes" etc. I wonder how commercial software handles these cruddy routers. Do they restrict themselves to a tiny part of the spec? Do they probe for problematic firmware, and maintain a list of unreliable versions? [I am very glad it is not our job to maintain such a list.] -- Nick

8 20

Relay Web Dashboard - Summer of Privacy
by Cristobal 31 Jul '15

31 Jul '15

Hello everyone, This summer I'll be working on a Relay Web Dashboard as part of Tor's Summer of Privacy [1]. The goal of this project is to develop a web application to monitor Tor relays. A good part of the back-end code will be based on what nyx [2] currently does (using stem [3]). You can take a look at my SOP proposal here [4] to get a better idea of what I'm planning to do. TL;DR : use Javascript to fetch information from a 'Relay API'. This 'Relay API' will use stem to collect data from the relay, keeping in mind that the idea is to follow a 'read-only' approach, in other words: just monitoring (at least for now). On the first half of SOP I'll be working on making this app based on nyx features. On the second half I'm planning to add new features if necessary, and for that I'd be very interested in hearing your feedback and ideas, specially if you are a relay operator and you think there are features that will help you with the task of running a relay. I hope this project would be of help to relay operators community by giving them a secure, useful and easy-to-use tool for their relays. I'll be informing you about my progress with bi-weekly reports to this list. I'm also on IRC by the nickname @clv Looking forward to hear from you! Best regards, Cristobal GPG: 2ACA 8DD4 444E 358F B72E 8C3C 7196 690D 9E03 CDFD [1] https://trac.torproject.org/projects/tor/wiki/doc/gsoc [2] https://gitweb.torproject.org/nyx.git [3] https://stem.torproject.org/ [4] https://leivaburto.github.io/sop-proposal

4 4

The Shadow Simulator will soon use cooperative threading to better support plug-ins
by Rob Jansen 31 Jul '15

31 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hello Shadow! Hello Tor! For the last couple of weeks, I have been overhauling the way Shadow interacts with plug-ins, and I want to make sure you (and others developing/using Shadow plug-ins) are aware of the upcoming changes. I am enhancing Shadow to support the following two capabilities: 1. Run plug-ins that make blocking system calls 2. Run plug-ins that create and use pthreads To achieve this, I am using a user-space cooperative threading library called pth: http://www.gnu.org/software/pth/ Pth works by storing and swapping stack states using makecontext and swapcontext (see man pages), so that we can intelligently switch back to Shadow's execution stack when the plug-in makes blocking libc function calls, and switch back to the plug-in stack only when its file descriptors are waiting for I/O or timers have expired. This same technique allows us to support would-be plug-in pthreads as application layer pth threads instead, which will help us avoid expensive kernel context switches. This will be especially useful/efficient when running thousands of nodes simultaneously, which normally would consume thousands of pthreads which are not Shadow aware. I have modified the pth library to be reentrant and thread-safe using caller-supplied states and thread-specific data, which I needed because Shadow itself runs with multiple simulator worker threads. Cool, huh!? So what does this mean? With this fancy new approach, we must still build each plug-in using LLVM/Clang in order to make use of Shadow's cool state-swapping trick, but we may no longer need to do other customization, particularly those historically needed for programs that block or use threads. Also, with these changes, Shadow's "API" has been removed, and Shadow now just calls the "main" function symbol in each plug-in directly, and the plug-in need not know anything about Shadow. I'm excited about these changes, and hope they will improve support for new plug-ins while also reducing maintenance in the long term. So far I have Shadow's built-in traffic generator running correctly with the new approach, and am now working on Tor. None of this code is public yet, but if you are interested in an example of how this would work or want some early access to the code, drop me a message off-list. Cheers! Rob - -- Rob Jansen, Ph.D. Computer Scientist Center for High Assurance Computer Systems U.S. Naval Research Laboratory rob.g.jansen(a)nrl.navy.mil robgjansen.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCgAGBQJVu4lYAAoJEO3Z5w0UVGXoASAP/3qjW0551xxnrLwWQcrXhbiN cf7hryzU7vDzKRkdzzNbEive0JFZhDONbruJ5Lcx1BW4/uvFZ1emW0FuZFri1m7b xWW6Pp43cMDGeIaMzmWZ9UAea59wirFk/C47R/jDBkXWnZpz5Arc1DjryPbj2LqN oPcfd9JDyE9zk2Y0sSytYt3IulKA62ZOV7+WbIwhHUTj70OgXFvVLhLMMVgOBEDO rHWxDg/+5mC8WqEg4L7lS8/ivAjr1YboA4PV0uSPzgZ+208iQBbsF3DoCG/8D9kY 83xpX/+3aGdWl7XjrncpA5x7gt392fX/3EL9k8lUYu2PKPaOWmJ0/xWbGoXGJOhA epAWPr95X7DFzpcsS59sq6usDeiltFllzfxgA5TbDsUgVwOa9227ygYtK/MVifi6 gYkpZzC6YEMDR6id0ZOwwqYKTsx1Sxomyz30w5iAp1l+Grb10bhUZlfqsNx9PkuI HdBMfvNziApSOQBge36QwskGFlJobfGdpL5w0yep1MSmBJCLVSCopCllLVcbDzQX ipbjmHjJXHNaIL7DHm073VJe2XAjcoMPHHs/nhq0vTgrMFGMDsNvMJ7+66xz3iwZ V3WUjEP0fPjgGZ8MTWj2HjGOQyENa5yAmAiBJUosg1fzgeinushWfE+Zm4m0sVEJ BJttAL/8SnI4DpbIqq6e =c1wf -----END PGP SIGNATURE-----

1 0

Unreachable hidden service by some clients
by Liste 31 Jul '15

31 Jul '15

I was open some hidden service to do an experiment: All work fine. After a day all hidden service on a single server are unreachable from another server used as client. After some hous all hidden server become reachable. Some time when the hidden services are unreachable, they return to work when tor restart. What's going on? We are under attack?

2 1

onionoo.tpo stuck at 2015-07-29 07:00
by nusenu 30 Jul '15

30 Jul '15

2 1

Roster introduction
by Virgil Griffith 29 Jul '15

29 Jul '15

Hello everyone. This is my first report on the Roster project and I wanted to give you all an introduction what it is and where it's going. I'm interested in seeing Tor grow. Current work towards this is tor2web and now Roster. Roster is the rebranded continuation of the "Torati" proposal which I first mentioned here: https://lists.torproject.org/pipermail/tor-dev/2014-June/006975.html In short the gist is to: * Socially incentivize operators to run relays. * Make the operator experience more compelling. * Gamify operating a Tor relay. The main difference between Roster and Globe/Atlas is that Roster provides information at the level of *operators*, not individual relays. In Roster information is aggregated and metrics are summed across all relays of a family. The idea is that when you run a relay, you generate nerd points and badges. And then when people Google your name/alias, they will see how many nerd points+badges you have and they will be in awe of your epic geek-street-cred. In addition to improving operator experience, there are proposals for encouraging operators to do things Tor management like them to do such as upgrading to the latest Tor version and/or probably setting up their Families. E.g., a badge for using the latest/recommended Tor software. That's the gist of it. I will be spending most of my time on tor-relays@ and talking with operators there. So if you want to follow day-to-day stuff subscribe there. Main things accomplished so far: * Setup the basic website at: http://www.tor-roster.org/ * I have an assistant whom I've taken under my wing. His name is Sean Saito <saitosean(a)ymail.com>. He's a Computer Science student at Yale-NUS here in Singapore. He wishes to become more involved in The Tor Project and this is his first Tor-related experience. * We also have discussed with the tor-relays@ list candidate badges. Are interested in additional badge proposals. --- https://lists.torproject.org/pipermail/tor-relays/2015-June/007228.html That's it so far. -Virgil

3 5

[Measurement Team] Next IRC meeting happens tomorrow, Jul 29, 14:00 UTC in #tor-project
by Karsten Loesing 28 Jul '15

28 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello list, This is an announcement/reminder that there will be an IRC meeting of the Measurement Team tomorrow on Wednesday, July 29, 2015, 14:00 UTC in #tor-project http://www.timeanddate.com/worldclock/fixedtime.html?iso=20150729T14 Two weeks ago we created a list of existing measurement-related products and last week we brainstormed what tools we would want to exist if none of these existed yet. Logs of all past meetings are available here: http://meetbot.debian.net/tor-project/2015/tor-project.2015-07-08-14.06.log… http://meetbot.debian.net/tor-project/2015/tor-project.2015-07-15-14.00.log… http://meetbot.debian.net/tor-project/2015/tor-project.2015-07-22-14.00.log… While I found the last two meetings extremely productive, I want to suggest a slightly looser-defined agenda this week: - Introductions, problems, products, priorities: any feedback, remarks, comments on what we did in the previous three weeks (for example, I'm planning to present a roadmap document based on the two pads we worked on in the past two weeks) - 1-1-1 task exchange: you get 1 minute to describe a task that would take somebody else roughly 1 hour and that they will do for you within 1 week (review a document, write some analysis code, fix a small bug, etc.; better come prepared to get the most out of this; give 1, take 1) - Misc/discussion: whatever comes up during the meeting If you think this can be fun, please join us at the meeting! Talk to you tomorrow! All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJVt5O9AAoJEJD5dJfVqbCrqVUH/jKEjckSOEY+1hOpz7cyPXFr E6Z7L64gDSUlCJ68YeADA5C8Y1kG6Pzkhb+Z7zGigJp7ZdEWI8xLbhUtXcgE6cQY 9pyqYd+IK/zNrSdUTgHEfpfHF0sAyHCgMcUMuNQ0Vt82IFxyWef30MJm/w6jvis8 v5iFRLbDut1YZ+xXmm/Y6YyyXxCuSPE/rKxQUbIXFAqf3Nz4207tC4VF9h53vKeF aZr5fnZC6j3cdt0QbPi3vBYMsw9oU0Z9HDpleEh639uEM07wq7wamBvvOKJioNtT 0IcwlDoCDx5ldyZsCNE9aOdVzwkzDjYBRytgNJUHzTyh+yjmFzlIR3a5H8KTVrI= =FCO3 -----END PGP SIGNATURE-----

1 0