November 2015 - tor-dev - lists.torproject.org

Re: [tor-dev] stale entries in bwscan.20151029-1145
by starlight.2015q3＠binnacle.cx 06 Nov '15

06 Nov '15

aAt 20:56 11/5/2015 -0600, you wrote: >On 5 November 2015 at 16:37, <starlight.2015q3(a)binnacle.cx> wrote: >> By having a single thread handle >> consensus retrieval and sub-division, >> issues of "lost" relays should >> go away entirely. > >So I'm coming around to this idea, after spending >an hour trying to explain why it was bad. . . :-D happy to hear that It can be expressed and implemented various ways, but the core idea of centralizing this … [View More]function is hard to escape. >So the >main problem as I see it is that it's easy to move >relays between slices that haven't happened yet - >but how do you do this when some slices are >completed and some aren't? >. . .if you used a work queue instead of a scanner. . . I'm thinking of it more as a set of circular work lists, since scanning never ends. Not then a question of "haven't happened yet" for new relays since regardless of where they're inserted they will be measured. More a question of giving priority to new relays and giving priority to relays that have exhibited a large change of bandwidth, particularly in the downward direction. If radical restructuring is on the menu (as the remainder of your comments imply) let me put out an idea I've come up with: I'm not enthusiastic about the current design where nine scanners blindly compete with each other for the local Tor client-router and link bandwidth. The goal should be to figure out what percent of the BWauth link's bandwidth should be consumed to minimize scanning time while ensuring quality measurement. Nine scanners could be replaced with one scanner that runs the entire list of relays sorted from fastest to slowest. This scanner would keep the slice approach of firing-up scans until a quota is filled. But instead of that quota being a somewhat arbitrary fixed value, the quota would be a minimum bandwidth target per the output of 'SETEVENTS BW'. A rough guess is that whatever bandwidth is available should be 35% subscribed, though the actual setting would require empirical determination. On a 100mbps link the target might then be a continuous 4.5 million bytes/sec of scanning. This might result in the 100 or so fastest relays being measured one-at-time, but of course it would happen rather quickly. As the scanner works down the list the number of concurrent scans would rise rapidly as the bandwidth of the targets decline and so perhaps several hundred of the slowest relays would be under measurement near the end of each pass. My belief is the end result would be dramatically faster than the current approach, have a low circuit-fail rate, and that the measurements would be far more reliable and consistent. >FWIW, looking at >https://bwauth.ritter.vg/bwauth/AA_scanner_loop_times.txt , it >seems like (for whatever weird reason) scanner1 took way longer than >the others. (Scanner 9 is very different, so ignore that one.) > >Scanner 1 > 5 days, 11:07:27 Yes noticed this a few days ago, examined ticket 17482 more closely and saw it was 6.5 days for scanner-one just before the 10/30 restart. The scanner-one log shows huge numbers of circuit failures. Observation partly inspired the above restructuring idea. [View Less]

1 0

Re: [tor-dev] stale entries in bwscan.20151029-1145
by starlight.2015q3＠binnacle.cx 06 Nov '15

06 Nov '15

At 11:47 11/5/2015 -0600, Tom Ritter wrote: > . . . >So them falling between the slices would be my >best guess. . . Immediately comes to mind that dealing with the changing consensus while scanning might be handled in a different but nonetheless straightforward manner. Why not create a snapshot of the consensus at the time scanning commences then-- without disturbing the scanners-- pull each new update with an asynchronous thread or process. The consensus thread would diff against … [View More]

2 2

[proposal] New Entry Guard Selection Algorithm
by isis 06 Nov '15

06 Nov '15

Hello, For your reviewing pleasure, may I present you with a new (draft) proposal for a new entry guard selection algorithm, and associated data structures (with reference implementations!). [0] The bulk of it is based upon George Kadianakis' recent email and pseudocode. [1] [2] There are still a few XXXs which need elucidation and/or comment. Relevant tickets include: https://bugs.torproject.org/17261 https://bugs.torproject.org/17262 https://bugs.torproject.org/16120 https://… [View More]

4 3

Re: [tor-dev] stale entries in bwscan.20151029-1145
by starlight.2015q3＠binnacle.cx 05 Nov '15

05 Nov '15

At 11:47 11/5/2015 -0600, Tom Ritter wrote: > . . . >So them falling between the slices would be my >best guess. . . Immediately comes to mind that dealing with the changing consensus while scanning might be handled in a different but nonetheless straightforward manner. Why not create a snapshot of the consensus at the time scanning commences then-- without disturbing the scanners-- pull each new update with an asynchronous thread or process. The consensus thread would diff against … [View More]

1 0

Re: [tor-dev] stale entries in bwscan.20151029-1145
by Tom Ritter 05 Nov '15

05 Nov '15

[+tor-dev] So... weird. I dug into Onyx primarily. No, in scanner.1/scan-data I cannot find any evidence of Onyx being present. I'm not super familiar with the files torflow produces, but I believe the bws- files list what slice each relay is assigned to. I've put those files (concatted) here: https://bwauth.ritter.vg/bwauth/bws-data Those relays are indeed missing. Mike: is it possible that relays are falling in between _slices_ as well as _scanners_? I thought the 'stop listening for … [View More]consensus' commit would mean that for a single scanner would use the same consensus for all the slices in the scanner... -tom [0] https://gitweb.torproject.org/torflow.git/commit/NetworkScanners/BwAuthorit… On 5 November 2015 at 10:48, <starlight.2015q4(a)binnacle.cx> wrote: > Hi Tom, > > Scanner 1 finally finished the first pass. > > Of the list of big relays not checked > below, three are still not checked: > > *Onyx 10/14 > atomicbox1 10/21 > *naiveTorer 10/15 > > Most interesting, ZERO evidence of > any attempt to use the two starred > entries appears in the scanner log. > 'atomicbox1' was used to test > other relays but was not tested > itself. > > Can you look in the database files > to see if any obvious reason for > this exists? These relays are > very fast, Stable-flagged relays > that rank near the top of the > Blutmagie list. > > > > > >>Date: Thu, 29 Oct 2015 19:57:52 -0500 >>To: Tom Ritter <tom(a)ritter.vg> >>From: starlight.2015q4(a)binnacle.cx >>Subject: Re: stale entries in bwscan.20151029-1145 >> >>Tom, >> >>Looked even more closely. >> >>I flittered out all relays that are >>not currently active, ending up with >>a list of 6303 live relays. >> >>1065 or 17% of them have not be >>updated for five or more days, >>292 or 4% have not been updated >>for ten days, and 102 or 1% >>have not been updated for 15 >>days. >> >>In particular I know of a very fast >>high quality relay in a CDN-grade >>network that has not been measured >>in 13 days. My relay Binnacle >>is a well run relay in the >>high-quality Verizon FiOS network >>and has not been measured for 10 days. >> >>This does not seem correct. >> >> >>P.S. Here is a quick list of some >>top-30 relays that have have been >>seriously neglected: >> >>redjohn1 10/9 >>becks 10/15 >>aurora 10/20 >>Onyx 10/14 >>IPredator 10/15 >>atomicbox1 10/21 >>sofia 10/14 >>naiveTorer 10/15 >>quadhead 10/12 >>3cce3a91f6a625 10/13 >>apx2 10/14 >> >> >> >>>At 13:35 10/29/2015 -0400, you wrote: >>> >>>>The system is definetly active. . . .the most recent file has ten day old entries? >>> >>>Just looked more closely. About 2500 >>>of 8144 lines (30%) have "updated_at=" more >>>than five days ago or 2015/10/24 00:00 UTC. >>> >>>Seems like something that should have >>>an alarm check/monitor. > [View Less]

1 1

Tor dev meeting times this week and beyond!
by Nick Mathewson 05 Nov '15

05 Nov '15

Hello all! As a reminder, we have a weekly patch workshop and a weekly developer's meeting on the #tor-dev IRC channel on OFTC. With the ending of DST in the US, these times are now convenient for even fewer people than before. With that in mind, we're changing! The new times will be effective starting NEXT WEEK, November 9. (My apologies; it's too late for me to change my schedule this week.) Primary meeting and patch workshop: Wednesday 1830 UTC (1:30pm EST , 10:30am PST) Secondary … [View More]

3 2

An ANTLR 4 grammar for Tor bridge network statuses
by Karsten Loesing 05 Nov '15

05 Nov '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello developers, in the past few days I have been working on a grammar to parse Tor bridge network statuses and hopefully other Tor descriptors in the future. It's working, for some definition of working, but some issues remain and I need some help. I just uploaded my sources, consisting only of the grammar with a fair amount of documentation: https://people.torproject.org/~karsten/volatile/BridgeNetworkStatus.g4 Quoting from that file to … [View More]facilitate discussion here: There are multiple goals of having a grammar for Tor descriptors available on CollecTor: 1. Translate descriptors to JSON for statistical analysis: Some tools and databases require Tor descriptors in a standard format like JSON. This grammar and a parser generated from it can help making that translation as easy as possible, also to keep future maintenance as low as possible. 2. Provide a basis for descriptor-parsing libraries: As of late 2015, there are three libraries for parsing Tor descriptors: metrics-lib for Java, Stem for Python, and Zoossh for Go. It would be beneficial to place as much knowledge about the descriptor format into a grammar shared by all those libraries and then generate parsers for different languages from that grammar. 3. Serve as documentation for the Tor directory protocol specification: Tor descriptors are already documented using a hand-written grammar, but that may contain slight inaccuracies because it's not verified. This grammar could fix that by either detecting inaccuracies while trying to rewrite it to an executable grammar form or by replacing the grammar in the specification documentation with this executable grammar. Open issues and questions: - Was it smart to explicitly include all those SP tokens in the rules, or should those be discarded right away by the lexer? The main reason for keeping them was to stay as close to the specification as possible, but maybe that has downsides on the other goals. - If a bridge uses a nickname (or other token that's supposed to be a STRING) that is also a keyword like "r" or "published", things get confusing. Try editing the input bridge network status and observe the result. But those are perfectly valid nicknames, so what can we do? - It would be really nice to use regular expressions in the grammar to match input more thoroughly than just ~[ \n]+, if only we can fix the lexer troubles. It's a pity that all that verification work would need to be duplicated in each of the language-dependent parsers. That kinda defeats the purpose. - Is it easy to walk the parse tree and output a JSON format *without* having to write code for each of the rules? Ideally, the translator would be 20 lines of code and not grow at all if we add 10 more descriptor types. Do we need to change the grammar for that? - The following may turn out to be a non-issue, but some descriptors require lines to be ordered, e.g., "accept" and "reject" lines in server descriptors, and we'll have to retain that order in the parse tree. This should be similar to how we parse entries, starting with "r" lines, but who knows. Feedback much appreciated! All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWOcp+AAoJEJD5dJfVqbCrnQUH/2dp8ER6ZcEGBtHP+dPb/4p0 0tKb4eZobQhZNx3oQOc08nCJl3AEa+Vedep5Caa9MSNycopf7mBFEGtw2V5J3mKN w6D6cvSbSBoFhuh/+Q8oVj+6h0KkUaCVVMaTHefb63usM0EmjsEXvDjBXr+g5nhn q0RqM1Id3V38rs3pKi1JDwGU4w5X45gzUPOXbiNGig6wJuLN1e2cxfF4RdDmGzST JvjlH/KRV59NjMRvUAeTxZIXlz6fKwjTWWQ2PXUnuXAXNPVxYakzHNhiT7qXGro0 7ZFfIr7gwk9kZlF0oy6ltFC1mGgL4xk0vqlrOjvwrh+oAzIciurMcOddXEHwr3E= =DZ0Z -----END PGP SIGNATURE----- [View Less]

2 2

A quick and dirty UX evaluation of Tor Messenger
by Greg Norcie 04 Nov '15

04 Nov '15

Hi all, First of all, great work on Tor Messenger - it's awesome to see Tor pushing to be more accessible to non-technical folks. I decided to take some notes when installing + using for the first time, hopefully this will be helpful. While there are some minor issues, I'd like to stress that this is a great first step and considering this is the first general public release it's a very good design. /////// I'm on an OSX machine, so the first thing I see after clicking on the icon is a … [View More]message that Tor can't be opened because it's from an unidentified developer. This would raise an eyebrow at least for most users. (And lessens the effectiveness of the warning if we are going to train users to ignore it) I right click and select "open" and there is a longish pause (>10s) which could frustrate new users, then the wizard appears. I don't chat often - I usually text my friends, and I don't want to use FB chat since it's tied to my real name. I remember that Twitter now has no DM limit. But it's not on the main list. I don't have any data to support this intuition, but I suspect that after XMPP, Twitter would be a popular protocol. I click once on "show all protocols" and click continue. There's only a few additional, Twitter being one of them. Not sure why this list wasn't present from the beginning. I get options for a local alias and "tracked keywords". I have no clue what the latter is, but it sounds like the concept of a taboo in Harry Potter, so for a laugh I put in "Voldermort". Now I'm authorizing Instabird. Fairly straightforward. Now I'm receiving a texted 2f code from twitter and entering it. Now I'm in, and I can see my Twitter timeline, which is interesting. I was expecting just a buddy list to send DMs, not timeline access. I post a tweet to my timeline, then want to send my boss a test DM. But I follow >1500 people, and have to scroll a lot to find him. I locate his name and double click, but nothing happens. At this point, I suspect a less security minded user might abandon the app. (I end up digging around a bit to find the functionality I need) [1] ("Your security preferences allow installation of only apps from the Mac App Store and identified developers.") -- /***********************************/ *Greg Norcie (norcie(a)cdt.org <norcie(a)cdt.org>)* *Staff Technologist* *Center for Democracy & Technology* 1634 Eye St NW Suite 1100 Washington DC 20006 (p) 202-637-9800 PGP: http://norcie.com/pgp.txt Fingerprint: 73DF-6710-520F-83FE-03B5 8407-2D0E-ABC3-E1AE-21F1 /***********************************/ [View Less]

2 2

1-1-1 task exchange meeting on Thursday, Nov 5, 15:00 UTC in #tor-dev
by Karsten Loesing 04 Nov '15

04 Nov '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello devs, we'll have another 1-1-1 task exchange meeting on Thursday, November 5, 2015, 15:00 UTC in #tor-dev https://www.timeanddate.com/worldclock/fixedtime.html?iso=20151105T15 Quick reminder: The idea of the 1-1-1 task exchange meeting is that participants get 1 minute to describe a task that would take somebody else roughly 1 hour and that they will do within 1 week. It could be to review a document, write some analysis code, fix a … [View More]

1 0

running a BWauth
by starlight.2015q3＠binnacle.cx 04 Nov '15

04 Nov '15

I am considering starting up a passive BWauth in order to understand how they work and fix bugs. What is the minimum and ideal hardware configuration for a BWauth? Have my eye on an OVH config with unmetered 1G, a fuzzy promise of 500mbps minimum bandwidth and a 3.4GHz 4core/8thread AES-NI CPU: http://ark.intel.com/products/80910 The server could be turned up in any of their facilities: https://www.ovh.com/us/about-us/datacenters.xml Or is 10G a requirement? (out of my league) Is all … [View More]

4 4