- tor-dev - lists.torproject.org

Reminder for weekly OONI dev meeting Monday 2015-01-05 18:00 UTC (19:00 CET)
by Arturo Filastò 05 Jan '15

05 Jan '15

Hello Oonitarians, This is a reminder that today there will be the weekly OONI meeting. It will happen as usual on the #ooni channel on irc.oftc.net at 18:00 UTC (19:00 CET, 13:00 EST, 10:00 PST). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Arturo

1 0

Re: [tor-dev] Fwd: gettimeofday() Syscall Issues
by grarpamp 03 Jan '15

03 Jan '15

On Fri, Jan 2, 2015 at 12:24 PM, Konstantin Belousov <kostikbel(a)gmail.com> wrote: > On Fri, Jan 02, 2015 at 09:09:34AM -0500, grarpamp wrote: >> Some recent FreeBSD related questions in this app area. >> > What is the question ? > > As a background, I can repeat that FreeBSD implements syscall-less > gettimeofday() and clock_gettime() for x86 machines which have > usable RDTSC. The selection of the timecounter can be verified > by sysctl kern.timecounter.hardware, and enabled by default fast > gettimeofday(2) can be checked by sysctl kern.timecounter.fast_gettime. > > On some Nehalem machine, I see it doing ~30M calls/sec with enabled > fast_gettime, and ~6.25M calls/sec with disabled fast_gettime. This is > measured on 2.8GHz Core i7 930 with src/tools/tools/syscall_timing. > > Check your timecounter hardware. Since it was noted that the tests > were done in VM, check the quality of RDTSC emulation in your hypervisor. https://lists.torproject.org/pipermail/tor-dev/2015-January/thread.html http://docs.freebsd.org/mail/current/freebsd-performance.html Maybe I can just refer non subscribers out to the two lists above that way in case anyone sees anything interesting they can join/comment as desired. Background might be that Tor operators have some large relays on *BSD and were looking to validate, and ways to improve, performance there. Cheers. https://lists.torproject.org/pipermail/tor-relays/ https://lists.torproject.org/pipermail/tor-talk/

3 2

Re: [tor-dev] Tor Cloud Maintenance, revisited
by Jeremy Olexa 03 Jan '15

03 Jan '15

Hi Vlad, I agree. AWS pricing is not friendly for bandwidth users. However, it is free for the first year of use and the general population is not taking advantage of this. Which, AFACT, is one of the points of the Tor Cloud project. Once you are not free-tier eligible, I highly recommend using a cheaper service like DO, Linode, Vultr, etc. At least DO and Linode would be trivial to setup. In summary, there is no need to replace AWS, but instead to supplement it with other providers and make the Tor Cloud project more robust. -Jeremy On Fri, Jan 2, 2015 at 6:10 PM, Vlad Tsyrklevich <vlad(a)tsyrklevich.net> wrote: > Hi Jeremy, I've working on something related so I figured I'd comment. I've > been working on a TorCloud replacement using DigitalOcean's API, this has > the benefit of a simplified set-up process (one-click set-up) and the > pricing on bandwidth is a major win over AWS ($5 for 1TB U/L instead of $20 > for 40GB, though your move to t2.micro would move it down to $10.) The > back-end is there and pretty much ready to go, I've primarily been waiting > for my co-author to come back from vacation to finish the front-end. > > I haven't actually discussed sunsetting/replacing the current TorCloud with > the Tor Project so this project might as well be vaporware; however, I think > continuing to use AWS doesn't make sense for pricing and ease-of-use. All > that said, if you're taking this on, it'd be great to try to also address > TorCloud's currently susceptibility to discovery by internet-wide port > scanning as I mentioned in > https://lists.torproject.org/pipermail/tor-dev/2014-December/007957.html > > On Thu Jan 01 2015 at 1:23:49 PM Jeremy Olexa <jolexa(a)jolexa.net> wrote: >> >> Hi Everyone, Happy New Year, first post to the -dev list but I've been >> running some relays for months[1]. Overall a new user to Tor so feel >> free to point me elsewhere if I'm asking poor questions. >> >> I've noticed that the Tor Cloud project is dead in the water right >> now. The last post on this list is in June 2014[2] and the bugs have >> been neglected, especially the one I opened[3] which states that tor >> does not even start! I've seen that there is a new maintainer, but >> still don't see any [public] activity. >> >> There are a couple of issues that have opportunity to be handled >> better. A large roadblock seems to be building/publishing a new AMI so >> I've addressed that in a keeping it simple theme: >> - Use Amazon AMI instead of Ubuntu. Justification: more aligned to AWS >> and is a first rate citizen in the ecosystem, yum repos, security >> updates, etc >> - Use t2.micro instances. Justification: t1.micro (currently in use) >> are more expensive and less performant than t2.micro >> - Use cloud-init to fetch ec2-prep.sh and run the script to configure >> the instance[4]. Justification: Less likely to roll a new AMI just for >> ec2-prep.sh updates therefore more of a rolling update infrastructure. >> One example is adding a new bridge protocol, all newly launched >> instances would get the ec2-prep updates without rolling a new AMI. >> Justification 2: Less Tor Cloud maintainer work. Downside: AMI has >> dependency on gitweb.torproject.org's uptime - we could use S3 or some >> other CDN but this is starting to go against the simple theme. >> - Publishing AMI to us-east (N Virginia), us-west2 (Oregon), us-europe >> (Ireland). Justification: These are the lowest cost regions. >> - Not publishing a separate AMI for private bridges. Justification: IF >> the Tor Cloud project wants to provide configuration for private >> bridges, reusing the shared AMI makes more sense. I have an idea of >> using cloud-init's user-data field but need to test it. Justification >> 2: Simpler, less Tor Cloud maintainer work. >> - Not addressed (yet): Ubuntu's unattended upgrades concept. I have >> one idea (yum security plugin) but I haven't thought about all the >> implications. >> >> My AMI testing has proved that the above concepts work and I have ec2 >> bridges running in east and west[5]. I propose that the TOR project >> uses the above model and I'm willing to help facilitate that. I am >> willing to share the AMI with some select beta testers but I'm not >> ready to make it public yet (some changes are expected). It is my idea >> that once the final AMI is produced, the Tor Cloud project will not >> have to publish another AMI unless wanting to change the instance >> type, or other "infrastructure" changes. >> >> I look forward to hearing some feedback, thanks, >> Jeremy >> >> [1]: https://atlas.thecthulhu.com/#search/jolexa >> [2]: https://lists.torproject.org/pipermail/tor-dev/2014-June/007001.html >> [3]: https://trac.torproject.org/projects/tor/ticket/13391 >> [4]: https://github.com/jolexa/tor-cloud/blob/master/run-once.sh >> [5]: https://globe.thecthulhu.com/#/search/query=ec2bridgei >> _______________________________________________ >> tor-dev mailing list >> tor-dev(a)lists.torproject.org >> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev > > > _______________________________________________ > tor-dev mailing list > tor-dev(a)lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev >

1 0

Tor Cloud Maintenance, revisited
by Jeremy Olexa 03 Jan '15

03 Jan '15

Hi Everyone, Happy New Year, first post to the -dev list but I've been running some relays for months[1]. Overall a new user to Tor so feel free to point me elsewhere if I'm asking poor questions. I've noticed that the Tor Cloud project is dead in the water right now. The last post on this list is in June 2014[2] and the bugs have been neglected, especially the one I opened[3] which states that tor does not even start! I've seen that there is a new maintainer, but still don't see any [public] activity. There are a couple of issues that have opportunity to be handled better. A large roadblock seems to be building/publishing a new AMI so I've addressed that in a keeping it simple theme: - Use Amazon AMI instead of Ubuntu. Justification: more aligned to AWS and is a first rate citizen in the ecosystem, yum repos, security updates, etc - Use t2.micro instances. Justification: t1.micro (currently in use) are more expensive and less performant than t2.micro - Use cloud-init to fetch ec2-prep.sh and run the script to configure the instance[4]. Justification: Less likely to roll a new AMI just for ec2-prep.sh updates therefore more of a rolling update infrastructure. One example is adding a new bridge protocol, all newly launched instances would get the ec2-prep updates without rolling a new AMI. Justification 2: Less Tor Cloud maintainer work. Downside: AMI has dependency on gitweb.torproject.org's uptime - we could use S3 or some other CDN but this is starting to go against the simple theme. - Publishing AMI to us-east (N Virginia), us-west2 (Oregon), us-europe (Ireland). Justification: These are the lowest cost regions. - Not publishing a separate AMI for private bridges. Justification: IF the Tor Cloud project wants to provide configuration for private bridges, reusing the shared AMI makes more sense. I have an idea of using cloud-init's user-data field but need to test it. Justification 2: Simpler, less Tor Cloud maintainer work. - Not addressed (yet): Ubuntu's unattended upgrades concept. I have one idea (yum security plugin) but I haven't thought about all the implications. My AMI testing has proved that the above concepts work and I have ec2 bridges running in east and west[5]. I propose that the TOR project uses the above model and I'm willing to help facilitate that. I am willing to share the AMI with some select beta testers but I'm not ready to make it public yet (some changes are expected). It is my idea that once the final AMI is produced, the Tor Cloud project will not have to publish another AMI unless wanting to change the instance type, or other "infrastructure" changes. I look forward to hearing some feedback, thanks, Jeremy [1]: https://atlas.thecthulhu.com/#search/jolexa [2]: https://lists.torproject.org/pipermail/tor-dev/2014-June/007001.html [3]: https://trac.torproject.org/projects/tor/ticket/13391 [4]: https://github.com/jolexa/tor-cloud/blob/master/run-once.sh [5]: https://globe.thecthulhu.com/#/search/query=ec2bridgei

2 1

Re: [tor-dev] gettimeofday() Syscall Issues
by teor 02 Jan '15

02 Jan '15

> From: Yawning Angel <yawning(a)schwanenlied.me> > Subject: Re: [tor-dev] gettimeofday() Syscall Issues > > On Thu, 01 Jan 2015 23:42:42 -0500 > Libertas <libertas(a)mykolab.com> wrote: > >> The first two account for the bulk of the calls, as they are in the >> core data relaying logic. >> >> Ultimately, the problem seems to be that the caching is very weak. At >> most, only half of the calls to tor_gettimeofday_cached_monotonic() >> use the cache. It appears in the vomiting print statements that >> loading a single simple HTML page >> (http://www.openbsd.org/faq/ports/guide.html to be exact) will cause >>> 30 gettimeofday() syscalls. You can imagine how that would >>> accumulate for an exit carrying 800 KB/s if the caching >> doesn't improve much with additional circuits. > > So while optimization is cool and all, I'm not seeing why this > specifically is the underlying issue. > > Each cell can contain 498 bytes of user payload. Looking at things > simplistically this is 800 KiB/s -> 1644 cells/sec, leaving you with > approximately 608 microseconds of processing time per cell. > > On my i5-4250U box, gettimeofday() takes 22 ns on Linux, and 2441 ns on > FreeBSD. I'm not sure how accurate the FreeBSD results are as it was > in a VirtualBox VM (getpid() on the same VM takes 124 ns). If someone > has a OpenBSD box they should benchmark gettimeofday() and see how long > the call takes. > > Taking the FreeBSD case (since we know that tor works fine on Linux), a IPredator has complained that tor on Linux spends too much time calling time() when pushing 500Mbit/s, which is an issue for them under 3.x series kernels, but not kernel 2.6. https://ipredator.se/guide/torserver#performance > single gettimeofday() call takes approximately, 0.39% of the per-cell > processing budget. > > For reference (assuming gettimeofday() in *BSD really is this shit > performance wise), 7000 calls to gettimeofday() is 17.09 ms worth of > calls. > > The clock code in tor does need love, so I wouldn't object to cleanup, > but I'm not sure it's in the state where it's causing the massive > performance degradation that you are seeing. > Yawning/Libertas, I just reviewed my profiling of an exit relay running chutney verify with 200MB of random data. This is on OS X 10.9.5 with tor 0.2.6.2-alpha-dev running the chutney basic-min network. The three leaf functions that take the most time in the call graph are: * channel_timestamp_recv * channel_timestamp_active * time Each of these functions takes around 16% of the execution time, the next nearest function is sha1_block_data_order_avx on 4%. While I understand that OS X, BSD, and Linux syscalls aren't necessarily identical, we now have results for the following platforms suggesting that calling time() too often has a performance impact: * Linux kernel 3.x * OpenBSD * OS X 10.9 My results suggest a maximum performance improvement of 15% on OS X if we reduced the calls to time() to a reasonable number per second. teor pgp 0xABFED1AC hkp://pgp.mit.edu/ https://gist.github.com/teor2345/d033b8ce0a99adbc89c5 http://0bin.net/paste/Mu92kPyphK0bqmbA#Zvt3gzMrSCAwDN6GKsUk7Q8G-eG+Y+BLpe7w…

2 2

Deadline Approaching: EECEA2015 (Electrical, Electronics, Computer Engineering and their Applications)
by Jackie Blanco 02 Jan '15

02 Jan '15

The Second International Conference on Electrical, Electronics, Computer Engineering and their Applications (EECEA2015) World Trade Center, Manila Philippines February 11, 2015 University of Perpetual Help System Dalta, Las Piñas - Manila, Philippines February 12-14, 2015 http://sdiwc.net/conferences/eecea2015/ All registered papers will be included in SDIWC Digital Library. =========================================================== The conference aims to enable researchers build connections between different digital applications. The event will be held over four days, with presentations delivered by researchers from the international community, including presentations from keynote speakers and state-of-the-art lectures. Trainings will be conducted by companies namely Microchip, Dassault Systèmes and TechSource in parallel with the paper presentations where delegates are encouraged to participate. RESEARCH TOPICS ARE NOT LIMITED TO: * Electronics Engineering * Electrical Engineering * Computer Engineering SUBMISSION GUIDELINES: - Researchers are encouraged to submit their work electronically. - Full paper must be submitted (Abstracts are not acceptable). - Submitted paper should not exceed 15 pages, including illustrations. All papers must be without page numbers. - Papers should be submitted electronically as pdf format without author(s) name. - Paper submission link: http://sdiwc.net/conferences/eecea2015/openconf/openconf.php IMPORTANT DATES: Submission Deadline: January 12, 2015 Notification of Acceptance: 2-3 weeks from the date of submission Camera Ready Submission: January 30, 2015 Registration Deadline: January 30, 2015 Seminar Date: February 11, 2015 Conference Dates: February 12-14, 2015

1 0

Re: [tor-dev] Tor BSD underperformance (was [Tor-BSD] Recognizing Randomness Exhaustion)
by teor 01 Jan '15

01 Jan '15

Libertas, As I wrote to you earlier today, IPredator on Linux complained of similar issues with excessive calls to time() under the Linux 3.x kernel series, but not 2.6. Let's track these issues in https://trac.torproject.org/projects/tor/ticket/14056 as they appear to be quite similar. I'd like to get a sense of how many calls per second this represents. (400,000 would seem to be 100 to 1000 per second, unless you were debugging for a long while.) I'd also like to know which function(s)/call stack(s) these calls are being made from. I wonder if TIME_IS_FAST is being defined as 1 in any of these Linux or BSD builds. This If TIME_IS_FAST is not defined (the default), we could change these calls to approx_time(), which is much faster. And in that case there should only be around 1 call to time() each second, which is clearly not the behaviour that either Libertas or IPredator are seeing. Libertas, can you search your tor binary or tor debug symbols for the function "approx_time"? grep will do this for you, or strings, or perhaps gdb if you feel like doing it the hard way. I'll add these questions to trac ticket #14056 as well. Libertas wrote: > I just ran ktrace/kdump (used for observing system calls) on the Tor > process of my exit node, which relays about 800 KB/s. It listed >400,000 > calls to gettimeofday(). The list was swamped with them. > > I think I remember reading somewhere that that sort of system call is > way slower in OpenBSD than Linux. Could this be related to the issue? > I've found a lot of similarly mysterious slowdowns related to *BSD > gettimeofday() on other projects' bug trackers, but nothing definitive. > > As you've likely noticed, although I started this discussion I'm very > new to system-level performance tuning. Let me know if I'm not making > sense, or if there's something else I should be focusing on. > >> <snip>

2 4

log files on windows
by Tony Willingham 01 Jan '15

01 Jan '15

Is there a trick to writing to log files in windows 7? The sample torrc file has this: Log debug file @LOCALSTATEDIR@/log/tor/debug.log I changed mine to this with no success: Log debug file C:\tor\tor_debug.log

1 0

Trac spam filter
by David Fifield 31 Dec '14

31 Dec '14

The Trac spam filter just caused me to lose a comment full of carefully researched external links. It's not the first time, and it's really frustrating. Can we whitelist usernames or something? Submission rejected as potential spam Maximum number of external links per post exceeded I tried to solve the recaptcha five times (they were really hard) and when I finally solved it, the comment wasn't posted anyway. David Fifield

1 0

Internet-wide scanning for bridges
by Vlad Tsyrklevich 31 Dec '14

31 Dec '14

Hello, while taking a looking at TorCloud I it used static OR and obfs2+obfs3 ports making them trivial to discover by scanning EC2's IP address space. This led me to consider the more general attack of internet-wide scanning to discover Tor bridges. Most Tor relays run their ORPorts on port 443 or 9001 so I began investigating there. I began by looking at Project Sonar's port 443 SSL certificate database looking for certificates matching the tor certificate pattern. In talking with Eric Wustrow about ZMap, I discovered that he and the other ZMap authors had already published about this attack in their original paper. Eric mentioned that in his discussion with Roger that the response to the attack was going to be the soon-to-be-deployed obfuscated transports which were intended to be run on random high ports; however, the eventual roll out of obfsproxy did not change the fact that bridge's ORPorts continue to be publicly accessible. While I was analyzing Internet-wide survey-data for port 443 and 9001 (helpfully provided by Project Sonar/H D Moore) I found the Onionoo data set. I was able to quantify the impact of the attack using the Onionoo data and verify it using the ZMap scans. There are 4267 bridges, of them 1819 serve their ORPort on port 443 and 383 serve on port 9001. That's 52% of tor bridges. There are 1926 pluggable-transports enabled bridges, 316 with ORPort 443 and 33 with ORPort 9001. That's 18% of Tor bridges. 203 (64%) of PT-enabled bridges with ORPort 443 were TorCloud. I was able to approximately verify these figures using the Internet-wide scans. By grabbing bridge's certificates and calculating their hashed fingerprint I realized I was also discovering a fair amount of private bridges not included in the Onionoo data set. The results aren't too awful, just under 20% of PT-enabled bridges are affected. It's unclear to me if the low number of PT-enabled bridges (and corresponding high number of vulnerable non-PT-enabled bridges) is a historical artifact or a consequence of out-of-date documentation like https://www.torproject.org/docs/bridges.html.en not setting up pluggable transports. Eliminating the ORPort entirely for PT-enabled bridges seems like the ultimate solution to this attack, but the implications of such a change are unclear to me. Another change to mitigate this issue is changing the behavior of ports specified as 'auto' and then changing defaults and documentation to use auto by default. Currently specifying 'auto' in your torrc lets the OS decide the port; however, if tor were to generate a random port on first use that it continued to use across restarts it would it possible to use this for ORPorts and pluggable transports by default (as opposed to the current situation where for bridges you want ORPort auto but constant PT ports and a constant ORPort for relays which is confusing.) I've attached a patch to warn bridge operators running with ORPort set to 443 or 9001 as a stop-gap measure. - Vladislav

8 12