tor-dev

tor-dev@lists.torproject.org

3581 discussions

Brainstorming ideas for controller features for improved testing; want feedback
by Nick Mathewson 09 Apr '15

09 Apr '15

Hi! I've got an end-of-month deliverable to flesh out as many good ideas here as I can, and I'd appreciate feedback on what kind of features it would be good to add to the controller protocol in order to better support testing. More ideas would be most welcome. Yes, some of these ideas are probably foolish or pointless or half-baked or useless or even dangerous; this is a brainstorming exercise, not a declaration of intent. The goal right now is to generate a lot of ideas and thoughts now, and to make decisions about what to build later. IDEAS ===== 1. Step-by-step hidden service connections Add the ability to create connections to hidden services step by step, to best What's necessary here is commands to: * Establish a rendezvous point on a given circuit. * Construct and send an introduce2 cell on a given circuit. * Realize that a rendezvous circuit has been constructed. 2. Send a single cell on a circuit (TESTING ONLY) For fuzzing and low-level testing purposes, it would be handy to be able to send a single cell on a tor circuit. This might be better to expose via a low-level modular API than via the control port. 3. Intercept cell by cell on a circuit (TESTING ONLY) For fuzzing, testing, and debugging purposes, it might be handy for a controller to be able to observe data cell by cell on a circuit of interest. This might be better to expose via a low-level modular API than via the control port. 4. Send a single cell on a connection. (TESTING ONLY) As 2, but for connections. Note that we might even, for testing, expose this at a sub-cell level. 5. Intercept all cells on a connection (TESTING ONLY) As 2, but for connections. 6. Plug-in to handle a relay or other command. Right now, all Tor's features need to be baked into Tor; it's not easy to write extensions. We could change that by having the controller able to intersect particular relay or extension commands and act accordingly. This could be used for prototyping new features, etc. 7. Force a given protocol on a given connection We could add a feature to restrict what protocols can be negotiated on a given connection we create. This could help us better test our protocols for interoperatbility. 8. Examine fine-grained connection detail. There are many data available for a given connection (such as fine-grained TLS information) that are not currently exposed on the GETINFO interface. We could make most of this available for testing, pending security analysis. 9. Examine cache in detail In the past we've seen crazy issues with our descriptor caching code. It might be good to expose for testing information about where exactly descriptors are stored, what attributes are set on them, and so on. We could also expose events for cache compaction and discarded expired descriptors. 10. Fetch literal documents Currently there's no way for a controller to ask Tor to download a given descriptor or microdescriptor or networkstatus. That could change. 11. OOM stats To resist out-of-memory attacks, we track our memory usage and kill off circuits as needed when memory gets low. We could expose the memory thresholds and current sizes via one or more controller commands. 12. Timeout values Tor has a truly huge variety of internal timers to ensure that given periodic events happen enough; we could expose those, and (TESTING ONLY) allow controllers to adjust them or trigger their corresponding events. 13. Detailed connection debugging info Current connection events expose only large-scale state changes in connections; we could instead expose every state transition at the cell-by-cell handshake level. 14. Detailed circuit debugging info As 13 but for circuits. 15. Halt main loop except for control layer. (TESTING ONLY) For inspection/debugging purposes, it might be clever to have Tor be able to freeze itself, except for the control layer, and let the controller inquire about information. This presents implementation challenges, and is probably not a great idea to do before a _big_ refactoring. 16. Service a single connection (TESTING ONLY) Currently controllers can disable circuit construction or stream attachment, and do them manually. We might also do this for connections, allowing a testing controller to trace what Tor does cell by cell on a single connection. 17. All rephist data There are many data about history and usage in rephist.c (which stands for 'reputation and history!'). We could expose them, to let us better test them. Some of this might be useful for Seth (previously arm) users. Spec: This would use GETINFO extensions, and probably some new events. 18. All ratelim data Sometimes our rate-limiting code can get wonky. It would be great to expose it to Tor controllers in order to help ensure it's behaving correctly. This would include send/receive windows and bw stuff. Spec: This would use GETINFO extensions, and probably some new events. 19. All accounting data As 17, but for hibernate.c, which performs bandwidth accounting. 20. All guard transitions Our guard node state logic is very complicated, and much in need of testing and refactoring. Exposing more state transitions and guard selection transitions to the controller might help. (We have a "GUARD" event now, but it is a bit out of sync with the main implementation) 21. All key transitions We could generate events every time we change keys, and (TESTING ONLY) allow a controller to time-out a key early. 22. Examine mux settings (TESTING ONLY) The circuitmux code that we use to decide which cell to send next is very complex; it would be good to expose its thinking and decisionmaking to a testing controller for better observation. 23. Examine pathbias settings Our pathbias code is also complex, and a bit flakier than the circuitmux code. We could do for it as with 22 above. 24. Examine cpuworker queues As of 0.2.6, we have a new cpuworker infrastructure that better sends data to worker threads, but not much visibility into how well it's working. Exposing some information about this to the controller could help us tune better. 25. All geoip data As 17, but for any geoip information we aren't currently exporting. 26. Replay detection For hidden service security (and maybe eventually for secure ntor Y reuse) we have to keep replay caches to prevent us from being tricked into handling the same value twice. We don't expose the load on these caches to the controller, however. And we could, to help us better tune them into using a good memory/error-rate trade-off. 27. Hidden service intropoint changes, desc changes, uploads Many hidden service transitions currently generate no events. We could at minimum generate events for changed inroduction points, changed hidden service descriptors, uploading our own HS descriptor. 28. Descriptor uploads. We have an event for when our descriptor has changed, but not for a successful upload for it. We could fix that. 29. Path generation logic -- expose, allow. Currently a controller's only visibility into path selection logic is in its outputs, and in the opportunity to replace path selection logic entirely. We could expose more details about the algorithm's operation in order to help better test our path selection. 30. All PT status information. Pluggable transport feedback is, at present, very coarse-grained. For testing we might expose more. 31. Crypto operation counts. We ought to keep count of our various cryptographic operations, and expose them to the controller. This would help us know where to spend our optimization efforts. 32. Forget cached information To better test our download logic, it would be helpful to have a way to drop items from our caches. IMPLEMENTATION NOTES: ===================== Not lightly does one list 31 controller improvement areas. If we're hoping to do these without too much programmer time and , we need to take a much more principled approach to implementing controller commands. For #8351 I worked on a branch called 'ticket8351' that has some code we could use here. COMPATIBILITY NOTES: ==================== Many of the features here are ones that we might not want to promise to support indefinitely; we should gate them behind a USEFEATURE command, and maybe place them in an annex of the control spec.

7 10

Tagged and moved tickets based on triages spreadsheet
by Nick Mathewson 09 Apr '15

09 Apr '15

Hi, all! Since Isabela is ill (get well soon, Isabela!), I've tried to translate the results of 0.2.7 triage onto the bugtracker. To ensure that everything is reversible, I've tagged items with: 027-triaged-1-in 027-triaged-1-out 027-triaged-1-deferrable. I've then moved all of the 027-triaged-1-out items that didn't already have code into 0.2.???. Perhaps the rest should follow them soon. I've likely messed something up at some point; not too much, I hope. Here are the items that have found their way into the 0.2.7 milestone that I did not see on the triage spreadsheet: https://trac.torproject.org/projects/tor/query?status=accepted&status=assig… I suspect this isn't our last triage round, and that once Isa is back in action (get well soon!) she'll have improvements to make. cheers, -- Nick

1 0

Tor Summer of Privacy [ Incorporating ruleset testing into https-everywhere release process ]
by karan grover 09 Apr '15

09 Apr '15

Hello there! I am Karan Grover. I am a 2nd year computer science student at IIIT Delhi. I am a part of my college's Cybersecurity Education and Research Center, which is the first of its kind in the country. I have worked primarily in the field of Privacy and Security of Online Social Media (PSOSM). I am interested in contributing to Tor as a part of Tor Summer of Privacy. The project I am interested in working on is *Incorporating ruleset testing into https-everywhere release process*. I have been understanding the codebase of *https-everywhere-checker, *and am now quite familiar with it. I would like to discuss further about what the project entails. I contribute to open source every now and then, but am now looking to be an active open source contributor and Tor seems to align with my interests. Looking forward to your reply. Thank you.

2 1

Orfox (Firefox for Android) - Recent Updates
by Amogh Pradeep 09 Apr '15

09 Apr '15

Hey guys, At the Valencia meet I got Orfox building again with the latest mozilla code and since then, I've been trying to get the project back on track. The current repo can be found at[0] After going through the source code for firefox for andoroid, these were the network related findings that I've made. The application uses maily 3 layers to work. 1) The core layer, written in C++. 2) A JS layer with some more functional code. 3) A Java layer that makes uses the Android API's and other code used for android. Out of the 3 layers, 2 of them make network calls. 1) The C++ layer. There is an advantage of this layer making the network calls as all the calls made by this layer are proxied. They obey the fennec settings for proxy that has currently been set at mozilla-central/mobile/android/app/mobile.js . Hence, we do not need to bother about setting proxy to these connections and calls. 2) The Java layer. The Java code, the code run by android also has many network calls but the problem is that these calls are not done obeying the tor proxy and hence they are harmful. List of places where these calls are made are: a) any class which imports org.apache.* (commons.net.ftp.*, http.*): build/mobile/sutagent/android/DoCommand.java build/mobile/sutagent/android/SUTAgentAndroid.java mobile/android/base/distribution/Distribution.java mobile/android/base/favicons/LoadFaviconTask.java mobile/android/base/tests/BaseRobocopTest.java b) The browser also seems to be using another library for network communications and that lies at mobile/android/thirdparty/ch/boye/httpclientandroidlib This library seems to have proxy support but the calls made to this library have not been proxied, a list of those are [1] NOTE: Not all these classes listed there make network calls, I will make a shorter list of this in the next few days. [0] https://github.com/amoghbl1/gecko-dev/tree/dev_orfox [1] https://dxr.mozilla.org/mozilla-central/search?q=ch.boye.httpclientandroidl… Feedback on what you think would be very useful and much appreciated. Best, amoghbl1

1 0

Aspirant for Tor Summer of Privacy ( for Improving Tails)
by Saket Sinha 08 Apr '15

08 Apr '15

Hi, I would like to participate to Tor Summer of Privacy by contributing to "Improve Tails by working on Debian". I have worked on Debian packaging and build systems and find this project a match for my technical skills. Kindly let me know how to go ahead with this. Regards, Saket Sinha

2 3

Please vote on times for the Pluggable Transports, Bridges, and BridgeDB Meeting!
by isis 07 Apr '15

07 Apr '15

Hello! Did you have an interest in attending the Pluggable Transports Meeting v1.0? Well then, you will certainly be excited to here about the *BRAND NEW* Pluggable Transports Meeting v2.0!!!! *NOW INCLUDING DEVELOPMENT DISCUSSION OF BRIDGEDB AND BRIDGES IN GENERAL* That's 3 meetings for the price of 1!! Wow!!! Please take a moment away from your undeniable shock and ecstatic joy at this great news to vote for a new time for this combined meeting: http://doodle.com/tn28wgzw8iydpznp Thanks! -- ♥Ⓐ isis agora lovecruft _________________________________________________________ OpenPGP: 4096R/0A6A58A14B5946ABDE18E207A3ADB67A2CDB8B35 Current Keys: https://blog.patternsinthevoid.net/isis.txt

3 4

Contact Stem example authors.
by Lluís 06 Apr '15

06 Apr '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hello everybody, After completing the "Stem" tutorial I tried to run one of the examples it points to, the "RTT Prober". I succeeded. However, after patching, recompiling and running the tor version they recommend, I had to change the python sources. I would share my results, but I really don't know how to contact the authors. May someone put me in touch with them ? Thanks, Lluís -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVImFvAAoJEJOS0D3m/rNlr3gP/1OEQI2NBAvEEAdTVgRPpuoc VMDnKt1PdjWDy+HTPyXAqki5eVzh8dyOYAla8yg38bxr5wCL1relUhkQzFyQb61h wXBTqvPzgJQOfcVj/V7h3CBPzkbXgfyaJIuOD0+NvSBLVf9W4keigA7cZ7uUE7d8 UQKnj30pFjC98CZt9Uq5ziyQqxq3xzzNHQfu5YE9YpQdYy6fm5t37I4Qm7ZXxin5 JI3nxvL34CKKXgBkxsqf+zY8f+Yoo7uFl1n+Fm+wyXc0aqi4MTzjHY7tm0cvgg/i qtxcBxJlLQu73srrbMXW9T4+o4dgRtegN8/hvRPnSOw7c520G8/2HODJ+EtEHXGm eFnNnZDHtOCjzCq9R8QcHXlEtSdXRdfxh3uI5AhPzEIM+Mx3Vtl8Y8lSMgsR0anD 0B0mGd25zgS7aLk6tuPzTKJ4B7ae1k2jsWYY3z4mziriGr+s61SiS0uWDdH3cUJb EnKVii5N5w0Tn2pJwStJrlecUkgHe6gTIJ1U5YYY2Raa/+sKKRDABAOUH+j2qV3n TL5utyQ+koszVwEreRf6q6IadHH1PxmqNd6MG6WLtejYWEqeWJy8BhK90HeOShTG ckR9OxeO5ZIZBOO8+XmaPdF8mam+hQxDU079rmoH3X+dvXVt+jPpcW+vEBzzwiG1 yx21KPu21u/TiAR6KbDd =pYKC -----END PGP SIGNATURE-----

2 2

Tor Summer of Privacy
by Damian Johnson 06 Apr '15

06 Apr '15

Do you want to make Tor even better, and get paid in the process? If you are a student, we have good news for you: the Tor Project is pleased to announce our first Summer of Privacy! https://trac.torproject.org/projects/tor/wiki/org/TorSoP Thanks to Google, over the last eight years Tor has mentored 53 students over summers. This year the program was trimmed back to make room for new organizations. So, rather than wait another year we're starting a program of our own! The Tor Summer of Privacy is a pilot program for students that want to collaborate and develop privacy tools. So why apply? * You will work with a world-class team of anonymity experts and developers on an anonymity network that is already protecting millions of people daily. * We only write free (open source) software. The tools you make won't be locked down or rot on a shelf. * Your work could contribute to academic publications. Tor development raises many interesting research questions. * You can work your own hours in your own locations. As long as you get the job done, we don't care about the process. * ... oh, and of course there's also the $5,500 stipend for the summer. :P We are looking for great coders who are self-motivated and able to work independently. The deadline for applications is April 17th, 2015 which is right around the corner, so don't be shy - apply!

1 0

multi-instance tor with a single torrc
by Nusenu 03 Apr '15

03 Apr '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Pascal, (OpenBSD tor package maintainer) (splitting this part off to tor-dev, since it is not suited well for tor-relays I guess) For the interested reader this discussion started here [1]. Pascal Stumpf: >> Multi-instance support isn't something that's provided by >> OpenBSD's rc.d(8) subsystem, and I doubt it ever will be. Too >> much complexity for something that's more of a crutch to work >> around few applications' limitations than something universally >> needed and useful. > >> I think the real solution would be to talk to upstream about >> improving MP scalability. Even if it's just something as >> primitive as forking a child for every ORPort directive in >> /etc/tor/torrc, it would be much easier than having to keep track >> of a bazillion individually started tor instances, all with their >> own configuration files. I guess this (forking one tor instance per ORPort line) sounds a lot simpler than it actually would be. How do you handle ControlPort, HS, DataDir, ExitPolicy and basically any other config option, ... but since I need a solution in the near future I'll probably just go with multiple services on OpenBSD. thanks, Nusenu [1] https://lists.torproject.org/pipermail/tor-relays/2015-April/006741.html -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVHuIaAAoJEFv7XvVCELh03H0QAKpg05vfOS/DTJ6Qhl7TfQV/ pKiAHjVfbWOfBD4vfqMHX0H054l7WfRUUiSz6gZ9DRGwXj7uDXh59kjsNa2ipMvK 4YYvDXjUIh3UBZdDnjLrkRPI5Lws1PuCi6wvnFX+e/weWiWQ18N/2HALaluhyldI 3aiDWA3k0ZnZVkuLLpbkxvbol8HS0upMTkwWlba/QXdWflK3it6sydkIi8EMhMx4 cEfTRCvWjxxU5YguIQ7/NpNFpKvRCtUEoCKBLjXuUGKTvawQ+Xu+XJQeSqE/yOqO Z1lTSQqH4i7JjBGEMxXQP/U7B3gELxv2tRcpfImGQZArLim41VPTVXse2+dPhb4d 0Knl19yYxtd4gjtoMbnv3GEPNgj2xnBr8e2MEYDsRY+rGYFnHEtnPvmJmC5p/vtX NAcuReFLd+y3j2PuB0XJHlGYOLtzvtaXJdiqV6oq42Mu0VHpiGfbTfKi2k8ZMUTw PfLQjx9z3dQ6GleC1mGHdMeJ7bJ7Wk2dJ7dicLF0veY5/Gc7gkYtWibsLXggTwrQ K0IqXnGJ8fWM41ZO4hhpmkkywhXodLzlWlXCxgxDUBsuFci/mVvtfu6ylxh+LCqG y5TxIk1T4/i+atx7jvuMLgrZGf02YgFyCf15Swj+VoDxx8GAxJl9nEQx0jD5d+Zl 206/+0QAI5hksaedorvj =qUCf -----END PGP SIGNATURE-----

1 0

Summary of meek's costs, March 2015
by David Fifield 03 Apr '15

03 Apr '15

Here's the summary of meek's CDN fees for March 2015. Earlier reports: https://lists.torproject.org/pipermail/tor-dev/2014-August/007429.html https://lists.torproject.org/pipermail/tor-dev/2014-October/007576.html https://lists.torproject.org/pipermail/tor-dev/2014-November/007716.html https://lists.torproject.org/pipermail/tor-dev/2014-December/007916.html https://lists.torproject.org/pipermail/tor-dev/2015-January/008082.html https://lists.torproject.org/pipermail/tor-dev/2015-February/008235.html https://lists.torproject.org/pipermail/tor-dev/2015-March/008427.html App Engine + Amazon + Azure = total by month February 2014 $0.09 + -- + -- = $0.09 March 2014 $0.00 + -- + -- = $0.00 April 2014 $0.73 + -- + -- = $0.73 May 2014 $0.69 + -- + -- = $0.69 June 2014 $0.65 + -- + -- = $0.65 July 2014 $0.56 + $0.00 + -- = $0.56 August 2014 $1.56 + $3.10 + -- = $4.66 September 2014 $4.02 + $4.59 + $0.00 = $8.61 October 2014 $40.85 + $130.29 + $0.00 = $171.14 November 2014 $224.67 + $362.60 + $0.00 = $587.27 December 2014 $326.81 + $417.31 + $0.00 = $744.12 January 2015 $464.37 + $669.02 + $0.00 = $1133.39 February 2015 $650.53 + $604.83 + $0.00 = $1255.36 March 2015 $690.29 + $815.68 + $0.00 = $1505.97 -- total by CDN $2405.82 + $3007.42 + $0.00 = $5413.24 grand total The number of simultaneous users was up again in March, reaching 2000 simultaneous users some days. The fluctuation between weekdays and weekends was greater than we've seen before. https://metrics.torproject.org/userstats-bridge-transport.html?graph=userst… We moved something over 9 TB in March. We're getting close to where CDNs start offering price breaks :/ == App Engine a.k.a. meek-google == At the end of February, I tried tweaking some performance parameters to try and reduce instance hours. It seems like it worked, as instance hours were down 31% as bandwidth was up 29%. Here is how the Google costs broke down: 5316 GB $637.93 2173 instance hours $108.64 Compared to the previous month: 4114 GB $492.71 3226 instance hours $157.82 https://globe.torproject.org/#/bridge/88F745840F47CE0C6A4FE61D827950B06F9E4… == Amazon a.k.a. meek-amazon == meek-amazon was up quite a bit compared to the previous month. Requests were up 6% and bandwidth was up a whopping 42%. The big increases were in the Asia Pacific (Singapore) region and EU (Ireland) region. I think we're going to go ahead with removing meek-amazon from Tor Browser. This will take a little bit of work--I'd like there to be a blog post that explains how to set up your own Amazon instance if meek-amazon is currently working for you. I also would like to try and work out any performance issues with meek-azure in order to use it as a replacement. Weirdly, there were over 24,000 plain HTTP requests from the US East region. That is strange because meek is HTTPS-only. The number of HTTP requests is still small compared to the 44,000,000 HTTPS requests received from the same region, but it's far above other regions, which have at most a few dozen HTTP requests. Asia Pacific (Singapore) 171M requests $205.57 1433 GB $195.60 Asia Pacific (Sydney) 156K requests $0.20 1 GB $0.17 Asia Pacific (Tokyo) 10M requests $12.05 70 GB $9.07 EU (Ireland) 158M requests $190.03 1486 GB $118.34 South America (Sao Paulo) 3M requests $6.70 17 GB $4.03 US East (Northern Virginia) 44M requests $44.00 378 GB $29.92 -- total 386M requests $458.55 3385 GB $357.13 https://globe.torproject.org/#/bridge/3FD131B74D9A96190B1EE5D31E91757FADA1A… == Azure a.k.a. meek-azure == https://onionoo.torproject.org/bandwidth?fingerprint=AA033EEB61601B2B7312D8… Estimated bandwidth use this month: 2015-03 737 GB Compared to last month: 2015-02 614 GB https://globe.torproject.org/#/bridge/AA033EEB61601B2B7312D89B62AAA23DC3ED8… David Fifield

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-dev