tor-dev

tor-dev@lists.torproject.org

3581 discussions

Proposal 213: Remove stream-level sendmes from the design
by Roger Dingledine 06 Nov '12

06 Nov '12

Filename: 213-remove-stream-sendmes.txt Title: Remove stream-level sendmes from the design Author: Roger Dingledine Created: 4-Nov-2012 Status: Open 1. Motivation Tor uses circuit-level sendme cells to handle congestion / flow fairness at the circuit level, but it has a second stream-level flow/congestion/fairness layer under that to share a given circuit between multiple streams. The circuit-level flow control, or something like it, is needed because different users are competing for the same resources. But the stream-level flow control has a different threat model, since all the streams belong to the same user. When the circuit has only one active stream, the downsides are a) that we waste 2% of our bandwidth sending stream-level sendmes, and b) because of the circuit-level and stream-level window parameters we picked, we end up sending only half the cells we might otherwise send. When the circuit has two active streams, they each get to send 500 cells for their window, because the circuit window is 1000. We still spend the 2% overhead. When the circuit has three or more active streams, they're all typically limited by the circuit window, since the stream-level window won't kick in. We still spend the 2% overhead though. And depending on their sending pattern, we could experience cases where a given stream might be able to send more data on the circuit, but it chooses not to because its stream-level window is empty. More generally, we don't have a good handle on the interactions between all the layers of congestion control in Tor. It would behoove us to simplify in the case where we're not clear on what it buys us. 2. Design We should strip all aspects of this stream-level flow control from the Tor design and code. 2.1. But doesn't having a lower stream window than circuit window save room for new streams? It could be that a feature of the stream window is that there's always space in the circuit window for another begin cell, so new streams will open faster than otherwise. But first, if there are two or more active streams going, there won't be any extra space. Second, since begin cells are client-to-exit, and typical circuits don't fill their outbound circuit windows very often anyway, and also since we're hoping to move to a world where we isolate more activities between circuits, I'm not inclined to worry much about losing this maybe-feature. See also proposal 168, "reduce default circuit window" -- it's interesting to note that proposal 168 was unknowingly dabbling in exactly this question, since reducing the default circuit window to 500 or less made stream windows moot. It might be worth resurrecting the proposal 168 experiments once this proposal is implemented. 2.2. If we dump stream windows, we're effectively doubling them. Right now the circuit window starts at 1000, and the stream window starts at 500. So if we just rip out stream windows, we'll effectively change the stream window default to 1000, doubling the amount of data in flight and potentially clogging up the network more. We could either live with that, or we could change the default circuit window to 500 (which is easy to do even in a backward compatible way, since the edge connection can simply choose to not send as many cells). 3. Evaluation It would be wise to have some plan for making sure we didn't screw up the network too much with this change. The main trouble there is that torperf et al only do one stream at a time, so we really have no good baseline, or measurement tools, to capture network performance for multiple parallel streams. Maybe we should resolve task 7168 before the transition, so we're more prepared. 4. Transition Option one is to do a two-phase transition. In the first phase, edges stop enforcing the deliver window (i.e. stop closing circuits when the stream deliver goes negative, but otherwise they send and receive stream-level sendmes as now). In the second phase (once all old versions are gone), we can start disobeying the deliver window, and also stop sending stream-level sendmes back. That approach takes a while before it will matter. As an optimization, since clients can know which relay versions support the new behavior, we could have relays interpret violating the deliver window as signaling support for removed stream-level sendmes: the relay would then stop sending or expecting sendmes. That optimization is somewhat klunky though, first because web-browsing clients don't generally finish out a stream window in the upstream direction (so the klunky trick will probably never happen by accident), and second because if we lower the circuit window to 500 (see Sec 2.2), there's now no way to violate stream deliver windows. Option two is to introduce another relay cell type, which the client sends before opening any streams to let the other side know that it shouldn't use or expect stream-level sendmes. A variation here is to extend either the create cell or the begin cell (ha -- and they thought I was crazy when I included the explicit \0 at the end of the current begin cell payload), so we can specify our circuit preferences without any extra overhead. Option three is to wait until we switch to a new circuit protocol (e.g. when we move to ntor or ace), and use that as the signal to drop stream-level sendmes from the design. And hey, if we're lucky, by then we'll have sorted out the n23 questions (see ticket 4506) and we might be dumping circuit-level sendmes at that point too. Options two or three seem way better than option one. And since it's not super-urgent, I suggest we hold off on option two to see if option three makes sense.

3 7

Damian's Status Report - October 2012
by Damian Johnson 02 Nov '12

02 Nov '12

Ahhh, the beginning of winter. That very special time of the year when Seattle skies are perpetual overcast and gloomy. When crowds flock to the coffee shops and the best thing to do with your time is hook up to a caffeine IV and code. ... on a side note it has been a really productive month. ======================================== * Stem Website Stem now has a far more developer friendly website! https://stem.readthedocs.org/en/latest/index.html Barring the obvious new sections I revised vast swaths of stem's API documentation. Feedback welcome! The tutorial section is just starting, but I have some ideas for expanding it. ======================================== * Network Status Document Parsing After a couple months of work by Ravi and me the network status document handling is *finally* done. It turned out to be a... very big feature branch. Announcement: https://lists.torproject.org/pipermail/tor-dev/2012-October/004092.html Docs: https://stem.readthedocs.org/en/latest/api/descriptor/networkstatus.html Code: https://gitweb.torproject.org/stem.git/commitdiff/ea356b7?hp=5cd457b With this done we're very nearly at feature parity with metrics-lib (and I suspect a bit past it in terms of testing). In finishing this up I also spotted an undocumented oddity with microdescriptor 'directory-signature' lines. [1] ======================================== * GSoC Mentor Summit Met with the developers of numerous other open source projects. I was really impressed at how many worthwhile conversations were crammed into such a short conference. Highlights include... * Arc from Python had distutils suggestions, thanks to which stem and arm will soon have both Python 2.x and 3.x support. * OSU's Open Source Labs are entertaining the thought of running Tor relays. * Discussed TorBirdy and other Tor projects with Sukhbir. He mentioned an issue with trac permissions that is now fixed. [2] * Talked with Terri from Python about Mailman 3, which would be a great answer for the requests we get to have a Tor forum. [3] * Attended a talk led by Marina from Gnome about their outreach program for women. We'll likely be taking part in it this year. For the full trip report see... https://lists.torproject.org/pipermail/tor-dev/2012-October/004115.html ======================================== Other changes include... * Cleaning up orphaned pyc files as a part of running our tests (feature suggested by Ravi) [4] * Ported arm's str_tools module to stem and added unit tests [5][6] * Code reviewed Ravi's attach_stream addition, waiting on the revisions [7] * Arm troubleshooting on irc with zack and ultramage. Also fixed a broken link on arm's site spotted by wh6iQ. * A new contributor, Eoin, spotted some great issues and contributed patches. Tickets include... * python 2.5 compatibility bugs [8] * count of the number of skipped tests [9] * error with the whitespace checker [10] * spelling corrections [11] * Sathyanarayanan spotted some nice bugs including... * on python 2.5-2.6 a missing microdescriptor consensus wasn't causing the related test to be skipped [12] * integ tests for the process module would fail if tor was already running [13] ======================================== Next up: Ravi and I are working on tor event handling, the last major feature stem is missing as a controller library. After that we plan to port arm over, then tidy up loose ends in preparation for stem's initial release. Cheers! -Damian [1] https://trac.torproject.org/7072 [2] https://trac.torproject.org/7186 [3] https://trac.torproject.org/3592 [4] https://trac.torproject.org/7103 [5] https://trac.torproject.org/7158 [6] https://stem.readthedocs.org/en/latest/api/util/str_tools.html [7] https://trac.torproject.org/7152 [8] https://trac.torproject.org/7244 [9] https://trac.torproject.org/7246 [10] https://trac.torproject.org/7263 [11] https://trac.torproject.org/7245 [12] https://trac.torproject.org/7250 [13] https://trac.torproject.org/7251

1 0

circuitmux_ewma.c
by Gisle Vanem 01 Nov '12

01 Nov '12

These gcc extensions: /*** EWMA circuitmux_policy_t method table ***/ circuitmux_policy_t ewma_policy = { .alloc_cmux_data = ewma_alloc_cmux_data, .free_cmux_data = ewma_free_cmux_data, Doesn't work well with MSVC. Here is a patch: --- ..\..\Git-latest\src\or\circuitmux_ewma.c 2012-11-01 15:53:23.125000000 +0100 +++ circuitmux_ewma.c 2012-11-01 17:52:55.371093500 +0100 @@ -200,15 +200,22 @@ /*** EWMA circuitmux_policy_t method table ***/ -circuitmux_policy_t ewma_policy = { .alloc_cmux_data = ewma_alloc_cmux_data, - .free_cmux_data = ewma_free_cmux_data, - .alloc_circ_data = ewma_alloc_circ_data, - .free_circ_data = ewma_free_circ_data, - .notify_circ_active = ewma_notify_circ_active, - .notify_circ_inactive = ewma_notify_circ_inactive, - .notify_set_n_cells = NULL, /* EWMA doesn't need this */ - .notify_xmit_cells = ewma_notify_xmit_cells, - .pick_active_circuit = ewma_pick_active_circuit +#ifdef __GNUC__ + #define STRUCT_INIT(member,val) member = val +#else + #define STRUCT_INIT(member,val) val +#endif + +circuitmux_policy_t ewma_policy = { + STRUCT_INIT (.alloc_cmux_data, ewma_alloc_cmux_data), + STRUCT_INIT (.free_cmux_data, ewma_free_cmux_data), + STRUCT_INIT (.alloc_circ_data, ewma_alloc_circ_data), + STRUCT_INIT (.free_circ_data, ewma_free_circ_data), + STRUCT_INIT (.notify_circ_active, ewma_notify_circ_active), + STRUCT_INIT (.notify_circ_inactive, ewma_notify_circ_inactive), + STRUCT_INIT (.notify_set_n_cells, NULL), /* EWMA doesn't need this */ + STRUCT_INIT (.notify_xmit_cells, ewma_notify_xmit_cells), + STRUCT_INIT (.pick_active_circuit, ewma_pick_active_circuit) }; /*** EWMA method implementations using the below EWMA helper functions ***/ --gv

2 2

Reminder of November, December deadlines for Tor 0.2.4
by Nick Mathewson 01 Nov '12

01 Nov '12

Hi, all! >From the Tor 0.2.4 schedule at https://trac.torproject.org/projects/tor/wiki/org/roadmaps/Tor/024 : """November 20, 2012: Big feature proposal freeze. Any big feature which requires a design proposal must have its design proposal finished by this date. If it needs changes after this date, then maybe we didn't design it well enough, and we probably shouldn't merge it. December 10, 2012: Big feature merge freeze. No big features will be merged after this date. December 15, 2012: Small feature proposal freeze. Any small feature which requires a design proposal must have its tiny little design proposal finished by this date. January 10, 2013: Feature merge freeze. No features after this date. I mean it. ... The definition of "big patch" is deliberately vague. Note that the cutoffs above are merge freezes, not submission freezes.""" (I've just added 10 days to that first deadline, and 5 days to the third.) Also, I'm currently thinking that, to give these deadlines teeth, I shouldn't self-enforce, since I've veered between "pushover" and "insane zealot" in the past in ways that didn't make a lot of sense. Rather, I'm going to say that in cases of doubt ("Is this a little feature") and cases where I'd like to violate freezes ("surely this can go in on December 12"), I'm not going to merge patches on my sole discretion any more. Rather, I'm going to say that at least two of me, Roger, and Andrea ought to agree to violate a freeze before we do so. (I will try not to forget and mess this up.) With any luck, this will let us get closer to a reasonable 6 month release cycle. yrs, -- Nick

1 0

[Patch] tools\tor-fw-helper\tor-fw-helper-natpmp.c
by Gisle Vanem 01 Nov '12

01 Nov '12

The code in tools\tor-fw-helper\tor-fw-helper-natpmp.c does things wrong on Winsock where e.g. errno isn't set on socket-errors and 'fd' is (always?) >= FD_SETSIZE. Patch attached. --gv

2 1

gsathya's October
by Sathyanarayanan Gunasekaran 01 Nov '12

01 Nov '12

I was mostly busy with giving the required exams for grad school and generally worrying about. I did manage to do a bunch of Tor stuff - - Implemented DisableNetwork in Orbot. Orbot stops making new connections and drops existing ones once you lose network connectivity, saving battery life. Also, one of the features I implemented during GSoC11 is now finally merged into master -- Orbot now shows usage statistics(bandwidth). - Wrote a whole bunch of tests for Pyonionoo -- which in hindsight weren't really good. I will mostly need to refactor them and make them more modular. - Added more functionality to Pyonionoo -- /bandwidths and /details work now - Hacked on #1854[0] - Set up a pyobfsproxy bridge to help test. - Found some bugs in stem - #7133, #7250, #7251 - Generally hung around #tor providing support. Hopefully I can do this a lot more. [0] - https://trac.torproject.org/projects/tor/ticket/1854

1 0

use tor controller to detect circuit/path change
by esolve esolve 31 Oct '12

31 Oct '12

Hi, I'm new to Tor controller, Tor client usually open 3 circuits, one for current use and the other 2 for standby. Tor will change the current circuit/path in use from time to time, depending on the network load dynamics I'm wondering whether it is possible to use Tor controller to detect when the circuit/path in use is changed and obtain the list of IP addresses of those relay nodes. thanks!

4 6

[Stegotorus] Re: Do you have time to poke at ST
by vmonmoonshine＠gmail.com 25 Oct '12

25 Oct '12

Hello Zack, Sorry for being bit out of touch as I'm traveling and my access to Internet is sporadic. I'm still working on Stegonotorus. I developed the webpage_tester (now on f--webpage-tester branch on github), that is curl using Stegotorus to download anything on the web and compare it with the original. It is based on assumption that is my best approximation for a browser. I did that in a hope to catch the bug, but I haven't. Basically I failed to browse anything unreliably. I can download 20 megabytes or so with no error. It takes a long time, but It doesn't fail. My payload server is the apache payload server but that shouldn't make a huge difference. I think my payloads even have smaller capacity and less efficient than your fake ones. Anyway, I'm going to tweak the tester proxy to drops packets at random, to force the ack into play, I'll get back to you with more update. Cheers, vmon

1 0

New tech report: Counting daily bridge users
by Karsten Loesing 24 Oct '12

24 Oct '12

Hi everyone, I just finished a new tech report titled "Counting daily bridge users": https://research.torproject.org/techreports/counting-daily-bridge-users-201… >From the abstract: "As part of the Tor Metrics Project, we want to learn how many people use the Tor network on a daily basis. Counting users in an anonymity network is, obviously, a difficult task for which we cannot collect too sensitive usage data. We came up with a privacy-preserving approach for estimating directly connecting user numbers by counting requests to the directory mirrors and deriving approximate user numbers from there. In this report we describe a modified approach for estimating the number of users connecting via bridges by evaluating directory requests made to bridges. We compare this new approach to our current approach that estimates bridge user numbers from total unique IP addresses seen at bridges. We think that results from the new approach are closer to reality, even though that means there are significantly fewer daily bridge users than originally expected." Feedback is much appreciated and could influence our implementation of user number estimates in the future (#7154). Thanks, Karsten

1 0

Re: [tor-dev] Automating Bridge Reachability Testing (#6414)
by Isis 23 Oct '12

23 Oct '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Karsten! Oh sheesh. I did not see it...I will have to figure out why. That is slightly worrying. So, I am rushing to meet the final deadline, but I still think it is doable. I have mostly finished up my OONI work for the month, and I planned to spend the remainder of this month working on the bridge test. I have finished most of the actual Tor connection code, as well as one of the four basic packet level scans (the icmp8 one). Two of the other packet level scans, the TCP SYN and ACK ones, are pretty much copies of the icmp8 one with a couple lines changed, and they shouldn't be a big deal. There is still the vanilla TLS handshake test/scan/thing, which has not been started yet, and will take a bit more time than the others because Python notoriously has problems with SSL bindings and libraries, so I'll need to do a bit of research on newer ones and updates and see which is the best to use now. I hear that tlslite[1] is the current best choice; if anyone else has any input on this it would be very helpful. :) There were a couple minor hups: 1) When George asked me to test pluggable transports, this required significantly more refactoring than I previously thought was necessary. 2) Arturo redesigned the OONI testing framework API again to use a completely different structure, which was supposed to be backwards compatible and turned out not to be (though I believe that my recent OONI commits fixed that). However, I have been fighting the framework already, because the main scripts in OONI (/ooni/oonicli.py and /ooni/ooniprobe.py) control the reactor, and also expect static iterations through single test and single control functions for each asset (an asset in this case would be one bridge address). The bridge testing is rather dynamic (I would like it to be able to evaluate an approximate danger level to running the next test) and so the framework is kind of troublesome. Also, because the framework handles calling the reactor (in Twisted, the reactor is a sort of event scheduler), and it also expects a rather linear progression of defer.Deferreds (in Twisted, those are standin objects which execute callbacks when they get results from some previous deferred/callback), it would be nicer if I had full control of these myself without needing to hack around the parent scripts. I think it's wise that OONI deals with these things for the testwriter in most cases, because the testwriter shouldn't be expected to be an expert in using Twisted. However, I also think that, in the long term, OONI shouldn't prohibit people who know what they are doing or are doing odd things from being able to do so. As a result, I've decided (for now), to use bits are parts of the OONI code before the recent refactoring, and later (after the deliverable) I will work on adding flags to OONI to give the test script full control of the reactor and deferreds, as well as evaluating whether or not the bridge test is even compatible with the new API. I do not want to get caught up in dealing with this right now, I just want to have it all working and deployable in a way that I know will work. 3) The indirect scans are becoming quite complicated to automate in any sane fashion. I still would like to continue working on this, as I'm quite enjoying the difficulty, but due to their temporary and volatile nature (they will change frequently depending on the blocking methods of a particular country and the currently available in-country bounces/proxies/whatever-thing-the-indirect-scan-uses), as well as the fact that many of these methods are still undiscovered, I think it is safe to add them as specialty cases after the fact without impacting overall general testing. There is one in particular that I would like to finish before the deadline because I am quite proud of it and am having a lot of fun working on it, but I'm first going to concentrate on wrapping up the active scans. There are other things which I've marked as helpful things to do, but which are not necessarily part of this deliverable: 1) Having a parser for bridge descriptors to turn them into test inputs, and vice versa. 2) Having some undiscoverable method for setting up lots of IPv6 bridges on one OR (Tor currently only allows up to eight, I believe) and having these be discoverable by bridgedb and no one else. I was thinking of this while talking with Aaron, because he reminded me that people on IPv6 have tons of IPs available, and I was thinking that if we configured some type of one-way hash function, we could say that a bridge descriptor for 2001:db8::1:1 should actually mean multiple descriptors for 2001:db8::fa98:38d2 2001:db8::e099:2188 2001:db8::88aa:3b7 or something, derived from the output of hashing the original descriptor with the OR's key or something else. This would help distribute bridges in the future quite a bit, though it doesn't do much for the current bridge situation. Anyone wanting to help with the above two things, or with an idea for another indirect scan, or with feedback on anything I'm working on, should feel free to contact me and it will be greatly appreciated. :D .-- .. ... .... - -- . .-.. ..- -.-. -.- <(A)3 isis agora lovecruft .-.-. [1] https://github.com/trevp/tlslite On Fri 12 Oct 2012 at 14:56, thus spake Karsten Loesing: > Hi Isis, > > did you see this email? > > Sorry for being pushy, but we're less than three weeks away from the > deadline, which makes me kinda nervous. > > Thanks, > Karsten > > > On 10/8/12 9:14 AM, Karsten Loesing wrote: > > Hi Isis, > > > > can you give me a brief status update on the sponsor F year 2 > > deliverable "Automating Bridge Reachability Testing", ticket #6414? > > > > In particular, how did the two project-internal milestones on September > > 15 and October 7 go? > > > > Do you mind updating #6414 with the status? > > > > The overall deadline is on November 1, which is in 3.5 weeks. How > > realistic do you think it is that we'll have automated bridge > > reachability tests by then? > > > > Thanks, > > Karsten > > > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCgAGBQJQeJ4pAAoJEKOttnos24s1GlkP/1ODucFe9q9a7KaKisgVqBcx Cg1X3yGI+dsWvKtMwzgNqT/hcS+95GAYQiaa8fDlbiLY3nCWEsGYSOTjsbAmp04O qPCdBiSFE4VqiZy4PYUHUkf8qZTYkXqAiMnyuuLVJINJ+BHLAdwEmEWB77ncQUxV PvG6sxQ3GY64ZqRaSHwudf3H8mEPN/tNWRIXubnvAGM16EV0hWb51uORwo6CG86m lKwpBqbSX5uDgG+B6HCeZps0Zv7cDypI3dtdd8hJHncUpFmfuj2y+Oad1M5DVbMC 4UMvCv/4tMPoPyjTYjWgLQRHEFiMdMwQcdPntnF6XqUBMLvd6v1fk1IxaaqNCDah JWxK8ymUu90FLmmiyaSHQE/tfkFoEDCXz3XzR8xQA1zI2aq0hHdxA+G2JmW8yadQ wt0N4EeCzK154r4+YRC1eJLrsSOhmA7SaAE6A+/iEOgasdMY0YWchTDnlUbUB5iv acbMVITSGqmcRMos5xi450V98Qw1nW3UoBj5wi8uh0BWqnhqfomqL6oloQEdgAze MqtCZ91adKKXntIsEZW0C/klBYg5mZIxdtmAW5eO/j+ogLB/L0/hX4/TZ6OH5i/N 7jJWAF0MbSWI2Q1Gk+oFFulib5m3XWzqg5Yq/+8VfE5n1R5xM1FgBwC932SWLOFZ BV5c9ntZ+iJlbhpdL5yJ =jk13 -----END PGP SIGNATURE-----

4 5

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-dev