- tor-dev - lists.torproject.org

Re: [tor-dev] Proposal: Controller events to better understand connection/circuit usage
by Karsten Loesing 28 Feb '13

28 Feb '13

[Nick, can you merge my torspec proposal218 branch, please? Thanks!] On 2/28/13 2:08 AM, Rob Jansen wrote: > On Mon, Feb 25, 2013 at 10:28 AM, Karsten Loesing <karsten(a)torproject.org> > wrote: > >> On 2/23/13 11:20 PM, Rob Jansen wrote: >>> On Feb 23, 2013 4:22 PM, "Karsten Loesing" <karsten(a)torproject.org> >> wrote: >>>> Your understanding of n_circ_id and p_circ_id matches mine, but are you >>>> sure there's a UID for circuits other than origin circuits? I think you >>>> mean origin_circuit_t->global_identifier. But there's no such field for >>>> or_circuit_t or circuit_t. Or do you mean something else? >>>> >>> Ok. Though I thought my original patch moved the global Id to the base >>> circuit struct. Perhaps I didn't. Anyway, I'm not sure it matters... >> >> Your original patch did move the ID to circuit_t, but I thought we >> wanted to avoid numbering non-origin circuits (mostly because that >> affects relays in non-TestingTorNetwork mode and could lead to busy >> relays running out of IDs at some point), which is why I took this >> change out. >> > > Right, I remember this now. I could imagine ways to get around the 'running > out of ids' problem, like resetting the id counter after a given timeframe. > In fact, we could just let the counter overflow and loop back to 0 on its > own (assuming its a unsigned int type) since we really only need them to be > unique approximately for the life of a circuit. If we see the ID come up > again after an hour, we can assume its a new circuit. I'd rather not want to change behavior in non-simulation mode. If we really need a UID for non-origin circuits, we could add a separate uint64_t global_identifier to circuit_t and only use that in simulating mode. I'm not sure that we need it though. >> Also, even if we moved the field to circuit_t, the new CELL_STATS events >> would be the only ones using these UIDs, because all other events are >> for origin circuits only. I don't see how these IDs would help us. We >> never learn what UID the next or previous node in a circuit picks for a >> given circuit. > > Not currently, but couldn't we implement the functionality where relays log > or export their circuit UIDs and next/prev IDs over the control port? > Though I'm not sure if you'd want this in mainline Tor if its only useful > in simulation mode... We could have relays log local circuit UIDs together with p_circ_id, p_conn_id, n_circ_id, and n_conn_id in simulation mode. But I don't see how that would facilitate circuit tracking compared to the current approach using ORCONN and CELL_STATS events. Here's an example: - fileclient creates a circuit with - UID 14, - n_conn_id 15, and - n_circ_id 19403. - tokenglobal is the first hop in this circuit and reports - (new) UID 12345, - p_conn_id 32, - p_circ_id 19403, - n_conn_id 18, and - n_circ_id 6710. - tokenrelay is the second hop and reports - (new) UID 23456, - p_conn_id 17, - p_circ_id 6710, - n_conn_id 16, - n_circ_id 34402. - exit2 is the third and final hop and reports - (new) UID 34567, - p_conn_id 15, and - p_circ_id 34402. We need ORCONN events to know that fileclient's n_conn_id 15 is the same connection as tokenglobal's p_conn_id 32. How would the new UIDs make this easier? >>>> tl;dr: I _think_ it's possible to reconstruct circuits from ORCONN and >>>> CELL_STATS events as they are currently specified in proposal 218. >>>> >>> Great, but do we really expect every Tor controller parser to get this >>> right? It seems complicated enough that there should be an easier way. >>> Maybe it's just wishful thinking on my part. >> >> I agree that reconstructing circuits from ORCONN and CELL_STATS events >> is far from trivial. I don't really see how to make it simpler though. > > What about linking all of the IDs and UIDs as described above? (See above.) >> From an earlier mail in this thread: >>>> Finally, Rob, should I look into CIRC_BW events you suggested a while >>>> ago? If yes, what did you have in mind how that event would look like, >>>> and when/by whom would it be emitted? >>> >>> If we want to do this, it would likely be an aggregation of all STREAM_BW >>> events for a circuit, but only during the time when those streams >> belonged >>> to that circuit. I don't think it makes sense to emit it for every >>> STREAM_BW event though, so what if we aggregate and emit it once per >>> second? A format similar to the STREAM_BW format should work fine. >> >> Done. I specified and implemented such a CIRC_BW event. >> >> > Great! > > >> >> Here's the updated proposal 218 (Nick, please don't merge this yet): >> >> >> https://gitweb.torproject.org/user/karsten/torspec.git/blob/refs/heads/prop… > > > In section 5.3/5.4/5.5, these events are emitted in the > second_elapsed_callback(), right? I wanted to verify that a relay who > hasn't sent anything in a few seconds and then starts sending again will > emit the event at the end of the second after which it resumed sending, > rather than the first bytes after it resumed sending. Yes, all these events are emitted in second_elapsed_callback(). > The last word in 5.4 should be 'reading' instead of 'read'. Fixed. > Is the specification of 5.5 and 5.6 complex enough to warrant including > example outputs? Sure, can't hurt. Added a few examples. > In 5.6, were we planning on explaining how buckets can go negative and how > that affects the reporting of the TB_EMPTY events? I tried a better explanation: """ ReadBucketEmpty (WriteBucketEmpty) is the time in millis that the read (write) bucket was empty since the last refill. LastRefill is the time in millis since the last refill. If a bucket went negative and if refilling tokens didn't make it go positive again, there will be multiple consecutive TB_EMPTY events for each refill interval during which the bucket contained zero tokens or less. In such a case, ReadBucketEmpty or WriteBucketEmpty are capped at LastRefill in order not to report empty times more than once. """ > Here's the tor branch: >> >> >> https://gitweb.torproject.org/karsten/tor.git/shortlog/refs/heads/morestats3 >> >> Here's a Shadow log file containing all new events: >> >> https://people.torproject.org/~karsten/volatile/scallion.log.gz >> >> Here's the Java program that I used to parse the Shadow log file: >> >> https://people.torproject.org/~karsten/volatile/ParseProposal218Events.java >> >> > Ugh. I really don't look forward to writing parsers for this. I guess there > may be few projects that actually require this information, and those that > do can use your code:) I hope that some of the complexity goes away once we use a parsing library like Stem. And I think we should provide parsing code to other people looking into this. >> And finally, here's the output, which should be easier to understand now: >> >> https://people.torproject.org/~karsten/volatile/scallion.txt >> >> Search for "Circuit [fileclient-60.1.0.0]:14" to find the circuit I >> mentioned earlier in this thread. >> >> >> Can you review the proposal changes and tell me if they make sense to you? >> >> Best, >> Karsten >> >> > See comments above. Overall, it looks good. I'm still wondering about the > UIDs / IDs issue. It may be that we don't want to include that for other > reasons, in which case the current implementation is fine. Okay, in that case let's consider proposal 218 done for the moment, unless we come up with a better idea to solve circuit-tracking thing. I asked Nick above to merge my changes, so that I can put proposal 218 on the sponsor F year 3 wiki page as one result of the February 28 milestone. (That doesn't mean we cannot make it even better after February.) Thanks! Karsten

2 1

Re: [tor-dev] Proposal: Controller events to better understand connection/circuit usage
by Karsten Loesing 25 Feb '13

25 Feb '13

On 2/23/13 11:20 PM, Rob Jansen wrote: > On Feb 23, 2013 4:22 PM, "Karsten Loesing" <karsten(a)torproject.org> wrote: >> Your understanding of n_circ_id and p_circ_id matches mine, but are you >> sure there's a UID for circuits other than origin circuits? I think you >> mean origin_circuit_t->global_identifier. But there's no such field for >> or_circuit_t or circuit_t. Or do you mean something else? >> > Ok. Though I thought my original patch moved the global Id to the base > circuit struct. Perhaps I didn't. Anyway, I'm not sure it matters... Your original patch did move the ID to circuit_t, but I thought we wanted to avoid numbering non-origin circuits (mostly because that affects relays in non-TestingTorNetwork mode and could lead to busy relays running out of IDs at some point), which is why I took this change out. Also, even if we moved the field to circuit_t, the new CELL_STATS events would be the only ones using these UIDs, because all other events are for origin circuits only. I don't see how these IDs would help us. We never learn what UID the next or previous node in a circuit picks for a given circuit. >> tl;dr: I _think_ it's possible to reconstruct circuits from ORCONN and >> CELL_STATS events as they are currently specified in proposal 218. >> > Great, but do we really expect every Tor controller parser to get this > right? It seems complicated enough that there should be an easier way. > Maybe it's just wishful thinking on my part. I agree that reconstructing circuits from ORCONN and CELL_STATS events is far from trivial. I don't really see how to make it simpler though. >From an earlier mail in this thread: >> Finally, Rob, should I look into CIRC_BW events you suggested a while >> ago? If yes, what did you have in mind how that event would look like, >> and when/by whom would it be emitted? > > If we want to do this, it would likely be an aggregation of all STREAM_BW > events for a circuit, but only during the time when those streams belonged > to that circuit. I don't think it makes sense to emit it for every > STREAM_BW event though, so what if we aggregate and emit it once per > second? A format similar to the STREAM_BW format should work fine. Done. I specified and implemented such a CIRC_BW event. Here's the updated proposal 218 (Nick, please don't merge this yet): https://gitweb.torproject.org/user/karsten/torspec.git/blob/refs/heads/prop… Here's the tor branch: https://gitweb.torproject.org/karsten/tor.git/shortlog/refs/heads/morestats3 Here's a Shadow log file containing all new events: https://people.torproject.org/~karsten/volatile/scallion.log.gz Here's the Java program that I used to parse the Shadow log file: https://people.torproject.org/~karsten/volatile/ParseProposal218Events.java And finally, here's the output, which should be easier to understand now: https://people.torproject.org/~karsten/volatile/scallion.txt Search for "Circuit [fileclient-60.1.0.0]:14" to find the circuit I mentioned earlier in this thread. Can you review the proposal changes and tell me if they make sense to you? Best, Karsten

1 0

Re: [tor-dev] Proposal: Controller events to better understand connection/circuit usage
by Karsten Loesing 23 Feb '13

23 Feb '13

On 2/22/13 8:50 PM, Rob Jansen wrote: > On Fri, Feb 22, 2013 at 12:59 PM, Karsten Loesing <karsten(a)torproject.org>wrote: > [...] > >> If anything here sounds strange to you, please let me know. I'm not >> 100% certain that this is the best approach to track circuits from >> client to exit, or if it's even correct. >> >> For example, I assume here that circuit IDs are unique between two >> nodes, which I think is correct. But before working on this I also >> assumed that a circuit uses a single connection for both inbound and >> outbound directions (which is apparently not the case). >> >> > Whether or not your assumption about circuit ids is correct depends on > which circuit id you are referring to - the source relay circuit id, the > destination relay circuit id, or the id(s) written in the cells. Here is > how I understand the ids work: > > Suppose relayA and relayB are part of the same circuit, and relayA is > closer to the client than relayB. The id by which relayA refers to its > circuit_t state is only unique to relayA and is chosen when the circuit_t > struct is created. Similarly, the id by which relayB refers to its > circuit_t state is only unique to relayB and also chosen when the circuit_t > struct is created. Check this in the *circuit_new() functions in > circuitlist.c. Lets refer to these as circuit UIDs, as they are unique and > only known to individual relays. > > Now, relayA writes a circuit id into cells it sends to relayB, so that > relayB knows which circuit the cells belong to, but it does not necessarily > use its circuit UID. The id for this is computed in > get_unique_circ_id_by_conn() in circuitbuild.c and is stored in n_circ_id > in the circuit_t struct at relayA. Similarly, relayB uses p_circ_id from > the or_circuit_t struct when sending cells back to relayA. > > Upon receiving cells from relayA, relayB immediately uses the id written in > the cell (relayA's circ->n_circ_id) to look up relayB's UID for its > circuit_t state. There is a circuit id map that is kept for this purpose. > This works similarly from relayB to relayA. > > Now, the problem that prevents us from linking these is that all of the > controller events print out the UIDs, but not the n_circ_id or p_circ_id. I > tried printing out all of these various IDs but never verified that it > actually worked. > > Please let me know if my understanding is flawed in some way. Your understanding of n_circ_id and p_circ_id matches mine, but are you sure there's a UID for circuits other than origin circuits? I think you mean origin_circuit_t->global_identifier. But there's no such field for or_circuit_t or circuit_t. Or do you mean something else? Anyway, your description made me realize that my previous attempt to link CELL_STATS events was flawed for (at least) two reasons: 1. I assumed that a circuit ID (n_circ_id or p_circ_id) is unique for a given pair of nodes, but it's really only unique for a given connection between two nodes. 2. I also assumed that a circuit uses two different connections for inbound and outbound queues, but it's actually the same connection that is known under different connection IDs by the involved nodes. Maybe this gets clearer when looking again at the example: [fileclient-60.1.0.0]:[tokenconn-59.1.0.0]:34760 00:23:16 [fileclient-60.1.0.0] >>> circ=34760 conn=12 create_fast=1;relay_early=2 create_fast=1;relay_early=2 00:23:16 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=1;created_fast=1 relay=1;created_fast=1 Circuit ID 34760 is what fileclient picked as n_circ_id for this circuit and what tokenconn stored as p_circ_id. There's a single OR connection carrying this circuit which fileclient identifies as 12 and tokenconn as 31. The part where this gets tricky is when we need to find out (reliably) that connection IDs 12 and 31 refer to the same connection. fileclient never tells tokenconn that it locally refers to this connection as ID 12, and vice versa. However, there can be more than one connection between the two nodes, and it's perfectly valid for those connections to each have a circuit with ID 34760 on them. I modified my Java program to parse ORCONN events to match corresponding connection IDs based on state transitions from NEW or LAUNCHED to CONNECTED. This works in 99% of cases and only fails for OR connections that were launched before the controller registered for ORCONN events. But we probably don't care about that early bootstrapping phase anyway. Here's the same circuit from my original example, this time with explanations (IDs might differ, because this example comes from another Shadow run): [fileclient-60.1.0.0]:12:34761 This queue is for circuit 34761 which runs over connection 12 at fileclient. It's an outbound queue, though that isn't explicitly stated here. The ID 34761 is what fileclient picked as n_circ_id. 00:23:16 [fileclient-60.1.0.0] >>> circ=34761 conn=12 create_fast=1;relay_early=2 create_fast=1;relay_early=2 There were three cells in outbound direction reported in this CELL_STATS event. Leaving out other CELL_STATS events here. [tokenconn-59.1.0.0]:31:34761 This queue is also for circuit 34761, but running over connection 31 at tokenconn. It's an inbound queue, so 34761 is a p_circ_id, chosen by fileclient. 00:23:16 [tokenconn-59.1.0.0] <<< circ=34761 conn=31 relay=1;created_fast=1 relay=1;created_fast=1 Two cells in inbound direction. [tokenconn-59.1.0.0]:16:26405 Queue for circuit 26405 on connection 16 at relay tokenconn. [tokenglobal-57.1.0.0]:15:26405 See above. [tokenglobal-57.1.0.0]:16:53557 See above. [exit2-62.1.0.0]:15:53557 See above. tl;dr: I _think_ it's possible to reconstruct circuits from ORCONN and CELL_STATS events as they are currently specified in proposal 218. Best, Karsten

1 0

Re: [tor-dev] Proposal: Controller events to better understand connection/circuit usage
by Karsten Loesing 22 Feb '13

22 Feb '13

On 2/22/13 5:36 PM, Rob Jansen wrote: > On Tue, Feb 19, 2013 at 10:20 AM, Karsten Loesing <karsten(a)torproject.org>wrote: > >> On 2/12/13 4:54 PM, Damian Johnson wrote: >>>> AppWard and ExitWard indicate the direction of cell flow through the >>>> circuit. Previous cell events aggregated statistics for both directions >> and >>>> therefore did not capture information which I believe may be important. >>>> These are much more informative than N and P. Do you have suggestions >> for >>>> improving this further? >>> >>> Ahhh, gotcha. Personally I'd go for 'inbound' and 'outbound' since >>> they better connotate directionality. AppWard and ExitWard sound >>> weird, but they make sense now too and would be fine if CELL_STATS got >>> a quick description saying what they mean. >> >> Sure, I'm fine with calling it 'inbound' and 'outbound'. Will change >> that the next time I touch the proposal. Thanks! >> >> >> I also thought more about Rob's idea to link circuits from client to >> exit in a testing network. I think we should add two more fields >> InboundName and OutboundName to CELL_STATS events. These fields would >> contain LongName ($CC88FB3B78599580F1EE4F6F73E26A7EC3DF2CA1~tokenconn) >> or Target (like 60.1.0.0:10002) of the previous or next node in a >> circuit. While it may be possible to extract this information from >> ORCONN events, that seems somewhat fragile to me. (When testing this, I >> had a case when we were missing ORCONN events, because the controller >> was not yet connected, so it was unclear where the CELL_STATS events >> belonged.) >> >> Here's an example of the extended CELL_STATS event (still using >> positional arguments, because I didn't change the code there yet): >> >> CELL_STATS 48957 15 $9441269F5989487F07AE824063345A0A6BCAB279 >> created=1;relay=1 created=1;relay=1 created=0;relay=0 29201 16 >> $DC284D0012ECB812C0CA2DEB080CB2F839A77E1A create=1 create=1 create=0 >> >> And here's how we'd put together relevant CELL_STATS events to learn >> what happened in a given circuit: >> >> [fileclient-60.1.0.0]:[tokenconn-59.1.0.0]:34760 >> 00:23:16 [fileclient-60.1.0.0] >>> circ=34760 conn=12 >> create_fast=1;relay_early=2 create_fast=1;relay_early=2 >> 00:23:16 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 >> relay=1;created_fast=1 relay=1;created_fast=1 >> 00:23:17 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=1 relay=1 >> 00:25:01 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=1 relay=1 >> 00:25:01 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay_early=1 >> relay_early=1 >> 00:25:02 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay_early=1 >> relay_early=1 >> 00:25:02 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=6 relay=6 >> 00:25:03 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=40 relay=40 >> 00:25:04 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay_early=1 >> relay_early=1 >> 00:25:04 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=61 relay=61 >> 00:25:05 [fileclient-60.1.0.0] >>> circ=34760 conn=12 >> relay=1;relay_early=3 relay=1;relay_early=3 >> 00:25:06 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=1 relay=1 >> 00:25:06 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay=1 relay=1 >> 00:25:07 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=44 relay=44 >> 00:25:08 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=63 relay=63 >> 00:25:08 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay=3 relay=3 >> 00:25:09 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay=2 relay=2 >> 00:25:10 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=1 relay=1 >> 00:25:10 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay=1 relay=1 >> 00:25:11 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=94 relay=94 >> 00:25:12 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay=4 relay=4 >> 00:25:12 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=12 relay=12 >> [tokenconn-59.1.0.0]:[tokenglobal-57.1.0.0]:9272 >> 00:23:16 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 >> create=1;relay_early=1 create=1;relay_early=1 >> 00:23:16 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 >> created=1;relay=1 created=1;relay=1 >> 00:25:01 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay_early=1 >> relay_early=1 >> 00:25:01 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=1 relay=1 >> 00:25:02 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=6 relay=6 >> 00:25:02 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay_early=1 >> relay_early=1 >> 00:25:03 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=41 relay=41 >> 00:25:04 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=60 relay=60 >> 00:25:04 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay_early=1 >> relay_early=1 >> 00:25:05 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 >> relay=1;relay_early=3 relay=1;relay_early=3 >> 00:25:06 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay=1 relay=1 >> 00:25:06 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=1 relay=1 >> 00:25:07 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=52 relay=52 >> 00:25:08 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay=1 relay=1 >> 00:25:08 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=55 relay=55 >> 00:25:09 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay=4 relay=4 >> 00:25:10 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay=1 relay=1 >> 00:25:10 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=1 relay=1 >> 00:25:11 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=98 relay=98 >> 00:25:12 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=8 relay=8 >> 00:25:12 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay=4 relay=4 >> [exit1-61.1.0.0]:[tokenglobal-57.1.0.0]:38910 >> 00:23:16 [exit1-61.1.0.0] <<< circ=38910 conn=24 created=1 created=1 >> 00:23:16 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 create=1 create=1 >> 00:25:01 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=1 relay=1 >> 00:25:01 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay_early=1 >> relay_early=1 >> 00:25:02 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=30 relay=30 >> 00:25:02 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay_early=1 >> relay_early=1 >> 00:25:03 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=77 relay=77 >> 00:25:04 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay_early=1 >> relay_early=1 >> 00:25:05 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 >> relay=1;relay_early=3 relay=1;relay_early=3 >> 00:25:06 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=1 relay=1 >> 00:25:06 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay=1 relay=1 >> 00:25:07 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=80 relay=80 >> 00:25:08 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=27 relay=27 >> 00:25:08 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay=1 relay=1 >> 00:25:09 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=1 relay=1 >> 00:25:09 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay=4 relay=4 >> 00:25:10 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=27 relay=27 >> 00:25:10 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay=1 relay=1 >> 00:25:11 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=79 relay=79 >> 00:25:12 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay=4 relay=4 >> >> I think this works for all circuits. Rob, what do you think about >> extending the CELL_STATS events as described above? > > > I don't think I understand the format. Is this what would appear in the log > file? Including the ">>>" and "<<<"? Those seem a little strange. Also, I'm > not seeing where e.g. the ID of the circuit as the entry relay knows it is > mapped to the ID of the circuit as the middle relay knows it. Circuit IDs > are relay-specific, right? Am I missing something here? Ah, sorry, it's the output of my little Java tool that puts together CELL_STATS events from a scallion.log file. (I can share that code if you want, but it's mostly a hack.) Here's how you'd read that format: > [fileclient-60.1.0.0]:[tokenconn-59.1.0.0]:34760 Client fileclient and relay tokenconn have one or more connections that carry circuit 34760. IIUC, fileclient picked that circuit ID. The names [fileclient-60.1.0.0] and [tokenconn-59.1.0.0] are ordered alphabetically, not in the order of whoever opened the circuit. > 00:23:16 [fileclient-60.1.0.0] >>> circ=34760 conn=12 > create_fast=1;relay_early=2 create_fast=1;relay_early=2 In that second, fileclient sent three cells over circuit 34760 using connection 12, one CREATE_FAST cell and two RELAY_EARLY cells. It's unclear who opened that connection and picked number 12 for it, but it doesn't really matter. All cells were added to and removed from the queue in the same second. The ">>>" stands for outbound. > 00:23:16 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 > relay=1;created_fast=1 relay=1;created_fast=1 tokenconn sent two cells in inbound direction belonging to the same circuit 34760 but using connection 31, not 12 (which I didn't know was possible, but hey). > [...] > [tokenconn-59.1.0.0]:[tokenglobal-57.1.0.0]:9272 Relay tokenconn and relay tokenglobal use one or more connections for circuit 9772, which is the extended circuit from above. The format doesn't show this, but the program generating this format makes sure that these two circuit parts belong together. > 00:23:16 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 > create=1;relay_early=1 create=1;relay_early=1 Two cells from tokenconn in outbound direction. Similar to above. > [...] > [exit1-61.1.0.0]:[tokenglobal-57.1.0.0]:38910 Relay exit1 and tokenglobal use ID 38910 for the last part of this circuit. If anything here sounds strange to you, please let me know. I'm not 100% certain that this is the best approach to track circuits from client to exit, or if it's even correct. For example, I assume here that circuit IDs are unique between two nodes, which I think is correct. But before working on this I also assumed that a circuit uses a single connection for both inbound and outbound directions (which is apparently not the case). >> Also, do we still >> need the ID= in ORCONN events if CELL_STATS events are sufficient to >> reconstruct circuits? > > Would the ORCONN IDs still be useful to track state on a per-connection > basis, like bandwidth used per connection (from the CONN_BW event)? Maybe, yes. Okay, leaving it in. >> Finally, Rob, should I look into CIRC_BW events you suggested a while >> ago? If yes, what did you have in mind how that event would look like, >> and when/by whom would it be emitted? > > If we want to do this, it would likely be an aggregation of all STREAM_BW > events for a circuit, but only during the time when those streams belonged > to that circuit. I don't think it makes sense to emit it for every > STREAM_BW event though, so what if we aggregate and emit it once per > second? A format similar to the STREAM_BW format should work fine. Sounds doable. I'll see if I can build this over the weekend. Best, Karsten

1 0

Fwd: python topics
by Ingy dot Net 22 Feb '13

22 Feb '13

Greetings, This is Ingy döt Net, a friend of ioerror's and arma's. I've been following Tor since 2009, and was asked to help out with ooni-probe a bit. I've been researching the state of the ooni, and was on a call with the primaries yesterday. I hack from a coworking space in Seattle that has a lot of various talent. During the call I learned that we need a light-weight, cross-platform gui button for using ooni-probe as simply as possible, and it would be nice to have it on the usual common platforms. I asked a co-coworker who runs the seattle python interest group (SeaPIG) for any advice and he sent me this email. Not sure it's the best stuff available, but at least we have starting points. Cheers, Ingy ---------- Forwarded message ---------- From: Ingy dot Net <ingy(a)ingy.net> Date: Wed, Feb 20, 2013 at 6:44 PM Subject: Re: python topics To: Jonathan Mark <jhmark(a)xenops.com> Thanks Jonathan! I'll let you know how it goes. On Wed, Feb 20, 2013 at 3:58 PM, Jonathan Mark <jhmark(a)xenops.com> wrote: > Tkinter is how to do a portable GUI: > http://wiki.python.org/moin/**TkInter<http://wiki.python.org/moin/TkInter> > > For building a standalone executable file: > There seems to be this new "pyinstaller" thing which sounds nice: > http://www.pyinstaller.org/ > > The old way I knew to do this is to use py2exe and py2app: > http://www.py2exe.org/ > http://wiki.python.org/moin/**MacPython/py2app<http://wiki.python.org/moin/MacPython/py2app> > > "Freeze" does it for Linux: > http://wiki.python.org/moin/**Freeze<http://wiki.python.org/moin/Freeze> > > I don't know how to do iOS or Android (unless a Linux solution covers > Android also) > > cheers, > > J >

1 0

Tor Browser Launcher
by Micah Lee 21 Feb '13

21 Feb '13

In my spare time I've been working on a way to make the Tor Browser Bundle easier to use for GNU/Linux users. I've mostly been talking about it with other people on tor-talk and in this bug: https://trac.torproject.org/projects/tor/ticket/5236 I decided to write Tor Browser Launcher, a completely separate program that's in charge or adding a "Tor Browser" .desktop file (so you can open it from your desktop environment's menu), downloads the latest version for you, verifies it's signature (making it more secure than how much GNU/Linux users get TBB), and installs it in your home directory. I think that Tor Browser Launcher can get in Debian and Ubuntu. I just finished a first version of it. The code is here: https://github.com/micahflee/torbrowser-launcher And there are screenshots here: http://imgur.com/a/Mvpwl You can read details about how it works on my latest comment on that bug: https://trac.torproject.org/projects/tor/ticket/5236#comment:32 Before trying to get it in Debian I'd like to make it so it doesn't need to be updated each time TBB is updated. There are more details in the ticket, but this would require Tor to maintain a file on https://www.torproject.org/ that has the current version number of TBB in it and a timestamp, and possibly digital signature of this file too. Do you think this is doable? I also want to get it localized into all the languages TBB is localized into. Any thoughts or suggestions? -- Micah Lee https://twitter.com/micahflee

7 27

Re: [tor-dev] Switch from Obfsproxy to Pyobfspoxy
by Nathan Freitas 21 Feb '13

21 Feb '13

(moved to tor-dev) Sorry I really haven't commented on this until now, but I have a lot of balls in the air, and sometimes minor, but very important details, like this slip by. George Kadianakis: > My current plan is to keep maintaining the C-Obfsproxy code, if it > keeps being as stable as it is today. For example, I plan to fix two > new-found bugs related to the shared-secret support of obfs2 soon > (#8217, #8223). Excellent. So we do have some time. > That said, I don't plan to port any of the new transports (like obfs3, > and some other pluggable transports currently under development) to > C-Obfsproxy any time soon (even though obfs3 wouldn't be *extremely* > hard to port, and I might consider it if there is a serious reason). Okay, I will take that into consideration. My plan then is to focus on determining how difficult supporting Pyobfsproxy on Android will be, and otherwise, try to find funding and resources to maintain the C version and potential to port obfs3. Thanks! +n

3 2

Re: [tor-dev] Proposal: Controller events to better understand connection/circuit usage
by Damian Johnson 20 Feb '13

20 Feb '13

> AppWard and ExitWard indicate the direction of cell flow through the > circuit. Previous cell events aggregated statistics for both directions and > therefore did not capture information which I believe may be important. > These are much more informative than N and P. Do you have suggestions for > improving this further? Ahhh, gotcha. Personally I'd go for 'inbound' and 'outbound' since they better connotate directionality. AppWard and ExitWard sound weird, but they make sense now too and would be fine if CELL_STATS got a quick description saying what they mean.

2 6

[Patch] or/onion_ntor.c
by Gisle Vanem 19 Feb '13

19 Feb '13

I suggest that the whole body of this file gets included inside an "#ifdef CURVE25519_ENABLED" in-case someone (like me!) inadvertently tries to compile it w/o this define set. Patch: --- Git-latest\src\or\onion_ntor.c Tue Feb 12 18:40:04 2013 +++ or\onion_ntor.c Tue Feb 12 23:46:51 2013 @@ -9,6 +9,8 @@ #include "torlog.h" #include "util.h" +#ifdef CURVE25519_ENABLED /* Rest of file */ + /** Free storage held in an ntor handshake state. */ void ntor_handshake_state_free(ntor_handshake_state_t *state) @@ -292,4 +294,4 @@ memwipe(&s, 0, sizeof(s)); return bad ? -1 : 0; } - +#endif /* CURVE25519_ENABLED */ --gv

2 1

changes to vidalia / browser launch in TBB?
by Greg Norcie 14 Feb '13

14 Feb '13

Hi all, I'm working on a followup to my usability paper from HotPETS... (http://petsymposium.org/2012/papers/hotpets12-1-usability.pdf) and doing some fact checking for the write up. I'm trying to find out if the developers tweaked Vidalia since v2.2.35-7.1. In my tests it seems like there is much less of a lag between Vidalia launching and the TBB window opening. I was hoping to find out if this was an intentional code change on the developers part, since it elliminates one of the isssues mentioned in my paper. -- -- Greg Norcie (greg(a)norcie.com) GPG key: 0x1B873635

1 0