[tor-scaling] Inprocess Cell Time Tracing

[Rob Jansen]

Hello David,

Wow, mega-post! Lot's of great info :)

I still haven't properly digested all of this yet. But I'll pretend like I know what I am talking about anyway ;)

> On Aug 7, 2019, at 2:51 PM, David Goulet <dgoulet at torproject.org> wrote:
> 
> So yeah, would love to hear anyone's opinion on all this.
> 

I believe understanding kernel buffering and TCP effects will be fundamental to understanding Tor's behavior.

One possible thing holding Tor back from writing to the kernel is an overloaded kernel outbuf (buffer bloat), but KIST should help to reduce this.

Another possible issue is that the quiet circuits are also on quiet sockets (no other circuit is sending on that socket). Then when the some data comes in, Tor sends and fills up the kernel socket out buffer. But because the socket was previously quiet, the TCP congestion window will be reset to 10 packets, and it will take a few round trips with the other end of the socket for the congestion window to ramp up. When the circuit becomes quiet again, the congestion window resets to 10 packets again and we start over. In order for this to be a real issue, a quiet circuit would need to get a burst of traffic greater than the socket outbuf size so that it would cause the outbuf to fill and Tor to queue it. We are talking more than 200 cells ~ 100kb to fill the kernel out buffer. The circuit window allows up to 1000 cells, so this phenomena is possible, though I don't know how likely.

> One of the next thing I want to learn is _where_ is that time all spent. As in
> from inbuf->queue or queue->outbuf or outbuf->flush. My instincts tells me
> outbuf->flush especially with KIST considering TCP congestion.
> 
> Food for thoughts here! Please let me know if you would like to see specific
> things, I now have a set of scripts to analyze the tracing data and plot it.

I think it would be useful to gather kernel in/out buffer times in order to get a more complete picture of queuing times through the entire machine (rather than just in Tor). Relatively high kernel out buffer time was the primary motivation for KIST, and clearly the kernel will affect Tor's ability to write. Can your tools be made to measure kernel in/out buffer times?

If not, and in case it's useful, I developed a multi-threaded library to measure kernel in/out buffer times [0]. It uses libpcap to measure kernel queuing times of packets.

Peace, love, and positivity,
Rob

[0] https://github.com/robgjansen/libkqtime