[tor-scaling] Inprocess Cell Time Tracing

[David Goulet]

On 19 Aug (11:04:11), Rob Jansen wrote:
> 
> 
> > On Aug 8, 2019, at 8:10 AM, David Goulet <dgoulet at torproject.org> wrote:
> > 
> >> 
> >> If not, and in case it's useful, I developed a multi-threaded library to
> >> measure kernel in/out buffer times [0]. It uses libpcap to measure kernel
> >> queuing times of packets.
> > 
> > Does this has good performance? Thing with libpcap is that they do not provide
> > any guarantee that all the packets will be seen by libpcap. Under pressure,
> > you can miss some.
> > 
> > But also, on a 25MB/s relay that is capping the CPU, I worry about adding an
> > intrusive library that could slow things down which changes drastically the
> > behavior of the application leading to very different results.
> > 
> > That is the primary reason why I'm not doing this whole thing with log_*() in
> > tor for instance due to the performance hit and behavior change.
> 

> Regarding performance, the tracing code mostly runs in helper threads not in
> the main thread, though there is some unavoidable overhead on the main
> thread in sending the pointer to new data over to the helper thread.

That portion is where LTTng tracer high performance comes into play. This is
called a "tracepoint execution" and it is on average measured at around
110nsec of overhead.

It has been often measured but this one is a good blog post about it:

https://dev.framing.life/tracing/lttng-ust-benchmark/#quantifying-performance-loss

So the performance hit is extremely low compared to a "log_debug()" event that
stalls the mainthread for a much longer time. But also, another big
difference, is that it changes the overall state of "tor" because it is simply
a tor subsystem. With tracing like LTTng userspace that I used, that is not
changed one bit.

Thus, using a tracer there changes as little as possible of the "execution
behavior" with the smallest impact on CPU overhead it can.

It is really one of the main reason why I would love to have stable
tracepoints in tor code so we can hook whatever tracers we want including
LTTng userspace support because it really allow us to deep dive into low
layers with the smallest interference possible.

> 
> I didn't notice any slowdowns with running on a fast exit, but I wasn't
> trying to fully load it in order to test the limits of my code either.
> 
> Regarding missing some packets; that was OK in my case because I was only
> sampling cells anyway. Missing some packets would just cause a timeout,
> which then would cause the code to evict the "stuck" event and move to the
> next.
> 
> Anyway, sounds like you have your own methods and learning something new
> wouldn't be worthwhile. Good luck!

Thanks Rob again for the feedbacks!
David

-- 
QXF1g9W9a1T5QckogO+BlzeqJD2tPQMwiIPR6Yj/umU=
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-scaling/attachments/20190819/ef3abe12/attachment.sig>