Hi Karsten,
we still believe that the statistics are useful. However we also agree with Rob that since more and more relays report data the scatter plot becomes confusing. I think some kind of aggregation would be helpful.
In one of our previous papers we assessed the performance impact of simultaneous TCP connections in overlay networks [1]. There we found that the occurring performance degradation is hard(er) to solve when connections are used bidirectionally -- which is the motivation for these statistics. From the current results I would presume that this is the case in Tor most of the time.
In Future developments the statistics could become helpful.
Cheers, Florian.
[1] D. Marks, F. Tschorsch, and B. Scheuermann, "Unleashing Tor, BitTorrent Co.: How to relieve TCP deficiencies in overlays," in 35th IEEE Conference on Local Computer Networks (LCN'10), 2010, pp. 320–323.
On 16/12/13 05:34, Rob Jansen wrote:
Hey Karsten,
I think the statistics could be useful, though I don't currently utilize them. I think the current presentation is somewhat confusing. Perhaps we can try to brainstorm some alternative ways to present the data if the decision is that we should keep it around.
Best, Rob
On Dec 9, 2013, at 12:43 PM, Karsten Loesing wrote:
Björn, Florian,
a few years back (in 2010, to be precise) we added statistics to little-t-tor reporting what fraction of connections is used uni-/bidirectionally. Quoting dir-spec.txt:
"conn-bi-direct" YYYY-MM-DD HH:MM:SS (NSEC s) BELOW,READ,WRITE,BOTH NL [At most once]
Number of connections, split into 10-second intervals, that are used uni-directionally or bi-directionally as observed in the NSEC seconds (usually 86400 seconds) before YYYY-MM-DD HH:MM:SS. Every 10 seconds, we determine for every connection whether we read and wrote less than a threshold of 20 KiB (BELOW), read at least 10 times more than we wrote (READ), wrote at least 10 times more than we read (WRITE), or read and wrote more than the threshold, but not 10 times more in either direction (BOTH). After classifying a connection, read and write counters are reset for the next 10-second interval.
These statistics are disabled by default, but when they are enabled, relays publish them in their extra-info descriptors. And quite a few relays do that. Here's a (bad) visualization (that used to be slightly less bad when fewer relays published these statistics):
https://metrics.torproject.org/performance.html#connbidirect
Here's the question: Is there still value in having these statistics? I recall that they were useful in 2010, but will that still be the case in 2013?
If the answer is "yes", never mind.
If the answer is "no", I'd create a ticket and submit a patch to remove code parts from little-t-tor, and I'd remove the not-really-useful graph from the metrics website.
Cc'ing Rob, Aaron, and Roger as the people who typically have an interest in these kinds of statistics. If other tor-dev@ people have an opinion on this, please raise your voice!
All the best, Karsten
Hi Rob, Florian,
thanks for your replies! If you say these statistics may still be useful, then let's leave them in!
I just worked on a slightly better visualization of the available data. The idea is that the most interesting piece of information, AIUI, is what fraction of connections is used bidirectionally; whether the rest is used mostly for writing or reading doesn't really matter. I also aggregated observations similar to Torperf measurements, by plotting only median and interquartile range. Here's the result:
https://people.torproject.org/~karsten/volatile/connbidirect-2013-09-19-2013...
The old graph containing the same data is still there:
https://metrics.torproject.org/performance.html?graph=connbidirect&start...
Do you like the new graph? Do you have further ideas for improving it?
This graph is only there to show what kind of data we have. If somebody is really interested in the data, they'll have to download the CSV file and do their own analysis. Here's the specification of the file format:
https://metrics.torproject.org/stats.html#connbidirect
All the best, Karsten
On 12/16/13 2:43 PM, Florian Tschorsch wrote:
Hi Karsten,
we still believe that the statistics are useful. However we also agree with Rob that since more and more relays report data the scatter plot becomes confusing. I think some kind of aggregation would be helpful.
In one of our previous papers we assessed the performance impact of simultaneous TCP connections in overlay networks [1]. There we found that the occurring performance degradation is hard(er) to solve when connections are used bidirectionally -- which is the motivation for these statistics. From the current results I would presume that this is the case in Tor most of the time.
In Future developments the statistics could become helpful.
Cheers, Florian.
[1] D. Marks, F. Tschorsch, and B. Scheuermann, "Unleashing Tor, BitTorrent Co.: How to relieve TCP deficiencies in overlays," in 35th IEEE Conference on Local Computer Networks (LCN'10), 2010, pp. 320–323.
On 16/12/13 05:34, Rob Jansen wrote:
Hey Karsten,
I think the statistics could be useful, though I don't currently utilize them. I think the current presentation is somewhat confusing. Perhaps we can try to brainstorm some alternative ways to present the data if the decision is that we should keep it around.
Best, Rob
On Dec 9, 2013, at 12:43 PM, Karsten Loesing wrote:
Björn, Florian,
a few years back (in 2010, to be precise) we added statistics to little-t-tor reporting what fraction of connections is used uni-/bidirectionally. Quoting dir-spec.txt:
"conn-bi-direct" YYYY-MM-DD HH:MM:SS (NSEC s) BELOW,READ,WRITE,BOTH NL [At most once]
Number of connections, split into 10-second intervals, that are used uni-directionally or bi-directionally as observed in the NSEC seconds (usually 86400 seconds) before YYYY-MM-DD HH:MM:SS. Every 10 seconds, we determine for every connection whether we read and wrote less than a threshold of 20 KiB (BELOW), read at least 10 times more than we wrote (READ), wrote at least 10 times more than we read (WRITE), or read and wrote more than the threshold, but not 10 times more in either direction (BOTH). After classifying a connection, read and write counters are reset for the next 10-second interval.
These statistics are disabled by default, but when they are enabled, relays publish them in their extra-info descriptors. And quite a few relays do that. Here's a (bad) visualization (that used to be slightly less bad when fewer relays published these statistics):
https://metrics.torproject.org/performance.html#connbidirect
Here's the question: Is there still value in having these statistics? I recall that they were useful in 2010, but will that still be the case in 2013?
If the answer is "yes", never mind.
If the answer is "no", I'd create a ticket and submit a patch to remove code parts from little-t-tor, and I'd remove the not-really-useful graph from the metrics website.
Cc'ing Rob, Aaron, and Roger as the people who typically have an interest in these kinds of statistics. If other tor-dev@ people have an opinion on this, please raise your voice!
All the best, Karsten
On Dec 18, 2013, at 4:51 AM, Karsten Loesing wrote:
Hi Rob, Florian,
thanks for your replies! If you say these statistics may still be useful, then let's leave them in!
Great!
I just worked on a slightly better visualization of the available data. The idea is that the most interesting piece of information, AIUI, is what fraction of connections is used bidirectionally; whether the rest is used mostly for writing or reading doesn't really matter.
Perhaps your are correct for the original reason for collecting this data. But actually, I find it very important and interesting that connections tend to have bursty writing more often than bursty reading!
I also aggregated observations similar to Torperf measurements, by plotting only median and interquartile range. Here's the result:
https://people.torproject.org/~karsten/volatile/connbidirect-2013-09-19-2013...
The old graph containing the same data is still there:
https://metrics.torproject.org/performance.html?graph=connbidirect&start...
Do you like the new graph? Do you have further ideas for improving it?
I do like the new graph, its much cleaner than the old one. But I like the mostly reading/writing parts of the old one too. Maybe we can create two more graphs like the new one (1 for mostly reading and 1 for mostly writing). I also think a stacked percentage area graph (e.g. http://www.highcharts.com/demo/area-stacked-percent) could work here, as a way to get all the data on the same chart.
This graph is only there to show what kind of data we have. If somebody is really interested in the data, they'll have to download the CSV file and do their own analysis. Here's the specification of the file format:
https://metrics.torproject.org/stats.html#connbidirect
All the best, Karsten
If the main goal is to show the data that exists, I think the old graph does that fine. But I think an important subgoal is also to have graphs that make it clear how the data is useful, not only that it exists. Perhaps keep both/all versions?
Best, Rob
On 12/18/13 2:03 PM, Rob Jansen wrote:
On Dec 18, 2013, at 4:51 AM, Karsten Loesing wrote:
I also aggregated observations similar to Torperf measurements, by plotting only median and interquartile range. Here's the result:
https://people.torproject.org/~karsten/volatile/connbidirect-2013-09-19-2013...
The old graph containing the same data is still there:
https://metrics.torproject.org/performance.html?graph=connbidirect&start...
Do you like the new graph? Do you have further ideas for improving it?
I do like the new graph, its much cleaner than the old one. But I like the mostly reading/writing parts of the old one too. Maybe we can create two more graphs like the new one (1 for mostly reading and 1 for mostly writing).
Ah okay, then let's put the unidirectional parts back into the graph. I made another graph with all three parts (both reading and writing, mostly writing, and mostly reading) displayed with medians and interquartile ranges on the same y axis. I find it easier to compare the three parts in this graph than in three separate graphs with possibly different y axis scales.
https://people.torproject.org/~karsten/volatile/connbidirect-2-2013-09-19-20...
How's this one compared to the other two?
I also think a stacked percentage area graph (e.g. http://www.highcharts.com/demo/area-stacked-percent) could work here, as a way to get all the data on the same chart.
I'm not really sure how that would work with our data. We could only display medians, not interquartile ranges. And our three medians don't even add up to 100%; using means instead of medians might fix this, though I didn't check.
Do you think this graph would be easier to understand than the one I posted above?
This graph is only there to show what kind of data we have. If somebody is really interested in the data, they'll have to download the CSV file and do their own analysis. Here's the specification of the file format:
https://metrics.torproject.org/stats.html#connbidirect
All the best, Karsten
If the main goal is to show the data that exists, I think the old graph does that fine. But I think an important subgoal is also to have graphs that make it clear how the data is useful, not only that it exists. Perhaps keep both/all versions?
Agreed, the graph should be useful, not just show that we have the data. Though I'd want to avoid adding a second or third graph and instead pick the most useful one we can come up with here.
Thanks for your input! Much appreciated.
All the best, Karsten
On Dec 21, 2013, at 4:13 AM, Karsten Loesing wrote:
On 12/18/13 2:03 PM, Rob Jansen wrote:
On Dec 18, 2013, at 4:51 AM, Karsten Loesing wrote:
I also aggregated observations similar to Torperf measurements, by plotting only median and interquartile range. Here's the result:
https://people.torproject.org/~karsten/volatile/connbidirect-2013-09-19-2013...
The old graph containing the same data is still there:
https://metrics.torproject.org/performance.html?graph=connbidirect&start...
Do you like the new graph? Do you have further ideas for improving it?
I do like the new graph, its much cleaner than the old one. But I like the mostly reading/writing parts of the old one too. Maybe we can create two more graphs like the new one (1 for mostly reading and 1 for mostly writing).
Ah okay, then let's put the unidirectional parts back into the graph. I made another graph with all three parts (both reading and writing, mostly writing, and mostly reading) displayed with medians and interquartile ranges on the same y axis. I find it easier to compare the three parts in this graph than in three separate graphs with possibly different y axis scales.
https://people.torproject.org/~karsten/volatile/connbidirect-2-2013-09-19-20...
How's this one compared to the other two?
Awesome! This is even better than have 3 separate graphs. I think this achieves the best balance between summarizing the data and showcasing the data that is available.
I also think a stacked percentage area graph (e.g. http://www.highcharts.com/demo/area-stacked-percent) could work here, as a way to get all the data on the same chart.
I'm not really sure how that would work with our data. We could only display medians, not interquartile ranges. And our three medians don't even add up to 100%; using means instead of medians might fix this, though I didn't check.
Ah, I see. I assumed they added to 100%.
Do you think this graph would be easier to understand than the one I posted above?
Likely not, given the above comment. I'd say ignore this suggestion.
This graph is only there to show what kind of data we have. If somebody is really interested in the data, they'll have to download the CSV file and do their own analysis. Here's the specification of the file format:
https://metrics.torproject.org/stats.html#connbidirect
All the best, Karsten
If the main goal is to show the data that exists, I think the old graph does that fine. But I think an important subgoal is also to have graphs that make it clear how the data is useful, not only that it exists. Perhaps keep both/all versions?
Agreed, the graph should be useful, not just show that we have the data. Though I'd want to avoid adding a second or third graph and instead pick the most useful one we can come up with here.
Thanks for your input! Much appreciated.
All the best, Karsten
I think your newest graph (the one with the three median+range plots on the same graph) is the best, and would be happy if we switched to that one.
Best, Rob
On 12/21/13 6:41 PM, Rob Jansen wrote:
I think your newest graph (the one with the three median+range plots on the same graph) is the best, and would be happy if we switched to that one.
Great! Glad you like the new graph. I just deployed it:
https://metrics.torproject.org/performance.html#connbidirect
Thanks again for your feedback!
All the best, Karsten
-
Florian Tschorsch -
Karsten Loesing -
Rob Jansen