[tor-relays] clarification on what Utah State University exit relays store ("360 gigs of log files")

[Mike Perry]

Mike Perry:
> grarpamp:
> > The questions were of a general "intro to netflow" nature, thus
> > the links, they and other resource describe all the data fields,
> > formation of records, timeouts, aggregation, IPFIX extensibility, etc.
> > Others and I on these lists know what "360 gigs" of netflow looks like.
> 
> Well, right, then. Let's get to the meat of it.
> 
> > *What* specific info are you looking for beyond that?
> 
> I am looking to understand what "360 gigs" aka "(3.2 billion records)"
> of netflow over 3 months looks like, and also if we can expect this to
> be standard practice, somewhat outside the norm, or indicative of
> someone who has specifically tuned their netflow config to attack Tor
> (should the opportunity arise).
> 
> Assuming the boingboing comment is accurate, and it's just one exit IP,
> then we're probably looking at two exits worth of data (either
> UtahStateExit0+UtahStateExit1, or UtahStateExit2+UtahStateExit3).
> 
> Each of these exit pairs appears to have averaged a little over
> 10Mbit/sec sustained over the most recent 3 month period according to
> https://globe.torproject.org. The exits are running some version of the
> Reduced Exit Policy, so there should be no bittorrent traffic. Likely
> mostly web traffic by connection count, and probably even byte count.
> 
> In three months, there are 7,776,000 seconds. So we're looking at 441
> records per second in this dataset.
> 
> For 10Mbit/sec worth of sustained web traffic, that sounds about
> connection-level resolution to me. Do you agree?

(Yay! Thinking once and posting two posts at once to three different
lists. I'm like some kind of Internet champion! ;)

I think I needed to do one more division. This is roughly one record per
3KB of traffic (which I think you alluded to earlier). Rather high if we
expect this to be web traffic, even if there was only 1 web request per
connection.

So then, what is the most likely configuration that would generate this
many records? Is it indeed likely to be some BOFH scenario, or might
there be some common (if half-insane) policy that ends up producing this
many records?


Here's Globe for UtahStatExit2 and 3 for easy access:
https://globe.torproject.org/#/relay/B4E641BC42DDB6FD2526CFF80504AB5221B0EB82
https://globe.torproject.org/#/relay/7E4E1CC167300932F05AC70ECD2B9A298732C6E2

The bandwidth histories have no current data, but you can click on the 3
month tab to get the numbers I used.


-- 
Mike Perry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Digital signature
URL: <http://lists.torproject.org/pipermail/tor-relays/attachments/20150813/6ee6d587/attachment.sig>