Am Fri, 9 Dec 2011 19:56:23 -0800 schrieb Mike Perry <mikeperry@torproject.org>: Hi Mike,

I do not fully understand the cause of it yet, but I did find a rather nasty bug in the treatment of Guard nodes, where we were not properly using the "bwauthmercy" consensus param for them, and were punishing slow guards through multiple rounds of negative feedback. This could skew the metrics data, and could alter the flow of traffic through the rest of the network, but there may be other issues at work, too :/.

Im anxious to here more about that issue because i running a guard ;-)

I am going to wait until the Guard numbers recover from the bug, which should take 24-48hrs, and then dig deeper next week if the metrics numbers still persist to be bad.

Seems to get better in the last hours ... I want to suggest strongly a change for the metrics / performance site. The displayed default size is 50KB and should be changed to 1MB. 50 KB ist out of touch with reality for any service i can think of. Could you please encourage your colleagues to do that ? Thx. -- Mit freundlichen Grüßen / Yours sincerely Sebastian Urbach -------------------------------------------------------- Religion is something left over from the infancy of our intelligence, it will fade away as we adopt reason and science as our guidelines. -------------------------------------------------------- Bertrand Arthur William Russell (1872-1970), British philosopher, logician, mathematician, historian, and social critic.

Thus spake Jon (torance.ca@gmail.com):

In adding further info on the topic, I have noticed and looked back over the past couple of weeks and have seen a drop of of usage of about 43% s of today. I am not in the higher bracket as others, but I have been in the 3 bars bracket only up till recently.

Don't know if this will help, but if it does, add it to the rest of the info you have already received.

What helps more is your node nick and/or idhex string. If you're not comfortable talking about that publicly, knowing if you are Guard, Exit, Guard+Exit, or just Stable (Middle) node helps. I actually do expect that this system may cause some slower nodes (especially those with capacities close to or below the network stream average of ~70Kbytes/sec) to experience less traffic. This does not mean the nodes are useless. It just means that we should try to use them infrequently enough such that when they are used, they can provide enough capacity to not be a bottleneck in a circuit. We are still waiting for the effects of the Guard bug to fully dissipate. I am cautiously optimistic that things are getting better, but we'll need to keep an eye on things for a bit longer to be sure, I suspect. -- Mike Perry

Thus spake Sebastian Urbach (sebastian@urbach.org):

Seems to get better in the last hours ...

I want to suggest strongly a change for the metrics / performance site. The displayed default size is 50KB and should be changed to 1MB. 50 KB ist out of touch with reality for any service i can think of. Could you please encourage your colleagues to do that ?

We want to optimize primarily for the low-latency web case, not bulk downloading. The one exception to this is Youtube/user-generated video, which I agree does fall more in the 1MB case. However, the state of web browser tech for user-generated video still sucks in the absence of Flash, so it is still not yet our primary focus. -- Mike Perry

On 10.12.2011 11:30, Andy Isaacson wrote:

...

Please keep an eye on your relays and tell us if anything unexpected happens over the next week or so. we're seeing a pretty significant rise in throughput on noisetor01, we peaked at 530 Mbps at around 2011-12-09 17:00 PST, versus previous peaks:

I don't see any noticable change in bandwidth on our Gbit relays. ( http://nforce1.torservers.net/vnstat_d.png http://nforce2.torservers.net/vnstat_d.png http://axigy1.torservers.net/vnstat_d.png ) -- Moritz Bartl https://www.torservers.net/

On 12.12.2011 20:18, Tim Wilde wrote:

Any specific tips on how you're pushing 700Mbps from a single instance (I assume that's what that's indicating) beyond what's on your wiki? With AES-NI and most (maybe all) of the tweaks I've found on your pages (awesome resource, by the way, thanks for that!) the best I've been able to manage is ~250Mbps on a single Tor instance, I've had to use multiple instances on the same hardware to try to get any closer to filling Gbit.

Thanks! I try to keep everything I do documented on that wiki. All these servers run four instances of Tor each (one per core) and traffic is accounted for in total. Also, keep in mind that vnstat counts both incoming and outgoing traffic, so 700Mbps in vnstat are really only 375 per direction. -- Moritz Bartl https://www.torservers.net/

Thus spake Tim Wilde (twilde@cymru.com):

...

I try to keep everything I do documented on that wiki. All these servers run four instances of Tor each (one per core) and traffic is accounted for in total. Also, keep in mind that vnstat counts both incoming and outgoing traffic, so 700Mbps in vnstat are really only 375 per direction.

Ah, okay, thanks for the clarification, I was thinking those numbers were for single Tor instances. That makes me feel a lot better then, especially with the combination of directions. :) I'm pushing around 600Mb/sec total in+out on my piece of bit iron so I'm much closer to the same ballpark than I thought. Thanks, and thanks again for your documentation!

Moritz, Andy, Tim, and others with Gbit+ Guards and/or Exits: Could you guys ensure you are not running into TCP socket exhaustion on any of your relays? It is a possibility, esp for Guard+Exits with gobs of CPU and gobs of throughput. I am curious if we will need to do this or not: https://trac.torproject.org/projects/tor/ticket/4709 -- Mike Perry

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/12/2011 9:14 PM, Mike Perry wrote:

It looks like Moritz is seeing some evidence of TCP sourceport exhaustion in his Tor logs: "[warn] Error binding network socket: Address already in use".

He's also monitoring TCP connection counts on each IP interface: netstat -ntap | grep $INTERFACE_IP | wc -l

It appears that right now, he's at only about ~10k connections per IP, and not experiencing any log lines at the moment. It is possible this is a transient condition caused by overly-agressive scrapers and/or torrenters who flock to the node for a short while and then move on?

Reports on the recent appearance or prevelance increase of that or other warns from others will be helpful.

Hey Mike, We're not seeing source port exhaustion, but we are seeing two warns, one of which I haven't been able to nail down: 2011 Dec 13 20:22:07.000|[notice] We stalled too much while trying to write 8542 bytes to address "[scrubbed]". If this happens a lot, either something is wrong with your network connection, or something is wrong with theirs. (fd 409, type Directory, state 2, marked at main.c:990). 2011 Dec 13 22:26:45.000|[warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [18 similar message(s) suppressed in last 60 seconds] The second warn I figure I should be tuning myself with MaxAdvertisedBandwidth, and it's happening on BigBoy, the relay on this box that's doing the majority of its bandwidth. So I'm not sure if it's anything that your feedback loop should be involved in or not. The first warn, however, is happening on all four of the other relays on the box, somewhat sporadically (GreenDragon, GoldDragon, WhiteDragon, RedDragon). It seems, though I can't say for sure, that this (or something else I can't quite find) is constraining those four relays and preventing from them growing nearly as high in bandwidth consumption as BigBoy. They're all configured identically (non-exit middle / eventually entry if/when they get Guard). Those four also are different from BigBoy in that they kept names/configs but lost fingerprints/keys (ie they're all specifically Unnamed in the consensus) during a rebuild when trying to milk more bandwidth out of them. My understanding is that this /shouldn't/ make a big difference, but perhaps it is? I should note that this all could be entirely unrelated to your PID feedback experiments; I coincidentally started experimenting with trying to bring these nodes up to their hardware potential at around the same time, so it could all be a red herring from my experimentation (but I must say, the fact that we're both experimenting together sure does make drawing valid conclusions difficult. :)). One other data point, I have seen (also sporadically) some indications in my system logs of hardware hangs on the ethernet interface all of this is running through, so I'm slightly suspicious that it's to blame for the *Dragon problems. It doesn't really explain why BigBoy isn't affected though, and I haven't been able to definitively prove anything yet, so I'm just not sure. Let me know if any more info would be of use (or if anyone has any ideas about that first message). Thanks, Tim - -- Tim Wilde, Senior Software Engineer, Team Cymru, Inc. twilde@cymru.com | +1-847-378-3333 | http://www.team-cymru.org/ -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAk7n5EUACgkQluRbRini9thmCgCdGMgAYyJ4+yGk7MMEi6IjVcyE sOMAn3e411v1h11asitncJl9abwNJ2YC =Zdc9 -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/13/2011 10:34 PM, Mike Perry wrote:

Thus spake Tim Wilde (twilde@cymru.com):

...

We're not seeing source port exhaustion, but we are seeing two warns, one of which I haven't been able to nail down:

2011 Dec 13 20:22:07.000|[notice] We stalled too much while trying to write 8542 bytes to address "[scrubbed]". If this happens a lot, either something is wrong with your network connection, or something is wrong with theirs. (fd 409, type Directory, state 2, marked at main.c:990).

Hrm.. Haven't seen this one before...

I've seen LOTS of them. :) When I turned off SafeLogging briefly to see what the scrubbed addresses were it turns out they seem to be the dir auths, if that helps: 2011 Dec 14 16:53:17.000|[notice] We stalled too much while trying to write 273 bytes to address "193.23.244.244". If this happens a lot, either something is wrong with your network connection, or something is wrong with theirs. (fd 4108, type Directory, state 2, marked at main.c:990). I haven't seen any examples of that error message that were not to a dir auth.

...

2011 Dec 13 22:26:45.000|[warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [18 similar message(s) suppressed in last 60 seconds]

Ah, we should be handling this issue with the fix for #1984: https://trac.torproject.org/projects/tor/ticket/1984

...

The second warn I figure I should be tuning myself with MaxAdvertisedBandwidth, and it's happening on BigBoy, the relay on this box that's doing the majority of its bandwidth. So I'm not sure if it's anything that your feedback loop should be involved in or not.

It's a shame this log message makes such a crazy recommendation wrt MaxAdvertisedBandwidth. But I guess some tweak is better than no tweak. Hopefully we can make this go away without you needing to lower it, though. Can you ping me on IRC if you keep getting these warns after leaving MaxAdvertisedBandwidth alone?

Will do.

This sounds incredibly familiar. What ethernet card + driver version do you have? Some combos of are pretty abysmal about IRQ load balancing and interrupt optimizations, or at least they were on old kernels (which may still apply if you are CentOS).

Yeah, some further investigation today indicates that may be the case. :| Running Intel PRO/1000s with the latest E1000E driver (1.6.3-NAPI), I do in fact see what looks like potential interrupt load issues. I've split the relays across to another NIC to see if that helps at all in the relatively short term, long term it looks like a migration away from CentOS for this is called for (it's good for some things, but not this :)). I also rediscovered that the receive packet/flow steering in 2.6.35+ kernels is one of the torservers.net optimizations I haven't done (and can't do on CentOS without a manual kernel installation due to the 2.6.32 kernel it ships with even in 6.0). So it looks like Debian is in this box's future (though I'll try to remember to keep my keys this time :)). Thanks for the help and suggestions, if I can provide any more info with regards to the stalled writes to the dir auths, let me know. Thanks, Tim - -- Tim Wilde, Senior Software Engineer, Team Cymru, Inc. twilde@cymru.com | +1-847-378-3333 | http://www.team-cymru.org/ -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAk7o1i4ACgkQluRbRini9tggqwCeNVW/af9eNz/TGh7Pe40tQpVj FQQAn1/UKmRHK/w3ctf0iERdgNVfPIM7 =Pq3p -----END PGP SIGNATURE-----

Thus spake Sebastian Urbach (sebastian@urbach.org):

Around the 24 of November the performance graph reported a value below 4 sec. per 50kb, and right now including the last few days we can barely reach 6 sec. per 50kb.

Or in other words, we lost way more than 50 % peformance since that day. Unless you guys are about to really step up the game i strongly suggest to shred the chances made in the last weeks.

The original plan was to speed up things with the modified bandwidth scanners. Its probably safe to say that this experiment is a complete bust.

Yeah, I agree. I think that it's now clear that both the variance and the mean of the torperf graphs are way above norm, and we don't have much other explaination for it other than the feedback experiment not working. The question is why? Intuitively, feedback makes a lot of sense. If there is a ton of spare capacity on fast nodes, why shouldn't we try to use it? Similarly, if slow nodes can't keep up with the network throughput average, clearly they are a bottleneck and should have reduced traffic directed at them until they have more spare capacity. What is causing it to break so badly on the torperf graphs, then? Is it bugs? Is it bad choice of setpoints? Is it that we are using the wrong metric? Until we have answers for at least some of these questions, I am inclined to keep playing with it. I've made the following changes this afternoon: 1. On the assumption that we're seeing the huge increase in variance on torperf because faster nodes are only *sometimes* at max capacity, but most of the time have enough spare capacity to get good measurements, I have stopped "filtering" measurements for them (ie I have stopped selecting only the fast measurements). I am still applying filtering to overly punished nodes, so that they don't get punished too much by being paired with slow peers. 2. I have changed the circ failure target from the network average to 0.0% failure. 3. I have changed the Guard feedback interval from 2 weeks to 1 week. My plan is to let these changes run for another couple days, and if they don't seem to change anything, I plan to try 1 week on, 1 week off cycles of the experiment, to see if we can detect any patterns in exactly when and why torperf starts to go south. -- Mike Perry

Am Thu, 15 Dec 2011 12:43:25 -0800 schrieb Mike Perry <mikeperry@torproject.org>: Hi Mike,

Yeah, I agree. I think that it's now clear that both the variance and the mean of the torperf graphs are way above norm, and we don't have much other explaination for it other than the feedback experiment not working. The question is why?

I don't want to be the guy who posts always the bad news ... I can happily report that the load, since your last mail, increased rapidly to max values and also the performance from the metrics page seems to be increasing very well right now. If you want to take a look for yourself: https://metrics.torproject.org/routerdetail.html?fingerprint=0aff5440ae93f2e... Coincidink ;-) ? I dont think so ... I hope that this will relieve a little bit pressure from you, Mike. -- Mit freundlichen Grüßen / Yours sincerely Sebastian Urbach -------------------------------------------------------- Religion is something left over from the infancy of our intelligence, it will fade away as we adopt reason and science as our guidelines. -------------------------------------------------------- Bertrand Arthur William Russell (1872-1970), British philosopher, logician, mathematician, historian, and social critic.

Am Mon, 19 Dec 2011 16:32:19 -0800 schrieb Mike Perry <mikeperry@torproject.org>: Hi,

Just to be sure, I decided to shut off the experiment on Sunday at 3pm US Pacific time anyways. It looks like the performance has in fact gotten worse since then.

Does this mean the feedback was definitely working? Who the hell knows. But it does seem clear that the transition between the two states is definitely a rocky one that requires patience and repeated trials to sniff out.

Thats exactly what i meant. My impression is that the feedback seems to be working after all. Why do you want to investigate the transition any further ? If the feedback is finally working we probably don't need the transition in the future anymore :-)

I am going to leave it disabled for another week, just to see how it recovers, then turn it back on again, possibly with the addition of https://trac.torproject.org/projects/tor/ticket/4730 if I get a hot minute, but possibly without it, just to watch the transition again.

Robert Ransom is quite insistent on the need for this as well: https://trac.torproject.org/projects/tor/ticket/4708, but that system doesn't really need me to build it, as it can be done independent of the bwauths. I also think #4730 is more likely to capture the problem we were seeing with fast nodes being fast only sometimes, and it will be way less work.

Thanks for bearing with me, but we still might have a ways to go before we really get to the bottom of this one.

Its way less frustrating if you have at least a hunch. But if you are stuck in the middle of nowhere ... -- Mit freundlichen Grüßen / Yours sincerely Sebastian Urbach -------------------------------------------------------- Religion is something left over from the infancy of our intelligence, it will fade away as we adopt reason and science as our guidelines. -------------------------------------------------------- Bertrand Arthur William Russell (1872-1970), British philosopher, logician, mathematician, historian, and social critic.

I've been seeing these messages ("hammer" is routine disk maintenance, which makes other things run slowly): Dec 17 05:09:27 darner Tor[1138]: Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth conf ig option or choosing a more restricted exit policy. [159 similar message(s) sup pressed in last 60 seconds] Dec 17 05:09:27 darner Tor[1138]: Failed to hand off onionskin. Closing. [159 si milar message(s) suppressed in last 21600 seconds] Dec 17 05:10:22 darner Tor[1138]: Closing stream for '[scrubbed].onion': hidden service is unavailable (try again later). Dec 17 05:16:39 darner kernel: hammer: System has insufficient buffers to rebala nce the tree. nbuf < 3969 Dec 17 05:17:07 darner last message repeated 5 times I also saw this: Dec 16 22:11:27 darner Tor[1138]: Tor has not observed any network activity for the past 65 seconds. Disabling circuit build timeout recording. Dec 16 22:11:42 darner Tor[1138]: Tor now sees network activity. Restoring circu it build timeout recording. Network was down for 80 seconds during 3 circuit att empts. but checking auth.log on my other box shows that this is when I was either sending a huge email or rsyncing some files on an ssh connection from overseas.