- tor-relays - lists.torproject.org

Guard flag flapping
by starlight.2015q3＠binnacle.cx 08 Aug '15

08 Aug '15

[apologies to all for thread-breaking, am really going on hiatus but the horror-show performance GBE topic was too darn interesting--last post I promise!] >I would call it a dedicated gigabit link. This is >probably up for debate. The provider's overall >capacity is very likely not [number of customers] >x [1 Gb/s] but I've never witnessed signs of >throttling or over-subscription. Pulling a whois on the IP 184.100.166.110 and a quick Google turned up that you are probably a bleeding-edge subscriber to this new Quest service: https://www.centurylink.com/fiber/ Running a filter on Blutmagie hostname contains quest shows that your relays are, by a huge margin, the fastest of about a dozen. My advice is that this QWest service is third-rate and rather than bleeding for the length of the contract, run for the hills! If you are still in the 1-month cancellation period (relays are about a month old), terminate the service immediately and place an order for Verizon FiOS. FiOS costs more, but sells real bandwidth instead of imaginary bandwidth. You can run a relay on a 75/75 MBit link (like mine) and obtain respectable bandwidth ranking. FiOS goes up to 1/2 GB for about $300-400 per month but the bandwidth is real and the network is good-to-excellent. Either that or switch to leasing a colo server in Germany or some other country where bandwidth is extremely cheap, or perhaps finding a cheap bandwidth colo in the US. QWest GBE looks like a turkey.

1 0

Guard flag flapping
by starlight.2015q3＠binnacle.cx 08 Aug '15

08 Aug '15

>The problem is likely that your ISP is routing >some traffic via an overloaded peering point. Running traceroute -w 1 -q 5 -A <bwauth-IP> 1400 may provide more detail on this issue. The -A option causes 'traceroute' to show Autonomous System numbers with each route hop, where ASNs are Internet routing domains associated with different network providers. One can readily see where big increases in latency are occurring along the path and what peerings are associated with it. Note that some big increases are justified--such as with a hop over a transatlantic cable. Setting -q 5 (or bigger) causes traceroute to attempt each hop that many times. If you see asterisk characters, that's packet loss and may indicate congestion points. Setting the packet size to 1400 will increase the probability that network congestion will impact the results. Try pulling up https: // torstatus dot blutmagie dot de which shows IPs together with other relay attributes and experiment tracing to various major relays.

1 0

Guard flag flapping
by starlight.2015q3＠binnacle.cx 08 Aug '15

08 Aug '15

>. . .have physical access to the . . . ONT. > Doubt this is the case, but on the off chance that the ONT is configured to flow IP traffic over a coaxial cable attached to a CPE router (customer premise equipment), the configuration should be changed to deliver data over a Gigabit Ethernet directly to a good-quality GBE switch provided by you. New eight-port Netgears are surprisingly good managed switches for around $100. For under $600 you can purchase used 24-or-48-port Nortel/Avaya 5510/5520 switches on eBay. Though old, these enterprise grade switches have astounding performance characteristics due to the Broadcom silicon switch-fabric chipset. Were around $7000 new when they were current-gen. All above switches have VLAN support so one switch can operate as multiple isolated networks. I run a Verizon FiOS 75/75 and the default setup is for Internet traffic to pass to a Verizon supplied Actiontec router over a coax cable. Fortunately I hung out with the installer and found out up front--had it switched to the GBE from the outset. For any VZ customers reading this, note that you can connect the Actiontec to your network so it can obtain TV programming information. The coax will continue to carry video and phone data which are on separate fiber optic wavelengths/channels.

1 0

onionoo.thecthulhu.com: 500 Server Error
by nusenu 08 Aug '15

08 Aug '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Thomas, in case this is not known to you yet: There appears to be a problem with your onionoo instance: https://onionoo.thecthulhu.com/summary?limit=1 Error 500 Server Error -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVxZ6AAAoJEFv7XvVCELh05cUQALASqTfj6gZTb8gd73WT/9mB gcUHqhl6scGBNW9Vl88X0uAk55HdO36nqOc/rTv9GaMSFQQ1bgO1nUIFWTvDcpNG /yLTHwWOshoaVNLmL6kJV4KiaLDtUjA3jY8knh59mIYBjL6gLaIKjp8c55i4HUzM UmYa9vPVVOmdWhAuIoIImbEfQqRcR61Ig0NBGheYWF7AAPBgQUv9c+c4fCYOBnla hs9pEMQJ/5u860RJqfETbEe/JVwD3zD+gJL9S7tshQNWKFAShPElgVMRvPu0aRK2 7IIG4szdinqXKd/7Ii9ZtllhWGu9vDGYeFP+9U/sHg4RGZ7bJhLwsMd5VTOgLGpb b5IEPxfTxfSN27FJig+bj6NNCtsVEmYKFayeWhAbml2Rip60a9fc+OBd9tGP6jlb pGCtox02kVUt+7zFJv1sqdf3Qp05RwvETr0it5d0V2+Lm68MRfVZDZ7O24IOTaYp eG6gBMnkmL2hNbgNgUMlvqOOiLYOyjlzhJx1toFLJcdWm9yfSTn90DrfaDFzMacF RRa7oW/mCzo0vseYhTWKJO8w4HJZJUbumZ0w11liqJbJQc1bg4nSVHurVf88Fb6H 5E80IsSFqNA1/vp15tTQsuAkTCK/ycLkKpPYd4Sz8ZEGiKXVqCNszG0QX8MCgk9Q QHz0wwggcg9wGiHFG13l =9H2Z -----END PGP SIGNATURE-----

2 2

Guard flag flapping
by starlight.2015q3＠binnacle.cx 08 Aug '15

08 Aug '15

>I had already run tests with both speedtest-cli >and iperf3. This server consistently achieves 200 >to 300 Mb/s in both directions, with both relays >still running, and on some runs is hitting over >800 Mb/s. Final caveats: If the server is on a shared gigabit link, performance may not improve. The difference between a dedicated gigabit and a shared gigabit is immense. One expects an ISP to load a shared GB links as heavily as possible--I.E. up to the point where customers complain and terminate service. True dedicated gigabit links are expensive. If it's running in a VPS that's showing a gigabit link speed, the number is fiction.

2 1

Guard flag flapping
by starlight.2015q3＠binnacle.cx 08 Aug '15

08 Aug '15

One more errata: The sense of this is negative, so the current setting you have is correct: net.ipv4.tcp_no_metrics_save = 0 The double-negative got me.

1 0

Guard flag flapping
by starlight.2015q3＠binnacle.cx 08 Aug '15

08 Aug '15

comments backward, but the sysctls are correct * 25 milliseconds or 3125000 (for write) * 40 milliseconds or 5000000 (for read)

1 0

Guard flag flapping
by starlight.2015q3＠binnacle.cx 08 Aug '15

08 Aug '15

Ah, forgot to calculate the default TCP windows for you link speed rather than mine. So that's 125000000 bytes/sec ( 1 gigabit / sec ) * 25 milliseconds or 3125000 (for read) * 40 milliseconds or 5000000 (for write) net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_wmem = 4096 3125000 16777216 net.ipv4.tcp_rmem = 4096 5000000 16777216 The 25 and 40 milliseconds are my guess at roughly the faster and more middle ping times within the domestic US Internet. This just gets each connection started net.ipv4.tcp_moderate_rcvbuf = 1 net.ipv4.tcp_timestamps = 1 cause the IP stack to very quickly establish precise values for the transmit and receive TCP windows.

1 0

Guard flag flapping
by starlight.2015q3＠binnacle.cx 08 Aug '15

08 Aug '15

First, I am assuming you are running bare-metal on a system and not in a virtualized server--everything below is premised on that. Do not expect a virtual server or Linux container to perform well as a high- capacity Tor relay. It's possible to configure a high-performance VM, but this is an esoteric art and one is better off renting a small dedicated physical server than going that route. Your story of a relay setup that should measure fast by all apparent metrics but is given terrible rankings by BWauths is common this year. BWauths scripts are known to be buggy, though supposedly have been improved very recently. 'longclaw' just came back online with the "latest" code, but after starting out with a failure to measure 2000 relays two days ago, it's still running 1000 shy of the full population: https://consensus-health.torproject.org/#bwauthstatus Scroll down a little and you will see 'longclaw' is unique in voting 976 relays not-guard and 1709 relays not-fast. That seems a more serious issue than cold start glitching IMO, and is not impressive if that is what it really is. A fifth BWauth is said to be arriving soon and it is said that it will help. Your relays currently are measured thusly: greendream848 longclaw-w Bandwidth=1694 Measured=986 gabelmoo-w Bandwidth=1694 Measured=347 maatuska-w Bandwidth=1694 Measured=874 moria1 -w Bandwidth=1694 Measured=1550 spacequeen974 longclaw-w Bandwidth=1698 Measured=493 gabelmoo-w Bandwidth=1698 Measured=970 maatuska-w Bandwidth=1698 Measured=1930 moria1 -w Bandwidth=1698 Measured=2130 You can see future and past reports of these in https://collector.torproject.org/recent/relay-descriptors/votes/ https://collector.torproject.org/archive/relay-descriptors/votes/ where longclaw is 23D15D9. . . gabelmoo is ED03BB6. . . maatuska is 49015F7. . . moria1 is D586D18. . . That the measurements are all in the same ballpark does indicate that some subtle issue with the network and/or equipment may be at work and the BWauths may not be at fault. But many have complained that nothing they do seems to work. If the firewall is performing stateful packet inspection or any kind of DPI (deep packet inspection) disable that for all incoming and outgoing Tor traffic. It's all encrypted anyway so there's no point, and DPI can drag down performance big-time. The directory traffic is unencrypted but I've never heard of a firewall with stateful rules for the Tor directory protocol. If you can put the system directly on the public IP address with no firewall or local-rack router I recommend doing this. Just make sure iptables are set to protect login and other non-tor access. Either that or disable iptables and strip the server down so that nothing but the 'tor' process and 'ssh' are running, and configure 'ssh' to accept only certificate authentication (be sure to set and test the cert auth before applying the setting). Check for minimized listeners with lsof -Pn | fgrep LISTEN The email daemon should stay up to handle alarms, just be sure it listens on 127.0.0.1. Likewise anything else that is absolutely necessary. Use *Port and *Policy settings in torrc to lock down control and socks access to the daemon. One notable sysctl that matters for high-capacity relays is net.netfilter.nf_conntrack_checksum = 0 though having this enabled would not cause the current poor measurements. You should change this setting: net.ipv4.tcp_no_metrics_save = 1 turning this off was to work around a very- long-ago kernel bug that is fixed everywhere. Turning it on improves performance. You might try net.ipv4.tcp_wmem = 4096 250000 4194304 net.ipv4.tcp_rmem = 4096 375000 4194304 which will cause the congestion window to get to full size a bit quicker, and these net.core.somaxconn = 1024 net.core.netdev_max_backlog = 524288 net.ipv4.tcp_slow_start_after_idle = 0 net.ipv4.tcp_keepalive_time = 600 which increase various limits for fast networks, lots of connections. Make sure these defaults values are active and have not been changed to non-default by /etc/sysctl.conf: net.ipv4.tcp_moderate_rcvbuf = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_sack = 1 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_congestion_control = cubic And try adding TXQUEUELEN=100000 to the /etc/sysconfig/network-scripts/ifcfg-ethX for the interface(s) where tor runs. Manually activated with ip link set qlen 100000 dev ethX ip link show dev ethX Finally make sure the kernel is of a vintage with the Google-advocated connection-start congestion-window increase: https://lwn.net/Articles/427104/ http://samsaffron.com/archive/2012/03/01/why-upgrading-your-linux-kernel-wi… http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdi… If you end up implementing any of the above and it works please describe the results in tor-relays post.

1 0

Guard flag flapping
by starlight.2015q3＠binnacle.cx 08 Aug '15

08 Aug '15

Both relays are showing low BWauth-measured bandwidth and are below the 2000 threshold for the Guard flag. Recently BWauths were offline and the consensus algorithm reverted to self- measure. During that period the relays were above the 2000 threshold and were assigned Guard. But even the self-measure was very low for a gigabit link in a tier-1 network (Quest). Some more tuning work is probably needed to get performance up higher. Possibly network issues are at work. If a low-end consumer-grade router or firewall is in the picture, it could be the cause of the problem.

2 2