Hi, We're currently running 6 different 100-200Mbit relay/guard nodes, and are looking at some issues moving on towards high performant exit nodes.
There are some administrative issues ( needing another IP block due to the RIPE registration, our ISP doesn't want their name on the exit nodes that we are responsible for ) which are generally minor ( are being resolved anyhow ) and then the big stumbling block.
Right now, with iptables modifications ( raw tables hacks to disable conntrack, bucket increases, following the general best practices ) our firewall is running at high amounts of CPU, but coping. However, once we start introducing Exit Nodes into this equation, things turn sour.
So, since we do not want to trust only routing level separation between Exit Nodes and internal networks, we're going to have to invest into new hardware that can cope with this. Before this, we tried Ingate firewalls, and they weren't capable of coping with the load of guard nodes.
( The traditional "linux box in front" doesn't quite cut it due to networking hardware in most cases. )
So, in summary, when you get to the point of actively dealing with 8-900Mbps of Tor traffic ( on top of normal users and others) what hardware is needed to cope with firewalling?
Regards, D.S. Ljungmark
Dennis Ljungmark:
Hi, We're currently running 6 different 100-200Mbit relay/guard nodes, and are looking at some issues moving on towards high performant exit nodes.
There are some administrative issues ( needing another IP block due to the RIPE registration, our ISP doesn't want their name on the exit nodes that we are responsible for ) which are generally minor ( are being resolved anyhow ) and then the big stumbling block.
Right now, with iptables modifications ( raw tables hacks to disable conntrack, bucket increases, following the general best practices ) our firewall is running at high amounts of CPU, but coping. However, once we start introducing Exit Nodes into this equation, things turn sour.
So, since we do not want to trust only routing level separation between Exit Nodes and internal networks, we're going to have to invest into new hardware that can cope with this. Before this, we tried Ingate firewalls, and they weren't capable of coping with the load of guard nodes.
( The traditional "linux box in front" doesn't quite cut it due to networking hardware in most cases. )
So, in summary, when you get to the point of actively dealing with 8-900Mbps of Tor traffic ( on top of normal users and others) what hardware is needed to cope with firewalling?
Hey Dennis,
What hardware are you using? In general iptables/netfilter should be able to handle more than 200Mb without any trouble at all.
I wonder if your network card is an issue? What CPUs are you using? What versions of OpenSSL and other relevant software are in use?
All the best, Jacob
Dennis Ljungmark:
Hi, We're currently running 6 different 100-200Mbit relay/guard nodes, and are looking at some issues moving on towards high performant exit nodes.
There are some administrative issues ( needing another IP block due to the RIPE registration, our ISP doesn't want their name on the exit nodes that we are responsible for ) which are generally minor ( are being resolved anyhow ) and then the big stumbling block.
Right now, with iptables modifications ( raw tables hacks to disable conntrack, bucket increases, following the general best practices ) our firewall is running at high amounts of CPU, but coping. However, once we start introducing Exit Nodes into this equation, things turn sour.
So, since we do not want to trust only routing level separation between Exit Nodes and internal networks, we're going to have to invest into new hardware that can cope with this. Before this, we tried Ingate firewalls, and they weren't capable of coping with the load of guard nodes.
( The traditional "linux box in front" doesn't quite cut it due to networking hardware in most cases. )
So, in summary, when you get to the point of actively dealing with 8-900Mbps of Tor traffic ( on top of normal users and others) what hardware is needed to cope with firewalling?
Hey Dennis,
What hardware are you using? In general iptables/netfilter should be able to handle more than 200Mb without any trouble at all.
I wonder if your network card is an issue? What CPUs are you using? What versions of OpenSSL and other relevant software are in use?
All the best, Jacob
Also tweaking a few sysctls and playing around with txqueuelen will help. See https://www.torservers.net/wiki/setup/server. I'll add some more stuff to the high bandwidth part of that page in a minute, also. I've done some more tweaking towards gbit that certainly helped, which I haven't documented yet.
Julian
On Thu, Jul 26, 2012 at 1:08 PM, Jacob Appelbaum jacob@appelbaum.netwrote:
Dennis Ljungmark:
Hi, We're currently running 6 different 100-200Mbit relay/guard nodes, and are looking at some issues moving on towards high performant exit nodes.
There are some administrative issues ( needing another IP block due to the RIPE registration, our ISP doesn't want their name on the exit nodes that we are responsible for ) which are generally minor ( are being resolved anyhow ) and then the big stumbling block.
Right now, with iptables modifications ( raw tables hacks to disable conntrack, bucket increases, following the general best practices ) our firewall is running at high amounts of CPU, but coping. However, once we start introducing Exit Nodes into this equation, things turn sour.
So, since we do not want to trust only routing level separation between Exit Nodes and internal networks, we're going to have to invest into new hardware that can cope with this. Before this, we tried Ingate
firewalls,
and they weren't capable of coping with the load of guard nodes.
( The traditional "linux box in front" doesn't quite cut it due to networking hardware in most cases. )
So, in summary, when you get to the point of actively dealing with
8-900Mbps
of Tor traffic ( on top of normal users and others) what hardware is
needed
to cope with firewalling?
Hey Dennis,
What hardware are you using? In general iptables/netfilter should be able to handle more than 200Mb without any trouble at all.
I wonder if your network card is an issue? What CPUs are you using? What versions of OpenSSL and other relevant software are in use?
All the best, Jacob
Hardware on the Firewall, or on the Tor nodes? Note here that the tor nodes are not our current bottleneck, so SSL Decoding/OpenSSL isn't part of the problems here. We're getting 200Mbps without trouble, but the network cards in the current firewall (separate from the Tor nodes) is capping out at ~800Mbps. ( Not good enough imo, but another issue )
The problem that I have is that the current i686 (32bit) firewall cannot cope with the connections once we move into exit node land.
Due to other network issues, we cannot "carte blanche" disable connection tracking ( Fex. Traffic from Tor exit nodes to other corporate networks need to be tracked, as well as corp net / public wifi need tracking and tracing ) ( Since it's all on a single fiber incoming, we don't have the option of physically separating them. )
//D.S.
tor-relays@lists.torproject.org
-
Dennis Ljungmark -
Jacob Appelbaum -
Julian Wissmann