Re: [tor-relays] Call for discussion: turning funding into more exit relays
On Mon, Jul 23, 2012 at 9:22 PM, Roger Dingledine <arma@mit.edu> wrote:
Hello Andrew,
I wanted to draw your attention to a thread I've started on the tor-relays list: https://lists.torproject.org/pipermail/tor-relays/2012-July/001433.html
Hi Roger, I'm not too familiar with mailing lists, so hopefully I'm not top posting or replying in the wrong place here. I'm simply replying to your email in Gmail with a CC to the tor-relays list. Sorry for the late reply, I've been away with no internet access for the past week.
In short, we have a funder who wants to sponsor more and faster Tor exits, and we're brainstorming about how to use the money in a way that makes the network stronger but also doesn't screw up the "community" side of the Tor relay operator community. The first step is collecting facts about the current fast Tor exit relays.
Great! I've not been vocal in the Tor community before now, but I'd like to start, and I'm certainly happy to share my experience / thoughts.
- What do you currently pay for hosting/bandwidth, and how much bandwidth do you get for that?
This is a complicated question, because I run a single Tor exit in a VPS on my company dedicated server. I run a local company doing computer repair and web development, and lease a single dedicated server from OVH (more specifically, Kimsufi) for a total of £64 a month (inc. VAT). That gets me the Kimsufi 16G dedicated server, a RIPE block of 4 extra IPs, and an external 2TB HDD. 100Mbit pipe, 10TB/month bandwidth. £0.87/TB if I go over that, so if I were to max out the bandwidth for an entire month, using around 30TB traffic, I would have to pay about £18 on top for the extra bandwidth. However, according to the OVH manager I never seem to go anywhere near the traffic limit, despite having had the exit set to use 50Mbit/s constantly for the past 3 months. As far as I'm concerned, it costs me nothing to run this exit node - my company needs the dedicated server regardless, and none of the ~50 websites I host use enough traffic to be affected by the fact that my server is using half it's available bandwidth for Tor. In an ideal world, I would rent a second Kimsufi server just for Tor purposes, which would cost £36/month (Kimsufi 16G) + £4/month (RIPE block). Therefore, if I were to participate in this experiment, I would say *£40 GBP / month* would get *10TB of 100 Mbit/s* exit traffic. Additional *20TB* traffic could be purchased for *£18 / month*, which would bring the maximum cost to *$92 USD / month.*
- Is it a stable hosting situation? For example, how do they handle abuse complaints so far?
This is an important one, because OVH are not Tor-friendly. In fact, they aren't very friendly in general! Several people told me it was very foolish to run an exit node on an OVH server, since as soon as OVH get even a whiff of a complaint, they are quick to suspend. All my company-hosted websites are hosted on a second redundant server with another ISP so I'm not worried if OVH do find out and take that route and cut me off. I'm also not worried about legal issues, as I have positioned myself as a web-hosting customer of my own PLC, with terms and conditions absolving my PLC of any legal risks. In the off chance some legal authority were to come to my flat in britain, they would find no electrical equipment to seize as I own only a single laptop and it usually lives in my car or partners house anyway! As I have only been running this exit for 3 months, I am far from an authoritative voice on the issue of abuse complaints, but the most important thing is definitely SWIP as far as I can see it - the IP address I use for the exit is one from the RIPE block I lease and as such the abuse email is my own. I've had about 20 abuse report emails so far, all automated (by the looks of things) from some system within the brazilian government, following an Sqlmap SQL injection scan/attack on a few government sites. I replied to the first one with the standard template, got no reply, haven't worried about it since. I reckon if OVH found out I was running an exit they would be likely to cut me off fairly swiftly, but they don't seem to pay much attention to Kimsufi customers since it is their budget range with very slow (week+ per ticket) support and presumably low margins. I think that and me as the primary abuse contact for the IP mean it's likely to stay up for a while. If I do get any hassle, I'll defend it as far as I can from a business perspective, but if they don't give in I'll likely just cancel it and open a new Kimsufi lease - I very much doubt anyone is checking new invoices for similar details to past customers.
- Is your hosting situation one where it could make sense for us to reimburse your bandwidth costs? (Some people have a deal through their employer, friend, etc where they don't pay for hosting.)
I don't think so - as explained above, at present I don't pay a penny, but I can only offer about half of my available bandwidth as the server is used for many purposes. If I were to participate in this experiment, it would probably mean purchasing another Kimsufi just for this, and the cost of the server itself would be what I would be looking for financial help with.
- Are you in a position to get more bandwidth if you pay more? At what rates? We're most interested in sponsoring >=100mbit relays.
Unfortunately the Kimsufi servers are capped at 100Mbit regardless of whether you want to pay more. OVH obviously have Gbit and 10Gbit servers available, but they are too expensive for this. There are obviously far better alternatives for higher bandwidth servers - a quick look tells me I could get a 1Gbit dedicated server with 100TB traffic from Leaseweb for €99 ($121) / month, so obviously if the money is there, more bandwidth and traffic can be had. I guess it boils down to how many people you can get interested in this - if plenty, lots of 100Mbit servers is presumably better than a few 10Gbit ones for the money as it aids network diversity, even if (worst case) they are all hosted by the same provider.
- Do you have other locations in mind where you would run another exit relay if you didn't have to pay for it?
Definitely! As far as I'm concerned, I am not worried about legal issues as long as I can purchase hosting through my business and SWIP the IP, and I have plenty of free time to spend configuring servers and responding to abuse emails, so if I had the money I would happily be running exit nodes in any country I could find a hosting provider in - money is the hurdle for me.
- What else should we be asking here? :)
One thing which I haven't seen discussed yet is how funding would actually be connected with operators - I'm not sure if you were thinking about the funder(s) directly sending money to operators, or if The Tor Project Inc would be acting as a middleman? What money transfer mechanisms would be safe to use, how would you verify that the money was going to the right person, would The Tor Project Inc receive invoices directly from hosting companies or would operators email copies of invoices to someone and then some money would turn up in their bank accounts? What about PayPal, etc? Just a few thoughts :) Thanks!
--Roger
Thanks for inviting me to share my thoughts on this! -Andrew
On Sun, Jul 29, 2012 at 03:05:32PM +0100, Andrew Beveridge wrote:
- What do you currently pay for hosting/bandwidth, and how much bandwidth do you get for that?
This is a complicated question, because I run a single Tor exit in a VPS on my company dedicated server. I run a local company doing computer repair and web development, and lease a single dedicated server from OVH (more specifically, Kimsufi) for a total of £64 a month (inc. VAT). That gets me the Kimsufi 16G dedicated server, a RIPE block of 4 extra IPs, and an external 2TB HDD. 100Mbit pipe, 10TB/month bandwidth. £0.87/TB if I go over that, so if I were to max out the bandwidth for an entire month, using around 30TB traffic, I would have to pay about £18 on top for the extra bandwidth. However, according to the OVH manager I never seem to go anywhere near the traffic limit, despite having had the exit set to use 50Mbit/s constantly for the past 3 months.
Sounds like you should bump it up to 100mbit then. ;) You can see on http://atlas.torproject.org/#details/FA02311AF49EB663CA2685A8604C403A9E10E6E... that there are periods where your rate limiting is bottlenecking traffic.
As far as I'm concerned, it costs me nothing to run this exit node - my company needs the dedicated server regardless, and none of the ~50 websites I host use enough traffic to be affected by the fact that my server is using half it's available bandwidth for Tor. In an ideal world, I would rent a second Kimsufi server just for Tor purposes, which would cost £36/month (Kimsufi 16G) + £4/month (RIPE block).
Therefore, if I were to participate in this experiment, I would say *£40 GBP / month* would get *10TB of 100 Mbit/s* exit traffic. Additional *20TB* traffic could be purchased for *£18 / month*, which would bring the maximum cost to *$92 USD / month.*
Sounds like a good price. And even though everybody says OVH is unsuitable, it looks worth continuing to try in this case. Maybe we can encourage you to run the new one as you describe, but also bump up the bandwidth on the first one. As you say, it's unlikely that you'll actually max out the transit all the time.
As I have only been running this exit for 3 months, I am far from an authoritative voice on the issue of abuse complaints, but the most important thing is definitely SWIP as far as I can see it - the IP address I use for the exit is one from the RIPE block I lease and as such the abuse email is my own.
Yes, a SWIP seems increasingly critical for stable exits.
Definitely! As far as I'm concerned, I am not worried about legal issues as long as I can purchase hosting through my business and SWIP the IP, and I have plenty of free time to spend configuring servers and responding to abuse emails, so if I had the money I would happily be running exit nodes in any country I could find a hosting provider in - money is the hurdle for me.
I say we set this one up and see how it goes. In your spare time, please do continue to look for other opportunities. You should probably compare notes with Moritz, Julian, et al here about places you find.
One thing which I haven't seen discussed yet is how funding would actually be connected with operators - I'm not sure if you were thinking about the funder(s) directly sending money to operators, or if The Tor Project Inc would be acting as a middleman? What money transfer mechanisms would be safe to use, how would you verify that the money was going to the right person, would The Tor Project Inc receive invoices directly from hosting companies or would operators email copies of invoices to someone and then some money would turn up in their bank accounts? What about PayPal, etc? Just a few thoughts :)
I definitely don't want Tor to be in the middle of the transactions -- if Tor pays the bills directly, that's too much like being the relay operator. One nice situation would be for you to produce receipts showing expenses, and then we reimburse those costs. It requires fronting a bit of money on your part, but that's part of saving the world, yes? :) Ultimately, we're also going to want to look into reducing overhead on Tor's side from sending out money. If we have to write and mail 50 checks every month, that's going to waste a lot of somebody's time. Maybe that means Paypal is the way to go. Maybe it means we send some money in bulk to Zwiebelfreunde, and they do intra-Europe wire transfers to the other Europeans (though I admit maybe that just shifts the time-wasting). Lots of options there. What would be best for you? All of this said, don't go out and start spending money quite yet. We should figure out these logistics first. And Tor should get a bit more of a handle on what this diversity thing should mean. And I should get buy-in from other Tor people for my plans here. :) --Roger
On Wed, Aug 01, 2012 at 12:03:07AM -0400, Roger Dingledine wrote:
All of this said, don't go out and start spending money quite yet. We should figure out these logistics first. And Tor should get a bit more of a handle on what this diversity thing should mean. And I should get buy-in from other Tor people for my plans here. :)
What constitutes a minimal useful exit policy? Mine is curently reject 0.0.0.0/8:* reject 169.254.0.0/16:* reject 127.0.0.0/8:* reject 192.168.0.0/16:* reject 10.0.0.0/8:* reject 172.16.0.0/12:* reject 78.46.119.2:* accept *:22 accept *:443 accept *:465 accept *:563 accept *:992-995 reject *:* which doesn't give me any complaints, but also no exit flag. On the other hand I would love to unthrottle it (this is a dual-core Atom, but on 1 GBit line) as long as someone pays for the extra traffic (this is 6.9 EUR/TByte). I think there are several people in my position. So a) what minimal exit policy would qualify us for applying for funding? b) when and who is the contact? Should I just talk to Zwiebelfreunde e.V.? Thanks.
On Wed, Aug 01, 2012 at 11:43:22AM +0200, Eugen Leitl wrote:
What constitutes a minimal useful exit policy? Mine is curently
reject 0.0.0.0/8:* reject 169.254.0.0/16:* reject 127.0.0.0/8:* reject 192.168.0.0/16:* reject 10.0.0.0/8:* reject 172.16.0.0/12:* reject 78.46.119.2:* accept *:22 accept *:443 accept *:465 accept *:563 accept *:992-995 reject *:*
which doesn't give me any complaints, but also no exit flag. On the other hand I would love to unthrottle it (this is a dual-core Atom, but on 1 GBit line) as long as someone pays for the extra traffic (this is 6.9 EUR/TByte).
6.9 EUR * 30 TB is quite pricy. At the current plan, we could pay for part of it, but we'd hope you, or the community around you, can pay for the other part.
I think there are several people in my position. So
a) what minimal exit policy would qualify us for applying for funding?
accept *:80 accept *:443 accept *:554 accept *:1755 That also happens to be enough to get you the Exit flag.
b) when and who is the contact? Should I just talk to Zwiebelfreunde e.V.?
The contact for what? I've been coordinating potential exit relay operators so far. Once things are up and running a bit more, I'm hoping Tor can contract part-time to somebody and I'll hand everything to him/her to keep it all going. --Roger
On 01.08.2012 06:03, Roger Dingledine wrote:
Maybe it means we send some money in bulk to Zwiebelfreunde, and they do intra-Europe wire transfers to the other Europeans (though I admit maybe that just shifts the time-wasting).
Don't even think about it. As far as I know, at least for us as a German charitable non-profit, we can only redistribute money to other charitable German non-profits for tax reasons. -- Moritz Bartl https://www.torservers.net/
On Wed, Aug 01, 2012 at 01:08:08PM +0200, Moritz Bartl wrote:
On 01.08.2012 06:03, Roger Dingledine wrote:
Maybe it means we send some money in bulk to Zwiebelfreunde, and they do intra-Europe wire transfers to the other Europeans (though I admit maybe that just shifts the time-wasting).
Don't even think about it. As far as I know, at least for us as a German charitable non-profit, we can only redistribute money to other charitable German non-profits for tax reasons.
Ok! In that case it sounds like we need some European partner who wants to help us handle the finances for doing bank transfers to the rest of the Europeans. Ideally one who's part of the community and well-respected. Let me know if one comes to mind. :) --Roger
CCC ? On 08/01/2012 10:58 PM, Roger Dingledine wrote:
On Wed, Aug 01, 2012 at 01:08:08PM +0200, Moritz Bartl wrote:
On 01.08.2012 06:03, Roger Dingledine wrote:
Maybe it means we send some money in bulk to Zwiebelfreunde, and they do intra-Europe wire transfers to the other Europeans (though I admit maybe that just shifts the time-wasting). Don't even think about it. As far as I know, at least for us as a German charitable non-profit, we can only redistribute money to other charitable German non-profits for tax reasons. Ok! In that case it sounds like we need some European partner who wants to help us handle the finances for doing bank transfers to the rest of the Europeans.
Ideally one who's part of the community and well-respected.
Let me know if one comes to mind. :)
--Roger
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On 02.08.2012 00:58, Roger Dingledine wrote:
Ok! In that case it sounds like we need some European partner who wants to help us handle the finances for doing bank transfers to the rest of the Europeans.
Ideally one who's part of the community and well-respected.
Let me know if one comes to mind. :)
I would be happy to do it personally, but I can't do it as Zwiebelfreunde. Not sure if that helps. -- Moritz Bartl https://www.torservers.net/
Roger Dingledine <arma@mit.edu> schrieb:
On Wed, Aug 01, 2012 at 01:08:08PM +0200, Moritz Bartl wrote:
On 01.08.2012 06:03, Roger Dingledine wrote:
Maybe it means we send some money in bulk to Zwiebelfreunde, and they do intra-Europe wire transfers to the other Europeans (though I admit maybe that just shifts the time-wasting).
Don't even think about it. As far as I know, at least for us as a German charitable non-profit, we can only redistribute money to other charitable German non-profits for tax reasons.
Ok! In that case it sounds like we need some European partner who wants to help us handle the finances for doing bank transfers to the rest of the Europeans.
Ideally one who's part of the community and well-respected.
Let me know if one comes to mind. :)
--Roger
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hi Roger, what about the CCC in Germany? They are not a charitable non-profit organization... best, sigi
The specs manufacturers publish for firewalls are typically best case and combined throughput. So if the specs say 1.2 Gbps throughput, this means 600mbps each way, best case (max MTU packets, etc.). In order to guarantee 1Gbps each direction real world, you should look for a firewall rated for at least 4Gbps. I'm a bit of a Cisco bigot, which means the ASA 5555 or ASA 5585 would be my recommendation. Even though you only have "a single fiber incoming", you still should be able to split the tor nodes from your corporate network. Simply put a switch on the fiber and connect your existing firewall and Tor nodes directly to it. If you really want a hardware firewall in front of your Tor nodes as well, you can use a less expensive firewall just for them. A Cisco PIX 535 can be picked up off of eBay for around $300, and is rated at 1.7Gbps (850mbps each way, real world maybe 5-600mbps). As your Tor nodes should not be talking to each other, only to Tor nodes not in your family, you could easily put a separate cheap firewall in front of each node rather than having one big firewall for everything. -Pascal On 7/26/2012 1:08 PM, Dennis Ljungmark wrote:
Dennis Ljungmark:
Hi, We're currently running 6 different 100-200Mbit relay/guard nodes, and are looking at some issues moving on towards high performant exit nodes.
Right now, with iptables modifications ( raw tables hacks to disable conntrack, bucket increases, following the general best practices ) our firewall is running at high amounts of CPU, but coping. However, once we start introducing Exit Nodes into this equation, things turn sour.
So, since we do not want to trust only routing level separation between Exit Nodes and internal networks, we're going to have to invest into new hardware that can cope with this. Before this, we tried Ingate firewalls, and they weren't capable of coping with the load of guard nodes.
( The traditional "linux box in front" doesn't quite cut it due to networking hardware in most cases. )
So, in summary, when you get to the point of actively dealing with 8-900Mbps of Tor traffic ( on top of normal users and others) what hardware is needed to cope with firewalling?
Note here that the tor nodes are not our current bottleneck, so SSL Decoding/OpenSSL isn't part of the problems here. We're getting 200Mbps without trouble, but the network cards in the current firewall (separate from the Tor nodes) is capping out at ~800Mbps. ( Not good enough imo, but another issue )
The problem that I have is that the current i686 (32bit) firewall cannot cope with the connections once we move into exit node land.
Due to other network issues, we cannot "carte blanche" disable connection tracking ( Fex. Traffic from Tor exit nodes to other corporate networks need to be tracked, as well as corp net / public wifi need tracking and tracing ) ( Since it's all on a single fiber incoming, we don't have the option of physically separating them. )
//D.S.
Excellent! Right now it seems they procured an (as of yet, for me) unknown piece of Juniper hardware for this case. It may be that I'll simply have to have that on the office net and route the Tor nodes outside it, if it cannot keep up with the load. Currently I don't know the version/kind of hardware, looks like I'll find that out on Monday when I get back to the office. Regards, D.S. On Thu, Aug 2, 2012 at 8:57 PM, Pascal <Pascal666@users.sourceforge.net>wrote:
The specs manufacturers publish for firewalls are typically best case and combined throughput. So if the specs say 1.2 Gbps throughput, this means 600mbps each way, best case (max MTU packets, etc.). In order to guarantee 1Gbps each direction real world, you should look for a firewall rated for at least 4Gbps. I'm a bit of a Cisco bigot, which means the ASA 5555 or ASA 5585 would be my recommendation.
Even though you only have "a single fiber incoming", you still should be able to split the tor nodes from your corporate network. Simply put a switch on the fiber and connect your existing firewall and Tor nodes directly to it. If you really want a hardware firewall in front of your Tor nodes as well, you can use a less expensive firewall just for them. A Cisco PIX 535 can be picked up off of eBay for around $300, and is rated at 1.7Gbps (850mbps each way, real world maybe 5-600mbps). As your Tor nodes should not be talking to each other, only to Tor nodes not in your family, you could easily put a separate cheap firewall in front of each node rather than having one big firewall for everything.
-Pascal
On 7/26/2012 1:08 PM, Dennis Ljungmark wrote:
Dennis Ljungmark:
Hi, We're currently running 6 different 100-200Mbit relay/guard nodes, and are looking at some issues moving on towards high performant exit nodes.
Right now, with iptables modifications ( raw tables hacks to disable conntrack, bucket increases, following the general best practices ) our firewall is running at high amounts of CPU, but coping. However, once we start introducing Exit Nodes into this equation, things turn sour.
So, since we do not want to trust only routing level separation between Exit Nodes and internal networks, we're going to have to invest into new hardware that can cope with this. Before this, we tried Ingate firewalls, and they weren't capable of coping with the load of guard nodes.
( The traditional "linux box in front" doesn't quite cut it due to networking hardware in most cases. )
So, in summary, when you get to the point of actively dealing with 8-900Mbps of Tor traffic ( on top of normal users and others) what hardware is needed to cope with firewalling?
Note here that the tor nodes
are not our current bottleneck, so SSL Decoding/OpenSSL isn't part of the problems here. We're getting 200Mbps without trouble, but the network cards in the current firewall (separate from the Tor nodes) is capping out at ~800Mbps. ( Not good enough imo, but another issue )
The problem that I have is that the current i686 (32bit) firewall cannot cope with the connections once we move into exit node land.
Due to other network issues, we cannot "carte blanche" disable connection tracking ( Fex. Traffic from Tor exit nodes to other corporate networks need to be tracked, as well as corp net / public wifi need tracking and tracing ) ( Since it's all on a single fiber incoming, we don't have the option of physically separating them. )
//D.S.
______________________________**_________________ tor-relays mailing list tor-relays@lists.torproject.**org <tor-relays@lists.torproject.org> https://lists.torproject.org/**cgi-bin/mailman/listinfo/tor-**relays<https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays>
participants (8)
-
Andrew Beveridge -
Dennis Ljungmark -
Eugen Leitl -
kupo＠damnfbi.tk -
Moritz Bartl -
Pascal -
Roger Dingledine -
sigi