Tor uses up all available memory and eventually quits
I am running a Tor (0.2.5.16 on Debian Linux) relay on a VPS with one gigabyte of RAM and no swap partition or swap file. The hosting company states that it doesn't support swap in order to prolong the life of its solid state drives. Tor gradually uses up more and more memory until its percentage usage is in the high 90s. Then Tor quits. This happens within about 48 hours of launch. What can I do to resolve this problem? Thanks! George W. Maschke https://georgemaschke.net
I think running on 1 GB of RAM with no swap is going to be difficult, especially if you have decent bandwidth and your node is busy. You can create a small swap file on the existing file-system like so: sudo fallocate -l 1G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile and to make permanent: echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab The swap file won't get used often, but will prevent your relay from going OOM and crashing. You can further reduce the likelihood of the swap getting used by setting something like "vm.swappiness = 10" in /etc/sysctl.conf. Also these two entries in your torrc might help: DisableOOSCheck 0 MaxMemInQueues 512 MB The first line will enable the "out of sockets" check, and although it aggressively drops connections, tends to prevent the node from falling over instead. The second reduces the amount of memory used.
I think the advice to create swap will get George kicked of VPS, as it goes right against the wishes of hosting company, and directly affects their hardware. A better advice is to tune tor process to work within memory boundaries. The "MaxMemInQueues 512 MB" is right direction, but from personal experience, I don't think it goes "far" enough. Will need to be limited further I think, but you could adjust further based on your observations. Seb On Sun, 28 Jan 2018 at 07:14 tor <tor@anondroid.com> wrote:
I think running on 1 GB of RAM with no swap is going to be difficult, especially if you have decent bandwidth and your node is busy.
You can create a small swap file on the existing file-system like so:
sudo fallocate -l 1G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile
and to make permanent:
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
The swap file won't get used often, but will prevent your relay from going OOM and crashing. You can further reduce the likelihood of the swap getting used by setting something like "vm.swappiness = 10" in /etc/sysctl.conf.
Also these two entries in your torrc might help:
DisableOOSCheck 0 MaxMemInQueues 512 MB
The first line will enable the "out of sockets" check, and although it aggressively drops connections, tends to prevent the node from falling over instead. The second reduces the amount of memory used.
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On 28 Jan 2018, at 20:51, r1610091651 <r1610091651@telenet.be> wrote:
I think the advice to create swap will get George kicked of VPS, as it goes right against the wishes of hosting company, and directly affects their hardware.
A better advice is to tune tor process to work within memory boundaries. The "MaxMemInQueues 512 MB" is right direction, but from personal experience, I don't think it goes "far" enough. Will need to be limited further I think, but you could adjust further based on your observations.
How fast is your relay? Try to make sure MaxMemInQueues allows 10-20s of traffic. Also, you can reduce the number of connections to your VPS (and therefore the RAM usage) using MaxAdvertisedBandwidth. Check your existing bandwidth, and reduce it by half. T
On Sun, 28 Jan 2018 at 07:14 tor <tor@anondroid.com> wrote:
I think running on 1 GB of RAM with no swap is going to be difficult, especially if you have decent bandwidth and your node is busy.
You can create a small swap file on the existing file-system like so:
sudo fallocate -l 1G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile
and to make permanent:
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
The swap file won't get used often, but will prevent your relay from going OOM and crashing. You can further reduce the likelihood of the swap getting used by setting something like "vm.swappiness = 10" in /etc/sysctl.conf.
Also these two entries in your torrc might help:
DisableOOSCheck 0 MaxMemInQueues 512 MB
The first line will enable the "out of sockets" check, and although it aggressively drops connections, tends to prevent the node from falling over instead. The second reduces the amount of memory used.
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Sun, 28 Jan 2018 at 11:06 teor <teor2345@gmail.com> wrote:
Try to make sure MaxMemInQueues allows 10-20s of traffic.
Hi teor
That advice is quite sensible in my opinion and should be incorporated into tor mainline. With the recent load spikes, I've always wanders why is there a need for that may MB or even GB of queue memory. If it can't be retransmitted in few seconds, it will be retransmitted by source and will increase the queues further... The current setting for maxmeminqueue is 256MB and will correct itself if lowered. I would suggest to make that value rate dependent, either configured or measured. To your knowledge, was there any thought put into such a dynamic config, instead of fixed percentage of free memory, independent of the actual throughput? Regards Seb
On 31 Jan 2018, at 10:45, r1610091651 <r1610091651@telenet.be> wrote:
On Sun, 28 Jan 2018 at 11:06 teor <teor2345@gmail.com> wrote:
Try to make sure MaxMemInQueues allows 10-20s of traffic.
Hi teor
That advice is quite sensible in my opinion and should be incorporated into tor mainline. With the recent load spikes, I've always wanders why is there a need for that may MB or even GB of queue memory. If it can't be retransmitted in few seconds, it will be retransmitted by source and will increase the queues further...
Tor doesn't work like TCP. Clients do give up and retry after about 10-20s. But if a circuit is broken by discarding cells, then the entire circuit needs to be rebuilt, and the request sent again. And the client will probably choose another path, that isn't as congested. A large MaxMemInQueues keeps circuits from breaking during traffic spikes. But you're right, if the whole network is overloaded, it leads to a lot of large buffers that don't do much.
The current setting for maxmeminqueue is 256MB and will correct itself if lowered. I would suggest to make that value rate dependent, either configured or measured.
To your knowledge, was there any thought put into such a dynamic config, instead of fixed percentage of free memory, independent of the actual throughput?
There's a ticket to make it lower: https://trac.torproject.org/projects/tor/ticket/24782 One problem with dynamic limits is that they don't handle traffic spikes well. So we don't want to make it too low, because if your relay has the RAM, it should use it. T
Thank you to all who responded to my question! The memory issue seems to be fixed now, and I would like to share what worked (and what didn’t). I first tried setting MaxMemInQueues to 512, then 384, while still running Tor 0.2.5.16. This didn’t help. Adding the option DisableOOSCheck 0 caused an error that prevented Tor from starting. I then upgraded Tor to 0.3.2.9. With this version, setting MaxMemInQeues to 384 seems to have fixed the problem, with no other changes to /etc/tor/torrc. Tor has been running for 29 hours with memory usage hovering around 85%, while average throughput is close to 10 megabytes per second. For any who are interested, additional details about the relay in question are available here: https://atlas.torproject.org/#details/0964EEEF3AEF8442F510F8A61370657BC6E0E0... George W. Maschke https://georgemaschke.net
On Jan 31, 2018, at 4:02 AM, teor <teor2345@gmail.com> wrote:
On 31 Jan 2018, at 10:45, r1610091651 <r1610091651@telenet.be <mailto:r1610091651@telenet.be>> wrote:
On Sun, 28 Jan 2018 at 11:06 teor <teor2345@gmail.com <mailto:teor2345@gmail.com>> wrote:
Try to make sure MaxMemInQueues allows 10-20s of traffic.
Hi teor
That advice is quite sensible in my opinion and should be incorporated into tor mainline. With the recent load spikes, I've always wanders why is there a need for that may MB or even GB of queue memory. If it can't be retransmitted in few seconds, it will be retransmitted by source and will increase the queues further...
Tor doesn't work like TCP. Clients do give up and retry after about 10-20s.
But if a circuit is broken by discarding cells, then the entire circuit needs to be rebuilt, and the request sent again. And the client will probably choose another path, that isn't as congested.
A large MaxMemInQueues keeps circuits from breaking during traffic spikes.
But you're right, if the whole network is overloaded, it leads to a lot of large buffers that don't do much.
The current setting for maxmeminqueue is 256MB and will correct itself if lowered. I would suggest to make that value rate dependent, either configured or measured.
To your knowledge, was there any thought put into such a dynamic config, instead of fixed percentage of free memory, independent of the actual throughput?
There's a ticket to make it lower: https://trac.torproject.org/projects/tor/ticket/24782 <https://trac.torproject.org/projects/tor/ticket/24782>
One problem with dynamic limits is that they don't handle traffic spikes well. So we don't want to make it too low, because if your relay has the RAM, it should use it.
T
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
georgemaschke@posteo.de hat am 28. Januar 2018 um 06:54 geschrieben:
I am running a Tor (0.2.5.16 on Debian Linux) relay on a VPS with one gigabyte of RAM and no swap partition or swap file. The hosting company states that it doesn't support swap in order to prolong the life of its solid state drives.
Before tweaking with the settings I would upgrade to the latest tor version.
participants (6)
-
George W. Maschke -
georgemaschke＠posteo.de
-
r1610091651 -
teor -
thomas＠akwgegner.de -
tor