MaxMemInQueues - per host, or per instance?
Hi, Is MaxMemInQueues parameter per-host (global) or per-instance? Say, there are 10 relays on the same 24 GB host. Should I set MaxMemInQueues to 20 GB, or 2 GB in each torrc? Thanks, Igor
It would expect it to be per instance. Instances are independent of each other. Further one can only run 2 instances max / ip. On Fri, 22 Dec 2017 at 20:40 Igor Mitrofanov <igor.n.mitrofanov@gmail.com> wrote:
Hi,
Is MaxMemInQueues parameter per-host (global) or per-instance? Say, there are 10 relays on the same 24 GB host. Should I set MaxMemInQueues to 20 GB, or 2 GB in each torrc?
Thanks, Igor _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Thanks. I do have the IP space. It is a pity multiple instances cannot watch the overall RAM remaining. I have quite a bit of RAM left, but there are large discrepancies in terms of how much RAM different relays are using (>3 GB for some, <1 GB for others), so it will be tricky to set MaxMemInQueues without making it too conservative. On Fri, Dec 22, 2017 at 11:46 AM, r1610091651 <r1610091651@telenet.be> wrote:
It would expect it to be per instance. Instances are independent of each other. Further one can only run 2 instances max / ip.
On Fri, 22 Dec 2017 at 20:40 Igor Mitrofanov <igor.n.mitrofanov@gmail.com> wrote:
Hi,
Is MaxMemInQueues parameter per-host (global) or per-instance? Say, there are 10 relays on the same 24 GB host. Should I set MaxMemInQueues to 20 GB, or 2 GB in each torrc?
Thanks, Igor _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
I'm wondering if it is necessary to have a lot of ram assigned to queues? Is there some rule of thumb to determine the proper sizing? Based on number of circuits maybe? Do the wise-minds have a guidance on this one? On Fri, 22 Dec 2017 at 21:08 Igor Mitrofanov <igor.n.mitrofanov@gmail.com> wrote:
Thanks. I do have the IP space.
It is a pity multiple instances cannot watch the overall RAM remaining. I have quite a bit of RAM left, but there are large discrepancies in terms of how much RAM different relays are using (>3 GB for some, <1 GB for others), so it will be tricky to set MaxMemInQueues without making it too conservative.
On Fri, Dec 22, 2017 at 11:46 AM, r1610091651 <r1610091651@telenet.be> wrote:
It would expect it to be per instance. Instances are independent of each other. Further one can only run 2 instances max / ip.
On Fri, 22 Dec 2017 at 20:40 Igor Mitrofanov < igor.n.mitrofanov@gmail.com> wrote:
Hi,
Is MaxMemInQueues parameter per-host (global) or per-instance? Say, there are 10 relays on the same 24 GB host. Should I set MaxMemInQueues to 20 GB, or 2 GB in each torrc?
Thanks, Igor _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On 22 Dec (20:37:37), r1610091651 wrote:
I'm wondering if it is necessary to have a lot of ram assigned to queues? Is there some rule of thumb to determine the proper sizing? Based on number of circuits maybe?
So there are probably many different answers to this or ways to look at it but I can speak on how "tor" is built and why it is important to have this memory limit assigned to queues. A tor relay gets cells in and most of the time will relay them so send them outbound. But for every cell that comes in, we need to do some processing on them that is mostly decryption work. So we get them, process then put them on a circuit queue. Then tor does its best to dequeue a "not too big amount of cells" from a circuit and puts them on the outbound connection buffers which, when the socket is writable, will be flushed onto the network (write to the socket). The MaxMemInQueues parameter basically tells the tor OOM handler when it is time to start cleaning up allocated memories. But here is the catch, it only handles cells on circuit queues, not connection's buffer (it actually handles other things but the majority of allocated data is in cells usually). For that reason, we are better off for now to keep relays with a sane value for MaxMemInQueues so the OOM is actually triggered before the load goes out of control. If that MaxMemInQueues value is not set in your torrc, tor will pick 3/4 of the total memory of your system. Usually, this is fine for most use cases but if you machine has 16GB of RAM but only 4GB are available, problem. So when setting it, it is not that easy to come up with a good value but a rule of thumb for now is look at how much memory you have available normally and estimate around it. It is also important to not go to low, a fast relay limited to 1GB for instance will start to degrade performance by killing cicuits more often if it sees 20MB/s (imperically speaking). I think we could do a better job at estimating it when not set, we could do a better job with the OOM, we could do lot more but unfortunately for now, this is the state of thing we need to deal with. We'll be trying to work on more DoS resistance feature hopefully in the near future. Hope this help! Cheers! David
Do the wise-minds have a guidance on this one?
On Fri, 22 Dec 2017 at 21:08 Igor Mitrofanov <igor.n.mitrofanov@gmail.com> wrote:
Thanks. I do have the IP space.
It is a pity multiple instances cannot watch the overall RAM remaining. I have quite a bit of RAM left, but there are large discrepancies in terms of how much RAM different relays are using (>3 GB for some, <1 GB for others), so it will be tricky to set MaxMemInQueues without making it too conservative.
On Fri, Dec 22, 2017 at 11:46 AM, r1610091651 <r1610091651@telenet.be> wrote:
It would expect it to be per instance. Instances are independent of each other. Further one can only run 2 instances max / ip.
On Fri, 22 Dec 2017 at 20:40 Igor Mitrofanov < igor.n.mitrofanov@gmail.com> wrote:
Hi,
Is MaxMemInQueues parameter per-host (global) or per-instance? Say, there are 10 relays on the same 24 GB host. Should I set MaxMemInQueues to 20 GB, or 2 GB in each torrc?
Thanks, Igor _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
-- tPcuU+9hl1BRjXh3xHhFgg22HULt2edIxY5kAKLBPPA=
So it will hurt Tor if we have too little buffers -> killing circuits -> unstable connections it will hurt Tor if we have too much buffers -> process out of control -> unstable instance Would it be possible to guestimate typical circuit buffer size? One could then estimate value not too big / small based on the number of circuit running through relay. If we where to say that a circuit would typically need 128k, one could estimate the setting for a 5K circuit relay to be ~600MB. Regards On Fri, 22 Dec 2017 at 22:55 David Goulet <dgoulet@torproject.org> wrote:
On 22 Dec (20:37:37), r1610091651 wrote:
I'm wondering if it is necessary to have a lot of ram assigned to queues? Is there some rule of thumb to determine the proper sizing? Based on number of circuits maybe?
So there are probably many different answers to this or ways to look at it but I can speak on how "tor" is built and why it is important to have this memory limit assigned to queues.
A tor relay gets cells in and most of the time will relay them so send them outbound. But for every cell that comes in, we need to do some processing on them that is mostly decryption work.
So we get them, process then put them on a circuit queue. Then tor does its best to dequeue a "not too big amount of cells" from a circuit and puts them on the outbound connection buffers which, when the socket is writable, will be flushed onto the network (write to the socket).
The MaxMemInQueues parameter basically tells the tor OOM handler when it is time to start cleaning up allocated memories. But here is the catch, it only handles cells on circuit queues, not connection's buffer (it actually handles other things but the majority of allocated data is in cells usually).
For that reason, we are better off for now to keep relays with a sane value for MaxMemInQueues so the OOM is actually triggered before the load goes out of control.
If that MaxMemInQueues value is not set in your torrc, tor will pick 3/4 of the total memory of your system. Usually, this is fine for most use cases but if you machine has 16GB of RAM but only 4GB are available, problem. So when setting it, it is not that easy to come up with a good value but a rule of thumb for now is look at how much memory you have available normally and estimate around it. It is also important to not go to low, a fast relay limited to 1GB for instance will start to degrade performance by killing cicuits more often if it sees 20MB/s (imperically speaking).
I think we could do a better job at estimating it when not set, we could do a better job with the OOM, we could do lot more but unfortunately for now, this is the state of thing we need to deal with. We'll be trying to work on more DoS resistance feature hopefully in the near future.
Hope this help!
Cheers! David
Do the wise-minds have a guidance on this one?
On Fri, 22 Dec 2017 at 21:08 Igor Mitrofanov <
igor.n.mitrofanov@gmail.com>
wrote:
Thanks. I do have the IP space.
It is a pity multiple instances cannot watch the overall RAM remaining. I have quite a bit of RAM left, but there are large discrepancies in terms of how much RAM different relays are using (>3 GB for some, <1 GB for others), so it will be tricky to set MaxMemInQueues without making it too conservative.
On Fri, Dec 22, 2017 at 11:46 AM, r1610091651 <r1610091651@telenet.be> wrote:
It would expect it to be per instance. Instances are independent of each other. Further one can only run 2 instances max / ip.
On Fri, 22 Dec 2017 at 20:40 Igor Mitrofanov < igor.n.mitrofanov@gmail.com> wrote:
Hi,
Is MaxMemInQueues parameter per-host (global) or per-instance? Say, there are 10 relays on the same 24 GB host. Should I set MaxMemInQueues to 20 GB, or 2 GB in each torrc?
Thanks, Igor _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
-- tPcuU+9hl1BRjXh3xHhFgg22HULt2edIxY5kAKLBPPA= _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On 23 Dec 2017, at 10:59, r1610091651 <r1610091651@telenet.be> wrote:
So it will hurt Tor if we have too little buffers -> killing circuits -> unstable connections it will hurt Tor if we have too much buffers -> process out of control -> unstable instance
Would it be possible to guestimate typical circuit buffer size?
Not really.
One could then estimate value not too big / small based on the number of circuit running through relay.
The number of circuits varies over time based on client load and relay capacity.
If we where to say that a circuit would typically need 128k, one could estimate the setting for a 5K circuit relay to be ~600MB.
On my machines, relays typically use 800 MB, and exits use about 1 GB. But it varies a lot. So it's better to tell Tor how much RAM you have available. Then you can let the network load balancing send the right amount of client circuits your way. For example, I have a relay with 32 GB of RAM. The RAM and CPU usage spike at times, but it's fine :-) T
participants (4)
-
David Goulet -
Igor Mitrofanov -
r1610091651 -
teor