I am running tor 4.0.1-alpha inside a docker container with a memory limit (6GB). Tor runs out of memory and aborts or crashes periodically.
Usually I see an error message like this: Could not mmap file "/var/lib/tor/data/diff-cache/1149": Out of memory ...(repeated some number of times) followed by segfault
Sometimes I see a message: Out of memory on malloc(). Dying followed by abort.
Am I correct to assume the diff-cache is the issue here? Looking at the files it seems they are all pretty small (~500K). Is some badly behaved client requesting 12,000 of these diffs causing my relay to mmap them all at once? or is it just expensive to generate and generating 30-60 at once is enough to use all the memory?
Are there any config options to reject expensive queries or otherwise limit concurrency?
Thanks
On January 27, 2019 1:07:12 PM UTC, notatorserver notatorserver@protonmail.com wrote:
I am running tor 4.0.1-alpha inside a docker container with a memory limit (6GB). Tor runs out of memory and aborts or crashes periodically.
Tor assumes that 64-bit machines have 8 GB of RAM, unless the relevant APIs return a lower value.
How does docker implement memory limits? Does it modify the values returned by the Linux RAM APIs?
Usually I see an error message like this: Could not mmap file "/var/lib/tor/data/diff-cache/1149": Out of memory ...(repeated some number of times)
Please paste your log on https://paste.debian.net
How many times are these messages repeated? Is the diff number the same, or different? How many files are in that directory?
followed by segfault
Sometimes I see a message: Out of memory on malloc(). Dying followed by abort.
Am I correct to assume the diff-cache is the issue here? Looking at the files it seems they are all pretty small (~500K). Is some badly behaved client requesting 12,000 of these diffs causing my relay to mmap them all at once?
How many times are these messages repeated?
or is it just expensive to generate and generating 30-60 at once is enough to use all the memory?
The diffs are generated once an hour, then cached for requests. Do the errors happen at generation time, or request time?
Are there any config options to reject expensive queries or otherwise limit concurrency?
Try MaxMemInQueues 4 GB
If that doesn't work, try NumCPUs 2 and please let us know, because we'd like to fix these kinds of bugs.
T
-- teor ----------------------------------------------------------------------
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Sunday, January 27, 2019 5:37 PM, teor teor@riseup.net wrote:
On January 27, 2019 1:07:12 PM UTC, notatorservernotatorserver@protonmail.com wrote:
I am running tor 4.0.1-alpha inside a docker container with a memory limit (6GB). Tor runs out of memory and aborts or crashes periodically.
Tor assumes that 64-bit machines have 8 GB of RAM, unless the relevant APIs return a lower value.
How does docker implement memory limits? Does it modify the values returned by the Linux RAM APIs?
Docker sets up a cgroup with a memory limit. I guess you could check the value: `cat /sys/fs/cgroup/memory/memory.limit_in_bytes`. The value reported in `cat /proc/meminfo` is the same for every container, i.e. host system value. So no, it does not modify the values.
Usually I see an error message like this: Could not mmap file "/var/lib/tor/data/diff-cache/1149": Out of memory ...(repeated some number of times)
Please paste your log onhttps://paste.debian.net
here's a sample of logs: http://paste.debian.net/1062943/ I can capture some more if these are helpful. Retention period is short
How many times are these messages repeated?
Depends but it can be anywhere from 1 to ~20.
Is the diff number the same, or different?
diff number is increasing
How many files are in that directory?
currently 259 files (~123MB)
followed by segfault Sometimes I see a message: Out of memory on malloc(). Dying followed by abort. Am I correct to assume the diff-cache is the issue here? Looking at the files it seems they are all pretty small (~500K). Is some badly behaved client requesting 12,000 of these diffs causing my relay to mmap them all at once?
How many times are these messages repeated?
or is it just expensive to generate and generating 30-60 at once is enough to use all the memory?
The diffs are generated once an hour, then cached for requests. Do the errors happen at generation time, or request time?
I guess at generation time if there is no way for a client to request to generate many diffs.
Looking at mtime of these files it seems they are created in bursts: / # stat /var/lib/tor/data/diff-cache/* | grep Modify | sort | uniq -c ... 24 Modify: 2019-01-28 17:15:30.000000000 6 Modify: 2019-01-28 17:15:31.000000000 15 Modify: 2019-01-28 17:15:32.000000000 6 Modify: 2019-01-28 17:15:33.000000000 12 Modify: 2019-01-28 17:15:34.000000000 21 Modify: 2019-01-28 17:15:35.000000000 1 Modify: 2019-01-28 17:20:27.000000000 2 Modify: 2019-01-28 18:03:27.000000000 21 Modify: 2019-01-28 18:03:28.000000000 6 Modify: 2019-01-28 18:03:29.000000000 9 Modify: 2019-01-28 18:03:30.000000000 18 Modify: 2019-01-28 18:03:31.000000000 3 Modify: 2019-01-28 18:03:32.000000000 6 Modify: 2019-01-28 18:03:33.000000000 18 Modify: 2019-01-28 18:03:34.000000000 18 Modify: 2019-01-28 18:03:35.000000000
Are there any config options to reject expensive queries or otherwise limit concurrency?
Try MaxMemInQueues 4 GB
I already have max mem set to 5GB but I will set to 4GB.
If that doesn't work, try NumCPUs 2 and please let us know, because we'd like to fix these kinds of bugs.
I will try this also.
Thanks
T
teor
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
tor-relays@lists.torproject.org
-
notatorserver -
teor