assign_to_cpuworker failed
Hello! What does this warning mean? "Jan 13 09:31:49.000 [warn] assign_to_cpuworker failed. Ignoring." Do I need to reduce the number of connection to the relay or could I ignore this message? Regards,
On 14 Jan 2017, at 06:18, diffusae <punasipuli@t-online.de> wrote:
Hello!
What does this warning mean?
"Jan 13 09:31:49.000 [warn] assign_to_cpuworker failed. Ignoring."
When tor tried to give a task (cell decryption) to a cpuworker, it didn't work. This can happen for a few different reasons.
Do I need to reduce the number of connection to the relay or could I ignore this message?
Check the other warnings/notices near this warning. If there aren't any, then there's no problem. T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
Thanks for your reply. On 13.01.2017 23:15, teor wrote:
Check the other warnings/notices near this warning. If there aren't any, then there's no problem.
The only warning I have found close to it: "Jan 13 11:08:46.000 [warn] Your system clock just jumped 216 seconds forward; assuming established circuits no longer work" That could be due to the IPv4 autodetection? Maybe I should explicitly set the Address option in torrc? Regards,
you have a ntp daemon running? Markus
On 14 Jan 2017, at 17:30, diffusae <punasipuli@t-online.de> wrote:
Thanks for your reply.
On 13.01.2017 23:15, teor wrote:
Check the other warnings/notices near this warning. If there aren't any, then there's no problem.
The only warning I have found close to it:
"Jan 13 11:08:46.000 [warn] Your system clock just jumped 216 seconds forward; assuming established circuits no longer work"
That could be due to the IPv4 autodetection? Maybe I should explicitly set the Address option in torrc?
Regards, _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Yes I do. Should I disable it? ;-) On 14.01.2017 17:32, niftybunny wrote:
you have a ntp daemon running?
Markus
On 14 Jan 2017, at 17:30, diffusae <punasipuli@t-online.de> wrote:
Thanks for your reply.
On 13.01.2017 23:15, teor wrote:
Check the other warnings/notices near this warning. If there aren't any, then there's no problem.
The only warning I have found close to it:
"Jan 13 11:08:46.000 [warn] Your system clock just jumped 216 seconds forward; assuming established circuits no longer work"
That could be due to the IPv4 autodetection? Maybe I should explicitly set the Address option in torrc?
Regards, _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
diffusae:
What does this warning mean?
"Jan 13 09:31:49.000 [warn] assign_to_cpuworker failed. Ignoring."
Do I need to reduce the number of connection to the relay or could I ignore this message?
As long as you don't have any warning around this one (clock jump is unrelated), I can say that if you had info log level you would see `circ->p_chan gone. Failing circ.` I guess that's just means that this circ got broken at the wrong time. This happens, but if you have many circuits it should be more visible. That's ok. btw, what is the number of connections on your relay (maybe I'm wrong)? -- Ivan Markin
Hi! Thanks for reply. On 14.01.2017 18:40, Ivan Markin wrote:
diffusae:
What does this warning mean?
"Jan 13 09:31:49.000 [warn] assign_to_cpuworker failed. Ignoring."
Do I need to reduce the number of connection to the relay or could I ignore this message?
As long as you don't have any warning around this one (clock jump is unrelated), I can say that if you had info log level you would see `circ->p_chan gone. Failing circ.` I guess that's just means that this circ got broken at the wrong time. This happens, but if you have many circuits it should be more visible. That's ok. btw, what is the number of connections on your relay (maybe I'm wrong)?
-- Ivan Markin
Yes, that depends. Normally it's around 400 or a bit more connections, but sometimes (with HSDir Flag) it jumps to 1000 or 2000 connections. Occasionally the relay gets unresponsive (very rare) or tor service stops working (only one time). Regards,
Got it too, Sooo many lines in my log file. [...] Jan 22 06:37:37.000 [warn] assign_to_cpuworker failed. Ignoring. Jan 22 06:37:37.000 [warn] assign_to_cpuworker failed. Ignoring. Jan 22 06:37:37.000 [warn] circuit_mark_for_close_(): Bug: Duplicate call to circuit_mark_for_close at ../src/or/onion.c:238 (first at ../src/or/command.c:579) (on Tor 0.2.9.8 ) Jan 22 06:37:37.000 [warn] circuit_mark_for_close_(): Bug: Duplicate call to circuit_mark_for_close at ../src/or/onion.c:238 (first at ../src/or/command.c:579) (on Tor 0.2.9.8 ) Jan 22 06:37:37.000 [warn] circuit_mark_for_close_(): Bug: Duplicate call to circuit_mark_for_close at ../src/or/onion.c:238 (first at ../src/or/command.c:579) (on Tor 0.2.9.8 ) [...] Then 1 Tor process has crashed and restarted himself... The other Tor instance has not been this log... NTP daemon is running too on this node. I'll keep an eye on this log file...
Hello!
What does this warning mean?
"Jan 13 09:31:49.000 [warn] assign_to_cpuworker failed. Ignoring."
Do I need to reduce the number of connection to the relay or could I ignore this message?
Regards,
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
-- Petrusko C0BF 2184 4A77 4A18 90E9 F72C B3CA E665 EBE2 3AE5
Hi! Yes, since I set the "Address "line in torrc, this message was gone. But ... I nevertheless get the "clock jumped" warning once a day: "Jan 21 18:34:53.000 [warn] Your system clock just jumped 398 seconds forward; assuming established circuits no longer work." 3-6 minutes difference is quite much. I also have a ntp daemon running and don't know why this happens. The clock is synced? Regards, On 22.01.2017 15:59, Petrusko wrote:
Got it too, Sooo many lines in my log file. [...] Jan 22 06:37:37.000 [warn] assign_to_cpuworker failed. Ignoring. Jan 22 06:37:37.000 [warn] assign_to_cpuworker failed. Ignoring. Jan 22 06:37:37.000 [warn] circuit_mark_for_close_(): Bug: Duplicate call to circuit_mark_for_close at ../src/or/onion.c:238 (first at ../src/or/command.c:579) (on Tor 0.2.9.8 ) Jan 22 06:37:37.000 [warn] circuit_mark_for_close_(): Bug: Duplicate call to circuit_mark_for_close at ../src/or/onion.c:238 (first at ../src/or/command.c:579) (on Tor 0.2.9.8 ) Jan 22 06:37:37.000 [warn] circuit_mark_for_close_(): Bug: Duplicate call to circuit_mark_for_close at ../src/or/onion.c:238 (first at ../src/or/command.c:579) (on Tor 0.2.9.8 ) [...]
Then 1 Tor process has crashed and restarted himself... The other Tor instance has not been this log...
NTP daemon is running too on this node.
I'll keep an eye on this log file...
Hello!
What does this warning mean?
"Jan 13 09:31:49.000 [warn] assign_to_cpuworker failed. Ignoring."
Do I need to reduce the number of connection to the relay or could I ignore this message?
Regards,
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
diffusae:
I nevertheless get the "clock jumped" warning once a day:
"Jan 21 18:34:53.000 [warn] Your system clock just jumped 398 seconds forward; assuming established circuits no longer work."
3-6 minutes difference is quite much. I also have a ntp daemon running and don't know why this happens. The clock is synced?
This may not be related to this issue. "clock jump" may not be a clock jump actually. E.g. one can just "pause" all TCP connections for some time (e.g. on a firewall/DPI/traffic shaper) and then resume them. In such case tor will regard this sleep/resume in the main loop as a "clock jump" thus breaking all circuits. This is because it produces ticks each second in the same thread as the main loop (which is blocking awaiting for data on a socket). There should be a ticket for this, maybe someone can arrange it. -- Ivan Markin
Hi! Thanks for your explanation. On 22.01.2017 16:54, Ivan Markin wrote:
This may not be related to this issue.
Yes, it looks like.
"clock jump" may not be a clock jump actually. E.g. one can just "pause" all TCP connections for some time (e.g. on a firewall/DPI/traffic shaper) and then resume them. In such case tor will regard this
That might be possible. There is no special FW rule, but the router also uses traffic shaping.
sleep/resume in the main loop as a "clock jump" thus breaking all circuits. This is because it produces ticks each second in the same thread as the main loop (which is blocking awaiting for data on a socket). There should be a ticket for this, maybe someone can arrange it.
It doesn't seem to hurt now. I will keep an eyes on it. Regards
-- Ivan Markin
Petrusko:
Got it too, Sooo many lines in my log file. [...] Jan 22 06:37:37.000 [warn] assign_to_cpuworker failed. Ignoring. Jan 22 06:37:37.000 [warn] assign_to_cpuworker failed. Ignoring. Jan 22 06:37:37.000 [warn] circuit_mark_for_close_(): Bug: Duplicate call to circuit_mark_for_close at ../src/or/onion.c:238 (first at ../src/or/command.c:579) (on Tor 0.2.9.8 ) Jan 22 06:37:37.000 [warn] circuit_mark_for_close_(): Bug: Duplicate call to circuit_mark_for_close at ../src/or/onion.c:238 (first at ../src/or/command.c:579) (on Tor 0.2.9.8 ) Jan 22 06:37:37.000 [warn] circuit_mark_for_close_(): Bug: Duplicate call to circuit_mark_for_close at ../src/or/onion.c:238 (first at ../src/or/command.c:579) (on Tor 0.2.9.8 ) [...]
Then 1 Tor process has crashed and restarted himself... The other Tor instance has not been this log...
On what hardware/OS this relay is running? It seems to be due to hitting resource limits. { src/or/onion.c:238 + "assign_to_cpuworker failed" error } Though neither circuit_mark_for_close() should be called twice nor tor process should crash. Do you see other warnings? Messages about the reason of tor termination? btw, there is a ticket for this very issue: https://trac.torproject.org/projects/tor/ticket/20059 As long as it happens at the same lines of code I guess all this has common reason. "assign_to_cpuworker" error is a kinda consequence of hitting rate-limits (likely RAM usage). Probably there is a memory leakage somewhere that makes everything fail and get process eventually killed by OS. -- Ivan Markin
You're right Ivan, my bad ! Swap has grown quickly and has been full... Ok, it was a test with another instance... so I'll kill this other instance :( Thx for your help Ivan, next time, I'll check my graphs :s Nice shot ;) Ivan Markin :
Probably there is a memory leakage somewhere that makes everything fail and get process eventually killed by OS.
-- Petrusko C0BF 2184 4A77 4A18 90E9 F72C B3CA E665 EBE2 3AE5
Petrusko:
Probably there is a memory leakage somewhere that makes everything fail and get process eventually killed by OS.
You're right Ivan, my bad ! Swap has grown quickly and has been full... Ok, it was a test with another instance... so I'll kill this other instance :(
There is nothing wrong at your side. You're probably experiencing the same issue as in ticket I've mentioned earlier. "a memory leakage somewhere" means that this "somewhere" is a place in tor code and probably triggered remotely. This definitely ought to be fixed since it may be a DoS vulnerability (process crash). So if you have some details on this issue please report them to the mentioned ticket. Thanks, -- Ivan Markin
Thx Ivan for your support. I got an eye on the logs and everything around. ps: updated to 2.9.9 some hours ago... looks like ok for now. Ivan Markin :
There is nothing wrong at your side. You're probably experiencing the same issue as in ticket I've mentioned earlier. "a memory leakage somewhere" means that this "somewhere" is a place in tor code and probably triggered remotely. This definitely ought to be fixed since it may be a DoS vulnerability (process crash). So if you have some details on this issue please report them to the mentioned ticket.
Thanks,
-- Petrusko C0BF 2184 4A77 4A18 90E9 F72C B3CA E665 EBE2 3AE5
Hi! Didn't update right now and got the same message today. So, it looks like, the address field wasn't the problem. Feb 05 15:01:25.000 [warn] assign_to_cpuworker failed. Ignoring. Feb 05 15:01:29.000 [warn] circuit_mark_for_close_: Bug: Duplicate call to circuit_mark_for_close at src/or/onion.c:238 (first at src/or/command.c:579) (on Tor 0.2.9.8 01ab67e38b358ae9) Feb 05 15:01:36.000 [warn] assign_to_cpuworker failed. Ignoring. Should I I update to 2.9.9, does it solve the issue. Regards, On 24.01.2017 20:06, Petrusko wrote:
Thx Ivan for your support. I got an eye on the logs and everything around.
ps: updated to 2.9.9 some hours ago... looks like ok for now.
Ivan Markin :
There is nothing wrong at your side. You're probably experiencing the same issue as in ticket I've mentioned earlier. "a memory leakage somewhere" means that this "somewhere" is a place in tor code and probably triggered remotely. This definitely ought to be fixed since it may be a DoS vulnerability (process crash). So if you have some details on this issue please report them to the mentioned ticket.
Thanks,
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
It's running fine since this last upgrade, on my case. (I've reduced RAM used by shutting down an instance... no problem, full bandwidth is used now!) Good luck ;) diffusae :
Hi!
Didn't update right now and got the same message today. So, it looks like, the address field wasn't the problem.
Feb 05 15:01:25.000 [warn] assign_to_cpuworker failed. Ignoring. Feb 05 15:01:29.000 [warn] circuit_mark_for_close_: Bug: Duplicate call to circuit_mark_for_close at src/or/onion.c:238 (first at src/or/command.c:579) (on Tor 0.2.9.8 01ab67e38b358ae9) Feb 05 15:01:36.000 [warn] assign_to_cpuworker failed. Ignoring.
Should I I update to 2.9.9, does it solve the issue.
Regards,
-- Petrusko C0BF 2184 4A77 4A18 90E9 F72C B3CA E665 EBE2 3AE5
Hi! Thanks for your answer. Just compiling the "new" version. If the problem persists, than I would also shut down one instance. The full configured bandwidth was used, but it really wasn't stable. I got this message once a week or so ... I will give a feedback soon. ;-) Regards, On 07.02.2017 18:52, Petrusko wrote:
It's running fine since this last upgrade, on my case. (I've reduced RAM used by shutting down an instance... no problem, full bandwidth is used now!)
Good luck ;)
diffusae :
Hi!
Didn't update right now and got the same message today. So, it looks like, the address field wasn't the problem.
Feb 05 15:01:25.000 [warn] assign_to_cpuworker failed. Ignoring. Feb 05 15:01:29.000 [warn] circuit_mark_for_close_: Bug: Duplicate call to circuit_mark_for_close at src/or/onion.c:238 (first at src/or/command.c:579) (on Tor 0.2.9.8 01ab67e38b358ae9) Feb 05 15:01:36.000 [warn] assign_to_cpuworker failed. Ignoring.
Should I I update to 2.9.9, does it solve the issue.
Regards,
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Ahh, I see ... I've used this for both instances: MaxMemInQueues 400 MB I guess, that I have to reduce it. Regards, On 07.02.2017 18:52, Petrusko wrote:
It's running fine since this last upgrade, on my case. (I've reduced RAM used by shutting down an instance... no problem, full bandwidth is used now!)
Good luck ;)
diffusae :
Hi!
Didn't update right now and got the same message today. So, it looks like, the address field wasn't the problem.
Feb 05 15:01:25.000 [warn] assign_to_cpuworker failed. Ignoring. Feb 05 15:01:29.000 [warn] circuit_mark_for_close_: Bug: Duplicate call to circuit_mark_for_close at src/or/onion.c:238 (first at src/or/command.c:579) (on Tor 0.2.9.8 01ab67e38b358ae9) Feb 05 15:01:36.000 [warn] assign_to_cpuworker failed. Ignoring.
Should I I update to 2.9.9, does it solve the issue.
Regards,
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
For me, I've set nothing, I'm leaving it automatic, and usually Tor understand how much RAM free there's in the machine ;) Have fun with source compil ! diffusae :
I've used this for both instances: MaxMemInQueues 400 MB I guess, that I have to reduce it.
Regards,
-- Petrusko C0BF 2184 4A77 4A18 90E9 F72C B3CA E665 EBE2 3AE5
Since, today I got this warning: Feb 07 20:01:54.000 [warn] Please upgrade! This version of Tor (0.2.9.8) is not recommended, according to the directory authorities. Recommended versions are: 0.2.4.27,0.2.4.28,0.2.5.12,0.2.5.13,0.2.7.6,0.2.7.7,0.2.8.9,0.2.8.10,0.2.8.11,0.2.8.12,0.2.9.9,0.3.0.2-alpha,0.3.0.3-alpha I guess everything would be fine after upgrading. Regards, On 07.02.2017 18:52, Petrusko wrote:
It's running fine since this last upgrade, on my case. (I've reduced RAM used by shutting down an instance... no problem, full bandwidth is used now!)
Good luck ;)
diffusae :
Hi!
Didn't update right now and got the same message today. So, it looks like, the address field wasn't the problem.
Feb 05 15:01:25.000 [warn] assign_to_cpuworker failed. Ignoring. Feb 05 15:01:29.000 [warn] circuit_mark_for_close_: Bug: Duplicate call to circuit_mark_for_close at src/or/onion.c:238 (first at src/or/command.c:579) (on Tor 0.2.9.8 01ab67e38b358ae9) Feb 05 15:01:36.000 [warn] assign_to_cpuworker failed. Ignoring.
Should I I update to 2.9.9, does it solve the issue.
Regards,
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
participants (5)
-
diffusae -
Ivan Markin -
niftybunny -
Petrusko -
teor