Debugging relays not showing online
Hi all, I am looking for some help in debugging a strange one.... EO runs ~30ish relays in our /24; but we currently have 4 relays that don't want to play nice and are reporting offline (although look to be online). The effected relay IPs are: 23.129.64.157 23.129.64.164 23.129.64.170 23.129.64.174 The first 3 do show on Metrics, and when they are restarted or rebuilt (we use relayor), metrics does pick it up.. in terms that its "uptime" is updated to reflect the restart, although still marked offline. The 4th - doesn't show at all... it is in the family members though. Looking more closely at: 23.129.64.157 (183C8C6727E2137AF278B3850AD5D9C2304B98C9); using nyx with debug messaging; 2 hours after a restart, I can still see active circuits, and I can see it doing "things" [logs bellow], and http://23.129.64.157 loads (we run the exit page). Is there anything I could be doing or look at to indicate what might be creating such a strange issue? All the other relays in the family seem to be working fine.. https://metrics.torproject.org/rs.html#search/family:183C8C6727E2137AF278B38... (and all built and managed with relayor using offline keys..). These have been online for about 10 days now. Help / guidance would be greatly appraised... Thanks, Jake Events (DEBUG-ERR, NYX NOTICE-ERR): x 10:48:54 [DEBUG] conn_read_callback: socket 18 wants to read. [181 duplicates hidden] x 10:48:50 [DEBUG] connection_or_process_cells_from_inbuf: 14: starting, inbuf_datalen 0 (0 pending in tls object). [83 duplicates hidden] x 10:48:50 [DEBUG] connection_or_process_cells_from_inbuf: 14: starting, inbuf_datalen 514 (0 pending in tls object). [83 duplicates hidden] x 10:48:50 [DEBUG] connection_buf_read_from_socket: After TLS read of 514: 543 read, 0 written [99 duplicates hidden] x 10:48:50 [DEBUG] connection_buf_read_from_socket: 14: starting, inbuf_datalen 0 (0 pending in tls object). at_most 16448. [105 duplicates hidden] x 10:48:50 [DEBUG] conn_read_callback: socket 14 wants to read. [83 duplicates hidden] x 10:48:43 [INFO] update_consensus_router_descriptor_downloads: 0 router descriptors downloadable. 0 delayed; 6722 present (0 of those were in old_routers); 0 would_reject; 0 wouldnt_use; 0 in progress. [9 duplicates hidden] x 10:48:39 [DEBUG] connection_or_process_cells_from_inbuf: 25: starting, inbuf_datalen 0 (0 pending in tls object). [1 duplicate hidden] x 10:48:39 [DEBUG] connection_or_process_cells_from_inbuf: 25: starting, inbuf_datalen 514 (0 pending in tls object). [1 duplicate hidden] x 10:48:39 [DEBUG] conn_read_callback: socket 25 wants to read. [1 duplicate hidden] x 10:48:39 [DEBUG] connection_or_process_cells_from_inbuf: 17: starting, inbuf_datalen 0 (0 pending in tls object). [1 duplicate hidden] x 10:48:39 [DEBUG] connection_or_process_cells_from_inbuf: 17: starting, inbuf_datalen 514 (0 pending in tls object). [1 duplicate hidden] x 10:48:39 [DEBUG] conn_read_callback: socket 17 wants to read. [1 duplicate hidden] x 10:48:39 [DEBUG] connection_handle_write_impl: After TLS write of 514: 0 read, 543 written [17 duplicates hidden] x 10:48:39 [DEBUG] flush_chunk_tls: flushed 514 bytes, 0 ready to flush, 0 remain. [17 duplicates hidden] x 10:48:39 [DEBUG] conn_write_callback: socket 25 wants to write. [1 duplicate hidden] x 10:48:39 [DEBUG] conn_write_callback: socket 17 wants to write. [1 duplicate hidden] x 10:48:39 [DEBUG] run_connection_housekeeping: Sending keepalive to (199.58.81.140:443) [1 duplicate hidden] x 10:48:39 [DEBUG] run_connection_housekeeping: Sending keepalive to (154.35.175.225:443) [1 duplicate hidden] x 10:48:16 [INFO] routerlist_remove_old_routers: We have 6869 live routers and 820 old router descriptors. [8 duplicates hidden] x 10:48:16 [DEBUG] resolve_my_address: Resolved Address to '23.129.64.157'. [8 duplicates hidden] x 10:48:13 [DEBUG] connection_or_process_cells_from_inbuf: 23: starting, inbuf_datalen 0 (0 pending in tls object). [1 duplicate hidden] x 10:48:13 [DEBUG] connection_or_process_cells_from_inbuf: 23: starting, inbuf_datalen 514 (0 pending in tls object). [1 duplicate hidden] x 10:48:13 [DEBUG] conn_read_callback: socket 23 wants to read. [1 duplicate hidden] x 10:48:13 [DEBUG] conn_write_callback: socket 23 wants to write. [1 duplicate hidden] x 10:48:13 [DEBUG] run_connection_housekeeping: Sending keepalive to (128.31.0.34:9101) [1 duplicate hidden] x 10:48:09 [DEBUG] conn_write_callback: socket 14 wants to write. [1 duplicate hidden] x 10:48:09 [DEBUG] run_connection_housekeeping: Sending keepalive to (217.20.130.72:9001) [1 duplicate hidden] x 10:46:53 [DEBUG] connection_or_process_cells_from_inbuf: 21: starting, inbuf_datalen 0 (0 pending in tls object). [1 duplicate hidden] x 10:46:53 [DEBUG] connection_or_process_cells_from_inbuf: 21: starting, inbuf_datalen 514 (0 pending in tls object). [1 duplicate hidden] x 10:46:53 [DEBUG] conn_read_callback: socket 21 wants to read. [1 duplicate hidden] x 10:46:53 [DEBUG] conn_write_callback: socket 21 wants to write. [1 duplicate hidden] x 10:46:53 [DEBUG] run_connection_housekeeping: Sending keepalive to (194.109.206.212:443) [1 duplicate hidden] x 10:46:40 [DEBUG] connection_or_process_cells_from_inbuf: 20: starting, inbuf_datalen 0 (0 pending in tls object). [1 duplicate hidden] x 10:46:40 [DEBUG] connection_or_process_cells_from_inbuf: 20: starting, inbuf_datalen 514 (0 pending in tls object). [1 duplicate hidden] x 10:46:40 [DEBUG] conn_read_callback: socket 20 wants to read. [1 duplicate hidden] x 10:46:40 [DEBUG] conn_write_callback: socket 20 wants to write. [1 duplicate hidden] x 10:46:40 [DEBUG] run_connection_housekeeping: Sending keepalive to (171.25.193.9:80) [1 duplicate hidden] x 10:46:26 [DEBUG] connection_or_process_cells_from_inbuf: 19: starting, inbuf_datalen 0 (0 pending in tls object). [1 duplicate hidden] x 10:46:26 [DEBUG] connection_or_process_cells_from_inbuf: 19: starting, inbuf_datalen 514 (0 pending in tls object). [1 duplicate hidden] x 10:46:26 [DEBUG] conn_read_callback: socket 19 wants to read. [1 duplicate hidden] x 10:46:26 [DEBUG] conn_write_callback: socket 19 wants to write. [1 duplicate hidden] x 10:46:26 [DEBUG] run_connection_housekeeping: Sending keepalive to (204.13.164.118:443) [1 duplicate hidden] x 10:46:23 [DEBUG] connection_or_process_cells_from_inbuf: 22: starting, inbuf_datalen 0 (0 pending in tls object). [1 duplicate hidden] x 10:46:23 [DEBUG] connection_or_process_cells_from_inbuf: 22: starting, inbuf_datalen 514 (0 pending in tls object). [1 duplicate hidden] x 10:46:23 [DEBUG] conn_read_callback: socket 22 wants to read. [1 duplicate hidden] x 10:46:22 [DEBUG] conn_write_callback: socket 22 wants to write. [1 duplicate hidden] x 10:46:22 [DEBUG] run_connection_housekeeping: Sending keepalive to (86.59.21.38:443) [1 duplicate hidden] x 10:46:10 [DEBUG] consdiffmgr_cleanup: Looking for consdiffmgr entries to remove qj 10:45:58 [DEBUG] connection_or_process_cells_from_inbuf: 15: starting, inbuf_datalen 0 (0 pending in tls object). [1 duplicate hidden]
On Thu, Apr 18, 2019 at 05:52:48PM +0000, Jake Visser wrote:
I am looking for some help in debugging a strange one.... EO runs ~30ish relays in our /24; but we currently have 4 relays that don't want to play nice and are reporting offline (although look to be online).
The effected relay IPs are:
23.129.64.157 23.129.64.164 23.129.64.170 23.129.64.174
Whenever I see strange issues like this, my first thought is whether the relay advertises an IPv6 address, yet it's unreachable there. Tor is pretty good at self-testing its IPv4 addresses, but it just trusts you that your IPv6 address works. And then only some of the directory authorities have IPv6 support, but these days it's a majority of them, so you won't get the Running flag (i.e. won't be in the consensus) if your advertised IPv6 address doesn't work. Hope that does it! --Roger
Thanks Roger - that was a palm facing moment for me... I thought I had all the ND issues with IPv6 fixed, but apparently not... those 4 were not accessible externally on V6. I just corrected and will see if they start registering 😊 Thanks again, Jake -----Original Message----- From: tor-relays <tor-relays-bounces@lists.torproject.org> On Behalf Of Roger Dingledine Sent: Thursday, April 18, 2019 11:11 AM To: tor-relays@lists.torproject.org Subject: Re: [tor-relays] Debugging relays not showing online On Thu, Apr 18, 2019 at 05:52:48PM +0000, Jake Visser wrote:
I am looking for some help in debugging a strange one.... EO runs ~30ish relays in our /24; but we currently have 4 relays that don't want to play nice and are reporting offline (although look to be online).
The effected relay IPs are:
23.129.64.157 23.129.64.164 23.129.64.170 23.129.64.174
Whenever I see strange issues like this, my first thought is whether the relay advertises an IPv6 address, yet it's unreachable there. Tor is pretty good at self-testing its IPv4 addresses, but it just trusts you that your IPv6 address works. And then only some of the directory authorities have IPv6 support, but these days it's a majority of them, so you won't get the Running flag (i.e. won't be in the consensus) if your advertised IPv6 address doesn't work. Hope that does it! --Roger _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Jake Visser wrote:
Thanks Roger - that was a palm facing moment for me... I thought I had all the ND issues with IPv6 fixed, but apparently not... those 4 were not accessible externally on V6.
I just corrected and will see if they start registering 😊
Thanks for running relays! Well, it happens. Wasn't your fault entirely, Tor currently doesn't have an IPv6 discovery and reach-ability testing mechanism, but hopefully will have soon. To make it even worse, Tor allows you to specify an advertised IPv6 address even if you have no IPv6 listening socket on the host running the Tor instance: https://trac.torproject.org/projects/tor/ticket/29570 Some think this is not a bug, and it should stay like this. I think it only makes such accidental configurations or mistakes harder to track down by operators.
Thanks again,
Jake
-----Original Message----- From: tor-relays <tor-relays-bounces@lists.torproject.org> On Behalf Of Roger Dingledine Sent: Thursday, April 18, 2019 11:11 AM To: tor-relays@lists.torproject.org Subject: Re: [tor-relays] Debugging relays not showing online
On Thu, Apr 18, 2019 at 05:52:48PM +0000, Jake Visser wrote:
I am looking for some help in debugging a strange one.... EO runs ~30ish relays in our /24; but we currently have 4 relays that don't want to play nice and are reporting offline (although look to be online).
The effected relay IPs are:
23.129.64.157 23.129.64.164 23.129.64.170 23.129.64.174
Whenever I see strange issues like this, my first thought is whether the relay advertises an IPv6 address, yet it's unreachable there.
Tor is pretty good at self-testing its IPv4 addresses, but it just trusts you that your IPv6 address works.
And then only some of the directory authorities have IPv6 support, but these days it's a majority of them, so you won't get the Running flag (i.e. won't be in the consensus) if your advertised IPv6 address doesn't work.
Finally -- this is very good! It was about time.
participants (3)
-
Jake Visser -
Roger Dingledine -
s7r