[tor-relays] Debugging relays not showing online

Thu Apr 18 17:52:48 UTC 2019

Hi all,

I am looking for some help in debugging a strange one....  EO runs ~30ish relays in our /24; but we currently have 4 relays that don't want to play nice and are reporting offline (although look to be online).

The effected relay IPs are:

23.129.64.157
23.129.64.164
23.129.64.170
23.129.64.174

The first 3 do show on Metrics, and when they are restarted or rebuilt (we use relayor), metrics does pick it up.. in terms that its "uptime" is updated to reflect the restart, although still marked offline. The 4th  - doesn't show at all... it is in the family members though.

Looking more closely at: 23.129.64.157 (183C8C6727E2137AF278B3850AD5D9C2304B98C9); using nyx with debug messaging; 2 hours after a restart, I can still see active circuits, and I can see it doing "things" [logs bellow], and http://23.129.64.157 loads (we run the exit page).
Is there anything I could be doing or look at to indicate what might be creating such a strange issue? All the other relays in the family seem to be working fine.. https://metrics.torproject.org/rs.html#search/family:183C8C6727E2137AF278B3850AD5D9C2304B98C9 (and all built and managed with relayor using offline keys..). These have been online for about 10 days now.

Help / guidance would be greatly appraised...

Thanks,

Jake

Events (DEBUG-ERR, NYX NOTICE-ERR):
x 10:48:54 [DEBUG] conn_read_callback: socket 18 wants to read.  [181 duplicates hidden]
x 10:48:50 [DEBUG] connection_or_process_cells_from_inbuf: 14: starting, inbuf_datalen 0 (0 pending in tls object).  [83 duplicates hidden]
x 10:48:50 [DEBUG] connection_or_process_cells_from_inbuf: 14: starting, inbuf_datalen 514 (0 pending in tls object).  [83 duplicates hidden]
x 10:48:50 [DEBUG] connection_buf_read_from_socket: After TLS read of 514: 543 read, 0 written  [99 duplicates hidden]
x 10:48:50 [DEBUG] connection_buf_read_from_socket: 14: starting, inbuf_datalen 0 (0 pending in tls object). at_most 16448.  [105 duplicates hidden]
x 10:48:50 [DEBUG] conn_read_callback: socket 14 wants to read.  [83 duplicates hidden]
x 10:48:43 [INFO] update_consensus_router_descriptor_downloads: 0 router descriptors downloadable. 0 delayed; 6722 present (0 of those were in old_routers); 0 would_reject; 0 wouldnt_use; 0 in progress.  [9 duplicates hidden]
x 10:48:39 [DEBUG] connection_or_process_cells_from_inbuf: 25: starting, inbuf_datalen 0 (0 pending in tls object).  [1 duplicate hidden]
x 10:48:39 [DEBUG] connection_or_process_cells_from_inbuf: 25: starting, inbuf_datalen 514 (0 pending in tls object).  [1 duplicate hidden]
x 10:48:39 [DEBUG] conn_read_callback: socket 25 wants to read.  [1 duplicate hidden]
x 10:48:39 [DEBUG] connection_or_process_cells_from_inbuf: 17: starting, inbuf_datalen 0 (0 pending in tls object).  [1 duplicate hidden]
x 10:48:39 [DEBUG] connection_or_process_cells_from_inbuf: 17: starting, inbuf_datalen 514 (0 pending in tls object).  [1 duplicate hidden]
x 10:48:39 [DEBUG] conn_read_callback: socket 17 wants to read.  [1 duplicate hidden]
x 10:48:39 [DEBUG] connection_handle_write_impl: After TLS write of 514: 0 read, 543 written  [17 duplicates hidden]
x 10:48:39 [DEBUG] flush_chunk_tls: flushed 514 bytes, 0 ready to flush, 0 remain.  [17 duplicates hidden]
x 10:48:39 [DEBUG] conn_write_callback: socket 25 wants to write.  [1 duplicate hidden]
x 10:48:39 [DEBUG] conn_write_callback: socket 17 wants to write.  [1 duplicate hidden]
x 10:48:39 [DEBUG] run_connection_housekeeping: Sending keepalive to (199.58.81.140:443)  [1 duplicate hidden]
x 10:48:39 [DEBUG] run_connection_housekeeping: Sending keepalive to (154.35.175.225:443)  [1 duplicate hidden]
x 10:48:16 [INFO] routerlist_remove_old_routers: We have 6869 live routers and 820 old router descriptors.  [8 duplicates hidden]
x 10:48:16 [DEBUG] resolve_my_address: Resolved Address to '23.129.64.157'.  [8 duplicates hidden]
x 10:48:13 [DEBUG] connection_or_process_cells_from_inbuf: 23: starting, inbuf_datalen 0 (0 pending in tls object).  [1 duplicate hidden]
x 10:48:13 [DEBUG] connection_or_process_cells_from_inbuf: 23: starting, inbuf_datalen 514 (0 pending in tls object).  [1 duplicate hidden]
x 10:48:13 [DEBUG] conn_read_callback: socket 23 wants to read.  [1 duplicate hidden]
x 10:48:13 [DEBUG] conn_write_callback: socket 23 wants to write.  [1 duplicate hidden]
x 10:48:13 [DEBUG] run_connection_housekeeping: Sending keepalive to (128.31.0.34:9101)  [1 duplicate hidden]
x 10:48:09 [DEBUG] conn_write_callback: socket 14 wants to write.  [1 duplicate hidden]
x 10:48:09 [DEBUG] run_connection_housekeeping: Sending keepalive to (217.20.130.72:9001)  [1 duplicate hidden]
x 10:46:53 [DEBUG] connection_or_process_cells_from_inbuf: 21: starting, inbuf_datalen 0 (0 pending in tls object).  [1 duplicate hidden]
x 10:46:53 [DEBUG] connection_or_process_cells_from_inbuf: 21: starting, inbuf_datalen 514 (0 pending in tls object).  [1 duplicate hidden]
x 10:46:53 [DEBUG] conn_read_callback: socket 21 wants to read.  [1 duplicate hidden]
x 10:46:53 [DEBUG] conn_write_callback: socket 21 wants to write.  [1 duplicate hidden]
x 10:46:53 [DEBUG] run_connection_housekeeping: Sending keepalive to (194.109.206.212:443)  [1 duplicate hidden]
x 10:46:40 [DEBUG] connection_or_process_cells_from_inbuf: 20: starting, inbuf_datalen 0 (0 pending in tls object).  [1 duplicate hidden]
x 10:46:40 [DEBUG] connection_or_process_cells_from_inbuf: 20: starting, inbuf_datalen 514 (0 pending in tls object).  [1 duplicate hidden]
x 10:46:40 [DEBUG] conn_read_callback: socket 20 wants to read.  [1 duplicate hidden]
x 10:46:40 [DEBUG] conn_write_callback: socket 20 wants to write.  [1 duplicate hidden]
x 10:46:40 [DEBUG] run_connection_housekeeping: Sending keepalive to (171.25.193.9:80)  [1 duplicate hidden]
x 10:46:26 [DEBUG] connection_or_process_cells_from_inbuf: 19: starting, inbuf_datalen 0 (0 pending in tls object).  [1 duplicate hidden]
x 10:46:26 [DEBUG] connection_or_process_cells_from_inbuf: 19: starting, inbuf_datalen 514 (0 pending in tls object).  [1 duplicate hidden]
x 10:46:26 [DEBUG] conn_read_callback: socket 19 wants to read.  [1 duplicate hidden]
x 10:46:26 [DEBUG] conn_write_callback: socket 19 wants to write.  [1 duplicate hidden]
x 10:46:26 [DEBUG] run_connection_housekeeping: Sending keepalive to (204.13.164.118:443)  [1 duplicate hidden]
x 10:46:23 [DEBUG] connection_or_process_cells_from_inbuf: 22: starting, inbuf_datalen 0 (0 pending in tls object).  [1 duplicate hidden]
x 10:46:23 [DEBUG] connection_or_process_cells_from_inbuf: 22: starting, inbuf_datalen 514 (0 pending in tls object).  [1 duplicate hidden]
x 10:46:23 [DEBUG] conn_read_callback: socket 22 wants to read.  [1 duplicate hidden]
x 10:46:22 [DEBUG] conn_write_callback: socket 22 wants to write.  [1 duplicate hidden]
x 10:46:22 [DEBUG] run_connection_housekeeping: Sending keepalive to (86.59.21.38:443)  [1 duplicate hidden]
x 10:46:10 [DEBUG] consdiffmgr_cleanup: Looking for consdiffmgr entries to remove
qj 10:45:58 [DEBUG] connection_or_process_cells_from_inbuf: 15: starting, inbuf_datalen 0 (0 pending in tls object).  [1 duplicate hidden]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-relays/attachments/20190418/4db35a50/attachment-0001.html>