onionoo.tpo stuck at 2016-05-13 12:00
Hi Karsten, I was surprised that ornetradar did not send a single email for yesterday's new relays. After looking into it, it turned out it is an onionoo problem. "relays_published":"2016-05-13 12:00:00" https://onionoo.torproject.org/details?limit=4 regards, nusenu
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 14/05/16 12:00, nusenu wrote:
Hi Karsten,
Hi Nusenu,
I was surprised that ornetradar did not send a single email for yesterday's new relays.
After looking into it, it turned out it is an onionoo problem.
"relays_published":"2016-05-13 12:00:00"
It looks like the Onionoo host has some serious load problems (again). It's at "relays_published":"2016-05-16 19:00:00" now, so it didn't stop entirely, but it's still behind. I just kicked it, maybe it'll catch up in the next few hours. As a short-term fix, feel free to switch to the Onionoo mirror: https://onionoo.thecthulhu.com/summary?limit=0 As a medium-term fix, extend your client to fetch both headers (/summary?limit=0) and get the full details from the host with more recent "relays_published" line. As a long-term fix we should fix most/all inconsistencies between Onionoo instances (different encodings and other minor problems) and add a hostname that round-robins requests to all available mirrors.
regards, nusenu
Thanks for letting me know! All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJXOt0JAAoJEC3ESO/4X7XBLqIH/RYSZrlcKE3+Z9tx0dvdeYRN OTvwX632AGnfn8Ej2x9mGzf12Qkv3ED6Of979z5Zu02uCgTpOfgKy8MxxjnW0qDu oWvAw1OeO0T+kw+tD5IbjKhqDX0dYgNzoiYS2W/SwkR+OmGEX1i5/nVZm/M7hJFo VWikP0th4ajVhIRwTEILjuMK3DS6UruHYYX7gCrLdvYOC3R11h3AXOWBgbblT8wz x0jPKmNiOSWSKdpPCD5DHQbHaifNWMTqvZEqURhDpSKpfqUDZ82i7b/rG5bXlYZM KdjMUdF0e5NIxzKhg0rQkaJFAeSdgXhj++yqJdGfNXYgtu4QDccTCaKdQKb67gs= =AmcN -----END PGP SIGNATURE-----
Hi Thomas, I'm using your onionoo instance to fetch details documents. Currently it is stuck at 2016-05-26 00:00, you might want to look into it? (even though its date does not change since a few days its file contents do) https://onionoo.thecthulhu.com/details?limit=0
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 31/05/16 11:24, nusenu wrote:
Hi Thomas,
Hi nusenu,
I'm using your onionoo instance to fetch details documents.
(Thomas is kindly providing the host, but I'm operating the Onionoo instance on that.)
Currently it is stuck at 2016-05-26 00:00, you might want to look into it?
Huh, you're right. Looks like the updater got stuck while downloading descriptors from CollecTor: 2016-05-26 01:19:15,175 INFO o.t.o.cron.Main:176 Downloading descriptors. 2016-05-26 01:19:15,176 INFO o.t.o.u.DescriptorSource:64 Loading: RELAY_CONSENSUSES 2016-05-26 01:25:05,420 INFO o.t.o.u.DescriptorSource:64 Loading: RELAY_SERVER_DESCRIPTORS That instance was offline for more than 3 days which means that I'll have to let it re-import the May 2016 tarballs that are currently produced. I need to wait for that and then let it chew on a lot of data. I hope it's back tomorrow.
(even though its date does not change since a few days its file contents do)
Not sure what you mean by "its file contents do change". Can you explain? All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJXTYHSAAoJEC3ESO/4X7XByHEH/iS9S7J1G3TKGpru1BJRKBaq BDKgw4VaRUUkFC6+yj6uJuk7O4sJGZyAr4SW7Q6voLEW3Mvoe36iBFyGP76Gu5TY Oxx4rez7ruCpP7nBTfrSsTjox8oB+S8vBFu2d/IsoKAfNgLxip0cRVeD60Pp+jjS v5jw5Ef0isorA4UEOxqya3Zq3Uj97MCrag7oZeNojzs19Qb0aROc3xF7tfJdEIk6 7/NyGlFzv9z8WIx34IIm7v9i0+oA/cYrjn7dAHiFhm4AVPc7WE/0mesg2s4Qh6jd oJROtnyBk5Kh8SFN0bk9kntyk+Fv+7TdbA4e8p9fh3MQBHr07AwgurKr63Qkoik= =dLOS -----END PGP SIGNATURE-----
That instance was offline for more than 3 days which means that I'll have to let it re-import the May 2016 tarballs that are currently produced. I need to wait for that and then let it chew on a lot of data. I hope it's back tomorrow.
Thanks for fixing it.
(even though its date does not change since a few days its file contents do)
https://onionoo.thecthulhu.com/details?limit=0 Not sure what you mean by "its file contents do change". Can you explain?
I was trying to say that the details records provided over the last few days changed even though relays_published timestamp didn't -- but it turned out that I was wrong so you can ignore it.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 31/05/16 14:21, Karsten Loesing wrote:
On 31/05/16 11:24, nusenu wrote:
Hi Thomas,
Hi nusenu,
I'm using your onionoo instance to fetch details documents.
(Thomas is kindly providing the host, but I'm operating the Onionoo instance on that.)
Currently it is stuck at 2016-05-26 00:00, you might want to look into it?
Huh, you're right. Looks like the updater got stuck while downloading descriptors from CollecTor:
2016-05-26 01:19:15,175 INFO o.t.o.cron.Main:176 Downloading descriptors. 2016-05-26 01:19:15,176 INFO o.t.o.u.DescriptorSource:64 Loading: RELAY_CONSENSUSES 2016-05-26 01:25:05,420 INFO o.t.o.u.DescriptorSource:64 Loading: RELAY_SERVER_DESCRIPTORS
That instance was offline for more than 3 days which means that I'll have to let it re-import the May 2016 tarballs that are currently produced. I need to wait for that and then let it chew on a lot of data. I hope it's back tomorrow.
Update on https://onionoo.thecthulhu.com/: I ran into an issue that I didn't resolve yet. It's probably something trivial, I just didn't find the time to look more closely: https://trac.torproject.org/projects/tor/ticket/19249 Update on https://onionoo.torproject.org/: This one should be running much more smoothly as of a few hours ago. There was a problem with Apache's disk cache which caused all ~1M/h requests to go through the Java process. That's what you can see in the last few weeks in the following graph. We're now back to a few 10k/h: https://people.torproject.org/~karsten/volatile/onionoo-total-requests-2016-... All the best, Karsten
(even though its date does not change since a few days its file contents do)
Not sure what you mean by "its file contents do change". Can you explain?
All the best, Karsten
-----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJXVH93AAoJEC3ESO/4X7XBVd0IAMoJZZUg49qRtzxPtSoZpgjS MHH+LEP151kIYDUSWEutOd8aFkZgIaenjGckYrouq1ksSLkhbVY8Mp9YDRNKySAk zxCd9yUydFDGZp0EaI+UDG3McKI+tMN9b+gp20LaXsg5TFibnyqL8y2zJ2LvZaRP t12m8agyzJFYXai7ZHM+/WPTMWtLey7yRxvYPVFGCmV/0P6H9SldSopm9ywhbJ7p gDF7yDTiK6UEEi1IsOsJICadjmZ+lLITvujn0pX8kH9kzGs8KiETVYhjV845+v15 2QMQI5RSJS5Q2qun1yYRL4x/D9bNG7irQ+1c06E6nO3YFVsQNnMeFthrRGOqTuQ= =BHEd -----END PGP SIGNATURE-----
Hi Karsten, could you look into onionoo.tpo, looks like it is stuck again. thanks, nusenu
"relays_published":"2016-10-30 23:00:00",
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi nusenu, should be resolved now. Turns out one of the CollecTor modules silently died. It's back now. If you notice similar problems in the future, be sure to let us know! We do have a few checks in place, but this issue slipped through somehow. Thanks! All the best, Karsten On 01/11/16 00:59, nusenu wrote:
Hi Karsten,
could you look into onionoo.tpo, looks like it is stuck again.
thanks, nusenu
"relays_published":"2016-10-30 23:00:00",
-----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBCAAGBQJYGHIXAAoJEC3ESO/4X7XBX0AIAKNd4iyWaIHsE0u35e6NQoYY dAHXJrgAYtPtHCZBJHNyBAI3KboKwO7GXEpX598IP03p7nnN24oxAMUG7sN57a6m Jea4R353PHoje+VDbTZSPuxH2YI8cveH/a9+wN13cHc9YtF+cCNGVHBnGq1wrkgt p3Fidr8XrCJiiBWoGdY6nHl7CHQ7F9bHcKTPVTg/wrG7jYtPhBwj6FRxuqWE2PLZ rBXRo64YZ/HRA8Oy92de6DvjbzJBkiVG/PtlPUbxTYOf/RNXUwZ/iwBni7Wj4hO3 OYmoMIvTDIQCFn/WHH2qvx8j1xmiUTgTZjrSn7KzUA5UEI55q+KIkHkN+8a1U2I= =eETR -----END PGP SIGNATURE-----
Karsten Loesing:
If you notice similar problems in the future, be sure to let us know! We do have a few checks in place, but this issue slipped through somehow.
I assume you are already aware that onionoo is currently a bit behind (2017-01-27 13:00). https://lists.torproject.org/pipermail/tor-relays/2017-January/011812.html
On 28/01/17 10:06, nusenu wrote:
Karsten Loesing:
If you notice similar problems in the future, be sure to let us know! We do have a few checks in place, but this issue slipped through somehow.
I assume you are already aware that onionoo is currently a bit behind (2017-01-27 13:00).
Yes, I'm upgrading to protocol version 3.2, which requires re-importing bridge descriptor archives since December 2014. Of course that didn't go without problems, first with memory running out, then disk space. I hope it'll be back later today. Thanks for the report! All the best, Karsten
https://lists.torproject.org/pipermail/tor-relays/2017-January/011812.html
On 28/01/17 10:23, Karsten Loesing wrote:
On 28/01/17 10:06, nusenu wrote:
Karsten Loesing:
If you notice similar problems in the future, be sure to let us know! We do have a few checks in place, but this issue slipped through somehow.
I assume you are already aware that onionoo is currently a bit behind (2017-01-27 13:00).
Yes, I'm upgrading to protocol version 3.2, which requires re-importing bridge descriptor archives since December 2014. Of course that didn't go without problems, first with memory running out, then disk space. I hope it'll be back later today.
It should be back since yesterday afternoon. Please shout if any issues remain. All the best, Karsten
Thanks for the report!
All the best, Karsten
https://lists.torproject.org/pipermail/tor-relays/2017-January/011812.html
participants (2)
-
Karsten Loesing -
nusenu