[tor-bugs] #32638 [Metrics/Onionoo]: Please remove onionoo-backend-01 from the backend pool

[Tor Bug Tracker & Wiki]

#32638: Please remove onionoo-backend-01 from the backend pool
-----------------------------+--------------------------
 Reporter:  irl              |          Owner:  irl
     Type:  task             |         Status:  assigned
 Priority:  High             |      Milestone:
Component:  Metrics/Onionoo  |        Version:
 Severity:  Normal           |     Resolution:
 Keywords:                   |  Actual Points:
Parent ID:                   |         Points:
 Reviewer:                   |        Sponsor:
-----------------------------+--------------------------
Changes (by anarcat):

 * status:  accepted => assigned
 * owner:  anarcat => irl
 * component:  Internal Services/Tor Sysadmin Team => Metrics/Onionoo


Comment:

 first off, thank you so much for the excellent documentation you wrote on
 this service. it makes my life (and i suspect everyone else who will have
 to deal with this in the future) so much easier!

 i think we're in the [https://help.torproject.org/metrics/ops/onionoo-
 ops/#index8h2 Single backend data corruption, no hardware failure]
 scenario, which says this:

 > {{{
 > sudo -u onionoo -i bash -c 'systemctl --user stop onionoo'
 > sudo -u onionoo-unpriv -i bash -c 'systemctl --user stop onionoo-web'
 > rm -rf /srv/onionoo.torproject.org/onionoo/home/{.,}\*
 > rm -rf /srv/onionoo.torproject.org/onionoo/home-unpriv/{.,}\*
 > rm -rf /srv/onionoo.torproject.org/onionoo/onionoo/{.,}\*
 > }}}
 >
 > Then pretend you are deploying a new backend from the instructions
 above.

 It seems the first two step, at least, haven't been ran, because (a) the
 services are still running on that backend and (b) the varnish frontend
 agrees that they are still running:

 {{{
 root at onionoo-frontend-01:~# varnishadm -S /etc/varnish/secret
 -Tlocalhost:6082 backend.list -p
 Backend name                   Admin   Probe      Last change
 boot.be_omeiense               probe   6/6 good   Tue, 19 Nov 2019
 14:21:35 GMT
 Current states  good:  6 threshold:  2 window:  6
   Average response time of good probes: 0.011608
   Oldest ================================================== Newest
   4444444444444444444444444444444444444444444444444444444444444444 Good
 IPv4
   XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good
 Xmit
   RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good
 Recv
   HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy

 boot.be_onionoo_backend_01     probe   6/6 good   Thu, 28 Nov 2019
 10:09:23 GMT
 Current states  good:  6 threshold:  2 window:  6
   Average response time of good probes: 0.002903
   Oldest ================================================== Newest
   4444444444444444444444444444444444444444444444444444444444444444 Good
 IPv4
   XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good
 Xmit
   RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good
 Recv
   HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy

 boot.be_oo_hetzner_03          probe   6/6 good   Tue, 19 Nov 2019
 14:04:28 GMT
 Current states  good:  6 threshold:  2 window:  6
   Average response time of good probes: 0.011816
   Oldest ================================================== Newest
   4444444444444444444444444444444444444444444444444444444444444444 Good
 IPv4
   XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good
 Xmit
   RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good
 Recv
   HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy

 boot.mybackends                probe   healthy    Tue, 19 Nov 2019
 12:40:16 GMT
 }}}

 Varnish is smart: if a backend goes down, it will stop serving from there.
 So you just need to shutdown the service in the backend, and it will stop
 being used in the frontend.

 At least, if I understand correctly what you want done here. :) I can also
 just destroy the box or shut it down if that's better for you. Or I can
 turn off the service myself, or i can remove it from the load balancer,
 but I think the latter can be done by the onionoo service operators. That
 might be better since you wouldn't need TSA to wake up to bring it back
 online...

 In short, I think you can do this yourself. :) Let me know if/how that is
 incorrect!

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/32638#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online