As with the previous month, I figured I would show a sign of life here and try to keep you up to date with what's happening in sysadmin-land, even though we're not having regular meetings. I'm still experimenting with structure here, and this is totally un-edited, so please bear with me.

# Important announcements

You might have missed this:

* Jenkins will be retired in December 2021, and it's time to move your jobs away * if you want old Trac wiki redirects to go to the right place, do let us know, see [ticket 40233][] * we do not have ARM 32 builders anymore, the last one was shut down recently ([ticket 32920][]) and they had been removed from CI (Jenkins) anyways before that. the core team is looking at alternatives for building Tor on armhf in the future, see [ticket 40347][] * we have setup a Prometheus Alertmanager during the hack week, which means we can do alerting based on Prometheus metrics, see the [altering documentation][] for more information

As usual, if you have any questions, comments, or issues, please do contact us following this "how to get help" procedure:

https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-2-support#how-to-get-help

Yes, that's a terrible URL. Blame GitLab. :)

[altering documentation]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/prometheus#alerting [ticket 40347]: https://gitlab.torproject.org/tpo/core/tor/-/issues/40347 [ticket 32920]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/32920 [ticket 40233]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40233

# Crash of the month

Your sysadmin crashed a Ganeti node, creating a split-brain scenario ([ticket 40229][]). He would love to say that was planned and a routine exercise to test the documentation but (a) it wasn't and (b) the document had to be made up as he went, so that was actually a stressful experience.

Remember kids: never start a migration before the weekend or going to bed unless you're willing and ready to stay up all night (or weekend).

[ticket 40229]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40229

# Metrics of the month

* hosts in Puppet: 86, LDAP: 89, Prometheus exporters: 140 * number of Apache servers monitored: 28, hits per second: 147 * number of Nginx servers: 2, hits per second: 2, hit ratio: 0.86 * number of self-hosted nameservers: 6, mail servers: 7 * pending upgrades: 1, reboots: 0 * average load: 0.68, memory available: 2.00 TiB/2.77 TiB, running processes: 552 * bytes sent: 276.43 MB/s, received: 162.75 MB/s * [GitLab tickets][]: ? tickets including... * open: 0 * icebox: 109 * backlog: 15 * next: 2 * doing: 2 * (closed: 2266)

[Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards

# Ticket analysis

Here's an update of the ticket table, which we last saw in February:

| date | open | icebox | backlog | next | doing | closed | delta | sum | new | spill | |------------|------|--------|---------|------|-------|--------|-------|--------|------|-------| | 2020-11-18 | 1 | 84 | 32 | 5 | 4 | 2119 | NA | 2245 | NA | NA | | 2020-12-02 | 0 | 92 | 20 | 9 | 8 | 2130 | 11 | 2259 | 14 | -3 | | 2021-01-19 | 0 | 91 | 20 | 12 | 10 | 2165 | 35 | 2298 | 39 | -4 | | 2021-02-02 | 0 | 96 | 18 | 10 | 7 | 2182 | 17 | 2313 | 15 | 2 | | 2021-03-02 | 0 | 107 | 15 | 9 | 7 | 2213 | 31 | 2351 | 38 | -7 | | 2021-04-07 | 0 | 106 | 22 | 7 | 4 | 2225 | 12 | 2364 | 13 | -1 | | 2021-05-03 | 0 | 109 | 15 | 2 | 2 | 2266 | 41 | 2394 | 30 | 11 | |------------|------|--------|---------|------|-------|--------|-------|--------|------|-------| | total | NA | NA | NA | NA | NA | NA | 147 | NA | 149 | -2 | |------------|------|--------|---------|------|-------|--------|-------|--------|------|-------| | mean | 0.1 | 97.9 | 20.3 | 7.7 | 6.0 | 2185.7 | 21.0 | 2317.7 | 21.3 | -0.3 |

I added a "delta" column which shows how many additional tickets were closed since the previous period. April is our record so far, with a record of 41 tickets closed in less than 30 days, more than one ticket per day!

I also added a "new" column that shows how many new tickets, in total, were created in the period. And the "spill" is the difference between the two. If positive, we're winning the ticket game, if negative, we're losing ground and more tickets are being created than we are closing. Overall, we're slightly behind (-2), but that's only because of the epic month of April.

And while I'm here, I went crazy with Emacs' `orgtbl-mode` and added totals and averages.

In other news, the Icebox keeps growing, which should keep us cool and breezy during the northern hemisphere summer that's coming up. ;) At least the Backlog is not growing too wildly, and the actual current queue (Next/Doing) is pretty reasonable. So things seem to be under control, but the new hiring process is taking significant time so this might upset our roadmap a little.

Regardless of those numbers: don't hesitate to make new tickets!

# Ticket of the month

Ticket [40218][] tracks the progress of the CI migration from Jenkins to GitLab CI. Jenkins is scheduled for retirement in December 2021, and progress has been excellent, with the network team actually *asking* for the Jenkins jobs to be disabled ([ticket 40225][]) which, if it gets completed, will means the retirement of 4 virtual machines already.

Exciting cleanup!

[40218]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40218 [ticket 40225]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40225