minutes from the sysadmin meeting

Hi! TPA had their weekly meeting, and here are its minutes. # Roll call: who's there and emergencies anarcat, kez, lavamind present. no emergencies. # "Star of the weeks" rotation anarcat has been the "star of the weeks" all of the last two months, how do we fix this process? We talked about a few options, namely having per-day schedules and per-week schedules. We settled on the latter because it gives us a longer "interrupt shield" and allows support to deal with a broader range, possibly more than short-term, set of issues. Let's set a schedule until the vacations: * Nov 1st, W45: lavamind * W46: kez * W47: anarcat * W48: lavamind * W49: kez * W50: etc So this week is lavamind, we need to remember to pass the buck at the end of the week. Let's talk about holidays at some point. We'll figure out what people have for a holiday and see if we can avoid overlapping holidays during the winter period. # Q4 roadmap review We did a quick review of the [quarterly roadmap][] to see if we're still on track to close our year! [quarterly roadmap]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2021#q4 We are clearly in a crunch: * Lavamind is prioritizing the blog launch because that's mid-november * Anarcat would love to finish the Jenkins retirement as well * Kez has been real busy with the year end campaign but hopes to complete the bridges rewrite by EOY as well There's also a lot of pressure on the GitLab infrastructure. So far we're throwing hardware at the problem but it will need a redesign at some point. See the [gitlab scaling ticket][] and [storage brainstorm][]. [storage brainstorm]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40478 [gitlab scaling ticket]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40479 # Dashboard triage We reviewed only this team dashboard, in a few minutes at the end of our meeting: * https://gitlab.torproject.org/tpo/tpa/team/-/boards/117 We didn't have time to process those: * https://gitlab.torproject.org/groups/tpo/web/-/boards (still overflowing) * https://gitlab.torproject.org/groups/tpo/tpa/-/boards (if time permits) # Other discussions The holidays discussion came up and should be addressed in the next meeting. # Next meeting First monday of the month in December is December 6th. Warning: 17:00UTC might mean a different time for you then, it then is equivalent to: 09:00 US/Pacific, 14:00 America/Montevideo, 12:00 US/Eastern, 18:00 Europe/Paris. # Metrics of the month * hosts in Puppet: 89, LDAP: 92, Prometheus exporters: 140 * number of Apache servers monitored: 27, hits per second: 161 * number of Nginx servers: 2, hits per second: 2, hit ratio: 0.81 * number of self-hosted nameservers: 6, mail servers: 8 * pending upgrades: 15, reboots: 0 * average load: 1.40, memory available: 3.52 TiB/4.47 TiB, running processes: 745 * bytes sent: 293.16 MB/s, received: 183.02 MB/s * [GitLab tickets][]: ? tickets including... * open: 0 * icebox: 133 * backlog: 22 * next: 5 * doing: 3 * needs information: 8 * (closed: 2484) [Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards Our backlog and `needs information` queues are at a record high since April, which confirms the crunch. -- Antoine Beaupré torproject.org system administration
participants (1)
-
Antoine Beaupré