[tor-project] minutes from the sysadmin meeting

[Antoine Beaupré]

Hi!

TPA had their weekly meeting, and here are its minutes.

# Roll call: who's there and emergencies

anarcat, kez, lavamind present. no emergencies.

# "Star of the weeks" rotation

anarcat has been the "star of the weeks" all of the last two months,
how do we fix this process?

We talked about a few options, namely having per-day schedules and
per-week schedules. We settled on the latter because it gives us a
longer "interrupt shield" and allows support to deal with a broader
range, possibly more than short-term, set of issues.

Let's set a schedule until the vacations:

 * Nov 1st, W45: lavamind
 * W46: kez
 * W47: anarcat
 * W48: lavamind
 * W49: kez
 * W50: etc

So this week is lavamind, we need to remember to pass the buck at the
end of the week.

Let's talk about holidays at some point. We'll figure out what people
have for a holiday and see if we can avoid overlapping holidays during
the winter period.

# Q4 roadmap review

We did a quick review of the [quarterly roadmap][] to see if we're
still on track to close our year!

[quarterly roadmap]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2021#q4

We are clearly in a crunch:

 * Lavamind is prioritizing the blog launch because that's
   mid-november
 * Anarcat would love to finish the Jenkins retirement as well
 * Kez has been real busy with the year end campaign but hopes to
   complete the bridges rewrite by EOY as well

There's also a lot of pressure on the GitLab infrastructure. So far
we're throwing hardware at the problem but it will need a redesign at
some point. See the [gitlab scaling ticket][] and [storage
brainstorm][].

[storage brainstorm]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40478
[gitlab scaling ticket]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40479

# Dashboard triage

We reviewed only this team dashboard, in a few minutes at the end of
our meeting:

 * https://gitlab.torproject.org/tpo/tpa/team/-/boards/117

We didn't have time to process those:

 * https://gitlab.torproject.org/groups/tpo/web/-/boards (still
   overflowing)
 * https://gitlab.torproject.org/groups/tpo/tpa/-/boards (if time
   permits)

# Other discussions

The holidays discussion came up and should be addressed in the next
meeting.

# Next meeting

First monday of the month in December is December 6th. Warning:
17:00UTC might mean a different time for you then, it then is
equivalent to: 09:00 US/Pacific, 14:00 America/Montevideo, 12:00
US/Eastern, 18:00 Europe/Paris.

# Metrics of the month

 * hosts in Puppet: 89, LDAP: 92, Prometheus exporters: 140
 * number of Apache servers monitored: 27, hits per second: 161
 * number of Nginx servers: 2, hits per second: 2, hit ratio: 0.81
 * number of self-hosted nameservers: 6, mail servers: 8
 * pending upgrades: 15, reboots: 0
 * average load: 1.40, memory available: 3.52 TiB/4.47 TiB, running
   processes: 745
 * bytes sent: 293.16 MB/s, received: 183.02 MB/s
 * [GitLab tickets][]: ? tickets including...
   * open: 0
   * icebox: 133
   * backlog: 22
   * next: 5
   * doing: 3
   * needs information: 8
   * (closed: 2484)
    
 [Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards

Our backlog and `needs information` queues are at a record high since
April, which confirms the crunch.

-- 
Antoine Beaupré
torproject.org system administration
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20211104/2998fc37/attachment.sig>