Hello! Here are the minutes from this month's sysadmin meeting. # Roll call: who's there and emergencies anarcat, hiro, qbi present, ln5 and weasel couldn't make it but still sent updates. # What has everyone been up to ## anarcat * blog service damage control (#32090) * new caching service (#32239) * try to kick cymru back into life (#29397) * jabber service shutdown (#31700) * prometheus/ipsec reliability issues (#31916) * bumped prometheus retention to 5m/365d, bumped back to 1m/365d after i realized it broke the graphs (#31244) * LDAP sudo transition (#6367) * finished director replacement (#31786) * archived public SVN (#15948) * shutdown SVN internal (#15949) * fix "ping on new VMs" bug on ganeti hosts (#31781) * review Fastly contracts and contacts * became a blog maintainer (#23007) * clarified hardware donation policy in FAQ (#32044) * tracking major upgrades progress (fancy graphs!), visible at https://help.torproject.org/tsa/howto/upgrades/ - current est: april 2020 * joined a call with giant rabbit about finances, security and cost, hiro also talked with them about upgrading their CiviCRM, some downtimes to be announced soon-ish * massive (~20%) trac ticket cleanup in the "trac" component * worked sysadmin onboarding process docs (ticket #29395) * drafted a template for service documentation in https://help.torproject.org/tsa/howto/template/ * daily grind: email aliases, pgp key updates, full disks, security upgrades, reboots, performance problems ## hiro * website maintenance and eoy campaign * retire getulum * make a new machine for gettor * crm stuff with giant rabbit * some security updates and service documentation. Testing out ansible for scripts. Happy with the current setup used for gettor with everything else in puppet. * some gettor updates and maintenance * started creating the dev website * survey update * nagios gettor status check * dip updates and maintenance ## weasel * moving onionoo forward to new VMs (#31659 and linked) * moved more things off metal we want to get rid of * includes preparing a new IRC host (#32281); the old one is not yet gone ## qbi * created tor-moderators@ * updated some machines (apt uprade) ## linus * followed up with nextcloud launch # What we're up to next ## anarcat New: * caching server launch and followup, missing stats (#32239) Continued/stalled: * followup on SVN shutdown, only corp missing (#17202) * upstreaming ganeti installer fix and audit of the others (#31781) * followup with email services improvements (#30608) * followup on SVN decomissionning (#17202) * send root@ emails to RT (#31242) * continue prometheus module merges ## hiro * Lektor package upgrade * More website maintenance * nagios bridgedb status check * investigating occasional websites build failures * move translations / majus out of moly * finish prometheus tasks w/ anticensorship-team * why is gitlab giving an error when creating a MR from a forked repository? ## ln5 * nextcloud migration ## qbi * Upgrade some hosts (<5) to buster # Other discussions No planned discussion. # Next meeting qbi can't on dec 2nd and we missed two people this time, so it make sense to do it a week earlier... november 25th 1500UTC, which is 1600CET and 1000EST # Metrics of the month Access and transfer rates are an average over the last 30 days. * hosts in Puppet: 75, LDAP: 79, Prometheus exporters: 120 * number of apache servers monitored: 32, hits per second: 203 * number of self-hosted nameservers: 5, mail servers: 10 * pending upgrades: 5, reboots: 0 * average load: 0.94, memory available: 303.76 GiB/946.18 GiB, running processes: 387 * bytes sent: 200.05 MB/s, received: 132.90 MB/s Now also available as the main Grafana dashboard. Head to <https://grafana.torproject.org/>, change the time period to 30 days, and wait a while for results to render. -- Antoine Beaupré torproject.org system administration