[tor-bugs] #32901 [Internal Services/Tor Sysadmin Team]: puppetize Nagios

[Tor Bug Tracker & Wiki]

#32901: puppetize Nagios
-------------------------------------------------+-------------------------
 Reporter:  anarcat                              |          Owner:  anarcat
     Type:  project                              |         Status:
                                                 |  assigned
 Priority:  Low                                  |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Major                                |     Resolution:
 Keywords:  tpa-roadmap-february                 |  Actual Points:
Parent ID:  #31239                               |         Points:  10
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------

Old description:

> one part of our install process is to configure Nagios, by hand, in the
> git repository. I usually do this by copy-pasting some similar blob of
> config from a possibly similar machine and hope for the best.
>
> this is a manual step, and as part of the automation of the install
> process, it should be made automatic.
>
> one way this could (and probably should) be done is by making Puppet
> automatically add its nodes into Nagios. this can be done using the
> [https://github.com/Icinga/puppet-icinga2 icinga2 module], for example.
> care should be taken to do a smooth transition, keeping existing
> configurations and just adding the Puppet ones on top, for new machines.
>
> but this could (eventually) be retroactively added to all nodes, removing
> all manual configuration.
>
> checklist:
>
> 1. [ ] audit and import the module in our monorepo
> 1. [ ] enable on the nagios server, without writing any config (hopefully
> a noop)
> 1. [ ] enable a single config from puppet, as a test
> 1. [ ] add a new host check configuration
> 1. [ ] add a new service check configuration
> 1. [ ] add all *base* service checks for the new host
> 1. [ ] convert legacy config into puppet (at this stage we only have the
> old hosts as legacy config)
> 1. [ ] convert old hosts into puppet
> 1. [ ] convert old *services* into puppet
>
> It's a long way there, but getting to the state where *new* hosts are
> covered would already be a great improvement.

New description:

 one part of our install process is to configure Nagios, by hand, in the
 git repository. I usually do this by copy-pasting some similar blob of
 config from a possibly similar machine and hope for the best.

 this is a manual step, and as part of the automation of the install
 process, it should be made automatic.

 one way this could (and probably should) be done is by making Puppet
 automatically add its nodes into Nagios. this can be done using the
 [https://github.com/Icinga/puppet-icinga2 icinga2 module], for example.
 care should be taken to do a smooth transition, keeping existing
 configurations and just adding the Puppet ones on top, for new machines.

 but this could (eventually) be retroactively added to all nodes, removing
 all manual configuration.

 checklist:

 1. [x] audit and import the module in our monorepo
 1. ~~[ ] enable on the nagios server, without writing any config
 (hopefully a noop)~~ not possible, config is overwritten by module,
 instead...
 1. [ ] move the base configuration (`config/static`) from git into Puppet
 (mostly icinga.cfg and so on, because they are overwritten by the module)
 1. [ ] enable a single config from puppet, as a test
 1. [ ] add a new host check configuration
 1. [ ] add a new service check configuration
 1. [ ] add all *base* service checks for the new host (e.g. the services
 defined for the `computers` hostgroup, equivalent of pieces of `from-
 git/generated/auto-services.cfg`)
 1. ~~[ ] convert legacy config into puppet (at this stage we only have the
 old hosts as legacy config)~~ done in third step
 1. [ ] convert NRPE service definitions (`puppet:///modules/nagios/tor-
 nagios/generated/nrpe_tor.cfg`, generated from the git repo)
 1. [ ] remove NRPE config sync from nagios to Puppet (the rsync to `pauli`
 in `config/Makefile`)
 1. [ ] convert old hosts checks into puppet
 1. [ ] convert old services checks into puppet
 1. [ ] remove git hook receiver on nagios server
 (`/etc/ssh/userkeys/nagiosadm` key, which calls `/home/nagiosadm/bin/from-
 git-rw`)

 It's a long way there, but getting to the state where *new* hosts are
 covered would already be a great improvement.

--

Comment (by anarcat):

 reorder checklist: we can't have nice things as the icinga module
 immediately rewrites the icinga.cfg, at the very least. also add items to
 convert NRPE, which I have overlooked.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/32901#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online