[tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-------------------------------------------------+-------------------------
 Reporter:  anarcat                              |          Owner:  anarcat
     Type:  project                              |         Status:
                                                 |  accepted
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:                                       |  Actual Points:
Parent ID:  #29387                               |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------
Changes (by anarcat):

 * status:  assigned => accepted


Old description:

> We currently use a custom-made YAML database for assigning roles to
> servers and other metadata. I started using Hiera for some hosts and it
> seems to be working well.
>
> Hiera is officially supported in Puppet and shipped by default in Puppet
> 5 and later. It's the standard way of specifying metadata and class
> parameters for hosts. I suspect it covers most of our needs in terms of
> metadata and should cover most if not all of what we're currently doing
> with the YAML stuff in Puppet.
>
> We should therefore switch to using Hiera instead of our homegrown
> solution.
>
> This involves converting:
>
>  * `if has_role('foo') { include foo }` into `classes: [ 'foo' ]` in
> hiera
>  * hardcoded macros in the ferm module's `me.conf.erb` into exported
> resources
>  * templates looping over allnodeinfo into exported resources
>  * the `$roles` array into Hiera
>  * the `$localinfo` into Hiera (assuming all the data is there)
>  * the `$nodeinfo` and `$allnodeinfo` arrays into Hiera (assuming we can
> switch from LDAP for host inventory)
>  * basically any other stuff of the kind
>
> Ideally, all YAML data should end up in the hiera/ directory somehow.
> This is the first step in making our repository public (#29387) but also
> using Hiera as a more elaborate inventory system (#30273).
>
> The idea of switching from LDAP to Hiera for host inventory will
> definitely need to be evaluated more thoroughly before going ahead with
> that part of the conversion, but YAML stuff in Puppet should definitely
> be converted.
>
> The general goal of this is both to allow for a better inventory system
> but also make it easier for people to get onboarded with Puppet. By using
> community standards like Hiera, we make it easier for new people to get
> familiar with the puppet infrastructures and do things meaningfully.

New description:

 We currently use a custom-made YAML database for assigning roles to
 servers and other metadata. I started using Hiera for some hosts and it
 seems to be working well.

 Hiera is officially supported in Puppet and shipped by default in Puppet 5
 and later. It's the standard way of specifying metadata and class
 parameters for hosts. I suspect it covers most of our needs in terms of
 metadata and should cover most if not all of what we're currently doing
 with the YAML stuff in Puppet.

 We should therefore switch to using Hiera instead of our homegrown
 solution.

 This involves converting:

  * `if has_role('foo') { include foo }` into `classes: [ 'foo' ]` in hiera
  * hardcoded macros in the ferm module's `me.conf.erb` into exported
 resources
  * templates looping over allnodeinfo into exported resources
  * the `$roles` array into Hiera
  * the `$localinfo` into Hiera (assuming all the data is there)
  * the `$nodeinfo` and `$allnodeinfo` arrays into Hiera (assuming we can
 switch from LDAP for host inventory)
  * basically any other stuff of the kind, including those files:
    {{{
 ./modules/torproject_org/misc/hoster.yaml
 ./modules/torproject_org/misc/local.yaml
 ./modules/ipsec/misc/config.yaml
 ./modules/roles/misc/static-components.yaml
 ./modules/roles/files/spec/spec-redirects.yaml
 }}}

 Ideally, all YAML data should end up in the hiera/ directory somehow. This
 is the first step in making our repository public (#29387) but also using
 Hiera as a more elaborate inventory system (#30273).

 The idea of switching from LDAP to Hiera for host inventory will
 definitely need to be evaluated more thoroughly before going ahead with
 that part of the conversion, but YAML stuff in Puppet should definitely be
 converted.

 The general goal of this is both to allow for a better inventory system
 but also make it easier for people to get onboarded with Puppet. By using
 community standards like Hiera, we make it easier for new people to get
 familiar with the puppet infrastructures and do things meaningfully.

--

Comment:

 we now have:

 * 3 `has_role` references
 * 4 roles left (`haproxy`, `mail_processing`, `natted`, `no_hw_clock`)
 * 2 `localinfo` references (in postfix, related to `mail_processing`)
 * 13 `allnodeinfo` references
 * 26 `nodeinfo` references

 That's on the stuff that I started working on at all. The `hoster.yaml`
 stuff, in particular, is a whole other ball game. It's less work than the
 larger `local.yaml`, but still impacts a lot of things, which are mostly
 visible in the `nodeinfo` calls:

 {{{
 anarcat at curie:tor-puppet(master)$ git grep -c nodeinfo
 manifests/site.pp:2
 modules/bind/templates/named.conf.puppet-shared-keys.erb:1
 modules/ferm/templates/defs.conf.erb:6
 modules/hosts/templates/etc-hosts.erb:2
 modules/motd/templates/motd.erb:11
 modules/ntp/templates/ntp.conf:1
 modules/postfix/templates/main.cf.erb:2
 modules/postgres/manifests/backup_server/register_backup_clienthost.pp:1
 modules/puppetmaster/lib/puppet/parser/functions/allnodeinfo.rb:2
 modules/puppetmaster/lib/puppet/parser/functions/nodeinfo.rb:22
 modules/resolv/templates/resolv.conf.erb:3
 modules/roles/manifests/onionoo_backend.pp:2
 modules/syslog_ng/templates/syslog-ng.conf.erb:1
 modules/torproject_org/manifests/init.pp:2
 modules/unbound/manifests/init.pp:4
 modules/unbound/templates/unbound.conf.erb:4
 }}}

 Similarly, the `allnodeinfo` construct imports a lot of stuff from LDAP
 into Puppet, which we might want to move into Hiera. That, however, could
 be left for a second phase as it would significantly disrupt the current
 host lifetime workflow.

 The status of the YAML file conversion is as follows:

  * `./modules/torproject_org/misc/hoster.yaml`: not started
  * `./modules/torproject_org/misc/local.yaml`: 53/57 roles done! almost
 finished, see below for the status of the remaining 4
  * `./modules/ipsec/misc/config.yaml`: will be phased out in favor of the
 new exported resource system built for the new networks on fsn-node-*
  * `./modules/roles/misc/static-components.yaml`: maybe easier to keep as
 such for now, or rewrite the static backend to read the file directly?
  * `./modules/roles/files/spec/spec-redirects.yaml`: unsure

 The remaining roles are:

  * `haproxy`: required for syslog-ng configuration, switching to rsyslog
 would make this easier
  * `mail_processing`: requires a refactoring of the postfix module
  * `natted`: small refactoring the `hosts` module, ignore the `nodeinfo`
 stuff, it's not used anywhere according to weasel
  * `no_hw_clock`: small refactoring of the NTP and torproject_org modules

 The bulk of the work will be with `mail_processing` and, obviously, with
 the syslog transition if we go that route.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/30020#comment:8>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online