[tor-bugs] #32937 [Internal Services/Tor Sysadmin Team]: install a new node in the gnt-fsn cluster (fsn-node-03)

[Tor Bug Tracker & Wiki]

#32937: install a new node in the gnt-fsn cluster (fsn-node-03)
-------------------------------------------------+-------------------------
 Reporter:  anarcat                              |          Owner:  anarcat
     Type:  project                              |         Status:
                                                 |  accepted
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:                                       |  Actual Points:
Parent ID:                                       |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------

Comment (by anarcat):

 Rerunning the install:

  1. login

  2. added an explicit step to set the hostname instead of hiding it in the
 disk partitionning

  3. partitionned the disks with the following configuration file:

     {{{
 # open questions
 # --align=optimal?
 # leave keys in /tmp/fai or specify passphrase?
 # use sameas: to set all disk names earlier?
 # bios_grub flag?

 disk_config nvme0n1 disklabel:gpt bootable:2 align-at:1M
 # bios grub second stage
 primary -       8MiB    -       -
 # /boot
 primary -       512MiB  -       -
 # rest is RAID+LUKS+LVM
 primary -       0-      -       -

 disk_config nvme1n1 disklabel:gpt bootable:2 align-at:1M
 # same as above
 primary -       8MiB    -       -
 primary -       512MiB  -       -
 primary -       0-      -       -

 disk_config sda disklabel:gpt align-at:1M
 primary -       0-      -       -

 disk_config sdb disklabel:gpt align-at:1M
 primary -       0-      -       -

 disk_config raid fstabkey:uuid
 raid1   /boot   nvme0n1p2,nvme1n1p2     ext4    rw,noatime,errors=remount-
 ro
 raid1   -       nvme0n1p3,nvme1n1p3     -       -
 raid1   -       sda1,sdb1       -       -

 # FAI defaults to -c aes-xts-plain64 -s 256
 disk_config cryptsetup
 luks    -       /dev/md1        -       -
 luks    -       /dev/md2        -       -

 disk_config lvm fstabkey:uuid
 # previous convention was "vg_$hostname"
 vg      vg_nvme md1
 vg_nvme-root    /       30G     ext4    rw
 vg_nvme-swap    swap    1G      swap    sw

 vg      vg_hdd  md2

 # HDD disks config intentionally left blank
 }}}

  4. install the system, modified version:

 {{{
 mkdir -p /target && mount /dev/vg_nvme/root /target &&
 mkdir -p /target/boot && mount /dev/md0 /target/boot &&
         mkdir -p /target/run && mount -t tmpfs tgt-run /target/run &&
         mkdir /target/run/udev && mount -o bind /run/udev /target/run/udev
 &&
         bootdisk=/dev/nvme1n1 &&
         ROOTPASSWORD=$(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 30) &&
         apt-get install -y grml-debootstrap && \
         sed -e 's/postfix//;
                 s/vlan//;
                 s/bridge-utils//;
                 s/ifenslave//;
                 s/resolvconf//;
                 s/zsh//;
                 s/strace//;
                 s/os-prober//;
                 s/bzip2//;
                 s/file//;
                 s/lsof//;
                 s/most//;
                 $adbus
                 $acryptsetup-initramfs
                 ' /etc/debootstrap/packages > /root/grml-packages &&
         grml-debootstrap --grub "$bootdisk" --target /target \
             --hostname `hostname` --release buster \
             --mirror https://mirror.hetzner.de/debian/packages/ \
             --packages /root/grml-packages \
             --password "$ROOTPASSWORD" \
             --remove-configs --defaultinterfaces &&
         umount /target/run/udev /target/run
 }}}

     I've also reset the LUKS passphrases with:

 {{{
 LUKS_PASSPHRASE=$(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 30) &&
 echo $LUKS_PASSPHRASE | cryptsetup luksAddKey /dev/md1 --key-
 file=/tmp/fai/crypt_dev_md1 &&
 echo $LUKS_PASSPHRASE | cryptsetup luksAddKey /dev/md2 --key-
 file=/tmp/fai/crypt_dev_md2 &&
 cryptsetup luksRemoveKey /dev/md1 --key-file=/tmp/fai/crypt_dev_md1 &&
 cryptsetup luksRemoveKey /dev/md2 --key-file=/tmp/fai/crypt_dev_md2
 }}}

  4. step 4 is replaced with:

     {{{
 ( cat /tmp/fai/fstab ; echo ; echo tmpfs /tmp tmpfs defaults,size=512m 0 0
 ) > /target/etc/fstab
 }}}

     that tmpfs stuff could probably be merged into the setup-storage
 configuration.

  5. this step was step 11 and moved up so we avoid regenerating the initrd
 for nothing

  6. i rewired the luks-setup script so that it correctly deals with
 multiple PVs setup, and hardcoded the "discard" option because i think
 it's fair to assume / is on SSD.

  7. now a noop

  8. done

  9. done, weirdly doesn't match the output of FAI

  10. I had to run this before step 9 to make grub happy:

     {{{
 parted --script /dev/nvme0n1 set 1 bios_grub on
 parted --script /dev/nvme1n1 set 1 bios_grub on
 }}}

  11. network looks good (DHCP)

  12. regen'd, need to figure out how to tell setup-storage to the
 bios_grub magic and fix its mdadm.conf so it matches

  13. unmounted everything

  14. documented in tor-passwords

  15. rebooted

 and it caaaame back! whoohoo! we have a base system installed with setup-
 storage!!!

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/32937#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online