Jump to content

Dumps/Dumpsdata hosts

From Wikitech
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

XML Dumpsdata hosts

Hardware

We have various hosts:

  • Dumpsdata1003 in eqiad, production misc dumps nfs:
    Hardware/OS: PowerEdge R730xd, Debian 11 (bullseye), 64GB RAM, 1 quad-core Xeon Silver 4112 cpu, HT enabled
    Disks: 12 4TB disks in 1 12-disk raid10 volume; two 1T disks in raid 1 for the OS
  • Dumpsdata1004 in eqiad, production xml/sql dumps nfs spare
    Hardware: PowerEdge R740XD, Debian 11 (bullseye), 64GB RAM, 1 Intel Xeon Silver 4112 cpu, HT enabled
    Disks: 12 4TB disks in 1 12-disk raid10 volume; two 1T disks in raid 1 for the OS
  • Dumpsdata1005 in eqiad, production xml/sql dumps nfs spare
    Hardware: PowerEdge R740XD, Debian 11 (bullseye), 64GB RAM, 1 Intel Xeon Silver 4112 cpu, HT enabled
    Disks: 12 4TB disks in 1 12-disk raid10 volume; two 1T disks in raid 1 for the OS
  • Dumpsdata1006 in eqiad, production xml/sql dumps nfs primary
    Hardware: PowerEdge R740XD, Debian 11 (bullseye), 64GB RAM, 1 Intel Xeon Gold 5220 cpu, HT enabled
    Disks: 12 4TB disks in 1 12-disk raid10 volume; two 1T disks in raid 1 for the OS
  • Dumpsdata1007 in eqiad, production xml/sql and misc dumps nfs fallback
    Hardware: PowerEdge R740XD, Debian 11 (bullseye), 64GB RAM, 1 Intel Xeon Gold 5220 cpu, HT enabled
    Disks: 12 4TB disks in 1 12-disk raid10 volume; two 1T disks in raid 1 for the OS

Services

The production host is nfs-mounted on the snapshot hosts; generated dumps are written there and rsynced from there to the web and rsync servers.

Deploying a new host

Now done by dc ops!

You'll need to set up the raid arrays by hand. The single LVM volume mounted on /data is ext4.

You'll want to make sure your host has the right partman recipe in netboot.cfg (configured in modules/profile/data/profile/installserver/preseed.yaml), at this writing dumpsdata100X.cfg. This will set up sda1 as /boot, one lvm as / and one as /data.

Install in the usual way (add to puppet, copying a pre-existing production dumpsdata host stanza, set up everything for PXE boot and go).

The inital role for the host can be dumps::generation::server::spare from the dc ops folks and we can then move it into production from there. This requires the creation of the file hieradata/hosts/dumpsdataXXXX.yaml (substitute in the right number) stealing from any existing spare or from a fallback host. They will also need to add the new host to the key profile::dumps::peer_hosts in the common/profile/dumps.yaml file.

Example patch (two patches because we make mistakes): https://gerrit.wikimedia.org/r/c/operations/puppet/+/893031 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/893519

If dc ops is uncomfortable with doing that, they may give it to us with the setup role applied, and we can set it up as dumps::generation::server::spare ourselves, doing the above.

The /data partition will likely be set up wrong and you'll have to go fiddle with the LVm and expand it and the filesystem to be a decent size; check the df when you get the host and see how it is first.

Doing the actual swap in of a spare dumpsdata host for a live one

This is extra fiddly because of NFS.

When you have the new dumpsdata host imaged and with the dumps::generation::server::spare role, you'll want to do roughly the following:

TL;DR version -- make a patch to move the live host to spare and the new one to the new role; turn off puppet on those and all snapshots that mount the live share; make sure any jobs running are gone (rsyncs on dumpsdata hosts, dumps on snapshots, etc); unmount the nfs share wherever it is mounted; run puppet on the live host to make it a spare, on the new host to make it the new role, enable puppet on a testbed snapshot and test; if all good, enable it on any other snapshots where the share was mounted and check them.

Long version:

  • Run rpcinfo -p on the new host and compare to the old host, make sure the ports are the same. If not, you might reboot the new host; there is a race condition which sometimes causes the various nfs daemons to start before the changes to defaults have been initially applied via puppet.
  • Make sure the current dumps run is complete, for whichever type of NFS share you are replacing (sql/xml dumps, or "misc" dumps. If you are replacing a fallback host that handles both, make sure the sql/xml dumps are complete.
  • Do a final rsync of all dumps data from the live production dumpsdata host to the new one; if you are swapping in a new fallback host, it may be fallback for both sql/xml dumps and 'other' dumps, and so you may ned rsyncs of both directory trees from the current fallback host for this. A fallback host for the sql/xml dumps should have the systemd timer and job dumps-stats-sender removed as well.
  • Stop puppet on all snapshot hosts with it mounted
  • If this is a new SQL/XML dumps primary NFS share, stop and disable the dumps monitor (systemd job) on the snapshot with the dumps::generation::worker::dumper_monitor role
  • Unmount the live share from the snapshots where it is mounted
  • disable puppet on the live and the new NFS shares
  • If this is a new SQL/XML dumps primary NFS share or a new "other dumps" primary NFS share, on the live host, stop and disable the systemd job that rsyncs the data elsewhere
  • Update puppet manifests to make the live share a dumps::generation::server::spare host and apply the appropriate role to the new dumpsdata host
  • On the new dumpsdata host, enable puppet and run it to make that the new live share
  • On the old host, enable puppet and run it to make that a dumpsdata spare
  • If this is a new SQL/XML dumps primary NFS share, on a snapshot testbed, enable and run puppet, check that the mount comes from the new host, and try a test run of dumps (using test config and test output dir)
  • Enable puppet on each snapshot host where it was disabled, run puppet, check that the mount comes from the new location
  • Update this page (the one you are reading right now) with the new info

= The puppet patch to swap in a new host

You will need to update the entry for the host in hieradata/hosts depending on what it is going to be doing; check the stanza for the existing live host.

If the server will be primary for SQL/XML dumps, it needs to replace the existing hostname in hiera/common.yaml for the dumps_nfs_server key. If the server will be primary for the "misc" dumps, it should replace the hostname in that file for the dumps_cron_nfs_server key instead.

You'll also need to update the host role in manifest/site.pp.

If it is a fallback host, data must be rsynced to it from the primary on a regular basis. To make this happen, add the hostname to the profile::dumps::internal in hieradata/common/profile/dumps.yaml.

If it is to be a primary host and the old primary is to go away, when you are ready to make the switch you will need to change profile::dumps::generation::worker::common so that the dumpsdatamount resource mounts the new server's filesystem on the snapshot hosts instead of mounting the old primary server.

An example of such a patch is here, swapping in a new host for the SQL/XML primary NFS share: https://gerrit.wikimedia.org/r/c/operations/puppet/+/924949/

Reimaging an old host

This assumes you are following the Server Lifecycle/Reimage procedure.

You likely want to preserve all the data on the /data filesystem. To make this happen, in netboot.cfg you'll want to set your host to use dumpsdata100X-no-data-format.cfg. This requires manual intervention during partitioning. Sepcifically, you'll need to select:

  • /dev/sda1 to use as ext4, mount at /boot, format
  • data vg to use as ext4, mount at /data, preserve all contents
  • root vg to use as ext4, mount at /, format

Write the changes to disk and the install will carry on as desired.

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Dumps/Dumpsdata_hosts&oldid=2167532"