Primary database switchover

The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

See Switch master/Archive for the previous version.

Updates

These parts need updating in case of a switchover:

dbctl (for mediawiki config)
puppet
DNS
heartbeat
orchestrator tags
zarcillio
query killer

Steps

There are previous steps before doing the topology change, use this task as reference: https://phabricator.wikimedia.org/T313383

The key part is to first change the topology to get all the hosts to replicate from the new master: Assuming old master: db1181 and new master: db1136

sudo db-switchover --timeout=25 --only-slave-move db1181 db1136

Once that is finished, we are ready to get the section into read-only and then change the master:

To interact with mediawiki dependencies we use dbctl:

sudo dbctl --scope eqiad section sX ro "Maintenance until 06:15 UTC - TXXXX"
sudo dbctl config commit -m "Set sX eqiad as read-only for maintenance - TXXXX"

Now we can perform the switch:

sudo db-switchover --skip-slave-move db1181 db1136

Assuming the above went fine, we can set the section back to WR and promote the new master

sudo dbctl --scope eqiad section sX set-master db1136
sudo dbctl --scope eqiad section sX rw
sudo dbctl config commit -m "Promote db1136 to sX primary and set section read-write TXXXX"

The rest of the steps provided db-switchover by the after the switchover need to be followed in order to ensure that the post-switchover tasks are completed successfully. Check this task for reference: https://phabricator.wikimedia.org/T313383

Please also see the main dbctl article