Jump to content

Server Admin Log/Archive 59

From Wikitech
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

2022-11-15

  • 23:54 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudmetrics[1001-1002].eqiad.wmnet
  • 23:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 23:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 23:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T318605)', diff saved to https://phabricator.wikimedia.org/P39860 and previous config saved to /var/cache/conftool/dbconfig/20221115-234056-ladsgroup.json
  • 23:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 23:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 23:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T321130)', diff saved to https://phabricator.wikimedia.org/P39859 and previous config saved to /var/cache/conftool/dbconfig/20221115-233253-marostegui.json
  • 23:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1118 (T318605)', diff saved to https://phabricator.wikimedia.org/P39858 and previous config saved to /var/cache/conftool/dbconfig/20221115-232600-ladsgroup.json
  • 23:25 brennen@deploy1002: Finished scap: Backport for Feed: Use DerivativeContext and not clone main RequestContext (T323153) (duration: 06m 26s)
  • 23:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P39857 and previous config saved to /var/cache/conftool/dbconfig/20221115-232550-ladsgroup.json
  • 23:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 23:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 23:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T318605)', diff saved to https://phabricator.wikimedia.org/P39856 and previous config saved to /var/cache/conftool/dbconfig/20221115-232532-ladsgroup.json
  • 23:21 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@7762e35]: import_cirrus_indexes: snapshot partitioning should not use dashes (duration: 02m 16s)
  • 23:19 brennen@deploy1002: brennen and tstarling: Backport for Feed: Use DerivativeContext and not clone main RequestContext (T323153) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 23:19 brennen@deploy1002: Started scap: Backport for Feed: Use DerivativeContext and not clone main RequestContext (T323153)
  • 23:18 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@7762e35]: import_cirrus_indexes: snapshot partitioning should not use dashes
  • 23:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P39855 and previous config saved to /var/cache/conftool/dbconfig/20221115-231746-marostegui.json
  • 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P39854 and previous config saved to /var/cache/conftool/dbconfig/20221115-231043-ladsgroup.json
  • 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P39853 and previous config saved to /var/cache/conftool/dbconfig/20221115-231025-ladsgroup.json
  • 23:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P39852 and previous config saved to /var/cache/conftool/dbconfig/20221115-230240-marostegui.json
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T318605)', diff saved to https://phabricator.wikimedia.org/P39851 and previous config saved to /var/cache/conftool/dbconfig/20221115-225537-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P39850 and previous config saved to /var/cache/conftool/dbconfig/20221115-225518-ladsgroup.json
  • 22:52 ejegg: re-enabled civicrm dedupe jobs
  • 22:51 mutante: phab1004 - running public_task_dump.py
  • 22:51 ejegg: civicrm upgraded from d85589e8 to fa71f219
  • 22:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T321130)', diff saved to https://phabricator.wikimedia.org/P39849 and previous config saved to /var/cache/conftool/dbconfig/20221115-224733-marostegui.json
  • 22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T318605)', diff saved to https://phabricator.wikimedia.org/P39848 and previous config saved to /var/cache/conftool/dbconfig/20221115-224011-ladsgroup.json
  • 22:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T321130)', diff saved to https://phabricator.wikimedia.org/P39847 and previous config saved to /var/cache/conftool/dbconfig/20221115-221247-marostegui.json
  • 22:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 22:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 22:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T321130)', diff saved to https://phabricator.wikimedia.org/P39846 and previous config saved to /var/cache/conftool/dbconfig/20221115-221225-marostegui.json
  • 21:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P39845 and previous config saved to /var/cache/conftool/dbconfig/20221115-215719-marostegui.json
  • 21:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P39844 and previous config saved to /var/cache/conftool/dbconfig/20221115-214212-marostegui.json
  • 21:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321126)', diff saved to https://phabricator.wikimedia.org/P39843 and previous config saved to /var/cache/conftool/dbconfig/20221115-214022-marostegui.json
  • 21:33 cjming: end of UTC late backport window
  • 21:27 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:27 cjming@deploy1002: Finished scap: Backport for tnwiki: Set timezone to Africa/Gaborone (UTC+2) (T318208) (duration: 06m 14s)
  • 21:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T321130)', diff saved to https://phabricator.wikimedia.org/P39842 and previous config saved to /var/cache/conftool/dbconfig/20221115-212706-marostegui.json
  • 21:25 cmooney@cumin1001: START - Cookbook sre.dns.netbox
  • 21:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39841 and previous config saved to /var/cache/conftool/dbconfig/20221115-212516-marostegui.json
  • 21:21 cjming@deploy1002: cjming and stang: Backport for tnwiki: Set timezone to Africa/Gaborone (UTC+2) (T318208) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 21:21 cjming@deploy1002: Started scap: Backport for tnwiki: Set timezone to Africa/Gaborone (UTC+2) (T318208)
  • 21:19 cjming@deploy1002: Finished scap: Backport for logos: Remove duplicated code (T307705) (duration: 04m 31s)
  • 21:15 cjming@deploy1002: cjming and stang: Backport for logos: Remove duplicated code (T307705) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 21:14 cjming@deploy1002: Started scap: Backport for logos: Remove duplicated code (T307705)
  • 21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2130 (T318605)', diff saved to https://phabricator.wikimedia.org/P39840 and previous config saved to /var/cache/conftool/dbconfig/20221115-211314-ladsgroup.json
  • 21:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 21:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T318605)', diff saved to https://phabricator.wikimedia.org/P39839 and previous config saved to /var/cache/conftool/dbconfig/20221115-211253-ladsgroup.json
  • 21:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39838 and previous config saved to /var/cache/conftool/dbconfig/20221115-211009-marostegui.json
  • 21:09 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:09 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:08 cjming@deploy1002: Finished scap: Backport for EditAttemptStep sampling rate to 1 everywhere (T312016) (duration: 04m 45s)
  • 21:07 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:07 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:04 cjming@deploy1002: cjming and phuedx: Backport for EditAttemptStep sampling rate to 1 everywhere (T312016) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 21:03 cjming@deploy1002: Started scap: Backport for EditAttemptStep sampling rate to 1 everywhere (T312016)
  • 21:02 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:01 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1107 (T318605)', diff saved to https://phabricator.wikimedia.org/P39837 and previous config saved to /var/cache/conftool/dbconfig/20221115-205929-ladsgroup.json
  • 20:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 20:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 20:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T318605)', diff saved to https://phabricator.wikimedia.org/P39836 and previous config saved to /var/cache/conftool/dbconfig/20221115-205907-ladsgroup.json
  • 20:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P39835 and previous config saved to /var/cache/conftool/dbconfig/20221115-205746-ladsgroup.json
  • 20:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321126)', diff saved to https://phabricator.wikimedia.org/P39834 and previous config saved to /var/cache/conftool/dbconfig/20221115-205503-marostegui.json
  • 20:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T321126)', diff saved to https://phabricator.wikimedia.org/P39833 and previous config saved to /var/cache/conftool/dbconfig/20221115-205222-marostegui.json
  • 20:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 20:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T321130)', diff saved to https://phabricator.wikimedia.org/P39832 and previous config saved to /var/cache/conftool/dbconfig/20221115-205214-marostegui.json
  • 20:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 20:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 20:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39831 and previous config saved to /var/cache/conftool/dbconfig/20221115-205201-marostegui.json
  • 20:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 20:49 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dbprov2004.mgmt.codfw.wmnet with reboot policy GRACEFUL
  • 20:44 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 20:44 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P39830 and previous config saved to /var/cache/conftool/dbconfig/20221115-204401-ladsgroup.json
  • 20:42 eileen: civicrm upgraded from 16167e9a to d85589e8
  • 20:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P39829 and previous config saved to /var/cache/conftool/dbconfig/20221115-204239-ladsgroup.json
  • 20:39 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy GRACEFUL
  • 20:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39828 and previous config saved to /var/cache/conftool/dbconfig/20221115-203654-marostegui.json
  • 20:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 20:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 20:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T321130)', diff saved to https://phabricator.wikimedia.org/P39827 and previous config saved to /var/cache/conftool/dbconfig/20221115-203521-marostegui.json
  • 20:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P39826 and previous config saved to /var/cache/conftool/dbconfig/20221115-202854-ladsgroup.json
  • 20:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T318605)', diff saved to https://phabricator.wikimedia.org/P39825 and previous config saved to /var/cache/conftool/dbconfig/20221115-202733-ladsgroup.json
  • 20:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39824 and previous config saved to /var/cache/conftool/dbconfig/20221115-202148-marostegui.json
  • 20:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P39823 and previous config saved to /var/cache/conftool/dbconfig/20221115-202015-marostegui.json
  • 20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T318605)', diff saved to https://phabricator.wikimedia.org/P39822 and previous config saved to /var/cache/conftool/dbconfig/20221115-201348-ladsgroup.json
  • 20:13 eileen: civicrm upgraded from 3eba6ad3 to 16167e9a
  • 20:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39821 and previous config saved to /var/cache/conftool/dbconfig/20221115-200641-marostegui.json
  • 20:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P39820 and previous config saved to /var/cache/conftool/dbconfig/20221115-200508-marostegui.json
  • 20:04 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39819 and previous config saved to /var/cache/conftool/dbconfig/20221115-200358-marostegui.json
  • 20:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 20:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 20:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T321126)', diff saved to https://phabricator.wikimedia.org/P39818 and previous config saved to /var/cache/conftool/dbconfig/20221115-200337-marostegui.json
  • 19:56 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:56 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:54 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:54 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:54 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:51 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2042.codfw.wmnet with OS bullseye
  • 19:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T321130)', diff saved to https://phabricator.wikimedia.org/P39817 and previous config saved to /var/cache/conftool/dbconfig/20221115-195002-marostegui.json
  • 19:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P39816 and previous config saved to /var/cache/conftool/dbconfig/20221115-194830-marostegui.json
  • 19:42 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:42 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T321130)', diff saved to https://phabricator.wikimedia.org/P39815 and previous config saved to /var/cache/conftool/dbconfig/20221115-194037-marostegui.json
  • 19:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T321130)', diff saved to https://phabricator.wikimedia.org/P39814 and previous config saved to /var/cache/conftool/dbconfig/20221115-194016-marostegui.json
  • 19:38 ejegg: turned off CiviCRM dedupe jobs for queue speed measurements
  • 19:38 ejegg: payments-wiki upgraded from a058fdbc to bba997aa
  • 19:34 jbond: updated pcc to 2.5.1
  • 19:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P39813 and previous config saved to /var/cache/conftool/dbconfig/20221115-193324-marostegui.json
  • 19:32 topranks: renumbering overlay vrf loopback interface lsw1-e3-eqiad
  • 19:28 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:28 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P39812 and previous config saved to /var/cache/conftool/dbconfig/20221115-192509-marostegui.json
  • 19:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T321126)', diff saved to https://phabricator.wikimedia.org/P39811 and previous config saved to /var/cache/conftool/dbconfig/20221115-191818-marostegui.json
  • 19:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T321126)', diff saved to https://phabricator.wikimedia.org/P39810 and previous config saved to /var/cache/conftool/dbconfig/20221115-191536-marostegui.json
  • 19:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 19:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 19:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39809 and previous config saved to /var/cache/conftool/dbconfig/20221115-191514-marostegui.json
  • 19:10 brennen@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.10 refs T320515
  • 19:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P39808 and previous config saved to /var/cache/conftool/dbconfig/20221115-191003-marostegui.json
  • 19:04 brennen: train 1.40.0-wmf.10 (T320515) - no current blockers, rolling to group0.
  • 19:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P39807 and previous config saved to /var/cache/conftool/dbconfig/20221115-190008-marostegui.json
  • 18:58 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp2042.codfw.wmnet with OS bullseye
  • 18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2116 (T318605)', diff saved to https://phabricator.wikimedia.org/P39806 and previous config saved to /var/cache/conftool/dbconfig/20221115-185619-ladsgroup.json
  • 18:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 18:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 18:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T318605)', diff saved to https://phabricator.wikimedia.org/P39805 and previous config saved to /var/cache/conftool/dbconfig/20221115-185558-ladsgroup.json
  • 18:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T321130)', diff saved to https://phabricator.wikimedia.org/P39804 and previous config saved to /var/cache/conftool/dbconfig/20221115-185457-marostegui.json
  • 18:49 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5032.eqsin.wmnet with OS buster
  • 18:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P39803 and previous config saved to /var/cache/conftool/dbconfig/20221115-184501-marostegui.json
  • 18:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P39802 and previous config saved to /var/cache/conftool/dbconfig/20221115-184051-ladsgroup.json
  • 18:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T318605)', diff saved to https://phabricator.wikimedia.org/P39801 and previous config saved to /var/cache/conftool/dbconfig/20221115-183053-ladsgroup.json
  • 18:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 18:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 18:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 18:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 18:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39800 and previous config saved to /var/cache/conftool/dbconfig/20221115-183025-ladsgroup.json
  • 18:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39799 and previous config saved to /var/cache/conftool/dbconfig/20221115-182955-marostegui.json
  • 18:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39798 and previous config saved to /var/cache/conftool/dbconfig/20221115-182712-marostegui.json
  • 18:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T321126)', diff saved to https://phabricator.wikimedia.org/P39797 and previous config saved to /var/cache/conftool/dbconfig/20221115-182640-marostegui.json
  • 18:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P39796 and previous config saved to /var/cache/conftool/dbconfig/20221115-182545-ladsgroup.json
  • 18:16 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage
  • 18:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P39795 and previous config saved to /var/cache/conftool/dbconfig/20221115-181519-ladsgroup.json
  • 18:13 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage
  • 18:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P39794 and previous config saved to /var/cache/conftool/dbconfig/20221115-181133-marostegui.json
  • 18:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T318605)', diff saved to https://phabricator.wikimedia.org/P39793 and previous config saved to /var/cache/conftool/dbconfig/20221115-181037-ladsgroup.json
  • 18:06 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=idwiki` at mwmaint1002 (T318457)
  • 18:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P39792 and previous config saved to /var/cache/conftool/dbconfig/20221115-180012-ladsgroup.json
  • 17:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P39791 and previous config saved to /var/cache/conftool/dbconfig/20221115-175627-marostegui.json
  • 17:54 jbond: move pcc to 2.5.0
  • 17:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T321130)', diff saved to https://phabricator.wikimedia.org/P39790 and previous config saved to /var/cache/conftool/dbconfig/20221115-174827-marostegui.json
  • 17:48 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39789 and previous config saved to /var/cache/conftool/dbconfig/20221115-174805-marostegui.json
  • 17:46 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 17:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39788 and previous config saved to /var/cache/conftool/dbconfig/20221115-174506-ladsgroup.json
  • 17:42 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS buster
  • 17:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T321126)', diff saved to https://phabricator.wikimedia.org/P39787 and previous config saved to /var/cache/conftool/dbconfig/20221115-174120-marostegui.json
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T321126)', diff saved to https://phabricator.wikimedia.org/P39786 and previous config saved to /var/cache/conftool/dbconfig/20221115-173841-marostegui.json
  • 17:38 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T321126)', diff saved to https://phabricator.wikimedia.org/P39785 and previous config saved to /var/cache/conftool/dbconfig/20221115-173804-marostegui.json
  • 17:37 hnowlan@cumin1001: END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0)
  • 17:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P39784 and previous config saved to /var/cache/conftool/dbconfig/20221115-173258-marostegui.json
  • 17:29 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 17:29 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 17:28 mutante: deploy1002:~] $ sudo systemctl start wmf_auto_restart_apache2.service
  • 17:25 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 17:25 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 17:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P39783 and previous config saved to /var/cache/conftool/dbconfig/20221115-172257-marostegui.json
  • 17:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P39782 and previous config saved to /var/cache/conftool/dbconfig/20221115-171752-marostegui.json
  • 17:16 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=enwiki` at mwmaint1002 (T318457)
  • 17:10 godog: add 150G to prometheus/ops in eqiad
  • 17:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P39781 and previous config saved to /var/cache/conftool/dbconfig/20221115-170751-marostegui.json
  • 17:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39780 and previous config saved to /var/cache/conftool/dbconfig/20221115-170245-marostegui.json
  • 17:02 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=fawiki` at mwmaint1002 (T318457)
  • 16:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39779 and previous config saved to /var/cache/conftool/dbconfig/20221115-165323-marostegui.json
  • 16:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 16:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 16:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T321130)', diff saved to https://phabricator.wikimedia.org/P39778 and previous config saved to /var/cache/conftool/dbconfig/20221115-165302-marostegui.json
  • 16:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T321126)', diff saved to https://phabricator.wikimedia.org/P39777 and previous config saved to /var/cache/conftool/dbconfig/20221115-165244-marostegui.json
  • 16:50 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T321126)', diff saved to https://phabricator.wikimedia.org/P39776 and previous config saved to /var/cache/conftool/dbconfig/20221115-165001-marostegui.json
  • 16:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 16:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 16:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T321126)', diff saved to https://phabricator.wikimedia.org/P39775 and previous config saved to /var/cache/conftool/dbconfig/20221115-164939-marostegui.json
  • 16:49 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 16:43 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry
  • 16:41 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry
  • 16:40 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 16:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P39774 and previous config saved to /var/cache/conftool/dbconfig/20221115-163755-marostegui.json
  • 16:37 ladsgroup:: Deployed security patch for T320987
  • 16:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2103 (T318605)', diff saved to https://phabricator.wikimedia.org/P39773 and previous config saved to /var/cache/conftool/dbconfig/20221115-163721-ladsgroup.json
  • 16:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 16:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 16:36 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wcqs-public
  • 16:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P39772 and previous config saved to /var/cache/conftool/dbconfig/20221115-163432-marostegui.json
  • 16:34 ladsgroup:: Deployed security patch for T320987
  • 16:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1014.eqiad.wmnet
  • 16:22 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wcqs-public
  • 16:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P39770 and previous config saved to /var/cache/conftool/dbconfig/20221115-162249-marostegui.json
  • 16:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1022.eqiad.wmnet with OS bullseye
  • 16:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet
  • 16:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1014.eqiad.wmnet
  • 16:20 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=arwiki` at mwmaint1002 (T318457)
  • 16:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P39769 and previous config saved to /var/cache/conftool/dbconfig/20221115-161925-marostegui.json
  • 16:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS bullseye
  • 16:15 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wdqs-all
  • 16:15 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 16:14 urandom: initiating Cassandra bootstrap, aqs1019-b -- T307802
  • 16:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet
  • 16:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet
  • 16:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39768 and previous config saved to /var/cache/conftool/dbconfig/20221115-160804-ladsgroup.json
  • 16:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 16:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 16:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T321130)', diff saved to https://phabricator.wikimedia.org/P39767 and previous config saved to /var/cache/conftool/dbconfig/20221115-160742-marostegui.json
  • 16:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1012.eqiad.wmnet
  • 16:06 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wdqs-all
  • 16:05 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 16:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet
  • 16:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T321126)', diff saved to https://phabricator.wikimedia.org/P39766 and previous config saved to /var/cache/conftool/dbconfig/20221115-160419-marostegui.json
  • 16:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
  • 16:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1022.eqiad.wmnet with reason: host reimage
  • 16:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1012.eqiad.wmnet
  • 16:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T321126)', diff saved to https://phabricator.wikimedia.org/P39765 and previous config saved to /var/cache/conftool/dbconfig/20221115-160140-marostegui.json
  • 16:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 16:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T321126)', diff saved to https://phabricator.wikimedia.org/P39764 and previous config saved to /var/cache/conftool/dbconfig/20221115-160010-marostegui.json
  • 15:59 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1022.eqiad.wmnet with reason: host reimage
  • 15:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T321130)', diff saved to https://phabricator.wikimedia.org/P39763 and previous config saved to /var/cache/conftool/dbconfig/20221115-155821-marostegui.json
  • 15:58 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
  • 15:58 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 15:58 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 15:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321130)', diff saved to https://phabricator.wikimedia.org/P39762 and previous config saved to /var/cache/conftool/dbconfig/20221115-155800-marostegui.json
  • 15:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
  • 15:54 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P39761 and previous config saved to /var/cache/conftool/dbconfig/20221115-155237-ladsgroup.json
  • 15:49 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=ptwiki` at mwmaint1002 (T318457)
  • 15:45 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1022.eqiad.wmnet with OS bullseye
  • 15:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P39760 and previous config saved to /var/cache/conftool/dbconfig/20221115-154504-marostegui.json
  • 15:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1022.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 15:44 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1022.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 15:43 moritzm: uploaded cas 6.6.2 to apt.wikimedia.org T311235
  • 15:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P39759 and previous config saved to /var/cache/conftool/dbconfig/20221115-154253-marostegui.json
  • 15:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P39758 and previous config saved to /var/cache/conftool/dbconfig/20221115-153731-ladsgroup.json
  • 15:33 taavi@deploy1002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 15:32 taavi@deploy1002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 15:32 taavi@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 15:31 taavi@deploy1002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 15:30 taavi@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 15:30 taavi@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 15:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P39757 and previous config saved to /var/cache/conftool/dbconfig/20221115-152957-marostegui.json
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P39756 and previous config saved to /var/cache/conftool/dbconfig/20221115-152747-marostegui.json
  • 15:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39755 and previous config saved to /var/cache/conftool/dbconfig/20221115-152224-ladsgroup.json
  • 15:15 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=bnwiki` at mwmaint1002 (T318457)
  • 15:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T321126)', diff saved to https://phabricator.wikimedia.org/P39754 and previous config saved to /var/cache/conftool/dbconfig/20221115-151451-marostegui.json
  • 15:14 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS bullseye
  • 15:14 moritzm: installing expat security updates
  • 15:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321130)', diff saved to https://phabricator.wikimedia.org/P39753 and previous config saved to /var/cache/conftool/dbconfig/20221115-151241-marostegui.json
  • 15:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T321126)', diff saved to https://phabricator.wikimedia.org/P39752 and previous config saved to /var/cache/conftool/dbconfig/20221115-151232-marostegui.json
  • 15:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 15:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 15:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T321126)', diff saved to https://phabricator.wikimedia.org/P39751 and previous config saved to /var/cache/conftool/dbconfig/20221115-151211-marostegui.json
  • 15:10 hnowlan@cumin1001: START - Cookbook sre.postgresql.postgres-init
  • 15:09 hnowlan@cumin1001: END (FAIL) - Cookbook sre.postgresql.postgres-init (exit_code=99)
  • 15:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318950)', diff saved to https://phabricator.wikimedia.org/P39750 and previous config saved to /var/cache/conftool/dbconfig/20221115-150901-ladsgroup.json
  • 15:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3003.esams.wmnet
  • 15:06 hnowlan@cumin1001: START - Cookbook sre.postgresql.postgres-init
  • 15:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T318955)', diff saved to https://phabricator.wikimedia.org/P39749 and previous config saved to /var/cache/conftool/dbconfig/20221115-150242-ladsgroup.json
  • 15:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 15:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti3003.esams.wmnet
  • 15:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 14:59 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=frwiki` at mwmaint1002 (T318457)
  • 14:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2032.codfw.wmnet
  • 14:57 urbanecm@deploy1002: Finished scap: Backport for updateIsActiveFlagForMentees: Process all mentees (T318457), MentorStore: Use $wgRCMaxAge instead of INACTIVITY_THRESHOLD (T318457) (duration: 05m 44s)
  • 14:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P39748 and previous config saved to /var/cache/conftool/dbconfig/20221115-145704-marostegui.json
  • 14:55 moritzm: installing tomcat security updates
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39747 and previous config saved to /var/cache/conftool/dbconfig/20221115-145355-ladsgroup.json
  • 14:53 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 14:52 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2032.codfw.wmnet
  • 14:52 urbanecm@deploy1002: urbanecm and urbanecm: Backport for updateIsActiveFlagForMentees: Process all mentees (T318457), MentorStore: Use $wgRCMaxAge instead of INACTIVITY_THRESHOLD (T318457) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 14:51 urbanecm@deploy1002: Started scap: Backport for updateIsActiveFlagForMentees: Process all mentees (T318457), MentorStore: Use $wgRCMaxAge instead of INACTIVITY_THRESHOLD (T318457)
  • 14:51 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp5032']
  • 14:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2031.codfw.wmnet
  • 14:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P39746 and previous config saved to /var/cache/conftool/dbconfig/20221115-144736-ladsgroup.json
  • 14:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2031.codfw.wmnet
  • 14:43 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5032']
  • 14:43 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cp5032']
  • 14:43 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 14:42 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2003-dev.codfw.wmnet with OS bullseye
  • 14:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P39745 and previous config saved to /var/cache/conftool/dbconfig/20221115-144158-marostegui.json
  • 14:40 moritzm: failover ganeti master in esams to ganeti3001
  • 14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39744 and previous config saved to /var/cache/conftool/dbconfig/20221115-143848-ladsgroup.json
  • 14:33 hnowlan@cumin1001: END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0)
  • 14:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P39743 and previous config saved to /var/cache/conftool/dbconfig/20221115-143229-ladsgroup.json
  • 14:31 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5032']
  • 14:27 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
  • 14:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3002.esams.wmnet
  • 14:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T321126)', diff saved to https://phabricator.wikimedia.org/P39742 and previous config saved to /var/cache/conftool/dbconfig/20221115-142652-marostegui.json
  • 14:25 aborrero@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
  • 14:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T321126)', diff saved to https://phabricator.wikimedia.org/P39741 and previous config saved to /var/cache/conftool/dbconfig/20221115-142432-marostegui.json
  • 14:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 14:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 14:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T321126)', diff saved to https://phabricator.wikimedia.org/P39740 and previous config saved to /var/cache/conftool/dbconfig/20221115-142411-marostegui.json
  • 14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318950)', diff saved to https://phabricator.wikimedia.org/P39739 and previous config saved to /var/cache/conftool/dbconfig/20221115-142342-ladsgroup.json
  • 14:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T318950)', diff saved to https://phabricator.wikimedia.org/P39738 and previous config saved to /var/cache/conftool/dbconfig/20221115-142130-ladsgroup.json
  • 14:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 14:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 14:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti3002.esams.wmnet
  • 14:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T318955)', diff saved to https://phabricator.wikimedia.org/P39737 and previous config saved to /var/cache/conftool/dbconfig/20221115-141723-ladsgroup.json
  • 14:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T318955)', diff saved to https://phabricator.wikimedia.org/P39736 and previous config saved to /var/cache/conftool/dbconfig/20221115-141513-ladsgroup.json
  • 14:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 14:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 14:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T321130)', diff saved to https://phabricator.wikimedia.org/P39735 and previous config saved to /var/cache/conftool/dbconfig/20221115-141218-marostegui.json
  • 14:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 14:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 14:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39734 and previous config saved to /var/cache/conftool/dbconfig/20221115-141157-marostegui.json
  • 14:09 aborrero@cumin1001: START - Cookbook sre.hosts.reimage for host cloudgw2003-dev.codfw.wmnet with OS bullseye
  • 14:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P39733 and previous config saved to /var/cache/conftool/dbconfig/20221115-140905-marostegui.json
  • 14:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp5032.mgmt.eqsin.wmnet with reboot policy FORCED
  • 14:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3001.esams.wmnet
  • 13:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P39732 and previous config saved to /var/cache/conftool/dbconfig/20221115-135650-marostegui.json
  • 13:56 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cp5032.mgmt.eqsin.wmnet with reboot policy FORCED
  • 13:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P39731 and previous config saved to /var/cache/conftool/dbconfig/20221115-135358-marostegui.json
  • 13:52 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti3001.esams.wmnet
  • 13:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P39730 and previous config saved to /var/cache/conftool/dbconfig/20221115-134144-marostegui.json
  • 13:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 13:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39729 and previous config saved to /var/cache/conftool/dbconfig/20221115-134036-ladsgroup.json
  • 13:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 13:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 13:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 13:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T321126)', diff saved to https://phabricator.wikimedia.org/P39728 and previous config saved to /var/cache/conftool/dbconfig/20221115-133852-marostegui.json
  • 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T321126)', diff saved to https://phabricator.wikimedia.org/P39727 and previous config saved to /var/cache/conftool/dbconfig/20221115-133631-marostegui.json
  • 13:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 13:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39726 and previous config saved to /var/cache/conftool/dbconfig/20221115-133610-marostegui.json
  • 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet
  • 13:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet
  • 13:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39725 and previous config saved to /var/cache/conftool/dbconfig/20221115-132637-marostegui.json
  • 13:22 sukhe: running homer for Gerrit: 856946 in cr*-ulsfo*
  • 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39724 and previous config saved to /var/cache/conftool/dbconfig/20221115-132103-marostegui.json
  • 13:20 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 13:20 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 13:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 13335
  • 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39723 and previous config saved to /var/cache/conftool/dbconfig/20221115-131710-marostegui.json
  • 13:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 13:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 13:08 moritzm: failover ganeti master in ulsfo to ganeti4005
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39722 and previous config saved to /var/cache/conftool/dbconfig/20221115-130557-marostegui.json
  • 13:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321130)', diff saved to https://phabricator.wikimedia.org/P39721 and previous config saved to /var/cache/conftool/dbconfig/20221115-125950-marostegui.json
  • 12:59 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 13335
  • 12:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
  • 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39720 and previous config saved to /var/cache/conftool/dbconfig/20221115-125050-marostegui.json
  • 12:49 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2003-dev.codfw.wmnet with OS bullseye
  • 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39719 and previous config saved to /var/cache/conftool/dbconfig/20221115-124830-marostegui.json
  • 12:48 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 12:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T321126)', diff saved to https://phabricator.wikimedia.org/P39718 and previous config saved to /var/cache/conftool/dbconfig/20221115-124808-marostegui.json
  • 12:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
  • 12:45 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on idp-test1002.wikimedia.org with reason: experiment with CAS 6.6
  • 12:45 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on idp-test1002.wikimedia.org with reason: experiment with CAS 6.6
  • 12:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P39717 and previous config saved to /var/cache/conftool/dbconfig/20221115-124443-marostegui.json
  • 12:43 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=varnish-fe
  • 12:43 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=ats-be
  • 12:43 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=ats-tls
  • 12:37 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39716 and previous config saved to /var/cache/conftool/dbconfig/20221115-123735-root.json
  • 12:36 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39715 and previous config saved to /var/cache/conftool/dbconfig/20221115-123326-root.json
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39714 and previous config saved to /var/cache/conftool/dbconfig/20221115-123302-marostegui.json
  • 12:31 aborrero@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
  • 12:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P39713 and previous config saved to /var/cache/conftool/dbconfig/20221115-122937-marostegui.json
  • 12:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
  • 12:22 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39712 and previous config saved to /var/cache/conftool/dbconfig/20221115-122230-root.json
  • 12:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
  • 12:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39711 and previous config saved to /var/cache/conftool/dbconfig/20221115-121821-root.json
  • 12:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39710 and previous config saved to /var/cache/conftool/dbconfig/20221115-121755-marostegui.json
  • 12:16 aborrero@cumin1001: START - Cookbook sre.hosts.reimage for host cloudgw2003-dev.codfw.wmnet with OS bullseye
  • 12:14 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321130)', diff saved to https://phabricator.wikimedia.org/P39709 and previous config saved to /var/cache/conftool/dbconfig/20221115-121431-marostegui.json
  • 12:13 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:11 hnowlan: resyncing maps2005 replica
  • 12:11 hnowlan@cumin1001: START - Cookbook sre.postgresql.postgres-init
  • 12:07 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39708 and previous config saved to /var/cache/conftool/dbconfig/20221115-120725-root.json
  • 12:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T321130)', diff saved to https://phabricator.wikimedia.org/P39707 and previous config saved to /var/cache/conftool/dbconfig/20221115-120502-marostegui.json
  • 12:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 12:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 12:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39706 and previous config saved to /var/cache/conftool/dbconfig/20221115-120316-root.json
  • 12:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T321126)', diff saved to https://phabricator.wikimedia.org/P39705 and previous config saved to /var/cache/conftool/dbconfig/20221115-120249-marostegui.json
  • 12:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T321126)', diff saved to https://phabricator.wikimedia.org/P39704 and previous config saved to /var/cache/conftool/dbconfig/20221115-120030-marostegui.json
  • 12:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 12:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 12:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T321126)', diff saved to https://phabricator.wikimedia.org/P39703 and previous config saved to /var/cache/conftool/dbconfig/20221115-120009-marostegui.json
  • 11:52 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39702 and previous config saved to /var/cache/conftool/dbconfig/20221115-115220-root.json
  • 11:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39701 and previous config saved to /var/cache/conftool/dbconfig/20221115-114812-root.json
  • 11:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 11:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 11:45 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti6003.drmrs.wmnet
  • 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P39700 and previous config saved to /var/cache/conftool/dbconfig/20221115-114502-marostegui.json
  • 11:42 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2002-dev.codfw.wmnet with OS bullseye
  • 11:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39699 and previous config saved to /var/cache/conftool/dbconfig/20221115-114216-marostegui.json
  • 11:37 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39698 and previous config saved to /var/cache/conftool/dbconfig/20221115-113715-root.json
  • 11:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
  • 11:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39697 and previous config saved to /var/cache/conftool/dbconfig/20221115-113307-root.json
  • 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P39696 and previous config saved to /var/cache/conftool/dbconfig/20221115-112956-marostegui.json
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39695 and previous config saved to /var/cache/conftool/dbconfig/20221115-112709-marostegui.json
  • 11:22 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
  • 11:22 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39694 and previous config saved to /var/cache/conftool/dbconfig/20221115-112210-root.json
  • 11:20 moritzm: failover ganeti master in drmrs/B12 to ganeti6001
  • 11:20 aborrero@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
  • 11:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39693 and previous config saved to /var/cache/conftool/dbconfig/20221115-111802-root.json
  • 11:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T321126)', diff saved to https://phabricator.wikimedia.org/P39692 and previous config saved to /var/cache/conftool/dbconfig/20221115-111449-marostegui.json
  • 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T321126)', diff saved to https://phabricator.wikimedia.org/P39691 and previous config saved to /var/cache/conftool/dbconfig/20221115-111229-marostegui.json
  • 11:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39690 and previous config saved to /var/cache/conftool/dbconfig/20221115-111203-marostegui.json
  • 11:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 11:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39689 and previous config saved to /var/cache/conftool/dbconfig/20221115-111150-marostegui.json
  • 11:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
  • 11:07 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39688 and previous config saved to /var/cache/conftool/dbconfig/20221115-110705-root.json
  • 11:05 aborrero@cumin1001: START - Cookbook sre.hosts.reimage for host cloudgw2002-dev.codfw.wmnet with OS bullseye
  • 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39687 and previous config saved to /var/cache/conftool/dbconfig/20221115-110257-root.json
  • 10:59 mfossati@deploy1002: Finished deploy [airflow-dags/platform_eng@3bb99c2]: (no justification provided) (duration: 00m 05s)
  • 10:59 mfossati@deploy1002: Started deploy [airflow-dags/platform_eng@3bb99c2]: (no justification provided)
  • 10:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39686 and previous config saved to /var/cache/conftool/dbconfig/20221115-105657-marostegui.json
  • 10:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P39685 and previous config saved to /var/cache/conftool/dbconfig/20221115-105644-marostegui.json
  • 10:52 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39684 and previous config saved to /var/cache/conftool/dbconfig/20221115-105200-root.json
  • 10:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39683 and previous config saved to /var/cache/conftool/dbconfig/20221115-104752-root.json
  • 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39682 and previous config saved to /var/cache/conftool/dbconfig/20221115-104435-marostegui.json
  • 10:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 10:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39681 and previous config saved to /var/cache/conftool/dbconfig/20221115-104402-marostegui.json
  • 10:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P39680 and previous config saved to /var/cache/conftool/dbconfig/20221115-104137-marostegui.json
  • 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39679 and previous config saved to /var/cache/conftool/dbconfig/20221115-102856-marostegui.json
  • 10:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39678 and previous config saved to /var/cache/conftool/dbconfig/20221115-102631-marostegui.json
  • 10:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39677 and previous config saved to /var/cache/conftool/dbconfig/20221115-102409-marostegui.json
  • 10:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39676 and previous config saved to /var/cache/conftool/dbconfig/20221115-102319-marostegui.json
  • 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39675 and previous config saved to /var/cache/conftool/dbconfig/20221115-101349-marostegui.json
  • 10:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P39674 and previous config saved to /var/cache/conftool/dbconfig/20221115-100812-marostegui.json
  • 09:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39673 and previous config saved to /var/cache/conftool/dbconfig/20221115-095843-marostegui.json
  • 09:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P39672 and previous config saved to /var/cache/conftool/dbconfig/20221115-095306-marostegui.json
  • 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39671 and previous config saved to /var/cache/conftool/dbconfig/20221115-094552-marostegui.json
  • 09:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39670 and previous config saved to /var/cache/conftool/dbconfig/20221115-093758-marostegui.json
  • 09:36 moritzm: draining ganeti1022 for eventual reimage T311687
  • 09:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39669 and previous config saved to /var/cache/conftool/dbconfig/20221115-093539-marostegui.json
  • 09:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 09:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 09:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39668 and previous config saved to /var/cache/conftool/dbconfig/20221115-093518-marostegui.json
  • 09:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P39667 and previous config saved to /var/cache/conftool/dbconfig/20221115-092011-marostegui.json
  • 09:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts bast5001.wikimedia.org
  • 09:14 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:12 jmm@cumin2002: START - Cookbook sre.dns.netbox
  • 09:07 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts bast5001.wikimedia.org
  • 09:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P39666 and previous config saved to /var/cache/conftool/dbconfig/20221115-090505-marostegui.json
  • 09:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2162', diff saved to https://phabricator.wikimedia.org/P39664 and previous config saved to /var/cache/conftool/dbconfig/20221115-090058-root.json
  • 08:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1138.eqiad.wmnet with reason: Maintenance
  • 08:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1138.eqiad.wmnet with reason: Maintenance
  • 08:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39663 and previous config saved to /var/cache/conftool/dbconfig/20221115-084959-marostegui.json
  • 08:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39662 and previous config saved to /var/cache/conftool/dbconfig/20221115-084637-marostegui.json
  • 08:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 08:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 08:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 08:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 08:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 08:06 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1021.eqiad.wmnet to cluster eqiad and group D
  • 08:05 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1021.eqiad.wmnet to cluster eqiad and group D
  • 08:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1021.eqiad.wmnet
  • 07:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1021.eqiad.wmnet
  • 07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39661 and previous config saved to /var/cache/conftool/dbconfig/20221115-072202-marostegui.json
  • 07:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P39660 and previous config saved to /var/cache/conftool/dbconfig/20221115-070655-marostegui.json
  • 06:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P39659 and previous config saved to /var/cache/conftool/dbconfig/20221115-065149-marostegui.json
  • 06:43 robh: all in rack work in eqsin is complete for today
  • 06:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39658 and previous config saved to /var/cache/conftool/dbconfig/20221115-063642-marostegui.json
  • 06:30 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1008.eqiad.wmnet
  • 06:26 robh@cumin1001: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5032
  • 06:26 robh@cumin1001: START - Cookbook sre.network.configure-switch-interfaces for host cp5032
  • 06:25 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 06:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39657 and previous config saved to /var/cache/conftool/dbconfig/20221115-062400-marostegui.json
  • 06:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 06:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 06:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321130)', diff saved to https://phabricator.wikimedia.org/P39656 and previous config saved to /var/cache/conftool/dbconfig/20221115-062339-marostegui.json
  • 06:23 robh@cumin1001: START - Cookbook sre.dns.netbox
  • 06:20 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1008.eqiad.wmnet
  • 06:19 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1007.eqiad.wmnet
  • 06:18 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1005.eqiad.wmnet
  • 06:15 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-tool1007.eqiad.wmnet
  • 06:13 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host karapace1001.eqiad.wmnet
  • 06:09 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host karapace1001.eqiad.wmnet
  • 06:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P39655 and previous config saved to /var/cache/conftool/dbconfig/20221115-060832-marostegui.json
  • 06:06 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1005.eqiad.wmnet
  • 06:05 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host stat1005.eqiad.wmnet
  • 06:05 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1005.eqiad.wmnet
  • 05:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P39654 and previous config saved to /var/cache/conftool/dbconfig/20221115-055326-marostegui.json
  • 05:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321130)', diff saved to https://phabricator.wikimedia.org/P39653 and previous config saved to /var/cache/conftool/dbconfig/20221115-053819-marostegui.json
  • 05:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2172 (T321130)', diff saved to https://phabricator.wikimedia.org/P39652 and previous config saved to /var/cache/conftool/dbconfig/20221115-052543-marostegui.json
  • 05:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 05:25 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 05:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321130)', diff saved to https://phabricator.wikimedia.org/P39651 and previous config saved to /var/cache/conftool/dbconfig/20221115-052521-marostegui.json
  • 05:13 robh: ~5AM UTC when plugging a new host into asw1-ulsfo, the virtual chassis crashed and rebooted, causing loss of connectivity to hosts for a very short period
  • 05:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P39650 and previous config saved to /var/cache/conftool/dbconfig/20221115-051015-marostegui.json
  • 04:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P39649 and previous config saved to /var/cache/conftool/dbconfig/20221115-045508-marostegui.json
  • 04:48 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 04:40 mwpresync@deploy1002: Pruned MediaWiki: 1.40.0-wmf.7 (duration: 02m 01s)
  • 04:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321130)', diff saved to https://phabricator.wikimedia.org/P39648 and previous config saved to /var/cache/conftool/dbconfig/20221115-044002-marostegui.json
  • 04:38 mwpresync@deploy1002: Finished scap: testwikis wikis to 1.40.0-wmf.10 refs T320515 (duration: 36m 14s)
  • 04:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 04:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2155 (T321130)', diff saved to https://phabricator.wikimedia.org/P39647 and previous config saved to /var/cache/conftool/dbconfig/20221115-042713-marostegui.json
  • 04:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 04:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 04:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 04:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 04:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39646 and previous config saved to /var/cache/conftool/dbconfig/20221115-042647-marostegui.json
  • 04:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 04:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 04:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 04:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 04:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P39645 and previous config saved to /var/cache/conftool/dbconfig/20221115-041140-marostegui.json
  • 04:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 04:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 04:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 04:03 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 04:02 mwpresync@deploy1002: Started scap: testwikis wikis to 1.40.0-wmf.10 refs T320515
  • 03:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P39644 and previous config saved to /var/cache/conftool/dbconfig/20221115-035634-marostegui.json
  • 03:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39643 and previous config saved to /var/cache/conftool/dbconfig/20221115-034127-marostegui.json
  • 03:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:32 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:32 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:32 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39642 and previous config saved to /var/cache/conftool/dbconfig/20221115-032929-marostegui.json
  • 03:29 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 03:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 03:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 03:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 03:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39641 and previous config saved to /var/cache/conftool/dbconfig/20221115-031826-marostegui.json
  • 03:16 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:12 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:12 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P39640 and previous config saved to /var/cache/conftool/dbconfig/20221115-030320-marostegui.json
  • 02:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P39639 and previous config saved to /var/cache/conftool/dbconfig/20221115-024813-marostegui.json
  • 02:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39638 and previous config saved to /var/cache/conftool/dbconfig/20221115-023307-marostegui.json
  • 02:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39637 and previous config saved to /var/cache/conftool/dbconfig/20221115-022052-marostegui.json
  • 02:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 02:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 02:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39636 and previous config saved to /var/cache/conftool/dbconfig/20221115-022030-marostegui.json
  • 02:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P39635 and previous config saved to /var/cache/conftool/dbconfig/20221115-020523-marostegui.json
  • 01:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P39634 and previous config saved to /var/cache/conftool/dbconfig/20221115-015017-marostegui.json
  • 01:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39633 and previous config saved to /var/cache/conftool/dbconfig/20221115-013510-marostegui.json
  • 01:28 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS buster
  • 01:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39632 and previous config saved to /var/cache/conftool/dbconfig/20221115-012313-marostegui.json
  • 01:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 01:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 01:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39631 and previous config saved to /var/cache/conftool/dbconfig/20221115-012251-marostegui.json
  • 01:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39630 and previous config saved to /var/cache/conftool/dbconfig/20221115-010745-marostegui.json
  • 01:05 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 01:02 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 00:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39629 and previous config saved to /var/cache/conftool/dbconfig/20221115-005238-marostegui.json
  • 00:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39628 and previous config saved to /var/cache/conftool/dbconfig/20221115-003732-marostegui.json
  • 00:33 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS buster
  • 00:29 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host arclamp2001.codfw.wmnet with OS bullseye
  • 00:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39627 and previous config saved to /var/cache/conftool/dbconfig/20221115-002514-marostegui.json
  • 00:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 00:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 00:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321130)', diff saved to https://phabricator.wikimedia.org/P39626 and previous config saved to /var/cache/conftool/dbconfig/20221115-002441-marostegui.json
  • 00:14 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on arclamp2001.codfw.wmnet with reason: host reimage
  • 00:13 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host puppetdb2003.codfw.wmnet with OS bullseye
  • 00:11 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on arclamp2001.codfw.wmnet with reason: host reimage
  • 00:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39625 and previous config saved to /var/cache/conftool/dbconfig/20221115-000935-marostegui.json

2022-11-14

  • 23:58 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on puppetdb2003.codfw.wmnet with reason: host reimage
  • 23:55 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on puppetdb2003.codfw.wmnet with reason: host reimage
  • 23:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39624 and previous config saved to /var/cache/conftool/dbconfig/20221114-235429-marostegui.json
  • 23:52 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host arclamp2001.codfw.wmnet with OS bullseye
  • 23:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321130)', diff saved to https://phabricator.wikimedia.org/P39623 and previous config saved to /var/cache/conftool/dbconfig/20221114-233922-marostegui.json
  • 23:36 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host puppetdb2003.codfw.wmnet with OS bullseye
  • 23:32 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov2004.codfw.wmnet with OS bullseye
  • 23:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39622 and previous config saved to /var/cache/conftool/dbconfig/20221114-232744-marostegui.json
  • 23:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2119 (T321130)', diff saved to https://phabricator.wikimedia.org/P39621 and previous config saved to /var/cache/conftool/dbconfig/20221114-232714-marostegui.json
  • 23:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 23:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 23:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321130)', diff saved to https://phabricator.wikimedia.org/P39620 and previous config saved to /var/cache/conftool/dbconfig/20221114-232653-marostegui.json
  • 23:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P39619 and previous config saved to /var/cache/conftool/dbconfig/20221114-231238-marostegui.json
  • 23:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P39618 and previous config saved to /var/cache/conftool/dbconfig/20221114-231146-marostegui.json
  • 23:10 eileen: civicrm upgraded from 93fa3f37 to 3eba6ad3
  • 22:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P39617 and previous config saved to /var/cache/conftool/dbconfig/20221114-225730-marostegui.json
  • 22:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P39616 and previous config saved to /var/cache/conftool/dbconfig/20221114-225638-marostegui.json
  • 22:56 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 22:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 22:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39614 and previous config saved to /var/cache/conftool/dbconfig/20221114-224224-marostegui.json
  • 22:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321130)', diff saved to https://phabricator.wikimedia.org/P39613 and previous config saved to /var/cache/conftool/dbconfig/20221114-224132-marostegui.json
  • 22:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39612 and previous config saved to /var/cache/conftool/dbconfig/20221114-224006-marostegui.json
  • 22:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 22:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 22:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39611 and previous config saved to /var/cache/conftool/dbconfig/20221114-223945-marostegui.json
  • 22:31 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 22:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2110 (T321130)', diff saved to https://phabricator.wikimedia.org/P39610 and previous config saved to /var/cache/conftool/dbconfig/20221114-222706-marostegui.json
  • 22:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 22:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 22:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T321130)', diff saved to https://phabricator.wikimedia.org/P39609 and previous config saved to /var/cache/conftool/dbconfig/20221114-222644-marostegui.json
  • 22:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P39608 and previous config saved to /var/cache/conftool/dbconfig/20221114-222438-marostegui.json
  • 22:21 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 22:19 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 22:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P39607 and previous config saved to /var/cache/conftool/dbconfig/20221114-221138-marostegui.json
  • 22:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P39606 and previous config saved to /var/cache/conftool/dbconfig/20221114-220932-marostegui.json
  • 22:09 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 22:09 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 22:03 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 21:58 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P39605 and previous config saved to /var/cache/conftool/dbconfig/20221114-215631-marostegui.json
  • 21:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39604 and previous config saved to /var/cache/conftool/dbconfig/20221114-215425-marostegui.json
  • 21:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39603 and previous config saved to /var/cache/conftool/dbconfig/20221114-215204-marostegui.json
  • 21:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 21:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 21:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39602 and previous config saved to /var/cache/conftool/dbconfig/20221114-215143-marostegui.json
  • 21:48 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T321130)', diff saved to https://phabricator.wikimedia.org/P39601 and previous config saved to /var/cache/conftool/dbconfig/20221114-214125-marostegui.json
  • 21:38 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P39600 and previous config saved to /var/cache/conftool/dbconfig/20221114-213636-marostegui.json
  • 21:35 mutante: phab2002 - systemctl start phd, debug why it still fails
  • 21:35 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2106 (T321130)', diff saved to https://phabricator.wikimedia.org/P39599 and previous config saved to /var/cache/conftool/dbconfig/20221114-212934-marostegui.json
  • 21:29 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 21:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 21:25 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P39598 and previous config saved to /var/cache/conftool/dbconfig/20221114-212130-marostegui.json
  • 21:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 21:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 21:11 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39597 and previous config saved to /var/cache/conftool/dbconfig/20221114-210853-marostegui.json
  • 21:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39596 and previous config saved to /var/cache/conftool/dbconfig/20221114-210623-marostegui.json
  • 21:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39595 and previous config saved to /var/cache/conftool/dbconfig/20221114-210503-marostegui.json
  • 21:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 21:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 21:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T321126)', diff saved to https://phabricator.wikimedia.org/P39594 and previous config saved to /var/cache/conftool/dbconfig/20221114-210430-marostegui.json
  • 21:01 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39593 and previous config saved to /var/cache/conftool/dbconfig/20221114-205347-marostegui.json
  • 20:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P39592 and previous config saved to /var/cache/conftool/dbconfig/20221114-204924-marostegui.json
  • 20:47 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 20:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39591 and previous config saved to /var/cache/conftool/dbconfig/20221114-203841-marostegui.json
  • 20:37 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P39590 and previous config saved to /var/cache/conftool/dbconfig/20221114-203417-marostegui.json
  • 20:34 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 20:34 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39589 and previous config saved to /var/cache/conftool/dbconfig/20221114-202334-marostegui.json
  • 20:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T321126)', diff saved to https://phabricator.wikimedia.org/P39588 and previous config saved to /var/cache/conftool/dbconfig/20221114-201911-marostegui.json
  • 20:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T321126)', diff saved to https://phabricator.wikimedia.org/P39587 and previous config saved to /var/cache/conftool/dbconfig/20221114-201650-marostegui.json
  • 20:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 20:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39586 and previous config saved to /var/cache/conftool/dbconfig/20221114-201556-marostegui.json
  • 20:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P39585 and previous config saved to /var/cache/conftool/dbconfig/20221114-200050-marostegui.json
  • 19:55 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: Upgrade horizon to Z to prepare for Openstack upgrades past Wallaby -- T305828 (duration: 04m 41s)
  • 19:50 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: Upgrade horizon to Z to prepare for Openstack upgrades past Wallaby -- T305828
  • 19:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P39583 and previous config saved to /var/cache/conftool/dbconfig/20221114-194543-marostegui.json
  • 19:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39582 and previous config saved to /var/cache/conftool/dbconfig/20221114-193037-marostegui.json
  • 19:29 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS bullseye
  • 19:28 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39581 and previous config saved to /var/cache/conftool/dbconfig/20221114-192816-marostegui.json
  • 19:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 19:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 19:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321126)', diff saved to https://phabricator.wikimedia.org/P39580 and previous config saved to /var/cache/conftool/dbconfig/20221114-192754-marostegui.json
  • 19:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39579 and previous config saved to /var/cache/conftool/dbconfig/20221114-192318-marostegui.json
  • 19:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 19:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 19:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321130)', diff saved to https://phabricator.wikimedia.org/P39578 and previous config saved to /var/cache/conftool/dbconfig/20221114-192257-marostegui.json
  • 19:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P39577 and previous config saved to /var/cache/conftool/dbconfig/20221114-191247-marostegui.json
  • 19:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P39576 and previous config saved to /var/cache/conftool/dbconfig/20221114-190750-marostegui.json
  • 18:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P39575 and previous config saved to /var/cache/conftool/dbconfig/20221114-185741-marostegui.json
  • 18:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P39574 and previous config saved to /var/cache/conftool/dbconfig/20221114-185244-marostegui.json
  • 18:50 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:45 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321126)', diff saved to https://phabricator.wikimedia.org/P39573 and previous config saved to /var/cache/conftool/dbconfig/20221114-184235-marostegui.json
  • 18:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T321126)', diff saved to https://phabricator.wikimedia.org/P39572 and previous config saved to /var/cache/conftool/dbconfig/20221114-184014-marostegui.json
  • 18:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 18:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 18:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321126)', diff saved to https://phabricator.wikimedia.org/P39571 and previous config saved to /var/cache/conftool/dbconfig/20221114-183952-marostegui.json
  • 18:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321130)', diff saved to https://phabricator.wikimedia.org/P39570 and previous config saved to /var/cache/conftool/dbconfig/20221114-183738-marostegui.json
  • 18:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1190 (T321130)', diff saved to https://phabricator.wikimedia.org/P39569 and previous config saved to /var/cache/conftool/dbconfig/20221114-182506-marostegui.json
  • 18:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 18:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 18:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P39568 and previous config saved to /var/cache/conftool/dbconfig/20221114-182446-marostegui.json
  • 18:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39567 and previous config saved to /var/cache/conftool/dbconfig/20221114-182445-marostegui.json
  • 18:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 18:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 18:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T322618)', diff saved to https://phabricator.wikimedia.org/P39566 and previous config saved to /var/cache/conftool/dbconfig/20221114-181700-ladsgroup.json
  • 18:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39565 and previous config saved to /var/cache/conftool/dbconfig/20221114-180938-marostegui.json
  • 18:08 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['arclamp2001']
  • 18:07 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['arclamp2001']
  • 18:07 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['arclamp2001']
  • 18:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P39564 and previous config saved to /var/cache/conftool/dbconfig/20221114-180153-ladsgroup.json
  • 17:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39563 and previous config saved to /var/cache/conftool/dbconfig/20221114-175432-marostegui.json
  • 17:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T321126)', diff saved to https://phabricator.wikimedia.org/P39562 and previous config saved to /var/cache/conftool/dbconfig/20221114-175213-marostegui.json
  • 17:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 17:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 17:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 17:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 17:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T321126)', diff saved to https://phabricator.wikimedia.org/P39561 and previous config saved to /var/cache/conftool/dbconfig/20221114-175129-marostegui.json
  • 17:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P39560 and previous config saved to /var/cache/conftool/dbconfig/20221114-174647-ladsgroup.json
  • 17:42 hashar: Restored CI caching mechanism which has been serving stalled caches since March 29th 2022 :-\ T323051
  • 17:42 hashar: Restored CI caching mechanism which has been serving stalled caches since March 29th 2022 :-\ T307334
  • 17:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39559 and previous config saved to /var/cache/conftool/dbconfig/20221114-173925-marostegui.json
  • 17:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P39558 and previous config saved to /var/cache/conftool/dbconfig/20221114-173622-marostegui.json
  • 17:34 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['arclamp2001']
  • 17:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T322618)', diff saved to https://phabricator.wikimedia.org/P39557 and previous config saved to /var/cache/conftool/dbconfig/20221114-173140-ladsgroup.json
  • 17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T322618)', diff saved to https://phabricator.wikimedia.org/P39556 and previous config saved to /var/cache/conftool/dbconfig/20221114-172929-ladsgroup.json
  • 17:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 17:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 17:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T322618)', diff saved to https://phabricator.wikimedia.org/P39555 and previous config saved to /var/cache/conftool/dbconfig/20221114-172846-ladsgroup.json
  • 17:25 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:24 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 17:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P39554 and previous config saved to /var/cache/conftool/dbconfig/20221114-172116-marostegui.json
  • 17:13 dancy@deploy1002: Installation of scap version "4.28.1" completed for 559 hosts
  • 17:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P39553 and previous config saved to /var/cache/conftool/dbconfig/20221114-171340-ladsgroup.json
  • 17:13 dancy@deploy1002: Installing scap version "4.28.1" for 559 hosts
  • 17:10 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 2:00:00 on wcqs1002.eqiad.wmnet with reason: Reboot for kernel update
  • 17:10 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 2:00:00 on wcqs1002.eqiad.wmnet with reason: Reboot for kernel update
  • 17:09 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs4005.ulsfo.wmnet
  • 17:09 sukhe@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:08 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wcqs[2001-2003].codfw.wmnet with reason: Reboot for kernel update
  • 17:07 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wcqs[2001-2003].codfw.wmnet with reason: Reboot for kernel update
  • 17:07 ryankemper@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on 12 hosts with reason: Reboot for kernel update
  • 17:07 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 12 hosts with reason: Reboot for kernel update
  • 17:07 sukhe@cumin2002: START - Cookbook sre.dns.netbox
  • 17:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T321126)', diff saved to https://phabricator.wikimedia.org/P39552 and previous config saved to /var/cache/conftool/dbconfig/20221114-170609-marostegui.json
  • 17:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1201 (T321126)', diff saved to https://phabricator.wikimedia.org/P39551 and previous config saved to /var/cache/conftool/dbconfig/20221114-170357-marostegui.json
  • 17:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 17:03 jdrewniak@deploy1002: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 03m 45s)
  • 17:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 17:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T321126)', diff saved to https://phabricator.wikimedia.org/P39550 and previous config saved to /var/cache/conftool/dbconfig/20221114-170325-marostegui.json
  • 17:03 sukhe@cumin2002: START - Cookbook sre.hosts.decommission for hosts lvs4005.ulsfo.wmnet
  • 16:59 jdrewniak@deploy1002: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 03m 58s)
  • 16:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P39549 and previous config saved to /var/cache/conftool/dbconfig/20221114-165833-ladsgroup.json
  • 16:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P39548 and previous config saved to /var/cache/conftool/dbconfig/20221114-164818-marostegui.json
  • 16:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T322618)', diff saved to https://phabricator.wikimedia.org/P39547 and previous config saved to /var/cache/conftool/dbconfig/20221114-164327-ladsgroup.json
  • 16:41 sukhe: depooled lvs4005
  • 16:41 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4005.ulsfo.wmnet with reason: downtimed, in the process of decom
  • 16:41 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on lvs4005.ulsfo.wmnet with reason: downtimed, in the process of decom
  • 16:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T322618)', diff saved to https://phabricator.wikimedia.org/P39546 and previous config saved to /var/cache/conftool/dbconfig/20221114-164015-ladsgroup.json
  • 16:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39545 and previous config saved to /var/cache/conftool/dbconfig/20221114-163954-ladsgroup.json
  • 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39544 and previous config saved to /var/cache/conftool/dbconfig/20221114-163910-marostegui.json
  • 16:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 16:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 16:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2166.codfw.wmnet with reason: Host crashed T323040
  • 16:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2166.codfw.wmnet with reason: Host crashed T323040
  • 16:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P39543 and previous config saved to /var/cache/conftool/dbconfig/20221114-163312-marostegui.json
  • 16:30 sukhe: cr4-ulsfo: set routing-options static route 198.35.26.96/28 next-hop 10.128.0.18 [lvs4005 decomm]
  • 16:29 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 16:28 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 16:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T321130)', diff saved to https://phabricator.wikimedia.org/P39542 and previous config saved to /var/cache/conftool/dbconfig/20221114-162851-marostegui.json
  • 16:28 sukhe: cr3-ulsfo: set routing-options static route 198.35.26.96/28 next-hop 10.128.0.18 [lvs4005 decomm]
  • 16:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P39541 and previous config saved to /var/cache/conftool/dbconfig/20221114-162448-ladsgroup.json
  • 16:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T321126)', diff saved to https://phabricator.wikimedia.org/P39540 and previous config saved to /var/cache/conftool/dbconfig/20221114-161804-marostegui.json
  • 16:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T321126)', diff saved to https://phabricator.wikimedia.org/P39539 and previous config saved to /var/cache/conftool/dbconfig/20221114-161553-marostegui.json
  • 16:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 16:15 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['dbprov2004']
  • 16:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 16:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39537 and previous config saved to /var/cache/conftool/dbconfig/20221114-161520-marostegui.json
  • 16:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P39536 and previous config saved to /var/cache/conftool/dbconfig/20221114-161344-marostegui.json
  • 16:13 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['puppetdb2003']
  • 16:12 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['puppetdb2003']
  • 16:12 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['puppetdb2003']
  • 16:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P39535 and previous config saved to /var/cache/conftool/dbconfig/20221114-160941-ladsgroup.json
  • 16:08 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dbprov2004']
  • 16:08 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['dbprov2004']
  • 16:03 sukhe: reprepro -C main include bullseye-wikimedia varnish-modules_0.15.0-2_amd64.changes: T321309
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'MySQL issues', diff saved to https://phabricator.wikimedia.org/P39534 and previous config saved to /var/cache/conftool/dbconfig/20221114-160140-ladsgroup.json
  • 16:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P39533 and previous config saved to /var/cache/conftool/dbconfig/20221114-160014-marostegui.json
  • 15:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P39532 and previous config saved to /var/cache/conftool/dbconfig/20221114-155838-marostegui.json
  • 15:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39531 and previous config saved to /var/cache/conftool/dbconfig/20221114-155435-ladsgroup.json
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39530 and previous config saved to /var/cache/conftool/dbconfig/20221114-155222-ladsgroup.json
  • 15:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39529 and previous config saved to /var/cache/conftool/dbconfig/20221114-155201-ladsgroup.json
  • 15:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P39528 and previous config saved to /var/cache/conftool/dbconfig/20221114-154507-marostegui.json
  • 15:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T321130)', diff saved to https://phabricator.wikimedia.org/P39527 and previous config saved to /var/cache/conftool/dbconfig/20221114-154331-marostegui.json
  • 15:42 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['puppetdb2003']
  • 15:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T322618)', diff saved to https://phabricator.wikimedia.org/P39526 and previous config saved to /var/cache/conftool/dbconfig/20221114-153903-ladsgroup.json
  • 15:38 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dbprov2004']
  • 15:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39525 and previous config saved to /var/cache/conftool/dbconfig/20221114-153654-ladsgroup.json
  • 15:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T321130)', diff saved to https://phabricator.wikimedia.org/P39524 and previous config saved to /var/cache/conftool/dbconfig/20221114-153030-marostegui.json
  • 15:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 15:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39523 and previous config saved to /var/cache/conftool/dbconfig/20221114-153001-marostegui.json
  • 15:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 15:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321130)', diff saved to https://phabricator.wikimedia.org/P39522 and previous config saved to /var/cache/conftool/dbconfig/20221114-152936-marostegui.json
  • 15:28 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1021.eqiad.wmnet with OS bullseye
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39521 and previous config saved to /var/cache/conftool/dbconfig/20221114-152749-marostegui.json
  • 15:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 15:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321126)', diff saved to https://phabricator.wikimedia.org/P39520 and previous config saved to /var/cache/conftool/dbconfig/20221114-152728-marostegui.json
  • 15:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39519 and previous config saved to /var/cache/conftool/dbconfig/20221114-152356-ladsgroup.json
  • 15:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39518 and previous config saved to /var/cache/conftool/dbconfig/20221114-152148-ladsgroup.json
  • 15:15 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 15:15 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 15:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39516 and previous config saved to /var/cache/conftool/dbconfig/20221114-151428-marostegui.json
  • 15:13 urandom: initiating Cassandra bootstrap, aqs1019-a -- T307802
  • 15:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P39515 and previous config saved to /var/cache/conftool/dbconfig/20221114-151222-marostegui.json
  • 15:11 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 15:10 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 15:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1021.eqiad.wmnet with reason: host reimage
  • 15:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39514 and previous config saved to /var/cache/conftool/dbconfig/20221114-150850-ladsgroup.json
  • 15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39513 and previous config saved to /var/cache/conftool/dbconfig/20221114-150642-ladsgroup.json
  • 15:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39512 and previous config saved to /var/cache/conftool/dbconfig/20221114-150531-ladsgroup.json
  • 15:05 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1021.eqiad.wmnet with reason: host reimage
  • 15:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T322618)', diff saved to https://phabricator.wikimedia.org/P39511 and previous config saved to /var/cache/conftool/dbconfig/20221114-150509-ladsgroup.json
  • 14:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39510 and previous config saved to /var/cache/conftool/dbconfig/20221114-145921-marostegui.json
  • 14:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P39509 and previous config saved to /var/cache/conftool/dbconfig/20221114-145715-marostegui.json
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T322618)', diff saved to https://phabricator.wikimedia.org/P39508 and previous config saved to /var/cache/conftool/dbconfig/20221114-145343-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T322618)', diff saved to https://phabricator.wikimedia.org/P39507 and previous config saved to /var/cache/conftool/dbconfig/20221114-145122-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 14:51 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1021.eqiad.wmnet with OS bullseye
  • 14:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39506 and previous config saved to /var/cache/conftool/dbconfig/20221114-145101-ladsgroup.json
  • 14:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39505 and previous config saved to /var/cache/conftool/dbconfig/20221114-145003-ladsgroup.json
  • 14:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1021.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 14:49 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1021.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 14:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321130)', diff saved to https://phabricator.wikimedia.org/P39504 and previous config saved to /var/cache/conftool/dbconfig/20221114-144415-marostegui.json
  • 14:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321126)', diff saved to https://phabricator.wikimedia.org/P39503 and previous config saved to /var/cache/conftool/dbconfig/20221114-144209-marostegui.json
  • 14:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T321126)', diff saved to https://phabricator.wikimedia.org/P39502 and previous config saved to /var/cache/conftool/dbconfig/20221114-143957-marostegui.json
  • 14:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 14:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 14:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321126)', diff saved to https://phabricator.wikimedia.org/P39501 and previous config saved to /var/cache/conftool/dbconfig/20221114-143936-marostegui.json
  • 14:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39500 and previous config saved to /var/cache/conftool/dbconfig/20221114-143554-ladsgroup.json
  • 14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39499 and previous config saved to /var/cache/conftool/dbconfig/20221114-143456-ladsgroup.json
  • 14:33 taavi: ^ correction, starting it on mwmaint1002, not deploy1002
  • 14:32 taavi: START taavi@deploy1002:~$ foreachwikiindblist group1 extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all | tee T315510.log # T315510
  • 14:30 taavi@deploy1002: Finished scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353) (duration: 07m 05s)
  • 14:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P39498 and previous config saved to /var/cache/conftool/dbconfig/20221114-142429-marostegui.json
  • 14:23 taavi@deploy1002: taavi and matmarex: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 14:23 taavi@deploy1002: Started scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353)
  • 14:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39497 and previous config saved to /var/cache/conftool/dbconfig/20221114-142048-ladsgroup.json
  • 14:20 taavi@deploy1002: Finished scap: Backport for Use legacy DiscussionTools heading markup except on beta cluster (T314714), ThreadItemStore: Handle race conditions when finding/inserting outside of transaction (T322701), persistRevisionThreadItems: Print time taken (duration: 06m 14s)
  • 14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T322618)', diff saved to https://phabricator.wikimedia.org/P39496 and previous config saved to /var/cache/conftool/dbconfig/20221114-141950-ladsgroup.json
  • 14:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T322618)', diff saved to https://phabricator.wikimedia.org/P39495 and previous config saved to /var/cache/conftool/dbconfig/20221114-141738-ladsgroup.json
  • 14:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 14:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T321130)', diff saved to https://phabricator.wikimedia.org/P39494 and previous config saved to /var/cache/conftool/dbconfig/20221114-141731-marostegui.json
  • 14:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 14:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 14:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T322618)', diff saved to https://phabricator.wikimedia.org/P39493 and previous config saved to /var/cache/conftool/dbconfig/20221114-141717-ladsgroup.json
  • 14:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 14:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39492 and previous config saved to /var/cache/conftool/dbconfig/20221114-141710-marostegui.json
  • 14:14 taavi@deploy1002: taavi and matmarex: Backport for Use legacy DiscussionTools heading markup except on beta cluster (T314714), ThreadItemStore: Handle race conditions when finding/inserting outside of transaction (T322701), persistRevisionThreadItems: Print time taken synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmn
  • 14:14 taavi@deploy1002: Started scap: Backport for Use legacy DiscussionTools heading markup except on beta cluster (T314714), ThreadItemStore: Handle race conditions when finding/inserting outside of transaction (T322701), persistRevisionThreadItems: Print time taken
  • 14:13 taavi@deploy1002: backport aborted: (duration: 01m 23s)
  • 14:13 taavi@deploy1002: prep aborted: (duration: 00m 06s)
  • 14:11 taavi@deploy1002: Finished scap: Backport for Separate identifiers from other statements for Lexemes (T318310) (duration: 06m 27s)
  • 14:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P39491 and previous config saved to /var/cache/conftool/dbconfig/20221114-140923-marostegui.json
  • 14:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39490 and previous config saved to /var/cache/conftool/dbconfig/20221114-140541-ladsgroup.json
  • 14:05 taavi@deploy1002: taavi and migr: Backport for Separate identifiers from other statements for Lexemes (T318310) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 14:05 taavi@deploy1002: Started scap: Backport for Separate identifiers from other statements for Lexemes (T318310)
  • 14:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39489 and previous config saved to /var/cache/conftool/dbconfig/20221114-140320-ladsgroup.json
  • 14:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 14:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 14:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T322618)', diff saved to https://phabricator.wikimedia.org/P39488 and previous config saved to /var/cache/conftool/dbconfig/20221114-140259-ladsgroup.json
  • 14:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P39487 and previous config saved to /var/cache/conftool/dbconfig/20221114-140210-ladsgroup.json
  • 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P39486 and previous config saved to /var/cache/conftool/dbconfig/20221114-140203-marostegui.json
  • 13:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321126)', diff saved to https://phabricator.wikimedia.org/P39485 and previous config saved to /var/cache/conftool/dbconfig/20221114-135416-marostegui.json
  • 13:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T321126)', diff saved to https://phabricator.wikimedia.org/P39484 and previous config saved to /var/cache/conftool/dbconfig/20221114-135204-marostegui.json
  • 13:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T321126)', diff saved to https://phabricator.wikimedia.org/P39483 and previous config saved to /var/cache/conftool/dbconfig/20221114-135114-marostegui.json
  • 13:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P39482 and previous config saved to /var/cache/conftool/dbconfig/20221114-134752-ladsgroup.json
  • 13:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P39481 and previous config saved to /var/cache/conftool/dbconfig/20221114-134704-ladsgroup.json
  • 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P39480 and previous config saved to /var/cache/conftool/dbconfig/20221114-134657-marostegui.json
  • 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P39479 and previous config saved to /var/cache/conftool/dbconfig/20221114-133608-marostegui.json
  • 13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P39478 and previous config saved to /var/cache/conftool/dbconfig/20221114-133246-ladsgroup.json
  • 13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T322618)', diff saved to https://phabricator.wikimedia.org/P39477 and previous config saved to /var/cache/conftool/dbconfig/20221114-133157-ladsgroup.json
  • 13:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39476 and previous config saved to /var/cache/conftool/dbconfig/20221114-133150-marostegui.json
  • 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P39475 and previous config saved to /var/cache/conftool/dbconfig/20221114-132101-marostegui.json
  • 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39474 and previous config saved to /var/cache/conftool/dbconfig/20221114-132008-marostegui.json
  • 13:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 13:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 13:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39473 and previous config saved to /var/cache/conftool/dbconfig/20221114-131946-marostegui.json
  • 13:19 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
  • 13:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T322618)', diff saved to https://phabricator.wikimedia.org/P39472 and previous config saved to /var/cache/conftool/dbconfig/20221114-131740-ladsgroup.json
  • 13:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T322618)', diff saved to https://phabricator.wikimedia.org/P39471 and previous config saved to /var/cache/conftool/dbconfig/20221114-131519-ladsgroup.json
  • 13:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 13:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 13:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39470 and previous config saved to /var/cache/conftool/dbconfig/20221114-131457-ladsgroup.json
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T321126)', diff saved to https://phabricator.wikimedia.org/P39469 and previous config saved to /var/cache/conftool/dbconfig/20221114-130555-marostegui.json
  • 13:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P39468 and previous config saved to /var/cache/conftool/dbconfig/20221114-130440-marostegui.json
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T321126)', diff saved to https://phabricator.wikimedia.org/P39467 and previous config saved to /var/cache/conftool/dbconfig/20221114-130343-marostegui.json
  • 13:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 13:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39466 and previous config saved to /var/cache/conftool/dbconfig/20221114-130322-marostegui.json
  • 12:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P39465 and previous config saved to /var/cache/conftool/dbconfig/20221114-125951-ladsgroup.json
  • 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P39464 and previous config saved to /var/cache/conftool/dbconfig/20221114-124934-marostegui.json
  • 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P39463 and previous config saved to /var/cache/conftool/dbconfig/20221114-124815-marostegui.json
  • 12:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P39462 and previous config saved to /var/cache/conftool/dbconfig/20221114-124444-ladsgroup.json
  • 12:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39461 and previous config saved to /var/cache/conftool/dbconfig/20221114-123427-marostegui.json
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P39460 and previous config saved to /var/cache/conftool/dbconfig/20221114-123309-marostegui.json
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T322618)', diff saved to https://phabricator.wikimedia.org/P39459 and previous config saved to /var/cache/conftool/dbconfig/20221114-123141-ladsgroup.json
  • 12:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:31 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 12:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 12:31 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 12:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39458 and previous config saved to /var/cache/conftool/dbconfig/20221114-123103-ladsgroup.json
  • 12:30 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 12:30 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 12:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39457 and previous config saved to /var/cache/conftool/dbconfig/20221114-122938-ladsgroup.json
  • 12:28 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39456 and previous config saved to /var/cache/conftool/dbconfig/20221114-122717-ladsgroup.json
  • 12:27 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 12:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T322618)', diff saved to https://phabricator.wikimedia.org/P39455 and previous config saved to /var/cache/conftool/dbconfig/20221114-122655-ladsgroup.json
  • 12:26 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:25 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39454 and previous config saved to /var/cache/conftool/dbconfig/20221114-122214-marostegui.json
  • 12:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 12:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 12:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39453 and previous config saved to /var/cache/conftool/dbconfig/20221114-121802-marostegui.json
  • 12:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P39452 and previous config saved to /var/cache/conftool/dbconfig/20221114-121556-ladsgroup.json
  • 12:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39451 and previous config saved to /var/cache/conftool/dbconfig/20221114-121547-marostegui.json
  • 12:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39450 and previous config saved to /var/cache/conftool/dbconfig/20221114-121525-marostegui.json
  • 12:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 12:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 12:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39449 and previous config saved to /var/cache/conftool/dbconfig/20221114-121202-marostegui.json
  • 12:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P39448 and previous config saved to /var/cache/conftool/dbconfig/20221114-121149-ladsgroup.json
  • 12:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P39447 and previous config saved to /var/cache/conftool/dbconfig/20221114-120043-ladsgroup.json
  • 12:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P39446 and previous config saved to /var/cache/conftool/dbconfig/20221114-120019-marostegui.json
  • 11:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39445 and previous config saved to /var/cache/conftool/dbconfig/20221114-115656-marostegui.json
  • 11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P39444 and previous config saved to /var/cache/conftool/dbconfig/20221114-115641-ladsgroup.json
  • 11:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39443 and previous config saved to /var/cache/conftool/dbconfig/20221114-114537-ladsgroup.json
  • 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P39442 and previous config saved to /var/cache/conftool/dbconfig/20221114-114512-marostegui.json
  • 11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39441 and previous config saved to /var/cache/conftool/dbconfig/20221114-114326-ladsgroup.json
  • 11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 11:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 11:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 11:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T322618)', diff saved to https://phabricator.wikimedia.org/P39440 and previous config saved to /var/cache/conftool/dbconfig/20221114-114244-ladsgroup.json
  • 11:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39439 and previous config saved to /var/cache/conftool/dbconfig/20221114-114150-marostegui.json
  • 11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T322618)', diff saved to https://phabricator.wikimedia.org/P39438 and previous config saved to /var/cache/conftool/dbconfig/20221114-114134-ladsgroup.json
  • 11:40 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 9231
  • 11:40 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 9231
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T322618)', diff saved to https://phabricator.wikimedia.org/P39437 and previous config saved to /var/cache/conftool/dbconfig/20221114-113913-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T322618)', diff saved to https://phabricator.wikimedia.org/P39436 and previous config saved to /var/cache/conftool/dbconfig/20221114-113837-ladsgroup.json
  • 11:34 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
  • 11:32 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
  • 11:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39435 and previous config saved to /var/cache/conftool/dbconfig/20221114-113006-marostegui.json
  • 11:29 ladsgroup@deploy1002: Finished scap: Backport for Re-add s11 in db config reload callback (T322598) (duration: 05m 01s)
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39434 and previous config saved to /var/cache/conftool/dbconfig/20221114-112750-marostegui.json
  • 11:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 11:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P39433 and previous config saved to /var/cache/conftool/dbconfig/20221114-112736-ladsgroup.json
  • 11:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39432 and previous config saved to /var/cache/conftool/dbconfig/20221114-112729-marostegui.json
  • 11:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39431 and previous config saved to /var/cache/conftool/dbconfig/20221114-112643-marostegui.json
  • 11:24 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for Re-add s11 in db config reload callback (T322598) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 11:24 ladsgroup@deploy1002: Started scap: Backport for Re-add s11 in db config reload callback (T322598)
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P39430 and previous config saved to /var/cache/conftool/dbconfig/20221114-112330-ladsgroup.json
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39429 and previous config saved to /var/cache/conftool/dbconfig/20221114-111434-marostegui.json
  • 11:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 11:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T321130)', diff saved to https://phabricator.wikimedia.org/P39428 and previous config saved to /var/cache/conftool/dbconfig/20221114-111412-marostegui.json
  • 11:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P39427 and previous config saved to /var/cache/conftool/dbconfig/20221114-111229-ladsgroup.json
  • 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P39426 and previous config saved to /var/cache/conftool/dbconfig/20221114-111222-marostegui.json
  • 11:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P39425 and previous config saved to /var/cache/conftool/dbconfig/20221114-110824-ladsgroup.json
  • 10:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P39424 and previous config saved to /var/cache/conftool/dbconfig/20221114-105906-marostegui.json
  • 10:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T322618)', diff saved to https://phabricator.wikimedia.org/P39423 and previous config saved to /var/cache/conftool/dbconfig/20221114-105723-ladsgroup.json
  • 10:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P39422 and previous config saved to /var/cache/conftool/dbconfig/20221114-105716-marostegui.json
  • 10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T322618)', diff saved to https://phabricator.wikimedia.org/P39421 and previous config saved to /var/cache/conftool/dbconfig/20221114-105512-ladsgroup.json
  • 10:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 10:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 10:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39420 and previous config saved to /var/cache/conftool/dbconfig/20221114-105450-ladsgroup.json
  • 10:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T322618)', diff saved to https://phabricator.wikimedia.org/P39419 and previous config saved to /var/cache/conftool/dbconfig/20221114-105317-ladsgroup.json
  • 10:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T322618)', diff saved to https://phabricator.wikimedia.org/P39418 and previous config saved to /var/cache/conftool/dbconfig/20221114-105056-ladsgroup.json
  • 10:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 10:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 10:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T322618)', diff saved to https://phabricator.wikimedia.org/P39417 and previous config saved to /var/cache/conftool/dbconfig/20221114-105034-ladsgroup.json
  • 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P39416 and previous config saved to /var/cache/conftool/dbconfig/20221114-104400-marostegui.json
  • 10:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39415 and previous config saved to /var/cache/conftool/dbconfig/20221114-104209-marostegui.json
  • 10:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39414 and previous config saved to /var/cache/conftool/dbconfig/20221114-103953-marostegui.json
  • 10:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P39413 and previous config saved to /var/cache/conftool/dbconfig/20221114-103944-ladsgroup.json
  • 10:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P39412 and previous config saved to /var/cache/conftool/dbconfig/20221114-103528-ladsgroup.json
  • 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T321130)', diff saved to https://phabricator.wikimedia.org/P39411 and previous config saved to /var/cache/conftool/dbconfig/20221114-102853-marostegui.json
  • 10:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P39410 and previous config saved to /var/cache/conftool/dbconfig/20221114-102437-ladsgroup.json
  • 10:21 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39409 and previous config saved to /var/cache/conftool/dbconfig/20221114-102155-root.json
  • 10:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P39408 and previous config saved to /var/cache/conftool/dbconfig/20221114-102021-ladsgroup.json
  • 10:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T321130)', diff saved to https://phabricator.wikimedia.org/P39407 and previous config saved to /var/cache/conftool/dbconfig/20221114-101659-marostegui.json
  • 10:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 10:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 10:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39406 and previous config saved to /var/cache/conftool/dbconfig/20221114-101637-marostegui.json
  • 10:12 vgutierrez: upgrading acme-chief on acmechief1001 to version 0.35 (requires disabling puppet on R:acme_chief::cert)
  • 10:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39405 and previous config saved to /var/cache/conftool/dbconfig/20221114-100931-ladsgroup.json
  • 10:09 ladsgroup@deploy1002: Finished scap: Backport for Rework SpecialPagesWithoutScans query (T322849) (duration: 11m 17s)
  • 10:07 vgutierrez: upload acme-chief 0.35 to apt.wm.o (buster-wikimedia) - T244232 T262251
  • 10:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39404 and previous config saved to /var/cache/conftool/dbconfig/20221114-100720-ladsgroup.json
  • 10:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 10:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 10:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 10:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39403 and previous config saved to /var/cache/conftool/dbconfig/20221114-100650-root.json
  • 10:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 10:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T322618)', diff saved to https://phabricator.wikimedia.org/P39402 and previous config saved to /var/cache/conftool/dbconfig/20221114-100515-ladsgroup.json
  • 10:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T322618)', diff saved to https://phabricator.wikimedia.org/P39401 and previous config saved to /var/cache/conftool/dbconfig/20221114-100254-ladsgroup.json
  • 10:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 10:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 10:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 10:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 10:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P39400 and previous config saved to /var/cache/conftool/dbconfig/20221114-100131-marostegui.json
  • 09:58 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for Rework SpecialPagesWithoutScans query (T322849) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 09:57 ladsgroup@deploy1002: Started scap: Backport for Rework SpecialPagesWithoutScans query (T322849)
  • 09:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 09:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39399 and previous config saved to /var/cache/conftool/dbconfig/20221114-095145-root.json
  • 09:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P39398 and previous config saved to /var/cache/conftool/dbconfig/20221114-094624-marostegui.json
  • 09:44 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
  • 09:39 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
  • 09:38 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 4788
  • 09:38 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 50083
  • 09:38 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 50083
  • 09:37 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 32934
  • 09:36 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39397 and previous config saved to /var/cache/conftool/dbconfig/20221114-093640-root.json
  • 09:35 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 32934
  • 09:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39396 and previous config saved to /var/cache/conftool/dbconfig/20221114-093118-marostegui.json
  • 09:21 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39395 and previous config saved to /var/cache/conftool/dbconfig/20221114-092135-root.json
  • 09:19 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39394 and previous config saved to /var/cache/conftool/dbconfig/20221114-091934-marostegui.json
  • 09:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 09:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 09:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39393 and previous config saved to /var/cache/conftool/dbconfig/20221114-091912-marostegui.json
  • 09:18 ayounsi@cumin1001: END (ERROR) - Cookbook sre.network.peering (exit_code=97) with action 'configure' for AS: 13335
  • 09:18 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 13335
  • 09:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39392 and previous config saved to /var/cache/conftool/dbconfig/20221114-090630-root.json
  • 09:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P39391 and previous config saved to /var/cache/conftool/dbconfig/20221114-090406-marostegui.json
  • 08:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39389 and previous config saved to /var/cache/conftool/dbconfig/20221114-085125-root.json
  • 08:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P39388 and previous config saved to /var/cache/conftool/dbconfig/20221114-084859-marostegui.json
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39387 and previous config saved to /var/cache/conftool/dbconfig/20221114-083620-root.json
  • 08:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39386 and previous config saved to /var/cache/conftool/dbconfig/20221114-083352-marostegui.json
  • 08:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2145 T322620', diff saved to https://phabricator.wikimedia.org/P39385 and previous config saved to /var/cache/conftool/dbconfig/20221114-082458-root.json
  • 08:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39384 and previous config saved to /var/cache/conftool/dbconfig/20221114-082205-marostegui.json
  • 08:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 08:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 08:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39383 and previous config saved to /var/cache/conftool/dbconfig/20221114-082144-marostegui.json
  • 08:20 moritzm: installing php7.4 security updates (as packaged in Debian)
  • 08:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P39381 and previous config saved to /var/cache/conftool/dbconfig/20221114-080637-marostegui.json
  • 08:02 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc1 master" (duration: 04m 34s)
  • 07:57 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc2014 to pc1 master" synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 07:57 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc1 master"
  • 07:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P39380 and previous config saved to /var/cache/conftool/dbconfig/20221114-075131-marostegui.json
  • 07:50 moritzm: draining ganeti1021 for eventual reimage T311687
  • 07:47 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
  • 07:47 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc2014 to pc1 master (T322295) (duration: 05m 14s)
  • 07:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
  • 07:42 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc2014 to pc1 master (T322295) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 07:42 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc2014 to pc1 master (T322295)
  • 07:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39379 and previous config saved to /var/cache/conftool/dbconfig/20221114-073624-marostegui.json
  • 07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39378 and previous config saved to /var/cache/conftool/dbconfig/20221114-072203-marostegui.json
  • 07:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 07:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 07:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 07:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 07:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1181.eqiad.wmnet with reason: Maintenance
  • 07:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1181.eqiad.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2118.codfw.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2118.codfw.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39377 and previous config saved to /var/cache/conftool/dbconfig/20221114-065620-marostegui.json
  • 06:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39376 and previous config saved to /var/cache/conftool/dbconfig/20221114-064113-marostegui.json
  • 06:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39375 and previous config saved to /var/cache/conftool/dbconfig/20221114-062607-marostegui.json
  • 06:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39374 and previous config saved to /var/cache/conftool/dbconfig/20221114-061100-marostegui.json
  • 06:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39373 and previous config saved to /var/cache/conftool/dbconfig/20221114-060847-marostegui.json
  • 06:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 06:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 06:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2173', diff saved to https://phabricator.wikimedia.org/P39372 and previous config saved to /var/cache/conftool/dbconfig/20221114-060207-root.json

2022-11-12

  • 23:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T318605)', diff saved to https://phabricator.wikimedia.org/P39371 and previous config saved to /var/cache/conftool/dbconfig/20221112-233420-ladsgroup.json
  • 23:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P39370 and previous config saved to /var/cache/conftool/dbconfig/20221112-231914-ladsgroup.json
  • 23:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P39369 and previous config saved to /var/cache/conftool/dbconfig/20221112-230407-ladsgroup.json
  • 22:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T318605)', diff saved to https://phabricator.wikimedia.org/P39368 and previous config saved to /var/cache/conftool/dbconfig/20221112-224900-ladsgroup.json
  • 22:46 urandom: initiating bootstrap, aqs1016-b -- T307802
  • 21:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T318605)', diff saved to https://phabricator.wikimedia.org/P39367 and previous config saved to /var/cache/conftool/dbconfig/20221112-210527-ladsgroup.json
  • 20:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39366 and previous config saved to /var/cache/conftool/dbconfig/20221112-205020-ladsgroup.json
  • 20:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39365 and previous config saved to /var/cache/conftool/dbconfig/20221112-203514-ladsgroup.json
  • 20:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T318605)', diff saved to https://phabricator.wikimedia.org/P39364 and previous config saved to /var/cache/conftool/dbconfig/20221112-202007-ladsgroup.json
  • off: uploaded python3-gjson_0.4.0 to apt.wikimedia.org bullseye-wikimedia
  • 17:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2179 (T318605)', diff saved to https://phabricator.wikimedia.org/P39363 and previous config saved to /var/cache/conftool/dbconfig/20221112-171705-ladsgroup.json
  • 17:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 17:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 17:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T318605)', diff saved to https://phabricator.wikimedia.org/P39362 and previous config saved to /var/cache/conftool/dbconfig/20221112-171643-ladsgroup.json
  • 17:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P39361 and previous config saved to /var/cache/conftool/dbconfig/20221112-170137-ladsgroup.json
  • 16:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P39360 and previous config saved to /var/cache/conftool/dbconfig/20221112-164630-ladsgroup.json
  • 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T318605)', diff saved to https://phabricator.wikimedia.org/P39359 and previous config saved to /var/cache/conftool/dbconfig/20221112-163124-ladsgroup.json
  • 14:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1199 (T318605)', diff saved to https://phabricator.wikimedia.org/P39358 and previous config saved to /var/cache/conftool/dbconfig/20221112-144302-ladsgroup.json
  • 14:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 14:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 14:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T318605)', diff saved to https://phabricator.wikimedia.org/P39357 and previous config saved to /var/cache/conftool/dbconfig/20221112-144240-ladsgroup.json
  • 14:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P39356 and previous config saved to /var/cache/conftool/dbconfig/20221112-142734-ladsgroup.json
  • 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P39355 and previous config saved to /var/cache/conftool/dbconfig/20221112-141227-ladsgroup.json
  • 13:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T318605)', diff saved to https://phabricator.wikimedia.org/P39354 and previous config saved to /var/cache/conftool/dbconfig/20221112-135721-ladsgroup.json
  • 10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2172 (T318605)', diff saved to https://phabricator.wikimedia.org/P39353 and previous config saved to /var/cache/conftool/dbconfig/20221112-105847-ladsgroup.json
  • 10:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 10:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T318605)', diff saved to https://phabricator.wikimedia.org/P39352 and previous config saved to /var/cache/conftool/dbconfig/20221112-105825-ladsgroup.json
  • 10:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P39351 and previous config saved to /var/cache/conftool/dbconfig/20221112-104319-ladsgroup.json
  • 10:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P39350 and previous config saved to /var/cache/conftool/dbconfig/20221112-102812-ladsgroup.json
  • 10:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T318605)', diff saved to https://phabricator.wikimedia.org/P39349 and previous config saved to /var/cache/conftool/dbconfig/20221112-101306-ladsgroup.json
  • 08:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1190 (T318605)', diff saved to https://phabricator.wikimedia.org/P39348 and previous config saved to /var/cache/conftool/dbconfig/20221112-082623-ladsgroup.json
  • 08:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 08:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 08:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T318605)', diff saved to https://phabricator.wikimedia.org/P39347 and previous config saved to /var/cache/conftool/dbconfig/20221112-082601-ladsgroup.json
  • 08:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39346 and previous config saved to /var/cache/conftool/dbconfig/20221112-081055-ladsgroup.json
  • 07:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39345 and previous config saved to /var/cache/conftool/dbconfig/20221112-075548-ladsgroup.json
  • 07:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T318605)', diff saved to https://phabricator.wikimedia.org/P39344 and previous config saved to /var/cache/conftool/dbconfig/20221112-074042-ladsgroup.json
  • 04:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2155 (T318605)', diff saved to https://phabricator.wikimedia.org/P39343 and previous config saved to /var/cache/conftool/dbconfig/20221112-043203-ladsgroup.json
  • 04:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 04:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 04:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 04:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 04:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39342 and previous config saved to /var/cache/conftool/dbconfig/20221112-043137-ladsgroup.json
  • 04:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P39341 and previous config saved to /var/cache/conftool/dbconfig/20221112-041631-ladsgroup.json
  • 04:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P39340 and previous config saved to /var/cache/conftool/dbconfig/20221112-040124-ladsgroup.json
  • 03:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39339 and previous config saved to /var/cache/conftool/dbconfig/20221112-034618-ladsgroup.json
  • 02:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T321130)', diff saved to https://phabricator.wikimedia.org/P39338 and previous config saved to /var/cache/conftool/dbconfig/20221112-022827-marostegui.json
  • 02:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T318605)', diff saved to https://phabricator.wikimedia.org/P39337 and previous config saved to /var/cache/conftool/dbconfig/20221112-022535-ladsgroup.json
  • 02:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 02:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 02:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P39336 and previous config saved to /var/cache/conftool/dbconfig/20221112-021321-marostegui.json
  • 01:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P39335 and previous config saved to /var/cache/conftool/dbconfig/20221112-015814-marostegui.json
  • 01:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T321130)', diff saved to https://phabricator.wikimedia.org/P39334 and previous config saved to /var/cache/conftool/dbconfig/20221112-014308-marostegui.json
  • 01:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2182 (T321130)', diff saved to https://phabricator.wikimedia.org/P39333 and previous config saved to /var/cache/conftool/dbconfig/20221112-013650-marostegui.json
  • 01:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 01:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 01:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39332 and previous config saved to /var/cache/conftool/dbconfig/20221112-013628-marostegui.json
  • 01:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39331 and previous config saved to /var/cache/conftool/dbconfig/20221112-012122-marostegui.json
  • 01:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39330 and previous config saved to /var/cache/conftool/dbconfig/20221112-010615-marostegui.json
  • 00:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39329 and previous config saved to /var/cache/conftool/dbconfig/20221112-005107-marostegui.json
  • 00:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39328 and previous config saved to /var/cache/conftool/dbconfig/20221112-004443-marostegui.json
  • 00:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 00:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 00:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39327 and previous config saved to /var/cache/conftool/dbconfig/20221112-004422-marostegui.json
  • 00:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39326 and previous config saved to /var/cache/conftool/dbconfig/20221112-002915-marostegui.json
  • 00:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39325 and previous config saved to /var/cache/conftool/dbconfig/20221112-001408-marostegui.json

2022-11-11

  • 23:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39324 and previous config saved to /var/cache/conftool/dbconfig/20221111-235902-marostegui.json
  • 23:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2168:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39323 and previous config saved to /var/cache/conftool/dbconfig/20221111-235235-marostegui.json
  • 23:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 23:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 23:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T321130)', diff saved to https://phabricator.wikimedia.org/P39322 and previous config saved to /var/cache/conftool/dbconfig/20221111-235214-marostegui.json
  • 23:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P39321 and previous config saved to /var/cache/conftool/dbconfig/20221111-233707-marostegui.json
  • 23:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P39320 and previous config saved to /var/cache/conftool/dbconfig/20221111-232201-marostegui.json
  • 23:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T321130)', diff saved to https://phabricator.wikimedia.org/P39319 and previous config saved to /var/cache/conftool/dbconfig/20221111-230654-marostegui.json
  • 23:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2159 (T321130)', diff saved to https://phabricator.wikimedia.org/P39318 and previous config saved to /var/cache/conftool/dbconfig/20221111-230037-marostegui.json
  • 23:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 23:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 23:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 23:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 23:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T321130)', diff saved to https://phabricator.wikimedia.org/P39317 and previous config saved to /var/cache/conftool/dbconfig/20221111-230000-marostegui.json
  • 22:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P39316 and previous config saved to /var/cache/conftool/dbconfig/20221111-224454-marostegui.json
  • 22:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P39315 and previous config saved to /var/cache/conftool/dbconfig/20221111-222948-marostegui.json
  • 22:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T321130)', diff saved to https://phabricator.wikimedia.org/P39314 and previous config saved to /var/cache/conftool/dbconfig/20221111-221441-marostegui.json
  • 22:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39313 and previous config saved to /var/cache/conftool/dbconfig/20221111-220939-ladsgroup.json
  • 22:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 22:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 22:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2150 (T321130)', diff saved to https://phabricator.wikimedia.org/P39312 and previous config saved to /var/cache/conftool/dbconfig/20221111-220820-marostegui.json
  • 22:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 22:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 22:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T321130)', diff saved to https://phabricator.wikimedia.org/P39311 and previous config saved to /var/cache/conftool/dbconfig/20221111-220758-marostegui.json
  • 21:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P39310 and previous config saved to /var/cache/conftool/dbconfig/20221111-215252-marostegui.json
  • 21:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P39309 and previous config saved to /var/cache/conftool/dbconfig/20221111-213745-marostegui.json
  • 21:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T321130)', diff saved to https://phabricator.wikimedia.org/P39308 and previous config saved to /var/cache/conftool/dbconfig/20221111-212239-marostegui.json
  • 21:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2122 (T321130)', diff saved to https://phabricator.wikimedia.org/P39307 and previous config saved to /var/cache/conftool/dbconfig/20221111-211611-marostegui.json
  • 21:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 21:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 21:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39306 and previous config saved to /var/cache/conftool/dbconfig/20221111-211550-marostegui.json
  • 21:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P39305 and previous config saved to /var/cache/conftool/dbconfig/20221111-210043-marostegui.json
  • 20:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 20:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 20:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T318605)', diff saved to https://phabricator.wikimedia.org/P39304 and previous config saved to /var/cache/conftool/dbconfig/20221111-205919-ladsgroup.json
  • 20:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P39303 and previous config saved to /var/cache/conftool/dbconfig/20221111-204536-marostegui.json
  • 20:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P39302 and previous config saved to /var/cache/conftool/dbconfig/20221111-204413-ladsgroup.json
  • 20:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39301 and previous config saved to /var/cache/conftool/dbconfig/20221111-203030-marostegui.json
  • 20:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P39300 and previous config saved to /var/cache/conftool/dbconfig/20221111-202906-ladsgroup.json
  • 20:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39299 and previous config saved to /var/cache/conftool/dbconfig/20221111-202413-marostegui.json
  • 20:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 20:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 20:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T321130)', diff saved to https://phabricator.wikimedia.org/P39298 and previous config saved to /var/cache/conftool/dbconfig/20221111-202351-marostegui.json
  • 20:21 mutante: phab1001,phab1004,phab2002 - systemctl reset-failed
  • 20:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T318605)', diff saved to https://phabricator.wikimedia.org/P39297 and previous config saved to /var/cache/conftool/dbconfig/20221111-201400-ladsgroup.json
  • 20:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P39296 and previous config saved to /var/cache/conftool/dbconfig/20221111-200845-marostegui.json
  • 19:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P39295 and previous config saved to /var/cache/conftool/dbconfig/20221111-195338-marostegui.json
  • 19:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T321130)', diff saved to https://phabricator.wikimedia.org/P39294 and previous config saved to /var/cache/conftool/dbconfig/20221111-193832-marostegui.json
  • 19:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2120 (T321130)', diff saved to https://phabricator.wikimedia.org/P39293 and previous config saved to /var/cache/conftool/dbconfig/20221111-193214-marostegui.json
  • 19:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 19:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 19:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T321130)', diff saved to https://phabricator.wikimedia.org/P39292 and previous config saved to /var/cache/conftool/dbconfig/20221111-193152-marostegui.json
  • 19:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P39291 and previous config saved to /var/cache/conftool/dbconfig/20221111-191646-marostegui.json
  • 19:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P39290 and previous config saved to /var/cache/conftool/dbconfig/20221111-190139-marostegui.json
  • 18:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T321130)', diff saved to https://phabricator.wikimedia.org/P39289 and previous config saved to /var/cache/conftool/dbconfig/20221111-184633-marostegui.json
  • 18:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2108 (T321130)', diff saved to https://phabricator.wikimedia.org/P39288 and previous config saved to /var/cache/conftool/dbconfig/20221111-184017-marostegui.json
  • 18:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 18:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 18:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 18:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T321130)', diff saved to https://phabricator.wikimedia.org/P39287 and previous config saved to /var/cache/conftool/dbconfig/20221111-182640-marostegui.json
  • 18:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39286 and previous config saved to /var/cache/conftool/dbconfig/20221111-181134-marostegui.json
  • 17:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39285 and previous config saved to /var/cache/conftool/dbconfig/20221111-175627-marostegui.json
  • 17:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T321130)', diff saved to https://phabricator.wikimedia.org/P39284 and previous config saved to /var/cache/conftool/dbconfig/20221111-174121-marostegui.json
  • 17:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T321130)', diff saved to https://phabricator.wikimedia.org/P39283 and previous config saved to /var/cache/conftool/dbconfig/20221111-173907-marostegui.json
  • 17:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T321130)', diff saved to https://phabricator.wikimedia.org/P39282 and previous config saved to /var/cache/conftool/dbconfig/20221111-173846-marostegui.json
  • 17:34 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=varnish-fe
  • 17:34 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-be
  • 17:34 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-tls
  • 17:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39281 and previous config saved to /var/cache/conftool/dbconfig/20221111-172339-marostegui.json
  • 17:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39280 and previous config saved to /var/cache/conftool/dbconfig/20221111-170833-marostegui.json
  • 16:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T321130)', diff saved to https://phabricator.wikimedia.org/P39279 and previous config saved to /var/cache/conftool/dbconfig/20221111-165326-marostegui.json
  • 16:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1194 (T321130)', diff saved to https://phabricator.wikimedia.org/P39278 and previous config saved to /var/cache/conftool/dbconfig/20221111-165113-marostegui.json
  • 16:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 16:50 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 16:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39277 and previous config saved to /var/cache/conftool/dbconfig/20221111-165051-marostegui.json
  • 16:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39275 and previous config saved to /var/cache/conftool/dbconfig/20221111-163545-marostegui.json
  • 16:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39274 and previous config saved to /var/cache/conftool/dbconfig/20221111-162038-marostegui.json
  • 16:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 16:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 16:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39273 and previous config saved to /var/cache/conftool/dbconfig/20221111-161528-ladsgroup.json
  • 16:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39272 and previous config saved to /var/cache/conftool/dbconfig/20221111-160532-marostegui.json
  • 16:05 vgutierrez: restart varnish in cp2042
  • 16:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P39271 and previous config saved to /var/cache/conftool/dbconfig/20221111-160022-ladsgroup.json
  • 15:58 vgutierrez: rolling restart of varnish in cp4045 - cp4050 - T322903
  • 15:57 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:56 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS buster
  • 15:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P39270 and previous config saved to /var/cache/conftool/dbconfig/20221111-154515-ladsgroup.json
  • 15:43 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS buster
  • 15:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39269 and previous config saved to /var/cache/conftool/dbconfig/20221111-153009-ladsgroup.json
  • 15:21 moritzm: installing node-end-of-stream security updates
  • 15:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39268 and previous config saved to /var/cache/conftool/dbconfig/20221111-150516-marostegui.json
  • 15:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 15:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 15:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T321130)', diff saved to https://phabricator.wikimedia.org/P39267 and previous config saved to /var/cache/conftool/dbconfig/20221111-150454-marostegui.json
  • 14:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P39266 and previous config saved to /var/cache/conftool/dbconfig/20221111-144948-marostegui.json
  • 14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T318605)', diff saved to https://phabricator.wikimedia.org/P39265 and previous config saved to /var/cache/conftool/dbconfig/20221111-144047-ladsgroup.json
  • 14:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 14:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T318605)', diff saved to https://phabricator.wikimedia.org/P39264 and previous config saved to /var/cache/conftool/dbconfig/20221111-144025-ladsgroup.json
  • 14:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P39263 and previous config saved to /var/cache/conftool/dbconfig/20221111-143441-marostegui.json
  • 14:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39262 and previous config saved to /var/cache/conftool/dbconfig/20221111-142519-ladsgroup.json
  • 14:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T321130)', diff saved to https://phabricator.wikimedia.org/P39261 and previous config saved to /var/cache/conftool/dbconfig/20221111-141935-marostegui.json
  • 14:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T321130)', diff saved to https://phabricator.wikimedia.org/P39260 and previous config saved to /var/cache/conftool/dbconfig/20221111-141721-marostegui.json
  • 14:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 14:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 14:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 14:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 14:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39259 and previous config saved to /var/cache/conftool/dbconfig/20221111-141233-marostegui.json
  • 14:12 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2003-dev.codfw.wmnet with OS bullseye
  • 14:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39258 and previous config saved to /var/cache/conftool/dbconfig/20221111-141012-ladsgroup.json
  • 13:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P39257 and previous config saved to /var/cache/conftool/dbconfig/20221111-135727-marostegui.json
  • 13:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T318605)', diff saved to https://phabricator.wikimedia.org/P39256 and previous config saved to /var/cache/conftool/dbconfig/20221111-135506-ladsgroup.json
  • 13:51 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 13:50 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 13:49 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2003-dev.codfw.wmnet with reason: host reimage
  • 13:47 aborrero@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2003-dev.codfw.wmnet with reason: host reimage
  • 13:45 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 13:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P39255 and previous config saved to /var/cache/conftool/dbconfig/20221111-134221-marostegui.json
  • 13:42 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:42 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:30 moritzm: installing procmail security updates
  • 13:30 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirt2003-dev.codfw.wmnet with OS bullseye
  • 13:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39254 and previous config saved to /var/cache/conftool/dbconfig/20221111-132714-marostegui.json
  • 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39253 and previous config saved to /var/cache/conftool/dbconfig/20221111-132105-marostegui.json
  • 13:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T321130)', diff saved to https://phabricator.wikimedia.org/P39252 and previous config saved to /var/cache/conftool/dbconfig/20221111-132043-marostegui.json
  • 13:20 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:13 jnuche@deploy1002: sync-world aborted: (no justification provided) (duration: 17m 49s)
  • 13:13 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
  • 13:13 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 13:13 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 13:13 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 13:13 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 13:13 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 13:13 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:12 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 13:10 jnuche@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 13:10 jnuche@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 13:08 jnuche@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 13:08 jnuche@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
  • 13:08 jnuche@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 13:07 jnuche@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
  • 13:06 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 13:06 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
  • 13:06 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 13:06 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 13:06 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 13:06 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 13:06 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:06 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 13:05 jnuche@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P39251 and previous config saved to /var/cache/conftool/dbconfig/20221111-130537-marostegui.json
  • 13:05 jnuche@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:01 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:01 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 12:55 jnuche@deploy1002: Started scap: (no justification provided)
  • 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P39249 and previous config saved to /var/cache/conftool/dbconfig/20221111-125030-marostegui.json
  • 12:42 moritzm: installing debootstrap bugfix updates from buster point release
  • 12:37 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 12:35 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T321130)', diff saved to https://phabricator.wikimedia.org/P39248 and previous config saved to /var/cache/conftool/dbconfig/20221111-123524-marostegui.json
  • 12:35 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:34 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host ganeti1033.eqiad.wmnet
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T321130)', diff saved to https://phabricator.wikimedia.org/P39247 and previous config saved to /var/cache/conftool/dbconfig/20221111-123310-marostegui.json
  • 12:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 12:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 12:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39246 and previous config saved to /var/cache/conftool/dbconfig/20221111-123232-marostegui.json
  • 12:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P39245 and previous config saved to /var/cache/conftool/dbconfig/20221111-121725-marostegui.json
  • 12:14 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 12:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet
  • 12:10 aborrero@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 12:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P39244 and previous config saved to /var/cache/conftool/dbconfig/20221111-120219-marostegui.json
  • 11:53 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 11:51 aborrero@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 11:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39243 and previous config saved to /var/cache/conftool/dbconfig/20221111-114712-marostegui.json
  • 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39242 and previous config saved to /var/cache/conftool/dbconfig/20221111-114458-marostegui.json
  • 11:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 11:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 11:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T321130)', diff saved to https://phabricator.wikimedia.org/P39241 and previous config saved to /var/cache/conftool/dbconfig/20221111-114437-marostegui.json
  • 11:42 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P39240 and previous config saved to /var/cache/conftool/dbconfig/20221111-112931-marostegui.json
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P39239 and previous config saved to /var/cache/conftool/dbconfig/20221111-111424-marostegui.json
  • 11:03 moritzm: installing wireshark security updates
  • 10:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T321130)', diff saved to https://phabricator.wikimedia.org/P39238 and previous config saved to /var/cache/conftool/dbconfig/20221111-105918-marostegui.json
  • 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T321130)', diff saved to https://phabricator.wikimedia.org/P39237 and previous config saved to /var/cache/conftool/dbconfig/20221111-105305-marostegui.json
  • 10:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 10:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 10:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39236 and previous config saved to /var/cache/conftool/dbconfig/20221111-105244-marostegui.json
  • 10:52 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 10:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P39235 and previous config saved to /var/cache/conftool/dbconfig/20221111-103738-marostegui.json
  • 10:22 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 10:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P39234 and previous config saved to /var/cache/conftool/dbconfig/20221111-102231-marostegui.json
  • 10:18 aborrero@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 10:15 elukey@cumin1001: END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES eqiad cluster: Roll restart of ORES's daemons.
  • 10:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39233 and previous config saved to /var/cache/conftool/dbconfig/20221111-100725-marostegui.json
  • 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39232 and previous config saved to /var/cache/conftool/dbconfig/20221111-100054-marostegui.json
  • 10:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 10:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39231 and previous config saved to /var/cache/conftool/dbconfig/20221111-100033-marostegui.json
  • 09:55 elukey@cumin1001: START - Cookbook sre.ores.roll-restart-workers for ORES eqiad cluster: Roll restart of ORES's daemons.
  • 09:54 elukey@cumin1001: END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES codfw cluster: Roll restart of ORES's daemons.
  • 09:45 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P39230 and previous config saved to /var/cache/conftool/dbconfig/20221111-094526-marostegui.json
  • 09:35 elukey@cumin1001: START - Cookbook sre.ores.roll-restart-workers for ORES codfw cluster: Roll restart of ORES's daemons.
  • 09:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P39229 and previous config saved to /var/cache/conftool/dbconfig/20221111-093020-marostegui.json
  • 09:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39228 and previous config saved to /var/cache/conftool/dbconfig/20221111-092503-ladsgroup.json
  • 09:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 09:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 09:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39227 and previous config saved to /var/cache/conftool/dbconfig/20221111-092441-ladsgroup.json
  • 09:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39226 and previous config saved to /var/cache/conftool/dbconfig/20221111-091514-marostegui.json
  • 09:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P39225 and previous config saved to /var/cache/conftool/dbconfig/20221111-090935-ladsgroup.json
  • 09:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39224 and previous config saved to /var/cache/conftool/dbconfig/20221111-090846-marostegui.json
  • 09:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:07 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1020.eqiad.wmnet to cluster eqiad and group D
  • 09:06 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1130.eqiad.wmnet with reason: Maintenance
  • 09:06 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1130.eqiad.wmnet with reason: Maintenance
  • 09:06 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1020.eqiad.wmnet to cluster eqiad and group D
  • 09:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2113.codfw.wmnet with reason: Maintenance
  • 09:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2113.codfw.wmnet with reason: Maintenance
  • 09:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1020.eqiad.wmnet
  • 09:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2112.codfw.wmnet with reason: Maintenance
  • 09:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2112.codfw.wmnet with reason: Maintenance
  • 08:55 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1020.eqiad.wmnet
  • 08:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P39223 and previous config saved to /var/cache/conftool/dbconfig/20221111-085428-ladsgroup.json
  • 08:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1020.eqiad.wmnet with OS bullseye
  • 08:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39222 and previous config saved to /var/cache/conftool/dbconfig/20221111-083922-ladsgroup.json
  • 08:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T318605)', diff saved to https://phabricator.wikimedia.org/P39221 and previous config saved to /var/cache/conftool/dbconfig/20221111-083611-ladsgroup.json
  • 08:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 08:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 08:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39220 and previous config saved to /var/cache/conftool/dbconfig/20221111-083549-ladsgroup.json
  • 08:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1020.eqiad.wmnet with reason: host reimage
  • 08:28 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1020.eqiad.wmnet with reason: host reimage
  • 08:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P39219 and previous config saved to /var/cache/conftool/dbconfig/20221111-082042-ladsgroup.json
  • 08:14 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1020.eqiad.wmnet with OS bullseye
  • 08:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1020.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 08:09 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1020.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 08:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P39218 and previous config saved to /var/cache/conftool/dbconfig/20221111-080536-ladsgroup.json
  • 07:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39217 and previous config saved to /var/cache/conftool/dbconfig/20221111-075028-ladsgroup.json
  • 06:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T321123)', diff saved to https://phabricator.wikimedia.org/P39216 and previous config saved to /var/cache/conftool/dbconfig/20221111-063240-marostegui.json
  • 06:22 vgutierrez: restart varnish on cp4047 to clear VarnishChildRestarted alert - T322903
  • 06:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P39215 and previous config saved to /var/cache/conftool/dbconfig/20221111-061733-marostegui.json
  • 06:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P39214 and previous config saved to /var/cache/conftool/dbconfig/20221111-060227-marostegui.json
  • 05:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T321123)', diff saved to https://phabricator.wikimedia.org/P39213 and previous config saved to /var/cache/conftool/dbconfig/20221111-054720-marostegui.json
  • 05:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2176 (T321123)', diff saved to https://phabricator.wikimedia.org/P39212 and previous config saved to /var/cache/conftool/dbconfig/20221111-054511-marostegui.json
  • 05:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 05:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 05:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T321123)', diff saved to https://phabricator.wikimedia.org/P39211 and previous config saved to /var/cache/conftool/dbconfig/20221111-054449-marostegui.json
  • 05:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P39210 and previous config saved to /var/cache/conftool/dbconfig/20221111-052943-marostegui.json
  • 05:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P39209 and previous config saved to /var/cache/conftool/dbconfig/20221111-051436-marostegui.json
  • 04:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T321123)', diff saved to https://phabricator.wikimedia.org/P39208 and previous config saved to /var/cache/conftool/dbconfig/20221111-045930-marostegui.json
  • 04:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2174 (T321123)', diff saved to https://phabricator.wikimedia.org/P39207 and previous config saved to /var/cache/conftool/dbconfig/20221111-045720-marostegui.json
  • 04:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 04:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 04:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T321123)', diff saved to https://phabricator.wikimedia.org/P39206 and previous config saved to /var/cache/conftool/dbconfig/20221111-045659-marostegui.json
  • 04:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P39205 and previous config saved to /var/cache/conftool/dbconfig/20221111-044152-marostegui.json
  • 04:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P39204 and previous config saved to /var/cache/conftool/dbconfig/20221111-042646-marostegui.json
  • 04:15 ejegg: civicrm upgraded from fd60273a to 93fa3f37
  • 04:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T321123)', diff saved to https://phabricator.wikimedia.org/P39203 and previous config saved to /var/cache/conftool/dbconfig/20221111-041139-marostegui.json
  • 04:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2173 (T321123)', diff saved to https://phabricator.wikimedia.org/P39202 and previous config saved to /var/cache/conftool/dbconfig/20221111-041030-marostegui.json
  • 04:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 04:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 04:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 04:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 04:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39201 and previous config saved to /var/cache/conftool/dbconfig/20221111-040953-marostegui.json
  • 03:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P39200 and previous config saved to /var/cache/conftool/dbconfig/20221111-035447-marostegui.json
  • 03:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P39199 and previous config saved to /var/cache/conftool/dbconfig/20221111-033940-marostegui.json
  • 03:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39198 and previous config saved to /var/cache/conftool/dbconfig/20221111-032434-marostegui.json
  • 03:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39197 and previous config saved to /var/cache/conftool/dbconfig/20221111-032224-marostegui.json
  • 03:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 03:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 03:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39196 and previous config saved to /var/cache/conftool/dbconfig/20221111-032203-marostegui.json
  • 03:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39195 and previous config saved to /var/cache/conftool/dbconfig/20221111-031358-ladsgroup.json
  • 03:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P39194 and previous config saved to /var/cache/conftool/dbconfig/20221111-030656-marostegui.json
  • 02:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P39193 and previous config saved to /var/cache/conftool/dbconfig/20221111-025851-ladsgroup.json
  • 02:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P39192 and previous config saved to /var/cache/conftool/dbconfig/20221111-025150-marostegui.json
  • 02:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P39191 and previous config saved to /var/cache/conftool/dbconfig/20221111-024345-ladsgroup.json
  • 02:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39190 and previous config saved to /var/cache/conftool/dbconfig/20221111-023643-marostegui.json
  • 02:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2167:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39189 and previous config saved to /var/cache/conftool/dbconfig/20221111-023534-marostegui.json
  • 02:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 02:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 02:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T321123)', diff saved to https://phabricator.wikimedia.org/P39188 and previous config saved to /var/cache/conftool/dbconfig/20221111-023513-marostegui.json
  • 02:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39187 and previous config saved to /var/cache/conftool/dbconfig/20221111-023252-ladsgroup.json
  • 02:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 02:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 02:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T318605)', diff saved to https://phabricator.wikimedia.org/P39186 and previous config saved to /var/cache/conftool/dbconfig/20221111-023231-ladsgroup.json
  • 02:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39185 and previous config saved to /var/cache/conftool/dbconfig/20221111-022838-ladsgroup.json
  • 02:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39184 and previous config saved to /var/cache/conftool/dbconfig/20221111-022619-ladsgroup.json
  • 02:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 02:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 02:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39183 and previous config saved to /var/cache/conftool/dbconfig/20221111-022557-ladsgroup.json
  • 02:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P39182 and previous config saved to /var/cache/conftool/dbconfig/20221111-022006-marostegui.json
  • 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39181 and previous config saved to /var/cache/conftool/dbconfig/20221111-021738-ladsgroup.json
  • 02:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39180 and previous config saved to /var/cache/conftool/dbconfig/20221111-021725-ladsgroup.json
  • 02:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39179 and previous config saved to /var/cache/conftool/dbconfig/20221111-021717-ladsgroup.json
  • 02:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39178 and previous config saved to /var/cache/conftool/dbconfig/20221111-021051-ladsgroup.json
  • 02:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P39177 and previous config saved to /var/cache/conftool/dbconfig/20221111-020500-marostegui.json
  • 02:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39176 and previous config saved to /var/cache/conftool/dbconfig/20221111-020218-ladsgroup.json
  • 02:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P39175 and previous config saved to /var/cache/conftool/dbconfig/20221111-020211-ladsgroup.json
  • 01:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39174 and previous config saved to /var/cache/conftool/dbconfig/20221111-015544-ladsgroup.json
  • 01:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T321123)', diff saved to https://phabricator.wikimedia.org/P39173 and previous config saved to /var/cache/conftool/dbconfig/20221111-014953-marostegui.json
  • 01:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2153 (T321123)', diff saved to https://phabricator.wikimedia.org/P39172 and previous config saved to /var/cache/conftool/dbconfig/20221111-014744-marostegui.json
  • 01:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 01:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 01:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T321123)', diff saved to https://phabricator.wikimedia.org/P39171 and previous config saved to /var/cache/conftool/dbconfig/20221111-014722-marostegui.json
  • 01:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T318605)', diff saved to https://phabricator.wikimedia.org/P39170 and previous config saved to /var/cache/conftool/dbconfig/20221111-014712-ladsgroup.json
  • 01:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P39169 and previous config saved to /var/cache/conftool/dbconfig/20221111-014704-ladsgroup.json
  • 01:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39168 and previous config saved to /var/cache/conftool/dbconfig/20221111-014037-ladsgroup.json
  • 01:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39167 and previous config saved to /var/cache/conftool/dbconfig/20221111-013818-ladsgroup.json
  • 01:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 01:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 01:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39166 and previous config saved to /var/cache/conftool/dbconfig/20221111-013756-ladsgroup.json
  • 01:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P39165 and previous config saved to /var/cache/conftool/dbconfig/20221111-013209-marostegui.json
  • 01:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39164 and previous config saved to /var/cache/conftool/dbconfig/20221111-013157-ladsgroup.json
  • 01:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39163 and previous config saved to /var/cache/conftool/dbconfig/20221111-012250-ladsgroup.json
  • 01:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P39162 and previous config saved to /var/cache/conftool/dbconfig/20221111-011703-marostegui.json
  • 01:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39161 and previous config saved to /var/cache/conftool/dbconfig/20221111-010743-ladsgroup.json
  • 01:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T321123)', diff saved to https://phabricator.wikimedia.org/P39160 and previous config saved to /var/cache/conftool/dbconfig/20221111-010156-marostegui.json
  • 00:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2146 (T321123)', diff saved to https://phabricator.wikimedia.org/P39159 and previous config saved to /var/cache/conftool/dbconfig/20221111-005947-marostegui.json
  • 00:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 00:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 00:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T321123)', diff saved to https://phabricator.wikimedia.org/P39158 and previous config saved to /var/cache/conftool/dbconfig/20221111-005925-marostegui.json
  • 00:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39157 and previous config saved to /var/cache/conftool/dbconfig/20221111-005237-ladsgroup.json
  • 00:50 jclark@cumin1001: START - Cookbook sre.hosts.provision for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2168:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39156 and previous config saved to /var/cache/conftool/dbconfig/20221111-005017-ladsgroup.json
  • 00:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 00:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 00:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T322618)', diff saved to https://phabricator.wikimedia.org/P39155 and previous config saved to /var/cache/conftool/dbconfig/20221111-004945-ladsgroup.json
  • 00:47 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 00:45 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 00:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P39154 and previous config saved to /var/cache/conftool/dbconfig/20221111-004419-marostegui.json
  • 00:43 jclark@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:43 jclark@cumin1001: START - Cookbook sre.hosts.provision for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:42 jclark@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:38 jclark@cumin1001: START - Cookbook sre.hosts.provision for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P39153 and previous config saved to /var/cache/conftool/dbconfig/20221111-003438-ladsgroup.json
  • 00:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39152 and previous config saved to /var/cache/conftool/dbconfig/20221111-003141-ladsgroup.json
  • 00:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 00:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 00:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P39151 and previous config saved to /var/cache/conftool/dbconfig/20221111-002913-marostegui.json
  • 00:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P39150 and previous config saved to /var/cache/conftool/dbconfig/20221111-001932-ladsgroup.json
  • 00:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T321123)', diff saved to https://phabricator.wikimedia.org/P39149 and previous config saved to /var/cache/conftool/dbconfig/20221111-001406-marostegui.json
  • 00:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2145 (T321123)', diff saved to https://phabricator.wikimedia.org/P39148 and previous config saved to /var/cache/conftool/dbconfig/20221111-001156-marostegui.json
  • 00:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2145.codfw.wmnet with reason: Maintenance
  • 00:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2145.codfw.wmnet with reason: Maintenance
  • 00:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 00:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 00:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T321123)', diff saved to https://phabricator.wikimedia.org/P39147 and previous config saved to /var/cache/conftool/dbconfig/20221111-001056-marostegui.json
  • 00:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T322618)', diff saved to https://phabricator.wikimedia.org/P39146 and previous config saved to /var/cache/conftool/dbconfig/20221111-000425-ladsgroup.json
  • 00:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2159 (T322618)', diff saved to https://phabricator.wikimedia.org/P39145 and previous config saved to /var/cache/conftool/dbconfig/20221111-000206-ladsgroup.json
  • 00:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 00:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 00:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 00:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 00:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T322618)', diff saved to https://phabricator.wikimedia.org/P39144 and previous config saved to /var/cache/conftool/dbconfig/20221111-000118-ladsgroup.json

2022-11-10

  • 23:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P39143 and previous config saved to /var/cache/conftool/dbconfig/20221110-235549-marostegui.json
  • 23:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P39142 and previous config saved to /var/cache/conftool/dbconfig/20221110-234612-ladsgroup.json
  • 23:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P39141 and previous config saved to /var/cache/conftool/dbconfig/20221110-234043-marostegui.json
  • 23:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P39140 and previous config saved to /var/cache/conftool/dbconfig/20221110-233105-ladsgroup.json
  • 23:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T321123)', diff saved to https://phabricator.wikimedia.org/P39139 and previous config saved to /var/cache/conftool/dbconfig/20221110-232536-marostegui.json
  • 23:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2130 (T321123)', diff saved to https://phabricator.wikimedia.org/P39138 and previous config saved to /var/cache/conftool/dbconfig/20221110-232327-marostegui.json
  • 23:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 23:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 23:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T321123)', diff saved to https://phabricator.wikimedia.org/P39137 and previous config saved to /var/cache/conftool/dbconfig/20221110-232305-marostegui.json
  • 23:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T322618)', diff saved to https://phabricator.wikimedia.org/P39136 and previous config saved to /var/cache/conftool/dbconfig/20221110-231558-ladsgroup.json
  • 23:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2150 (T322618)', diff saved to https://phabricator.wikimedia.org/P39135 and previous config saved to /var/cache/conftool/dbconfig/20221110-231339-ladsgroup.json
  • 23:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 23:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 23:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T322618)', diff saved to https://phabricator.wikimedia.org/P39134 and previous config saved to /var/cache/conftool/dbconfig/20221110-231306-ladsgroup.json
  • 23:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P39133 and previous config saved to /var/cache/conftool/dbconfig/20221110-230759-marostegui.json
  • 22:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P39132 and previous config saved to /var/cache/conftool/dbconfig/20221110-225759-ladsgroup.json
  • 22:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P39131 and previous config saved to /var/cache/conftool/dbconfig/20221110-225253-marostegui.json
  • 22:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P39130 and previous config saved to /var/cache/conftool/dbconfig/20221110-224253-ladsgroup.json
  • 22:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T321123)', diff saved to https://phabricator.wikimedia.org/P39129 and previous config saved to /var/cache/conftool/dbconfig/20221110-223746-marostegui.json
  • 22:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2116 (T321123)', diff saved to https://phabricator.wikimedia.org/P39128 and previous config saved to /var/cache/conftool/dbconfig/20221110-223537-marostegui.json
  • 22:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 22:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 22:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T321123)', diff saved to https://phabricator.wikimedia.org/P39127 and previous config saved to /var/cache/conftool/dbconfig/20221110-223515-marostegui.json
  • 22:27 eileen: thank you back on config revision changed from bbdd4315 to 2bb73bb1
  • 22:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T322618)', diff saved to https://phabricator.wikimedia.org/P39126 and previous config saved to /var/cache/conftool/dbconfig/20221110-222746-ladsgroup.json
  • 22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2122 (T322618)', diff saved to https://phabricator.wikimedia.org/P39125 and previous config saved to /var/cache/conftool/dbconfig/20221110-222526-ladsgroup.json
  • 22:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 22:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T322618)', diff saved to https://phabricator.wikimedia.org/P39124 and previous config saved to /var/cache/conftool/dbconfig/20221110-222505-ladsgroup.json
  • 22:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P39123 and previous config saved to /var/cache/conftool/dbconfig/20221110-222009-marostegui.json
  • 22:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P39122 and previous config saved to /var/cache/conftool/dbconfig/20221110-220958-ladsgroup.json
  • 22:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P39121 and previous config saved to /var/cache/conftool/dbconfig/20221110-220502-marostegui.json
  • 21:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P39120 and previous config saved to /var/cache/conftool/dbconfig/20221110-215452-ladsgroup.json
  • 21:53 jgleeson: payments updated from 17cd1956 to a058fdbc
  • 21:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T321123)', diff saved to https://phabricator.wikimedia.org/P39119 and previous config saved to /var/cache/conftool/dbconfig/20221110-214956-marostegui.json
  • 21:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2103 (T321123)', diff saved to https://phabricator.wikimedia.org/P39118 and previous config saved to /var/cache/conftool/dbconfig/20221110-214746-marostegui.json
  • 21:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 21:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 21:47 greg-g: 3:43:33 <eileen> !civicrm upgraded from 6c2e07e0 to fd60273a
  • 21:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 21:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 21:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 21:42 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 21:42 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 21:42 eileen: process-control config revision changed from 4e438cf5 to bbdd4315 (disable thank you)
  • 21:42 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 21:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T321123)', diff saved to https://phabricator.wikimedia.org/P39117 and previous config saved to /var/cache/conftool/dbconfig/20221110-214240-marostegui.json
  • 21:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T322618)', diff saved to https://phabricator.wikimedia.org/P39116 and previous config saved to /var/cache/conftool/dbconfig/20221110-213945-ladsgroup.json
  • 21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2121 (T322618)', diff saved to https://phabricator.wikimedia.org/P39115 and previous config saved to /var/cache/conftool/dbconfig/20221110-213726-ladsgroup.json
  • 21:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 21:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T322618)', diff saved to https://phabricator.wikimedia.org/P39114 and previous config saved to /var/cache/conftool/dbconfig/20221110-213704-ladsgroup.json
  • 21:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P39113 and previous config saved to /var/cache/conftool/dbconfig/20221110-212734-marostegui.json
  • 21:27 dancy@deploy1002: Installation of scap version "4.28.0" completed for 559 hosts
  • 21:26 dancy@deploy1002: Installing scap version "4.28.0" for 559 hosts
  • 21:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P39112 and previous config saved to /var/cache/conftool/dbconfig/20221110-212158-ladsgroup.json
  • 21:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P39111 and previous config saved to /var/cache/conftool/dbconfig/20221110-211227-marostegui.json
  • 21:10 mutante: deploy1002 - armed the keyholder with deployment keys - 2 hours ago alerts started that it was not armed (does it notify people?) - got pinged that deployers got scap problems - unknown why it was disarmed - now it is armed again
  • 21:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P39110 and previous config saved to /var/cache/conftool/dbconfig/20221110-210651-ladsgroup.json
  • 21:04 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:03 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 21:03 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕞🍵 sudo cumin -b 8 A:cp 'run-puppet-agent --enable T306580'
  • 20:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T321123)', diff saved to https://phabricator.wikimedia.org/P39109 and previous config saved to /var/cache/conftool/dbconfig/20221110-205720-marostegui.json
  • 20:56 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1196 (T321123)', diff saved to https://phabricator.wikimedia.org/P39108 and previous config saved to /var/cache/conftool/dbconfig/20221110-205613-marostegui.json
  • 20:56 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 20:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 20:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T321123)', diff saved to https://phabricator.wikimedia.org/P39107 and previous config saved to /var/cache/conftool/dbconfig/20221110-205552-marostegui.json
  • 20:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T322618)', diff saved to https://phabricator.wikimedia.org/P39106 and previous config saved to /var/cache/conftool/dbconfig/20221110-205145-ladsgroup.json
  • 20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2120 (T322618)', diff saved to https://phabricator.wikimedia.org/P39105 and previous config saved to /var/cache/conftool/dbconfig/20221110-204925-ladsgroup.json
  • 20:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 20:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T322618)', diff saved to https://phabricator.wikimedia.org/P39104 and previous config saved to /var/cache/conftool/dbconfig/20221110-204904-ladsgroup.json
  • 20:43 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:42 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 20:41 dancy@deploy1002: Installing scap version "4.28.0" for 559 hosts
  • 20:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P39103 and previous config saved to /var/cache/conftool/dbconfig/20221110-204045-marostegui.json
  • 20:40 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:39 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 20:36 cdanis: ✔️ cdanis@cp3053.esams.wmnet ~ 🕞🍵 sudo run-puppet-agent --enable T306580
  • 20:36 cdanis: ✔️ cdanis@cp3052.esams.wmnet ~ 🕞🍵 sudo run-puppet-agent --enable T306580
  • 20:35 cdanis: ✔️ cdanis@cp2027.codfw.wmnet ~ 🕞🍵 sudo run-puppet-agent --enable T306580
  • 20:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P39102 and previous config saved to /var/cache/conftool/dbconfig/20221110-203357-ladsgroup.json
  • 20:33 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕞🍵 sudo cumin A:cp 'disable-puppet T306580'
  • 20:32 dancy@deploy1002: Installing scap version "4.28.0" for 559 hosts
  • 20:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T321130)', diff saved to https://phabricator.wikimedia.org/P39101 and previous config saved to /var/cache/conftool/dbconfig/20221110-202744-marostegui.json
  • 20:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P39100 and previous config saved to /var/cache/conftool/dbconfig/20221110-202539-marostegui.json
  • 20:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P39099 and previous config saved to /var/cache/conftool/dbconfig/20221110-201851-ladsgroup.json
  • 20:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P39098 and previous config saved to /var/cache/conftool/dbconfig/20221110-201237-marostegui.json
  • 20:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T321123)', diff saved to https://phabricator.wikimedia.org/P39097 and previous config saved to /var/cache/conftool/dbconfig/20221110-201032-marostegui.json
  • 20:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1186 (T321123)', diff saved to https://phabricator.wikimedia.org/P39096 and previous config saved to /var/cache/conftool/dbconfig/20221110-200924-marostegui.json
  • 20:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 20:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 20:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T321123)', diff saved to https://phabricator.wikimedia.org/P39095 and previous config saved to /var/cache/conftool/dbconfig/20221110-200903-marostegui.json
  • 20:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T322618)', diff saved to https://phabricator.wikimedia.org/P39094 and previous config saved to /var/cache/conftool/dbconfig/20221110-200344-ladsgroup.json
  • 20:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2108 (T322618)', diff saved to https://phabricator.wikimedia.org/P39093 and previous config saved to /var/cache/conftool/dbconfig/20221110-200125-ladsgroup.json
  • 20:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 20:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 20:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 20:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 20:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 20:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 19:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 19:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 19:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T322618)', diff saved to https://phabricator.wikimedia.org/P39092 and previous config saved to /var/cache/conftool/dbconfig/20221110-195938-ladsgroup.json
  • 19:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P39091 and previous config saved to /var/cache/conftool/dbconfig/20221110-195731-marostegui.json
  • 19:57 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:55 dzahn@cumin2002: START - Cookbook sre.dns.netbox
  • 19:54 mutante: netbox - deleting special case phab2001-vcs.codfw.wmnet IPv4 (10.192.32.149) and IPv6 (2620:0:860:103:10:192:32:149) - T296022 - T322250
  • 19:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P39090 and previous config saved to /var/cache/conftool/dbconfig/20221110-195357-marostegui.json
  • 19:52 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:51 robh@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1033.eqiad.wmnet with OS bullseye
  • 19:51 dzahn@cumin2002: START - Cookbook sre.dns.netbox
  • 19:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39089 and previous config saved to /var/cache/conftool/dbconfig/20221110-194431-ladsgroup.json
  • 19:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T321130)', diff saved to https://phabricator.wikimedia.org/P39088 and previous config saved to /var/cache/conftool/dbconfig/20221110-194224-marostegui.json
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2178 (T321130)', diff saved to https://phabricator.wikimedia.org/P39087 and previous config saved to /var/cache/conftool/dbconfig/20221110-194031-marostegui.json
  • 19:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39086 and previous config saved to /var/cache/conftool/dbconfig/20221110-194009-marostegui.json
  • 19:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P39085 and previous config saved to /var/cache/conftool/dbconfig/20221110-193850-marostegui.json
  • 19:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2136 (T318605)', diff saved to https://phabricator.wikimedia.org/P39084 and previous config saved to /var/cache/conftool/dbconfig/20221110-193459-ladsgroup.json
  • 19:34 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1033.eqiad.wmnet with reason: host reimage
  • 19:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 19:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 19:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T318605)', diff saved to https://phabricator.wikimedia.org/P39083 and previous config saved to /var/cache/conftool/dbconfig/20221110-193437-ladsgroup.json
  • 19:32 damilare: civicrm upgraded from 07fdeed5 to 6c2e07e0
  • 19:31 robh@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1033.eqiad.wmnet with reason: host reimage
  • 19:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39082 and previous config saved to /var/cache/conftool/dbconfig/20221110-192925-ladsgroup.json
  • 19:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P39081 and previous config saved to /var/cache/conftool/dbconfig/20221110-192503-marostegui.json
  • 19:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T321123)', diff saved to https://phabricator.wikimedia.org/P39080 and previous config saved to /var/cache/conftool/dbconfig/20221110-192343-marostegui.json
  • 19:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1184 (T321123)', diff saved to https://phabricator.wikimedia.org/P39079 and previous config saved to /var/cache/conftool/dbconfig/20221110-192236-marostegui.json
  • 19:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 19:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 19:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T321123)', diff saved to https://phabricator.wikimedia.org/P39078 and previous config saved to /var/cache/conftool/dbconfig/20221110-192215-marostegui.json
  • 19:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39077 and previous config saved to /var/cache/conftool/dbconfig/20221110-191930-ladsgroup.json
  • 19:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 19:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 19:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39076 and previous config saved to /var/cache/conftool/dbconfig/20221110-191900-ladsgroup.json
  • 19:18 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1033.eqiad.wmnet with OS bullseye
  • 19:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T322618)', diff saved to https://phabricator.wikimedia.org/P39075 and previous config saved to /var/cache/conftool/dbconfig/20221110-191418-ladsgroup.json
  • 19:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T322618)', diff saved to https://phabricator.wikimedia.org/P39074 and previous config saved to /var/cache/conftool/dbconfig/20221110-191208-ladsgroup.json
  • 19:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 19:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 19:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P39073 and previous config saved to /var/cache/conftool/dbconfig/20221110-191146-ladsgroup.json
  • 19:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P39072 and previous config saved to /var/cache/conftool/dbconfig/20221110-190957-marostegui.json
  • 19:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P39071 and previous config saved to /var/cache/conftool/dbconfig/20221110-190708-marostegui.json
  • 19:05 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts phab2001.codfw.wmnet
  • 19:05 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39070 and previous config saved to /var/cache/conftool/dbconfig/20221110-190424-ladsgroup.json
  • 19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39069 and previous config saved to /var/cache/conftool/dbconfig/20221110-190353-ladsgroup.json
  • 19:03 dzahn@cumin2002: START - Cookbook sre.dns.netbox
  • 18:58 mutante: phabricator - running decom cookbook on phab2001 - T322250
  • 18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39068 and previous config saved to /var/cache/conftool/dbconfig/20221110-185640-ladsgroup.json
  • 18:55 dzahn@cumin2002: START - Cookbook sre.hosts.decommission for hosts phab2001.codfw.wmnet
  • 18:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39067 and previous config saved to /var/cache/conftool/dbconfig/20221110-185450-marostegui.json
  • 18:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P39066 and previous config saved to /var/cache/conftool/dbconfig/20221110-185202-marostegui.json
  • 18:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39065 and previous config saved to /var/cache/conftool/dbconfig/20221110-185135-marostegui.json
  • 18:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 18:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 18:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39064 and previous config saved to /var/cache/conftool/dbconfig/20221110-185103-marostegui.json
  • 18:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T318605)', diff saved to https://phabricator.wikimedia.org/P39063 and previous config saved to /var/cache/conftool/dbconfig/20221110-184917-ladsgroup.json
  • 18:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39062 and previous config saved to /var/cache/conftool/dbconfig/20221110-184847-ladsgroup.json
  • 18:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39061 and previous config saved to /var/cache/conftool/dbconfig/20221110-184133-ladsgroup.json
  • 18:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T321123)', diff saved to https://phabricator.wikimedia.org/P39060 and previous config saved to /var/cache/conftool/dbconfig/20221110-183655-marostegui.json
  • 18:36 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4008.ulsfo.wmnet with reason: downtimed as we are resolving issues with LVS configuration
  • 18:36 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on lvs4008.ulsfo.wmnet with reason: downtimed as we are resolving issues with LVS configuration
  • 18:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P39059 and previous config saved to /var/cache/conftool/dbconfig/20221110-183556-marostegui.json
  • 18:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1169 (T321123)', diff saved to https://phabricator.wikimedia.org/P39058 and previous config saved to /var/cache/conftool/dbconfig/20221110-183548-marostegui.json
  • 18:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 18:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 18:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T321123)', diff saved to https://phabricator.wikimedia.org/P39057 and previous config saved to /var/cache/conftool/dbconfig/20221110-183455-marostegui.json
  • 18:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39056 and previous config saved to /var/cache/conftool/dbconfig/20221110-183340-ladsgroup.json
  • 18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P39055 and previous config saved to /var/cache/conftool/dbconfig/20221110-182627-ladsgroup.json
  • 18:26 ladsgroup@deploy1002: Synchronized portals: (no justification provided) (duration: 03m 38s)
  • 18:22 ladsgroup@deploy1002: Synchronized portals/wikipedia.org/assets: (no justification provided) (duration: 03m 46s)
  • 18:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P39054 and previous config saved to /var/cache/conftool/dbconfig/20221110-182049-marostegui.json
  • 18:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P39053 and previous config saved to /var/cache/conftool/dbconfig/20221110-181948-marostegui.json
  • 18:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:18 ladsgroup@deploy1002: Finished scap: Backport for Bump portals to HEAD (T273179) (duration: 05m 14s)
  • 18:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:17 dcausse@deploy1002: Finished deploy [wikimedia/discovery/analytics@a030f5f]: T320656: convert_to_esbulk: fix typo in config (duration: 02m 22s)
  • 18:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:15 dcausse@deploy1002: Started deploy [wikimedia/discovery/analytics@a030f5f]: T320656: convert_to_esbulk: fix typo in config
  • 18:13 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for Bump portals to HEAD (T273179) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 18:12 ladsgroup@deploy1002: Started scap: Backport for Bump portals to HEAD (T273179)
  • 18:09 volans: upgrading spicerack to 5.0.0 on cumin hosts
  • 18:05 volans: uploaded spicerack_5.0.0 to apt.wikimedia.org bullseye-wikimedia
  • 18:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39052 and previous config saved to /var/cache/conftool/dbconfig/20221110-180543-marostegui.json
  • 18:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P39051 and previous config saved to /var/cache/conftool/dbconfig/20221110-180442-marostegui.json
  • 18:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39050 and previous config saved to /var/cache/conftool/dbconfig/20221110-180228-marostegui.json
  • 18:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 18:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 18:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39049 and previous config saved to /var/cache/conftool/dbconfig/20221110-180206-marostegui.json
  • 18:01 volans: uploaded python3-gjson_0.3.0 to apt.wikimedia.org bullseye-wikimedia,unstable-wikimedia
  • 17:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T321123)', diff saved to https://phabricator.wikimedia.org/P39048 and previous config saved to /var/cache/conftool/dbconfig/20221110-174935-marostegui.json
  • 17:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1135 (T321123)', diff saved to https://phabricator.wikimedia.org/P39047 and previous config saved to /var/cache/conftool/dbconfig/20221110-174828-marostegui.json
  • 17:48 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T321123)', diff saved to https://phabricator.wikimedia.org/P39046 and previous config saved to /var/cache/conftool/dbconfig/20221110-174806-marostegui.json
  • 17:47 dcausse@deploy1002: Finished deploy [wikimedia/discovery/analytics@84dd7b5]: T320656: image_suggestions: schedule ad hoc dataset to fix improper suggestions (duration: 02m 18s)
  • 17:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P39045 and previous config saved to /var/cache/conftool/dbconfig/20221110-174659-marostegui.json
  • 17:44 dcausse@deploy1002: Started deploy [wikimedia/discovery/analytics@84dd7b5]: T320656: image_suggestions: schedule ad hoc dataset to fix improper suggestions
  • 17:37 sukhe: [done] running sukhe@cumin2002:~$ homer "cr*-ulsfo*" commit "Gerrit 855583: sites.yaml: add lvs4008 (ulsfo hardware refresh)"
  • 17:36 sukhe: running sukhe@cumin2002:~$ homer "cr*-ulsfo*" commit "Gerrit 855583: sites.yaml: add lvs4008 (ulsfo hardware refresh)"
  • 17:35 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4008.ulsfo.wmnet with OS buster
  • 17:34 rzl: rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_growthexperiments-userImpactDelete.service # test run for T322706 T322541
  • 17:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P39044 and previous config saved to /var/cache/conftool/dbconfig/20221110-173300-marostegui.json
  • 17:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P39043 and previous config saved to /var/cache/conftool/dbconfig/20221110-173153-marostegui.json
  • 17:28 urandom: restarting bootstrap of aqs1016-a -- T307802
  • 17:26 urandom: increasing stream throughput to 400mbit, aqs1011-{a,b} & aqs1013-{a,b} -- T307802
  • 17:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P39042 and previous config saved to /var/cache/conftool/dbconfig/20221110-172611-ladsgroup.json
  • 17:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 17:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 17:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T322618)', diff saved to https://phabricator.wikimedia.org/P39041 and previous config saved to /var/cache/conftool/dbconfig/20221110-172549-ladsgroup.json
  • 17:23 rzl: rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_growthexperiments-userImpactUpdateRecentlyEdited.service # test run for T322706 T322541
  • 17:18 robh@cumin1001: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ganeti1033']
  • 17:18 rzl: rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_growthexperiments-userImpactUpdateRecentlyRegistered.service # test run for T322706 T322541
  • 17:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P39040 and previous config saved to /var/cache/conftool/dbconfig/20221110-171753-marostegui.json
  • 17:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39039 and previous config saved to /var/cache/conftool/dbconfig/20221110-171646-marostegui.json
  • 17:13 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 17:13 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39038 and previous config saved to /var/cache/conftool/dbconfig/20221110-171329-marostegui.json
  • 17:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 17:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 17:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T321130)', diff saved to https://phabricator.wikimedia.org/P39037 and previous config saved to /var/cache/conftool/dbconfig/20221110-171308-marostegui.json
  • 17:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39036 and previous config saved to /var/cache/conftool/dbconfig/20221110-171043-ladsgroup.json
  • 17:10 robh@cumin1001: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1033']
  • 17:09 robh@cumin1001: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ganeti1033']
  • 17:09 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 17:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T321123)', diff saved to https://phabricator.wikimedia.org/P39035 and previous config saved to /var/cache/conftool/dbconfig/20221110-170247-marostegui.json
  • 17:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1134 (T321123)', diff saved to https://phabricator.wikimedia.org/P39034 and previous config saved to /var/cache/conftool/dbconfig/20221110-170139-marostegui.json
  • 17:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1134.eqiad.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1134.eqiad.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1133.eqiad.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1133.eqiad.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 (T321123)', diff saved to https://phabricator.wikimedia.org/P39033 and previous config saved to /var/cache/conftool/dbconfig/20221110-170102-marostegui.json
  • 16:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P39032 and previous config saved to /var/cache/conftool/dbconfig/20221110-165802-marostegui.json
  • 16:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39031 and previous config saved to /var/cache/conftool/dbconfig/20221110-165536-ladsgroup.json
  • 16:53 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS buster
  • 16:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P39030 and previous config saved to /var/cache/conftool/dbconfig/20221110-164556-marostegui.json
  • 16:44 sukhe@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs4008.ulsfo.wmnet with OS buster
  • 16:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P39029 and previous config saved to /var/cache/conftool/dbconfig/20221110-164255-marostegui.json
  • 16:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T322618)', diff saved to https://phabricator.wikimedia.org/P39028 and previous config saved to /var/cache/conftool/dbconfig/20221110-164030-ladsgroup.json
  • 16:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T322618)', diff saved to https://phabricator.wikimedia.org/P39027 and previous config saved to /var/cache/conftool/dbconfig/20221110-163819-ladsgroup.json
  • 16:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 16:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 16:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T322618)', diff saved to https://phabricator.wikimedia.org/P39026 and previous config saved to /var/cache/conftool/dbconfig/20221110-163758-ladsgroup.json
  • 16:37 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 16:34 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 16:33 robh@cumin1001: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1033']
  • 16:33 robh@cumin1001: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti1033']
  • 16:32 robh@cumin1001: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1033']
  • 16:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P39025 and previous config saved to /var/cache/conftool/dbconfig/20221110-163049-marostegui.json
  • 16:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T321130)', diff saved to https://phabricator.wikimedia.org/P39024 and previous config saved to /var/cache/conftool/dbconfig/20221110-162749-marostegui.json
  • 16:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2128 (T321130)', diff saved to https://phabricator.wikimedia.org/P39023 and previous config saved to /var/cache/conftool/dbconfig/20221110-162453-marostegui.json
  • 16:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 16:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 16:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 16:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 16:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T321130)', diff saved to https://phabricator.wikimedia.org/P39022 and previous config saved to /var/cache/conftool/dbconfig/20221110-162416-marostegui.json
  • 16:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P39021 and previous config saved to /var/cache/conftool/dbconfig/20221110-162251-ladsgroup.json
  • 16:16 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS buster
  • 16:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 (T321123)', diff saved to https://phabricator.wikimedia.org/P39020 and previous config saved to /var/cache/conftool/dbconfig/20221110-161543-marostegui.json
  • 16:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1132 (T321123)', diff saved to https://phabricator.wikimedia.org/P39019 and previous config saved to /var/cache/conftool/dbconfig/20221110-161435-marostegui.json
  • 16:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1132.eqiad.wmnet with reason: Maintenance
  • 16:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1132.eqiad.wmnet with reason: Maintenance
  • 16:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128 (T321123)', diff saved to https://phabricator.wikimedia.org/P39018 and previous config saved to /var/cache/conftool/dbconfig/20221110-161413-marostegui.json
  • 16:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P39017 and previous config saved to /var/cache/conftool/dbconfig/20221110-160906-marostegui.json
  • 16:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P39016 and previous config saved to /var/cache/conftool/dbconfig/20221110-160745-ladsgroup.json
  • 16:03 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2002-dev.codfw.wmnet with OS bullseye
  • 15:59 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 15:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P39015 and previous config saved to /var/cache/conftool/dbconfig/20221110-155907-marostegui.json
  • 15:56 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-eqiad
  • 15:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P39014 and previous config saved to /var/cache/conftool/dbconfig/20221110-155400-marostegui.json
  • 15:53 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T322618)', diff saved to https://phabricator.wikimedia.org/P39013 and previous config saved to /var/cache/conftool/dbconfig/20221110-155238-ladsgroup.json
  • 15:50 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
  • 15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T322618)', diff saved to https://phabricator.wikimedia.org/P39012 and previous config saved to /var/cache/conftool/dbconfig/20221110-154827-ladsgroup.json
  • 15:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 15:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 15:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 15:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 15:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39011 and previous config saved to /var/cache/conftool/dbconfig/20221110-154746-ladsgroup.json
  • 15:47 aborrero@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
  • 15:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P39010 and previous config saved to /var/cache/conftool/dbconfig/20221110-154400-marostegui.json
  • 15:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T321130)', diff saved to https://phabricator.wikimedia.org/P39009 and previous config saved to /var/cache/conftool/dbconfig/20221110-153853-marostegui.json
  • 15:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2123 (T321130)', diff saved to https://phabricator.wikimedia.org/P39008 and previous config saved to /var/cache/conftool/dbconfig/20221110-153559-marostegui.json
  • 15:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 15:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 15:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T321130)', diff saved to https://phabricator.wikimedia.org/P39007 and previous config saved to /var/cache/conftool/dbconfig/20221110-153537-marostegui.json
  • 15:33 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudgw2002-dev.codfw.wmnet with OS bullseye
  • 15:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P39006 and previous config saved to /var/cache/conftool/dbconfig/20221110-153240-ladsgroup.json
  • 15:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128 (T321123)', diff saved to https://phabricator.wikimedia.org/P39005 and previous config saved to /var/cache/conftool/dbconfig/20221110-152854-marostegui.json
  • 15:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P39004 and previous config saved to /var/cache/conftool/dbconfig/20221110-152031-marostegui.json
  • 15:19 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1128 (T321123)', diff saved to https://phabricator.wikimedia.org/P39003 and previous config saved to /var/cache/conftool/dbconfig/20221110-151945-marostegui.json
  • 15:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1128.eqiad.wmnet with reason: Maintenance
  • 15:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1128.eqiad.wmnet with reason: Maintenance
  • 15:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T321123)', diff saved to https://phabricator.wikimedia.org/P39002 and previous config saved to /var/cache/conftool/dbconfig/20221110-151924-marostegui.json
  • 15:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P39001 and previous config saved to /var/cache/conftool/dbconfig/20221110-151733-ladsgroup.json
  • 15:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 15:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P39000 and previous config saved to /var/cache/conftool/dbconfig/20221110-150524-marostegui.json
  • 15:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P38999 and previous config saved to /var/cache/conftool/dbconfig/20221110-150417-marostegui.json
  • 15:03 kharlan@deploy1002: Finished scap: Backport for refreshUserImpactData: Add option to use job queue (T322706), refreshUserImpactData: Add feature flag (T313395) (duration: 04m 47s)
  • 15:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38998 and previous config saved to /var/cache/conftool/dbconfig/20221110-150226-ladsgroup.json
  • 15:02 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:01 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:01 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38997 and previous config saved to /var/cache/conftool/dbconfig/20221110-150015-ladsgroup.json
  • 15:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 14:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38996 and previous config saved to /var/cache/conftool/dbconfig/20221110-145953-ladsgroup.json
  • 14:58 kharlan@deploy1002: kharlan and kharlan: Backport for refreshUserImpactData: Add option to use job queue (T322706), refreshUserImpactData: Add feature flag (T313395) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 14:58 kharlan@deploy1002: Started scap: Backport for refreshUserImpactData: Add option to use job queue (T322706), refreshUserImpactData: Add feature flag (T313395)
  • 14:51 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-codfw
  • 14:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T321130)', diff saved to https://phabricator.wikimedia.org/P38995 and previous config saved to /var/cache/conftool/dbconfig/20221110-145018-marostegui.json
  • 14:49 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp6016.drmrs.wmnet
  • 14:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P38994 and previous config saved to /var/cache/conftool/dbconfig/20221110-144911-marostegui.json
  • 14:46 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet
  • 14:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2111 (T321130)', diff saved to https://phabricator.wikimedia.org/P38993 and previous config saved to /var/cache/conftool/dbconfig/20221110-144602-marostegui.json
  • 14:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 14:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 14:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P38992 and previous config saved to /var/cache/conftool/dbconfig/20221110-144447-ladsgroup.json
  • 14:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 14:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 14:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 14:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 14:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T321130)', diff saved to https://phabricator.wikimedia.org/P38991 and previous config saved to /var/cache/conftool/dbconfig/20221110-144121-marostegui.json
  • 14:39 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:38 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:38 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:38 kharlan@deploy1002: backport aborted: (duration: 02m 16s)
  • 14:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:35 kharlan@deploy1002: Finished scap: Backport for GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled (T313395) (duration: 04m 57s)
  • 14:34 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 14:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38990 and previous config saved to /var/cache/conftool/dbconfig/20221110-143404-marostegui.json
  • 14:33 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 14:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38989 and previous config saved to /var/cache/conftool/dbconfig/20221110-143256-marostegui.json
  • 14:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1119.eqiad.wmnet with reason: Maintenance
  • 14:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1119.eqiad.wmnet with reason: Maintenance
  • 14:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 (T321123)', diff saved to https://phabricator.wikimedia.org/P38988 and previous config saved to /var/cache/conftool/dbconfig/20221110-143235-marostegui.json
  • 14:31 kharlan@deploy1002: kharlan and kharlan: Backport for GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled (T313395) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 14:30 kharlan@deploy1002: Started scap: Backport for GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled (T313395)
  • 14:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P38987 and previous config saved to /var/cache/conftool/dbconfig/20221110-142938-ladsgroup.json
  • 14:29 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 14:28 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 14:28 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
  • 14:27 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
  • 14:27 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
  • 14:26 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 14:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P38986 and previous config saved to /var/cache/conftool/dbconfig/20221110-142614-marostegui.json
  • 14:25 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
  • 14:24 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 14:24 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 14:22 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 14:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:21 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:21 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:21 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 14:19 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 14:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P38985 and previous config saved to /var/cache/conftool/dbconfig/20221110-141728-marostegui.json
  • 14:14 daniel@deploy1002: Finished scap: Backport for mediawiki.org: set VE to new direct mode (duration: 08m 17s)
  • 14:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38984 and previous config saved to /var/cache/conftool/dbconfig/20221110-141431-ladsgroup.json
  • 14:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38983 and previous config saved to /var/cache/conftool/dbconfig/20221110-141220-ladsgroup.json
  • 14:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 14:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 14:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 14:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 14:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T322618)', diff saved to https://phabricator.wikimedia.org/P38982 and previous config saved to /var/cache/conftool/dbconfig/20221110-141141-ladsgroup.json
  • 14:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P38981 and previous config saved to /var/cache/conftool/dbconfig/20221110-141106-marostegui.json
  • 14:06 daniel@deploy1002: daniel and daniel: Backport for mediawiki.org: set VE to new direct mode synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 14:06 daniel@deploy1002: Started scap: Backport for mediawiki.org: set VE to new direct mode
  • 14:04 moritzm: rolling restart of FPM and Apache on mw canaries to pick up expat security update
  • 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P38980 and previous config saved to /var/cache/conftool/dbconfig/20221110-140222-marostegui.json
  • 14:00 moritzm: drain ganeti1020 for eventual reimage to bullseye T311687
  • 13:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P38979 and previous config saved to /var/cache/conftool/dbconfig/20221110-135635-ladsgroup.json
  • 13:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T321130)', diff saved to https://phabricator.wikimedia.org/P38978 and previous config saved to /var/cache/conftool/dbconfig/20221110-135600-marostegui.json
  • 13:53 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D
  • 13:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1200 (T321130)', diff saved to https://phabricator.wikimedia.org/P38977 and previous config saved to /var/cache/conftool/dbconfig/20221110-135334-marostegui.json
  • 13:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 13:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 13:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T321130)', diff saved to https://phabricator.wikimedia.org/P38976 and previous config saved to /var/cache/conftool/dbconfig/20221110-135313-marostegui.json
  • 13:53 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 13:52 moritzm: installing expat securiy updates
  • 13:50 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 13:48 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 (T321123)', diff saved to https://phabricator.wikimedia.org/P38975 and previous config saved to /var/cache/conftool/dbconfig/20221110-134715-marostegui.json
  • 13:46 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 13:46 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1118 (T321123)', diff saved to https://phabricator.wikimedia.org/P38974 and previous config saved to /var/cache/conftool/dbconfig/20221110-134608-marostegui.json
  • 13:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 13:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 13:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T321123)', diff saved to https://phabricator.wikimedia.org/P38973 and previous config saved to /var/cache/conftool/dbconfig/20221110-134546-marostegui.json
  • 13:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P38972 and previous config saved to /var/cache/conftool/dbconfig/20221110-134128-ladsgroup.json
  • 13:40 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D
  • 13:39 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 13:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38971 and previous config saved to /var/cache/conftool/dbconfig/20221110-133806-marostegui.json
  • 13:37 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 13:37 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 13:36 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 13:36 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 13:35 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 13:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P38970 and previous config saved to /var/cache/conftool/dbconfig/20221110-133040-marostegui.json
  • 13:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T322618)', diff saved to https://phabricator.wikimedia.org/P38969 and previous config saved to /var/cache/conftool/dbconfig/20221110-132622-ladsgroup.json
  • 13:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38968 and previous config saved to /var/cache/conftool/dbconfig/20221110-132300-marostegui.json
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T322618)', diff saved to https://phabricator.wikimedia.org/P38967 and previous config saved to /var/cache/conftool/dbconfig/20221110-132010-ladsgroup.json
  • 13:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 13:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T322618)', diff saved to https://phabricator.wikimedia.org/P38966 and previous config saved to /var/cache/conftool/dbconfig/20221110-131949-ladsgroup.json
  • 13:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P38965 and previous config saved to /var/cache/conftool/dbconfig/20221110-131533-marostegui.json
  • 13:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T321130)', diff saved to https://phabricator.wikimedia.org/P38964 and previous config saved to /var/cache/conftool/dbconfig/20221110-130753-marostegui.json
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1185 (T321130)', diff saved to https://phabricator.wikimedia.org/P38963 and previous config saved to /var/cache/conftool/dbconfig/20221110-130527-marostegui.json
  • 13:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 13:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T321130)', diff saved to https://phabricator.wikimedia.org/P38962 and previous config saved to /var/cache/conftool/dbconfig/20221110-130506-marostegui.json
  • 13:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38961 and previous config saved to /var/cache/conftool/dbconfig/20221110-130443-ladsgroup.json
  • 13:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D
  • 13:04 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D
  • 13:01 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet
  • 13:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T321123)', diff saved to https://phabricator.wikimedia.org/P38960 and previous config saved to /var/cache/conftool/dbconfig/20221110-130027-marostegui.json
  • 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1107 (T321123)', diff saved to https://phabricator.wikimedia.org/P38959 and previous config saved to /var/cache/conftool/dbconfig/20221110-125919-marostegui.json
  • 12:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38958 and previous config saved to /var/cache/conftool/dbconfig/20221110-125858-marostegui.json
  • 12:53 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet
  • 12:51 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 12:51 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 12:51 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:51 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38957 and previous config saved to /var/cache/conftool/dbconfig/20221110-124959-marostegui.json
  • 12:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38956 and previous config saved to /var/cache/conftool/dbconfig/20221110-124936-ladsgroup.json
  • 12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2119 (T318605)', diff saved to https://phabricator.wikimedia.org/P38955 and previous config saved to /var/cache/conftool/dbconfig/20221110-124805-ladsgroup.json
  • 12:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 12:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T318605)', diff saved to https://phabricator.wikimedia.org/P38954 and previous config saved to /var/cache/conftool/dbconfig/20221110-124743-ladsgroup.json
  • 12:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P38953 and previous config saved to /var/cache/conftool/dbconfig/20221110-124352-marostegui.json
  • 12:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38952 and previous config saved to /var/cache/conftool/dbconfig/20221110-123527-ladsgroup.json
  • 12:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38951 and previous config saved to /var/cache/conftool/dbconfig/20221110-123453-marostegui.json
  • 12:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T322618)', diff saved to https://phabricator.wikimedia.org/P38950 and previous config saved to /var/cache/conftool/dbconfig/20221110-123428-ladsgroup.json
  • 12:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38949 and previous config saved to /var/cache/conftool/dbconfig/20221110-123237-ladsgroup.json
  • 12:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T322618)', diff saved to https://phabricator.wikimedia.org/P38948 and previous config saved to /var/cache/conftool/dbconfig/20221110-123215-ladsgroup.json
  • 12:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 12:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38947 and previous config saved to /var/cache/conftool/dbconfig/20221110-123153-ladsgroup.json
  • 12:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P38946 and previous config saved to /var/cache/conftool/dbconfig/20221110-122845-marostegui.json
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P38945 and previous config saved to /var/cache/conftool/dbconfig/20221110-122720-ladsgroup.json
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T318605)', diff saved to https://phabricator.wikimedia.org/P38944 and previous config saved to /var/cache/conftool/dbconfig/20221110-122708-ladsgroup.json
  • 12:26 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:26 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:26 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 12:26 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 12:25 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 12:25 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 12:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38943 and previous config saved to /var/cache/conftool/dbconfig/20221110-122020-ladsgroup.json
  • 12:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T321130)', diff saved to https://phabricator.wikimedia.org/P38942 and previous config saved to /var/cache/conftool/dbconfig/20221110-121946-marostegui.json
  • 12:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38941 and previous config saved to /var/cache/conftool/dbconfig/20221110-121730-ladsgroup.json
  • 12:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P38940 and previous config saved to /var/cache/conftool/dbconfig/20221110-121647-ladsgroup.json
  • 12:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T321130)', diff saved to https://phabricator.wikimedia.org/P38939 and previous config saved to /var/cache/conftool/dbconfig/20221110-121601-marostegui.json
  • 12:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 12:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38938 and previous config saved to /var/cache/conftool/dbconfig/20221110-121339-marostegui.json
  • 12:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 12:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 12:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38937 and previous config saved to /var/cache/conftool/dbconfig/20221110-121313-marostegui.json
  • 12:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P38936 and previous config saved to /var/cache/conftool/dbconfig/20221110-121202-ladsgroup.json
  • 12:08 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry
  • 12:06 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry
  • 12:06 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:06 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 100%: Index rebuilt', diff saved to https://phabricator.wikimedia.org/P38935 and previous config saved to /var/cache/conftool/dbconfig/20221110-120537-ladsgroup.json
  • 12:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38934 and previous config saved to /var/cache/conftool/dbconfig/20221110-120513-ladsgroup.json
  • 12:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T318605)', diff saved to https://phabricator.wikimedia.org/P38933 and previous config saved to /var/cache/conftool/dbconfig/20221110-120224-ladsgroup.json
  • 12:02 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:02 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:01 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:01 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P38932 and previous config saved to /var/cache/conftool/dbconfig/20221110-120140-ladsgroup.json
  • 11:59 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 11:59 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 11:58 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 11:58 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 11:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P38931 and previous config saved to /var/cache/conftool/dbconfig/20221110-115807-marostegui.json
  • 11:57 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 11:57 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P38930 and previous config saved to /var/cache/conftool/dbconfig/20221110-115655-ladsgroup.json
  • 11:55 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 11:52 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 75%: Index rebuilt', diff saved to https://phabricator.wikimedia.org/P38929 and previous config saved to /var/cache/conftool/dbconfig/20221110-115032-ladsgroup.json
  • 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38928 and previous config saved to /var/cache/conftool/dbconfig/20221110-115007-ladsgroup.json
  • 11:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38927 and previous config saved to /var/cache/conftool/dbconfig/20221110-114634-ladsgroup.json
  • 11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38926 and previous config saved to /var/cache/conftool/dbconfig/20221110-114422-ladsgroup.json
  • 11:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 11:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38925 and previous config saved to /var/cache/conftool/dbconfig/20221110-114400-ladsgroup.json
  • 11:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P38924 and previous config saved to /var/cache/conftool/dbconfig/20221110-114300-marostegui.json
  • 11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T318605)', diff saved to https://phabricator.wikimedia.org/P38923 and previous config saved to /var/cache/conftool/dbconfig/20221110-114149-ladsgroup.json
  • 11:41 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wdqs-all
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38922 and previous config saved to /var/cache/conftool/dbconfig/20221110-113958-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38921 and previous config saved to /var/cache/conftool/dbconfig/20221110-113948-ladsgroup.json
  • 11:38 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38920 and previous config saved to /var/cache/conftool/dbconfig/20221110-113853-root.json
  • 11:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 25%: Index rebuilt', diff saved to https://phabricator.wikimedia.org/P38919 and previous config saved to /var/cache/conftool/dbconfig/20221110-113526-ladsgroup.json
  • 11:31 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wdqs-all
  • 11:31 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wcqs-public
  • 11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P38918 and previous config saved to /var/cache/conftool/dbconfig/20221110-112854-ladsgroup.json
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38917 and previous config saved to /var/cache/conftool/dbconfig/20221110-112753-marostegui.json
  • 11:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38916 and previous config saved to /var/cache/conftool/dbconfig/20221110-112441-ladsgroup.json
  • 11:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38915 and previous config saved to /var/cache/conftool/dbconfig/20221110-112403-marostegui.json
  • 11:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38914 and previous config saved to /var/cache/conftool/dbconfig/20221110-112348-root.json
  • 11:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38913 and previous config saved to /var/cache/conftool/dbconfig/20221110-112342-marostegui.json
  • 11:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 10%: Index rebuilt', diff saved to https://phabricator.wikimedia.org/P38912 and previous config saved to /var/cache/conftool/dbconfig/20221110-112022-ladsgroup.json
  • 11:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P38911 and previous config saved to /var/cache/conftool/dbconfig/20221110-111347-ladsgroup.json
  • 11:13 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wcqs-public
  • 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38910 and previous config saved to /var/cache/conftool/dbconfig/20221110-111323-marostegui.json
  • 11:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 11:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 11:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38909 and previous config saved to /var/cache/conftool/dbconfig/20221110-111244-marostegui.json
  • 11:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38908 and previous config saved to /var/cache/conftool/dbconfig/20221110-110935-ladsgroup.json
  • 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38907 and previous config saved to /var/cache/conftool/dbconfig/20221110-110843-root.json
  • 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P38906 and previous config saved to /var/cache/conftool/dbconfig/20221110-110835-marostegui.json
  • 11:00 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38905 and previous config saved to /var/cache/conftool/dbconfig/20221110-105841-ladsgroup.json
  • 10:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P38904 and previous config saved to /var/cache/conftool/dbconfig/20221110-105738-marostegui.json
  • 10:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38903 and previous config saved to /var/cache/conftool/dbconfig/20221110-105628-ladsgroup.json
  • 10:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 10:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 10:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38902 and previous config saved to /var/cache/conftool/dbconfig/20221110-105428-ladsgroup.json
  • 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38901 and previous config saved to /var/cache/conftool/dbconfig/20221110-105338-root.json
  • 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P38900 and previous config saved to /var/cache/conftool/dbconfig/20221110-105329-marostegui.json
  • 10:53 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 10:50 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38899 and previous config saved to /var/cache/conftool/dbconfig/20221110-104919-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 10:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 10:47 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 10:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P38898 and previous config saved to /var/cache/conftool/dbconfig/20221110-104231-marostegui.json
  • 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38897 and previous config saved to /var/cache/conftool/dbconfig/20221110-103827-root.json
  • 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38896 and previous config saved to /var/cache/conftool/dbconfig/20221110-103822-marostegui.json
  • 10:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38895 and previous config saved to /var/cache/conftool/dbconfig/20221110-103533-marostegui.json
  • 10:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T321130)', diff saved to https://phabricator.wikimedia.org/P38894 and previous config saved to /var/cache/conftool/dbconfig/20221110-103512-marostegui.json
  • 10:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38893 and previous config saved to /var/cache/conftool/dbconfig/20221110-102725-marostegui.json
  • 10:26 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38892 and previous config saved to /var/cache/conftool/dbconfig/20221110-102617-marostegui.json
  • 10:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 10:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 10:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38891 and previous config saved to /var/cache/conftool/dbconfig/20221110-102556-marostegui.json
  • 10:25 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1013.eqiad.wmnet to cluster eqiad and group B
  • 10:24 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1013.eqiad.wmnet to cluster eqiad and group B
  • 10:23 moritzm: installing libxml2 security updates
  • 10:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1013.eqiad.wmnet
  • 10:23 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 10:22 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 10:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P38890 and previous config saved to /var/cache/conftool/dbconfig/20221110-102005-marostegui.json
  • 10:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1013.eqiad.wmnet
  • 10:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P38889 and previous config saved to /var/cache/conftool/dbconfig/20221110-101050-marostegui.json
  • 10:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P38888 and previous config saved to /var/cache/conftool/dbconfig/20221110-100459-marostegui.json
  • 09:57 marostegui@cumin1001: dbctl commit (dc=all): 'Reduce es4 master weight', diff saved to https://phabricator.wikimedia.org/P38887 and previous config saved to /var/cache/conftool/dbconfig/20221110-095724-marostegui.json
  • 09:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P38886 and previous config saved to /var/cache/conftool/dbconfig/20221110-095543-marostegui.json
  • 09:53 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38885 and previous config saved to /var/cache/conftool/dbconfig/20221110-095313-root.json
  • 09:52 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 09:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T321130)', diff saved to https://phabricator.wikimedia.org/P38884 and previous config saved to /var/cache/conftool/dbconfig/20221110-094952-marostegui.json
  • 09:49 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 09:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T321130)', diff saved to https://phabricator.wikimedia.org/P38883 and previous config saved to /var/cache/conftool/dbconfig/20221110-094604-marostegui.json
  • 09:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1110.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1110.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100 (T321130)', diff saved to https://phabricator.wikimedia.org/P38882 and previous config saved to /var/cache/conftool/dbconfig/20221110-094542-marostegui.json
  • 09:44 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 09:43 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 09:43 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 09:42 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 09:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38881 and previous config saved to /var/cache/conftool/dbconfig/20221110-094037-marostegui.json
  • 09:39 marostegui@deploy1002: Finished scap: Backport for Revert "db-production.php: Disable es5 writes" (duration: 04m 20s)
  • 09:39 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 09:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38880 and previous config saved to /var/cache/conftool/dbconfig/20221110-093929-marostegui.json
  • 09:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 09:36 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-eqiad
  • 09:35 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "db-production.php: Disable es5 writes" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 09:35 marostegui@deploy1002: Started scap: Backport for Revert "db-production.php: Disable es5 writes"
  • 09:34 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 09:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es1023 T322187', diff saved to https://phabricator.wikimedia.org/P38879 and previous config saved to /var/cache/conftool/dbconfig/20221110-093354-root.json
  • 09:32 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es1024 to es5 primary T322187', diff saved to https://phabricator.wikimedia.org/P38878 and previous config saved to /var/cache/conftool/dbconfig/20221110-093243-root.json
  • 09:32 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 09:32 marostegui: Starting es5 eqiad failover from es1023 to es1024 T322187
  • 09:31 marostegui@deploy1002: Finished scap: Backport for db-production.php: Disable es5 writes (T322187) (duration: 04m 39s)
  • 09:31 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 09:31 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 09:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P38877 and previous config saved to /var/cache/conftool/dbconfig/20221110-093036-marostegui.json
  • 09:30 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 09:27 marostegui@deploy1002: marostegui and marostegui: Backport for db-production.php: Disable es5 writes (T322187) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 09:27 marostegui@deploy1002: Started scap: Backport for db-production.php: Disable es5 writes (T322187)
  • 09:23 marostegui@cumin1001: dbctl commit (dc=all): 'Set es1024 with weight 0 T322187', diff saved to https://phabricator.wikimedia.org/P38876 and previous config saved to /var/cache/conftool/dbconfig/20221110-092336-root.json
  • 09:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322187
  • 09:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322187
  • 09:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38875 and previous config saved to /var/cache/conftool/dbconfig/20221110-092215-marostegui.json
  • 09:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1013.eqiad.wmnet with OS bullseye
  • 09:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P38874 and previous config saved to /var/cache/conftool/dbconfig/20221110-091530-marostegui.json
  • 09:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38873 and previous config saved to /var/cache/conftool/dbconfig/20221110-090708-marostegui.json
  • 09:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1013.eqiad.wmnet with reason: host reimage
  • 09:00 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1013.eqiad.wmnet with reason: host reimage
  • 09:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100 (T321130)', diff saved to https://phabricator.wikimedia.org/P38872 and previous config saved to /var/cache/conftool/dbconfig/20221110-090023-marostegui.json
  • 08:58 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-codfw
  • 08:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1100 (T321130)', diff saved to https://phabricator.wikimedia.org/P38871 and previous config saved to /var/cache/conftool/dbconfig/20221110-085756-marostegui.json
  • 08:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 08:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 08:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38870 and previous config saved to /var/cache/conftool/dbconfig/20221110-085735-marostegui.json
  • 08:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38869 and previous config saved to /var/cache/conftool/dbconfig/20221110-085201-marostegui.json
  • 08:46 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1013.eqiad.wmnet with OS bullseye
  • 08:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P38868 and previous config saved to /var/cache/conftool/dbconfig/20221110-084229-marostegui.json
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38867 and previous config saved to /var/cache/conftool/dbconfig/20221110-083655-marostegui.json
  • 08:30 hashar@deploy1002: Finished deploy [gerrit/gerrit@84648b3]: Gerrit to 3.4.8 on gerrit1001 # T322724 (duration: 00m 08s)
  • 08:30 hashar@deploy1002: Started deploy [gerrit/gerrit@84648b3]: Gerrit to 3.4.8 on gerrit1001 # T322724
  • 08:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P38866 and previous config saved to /var/cache/conftool/dbconfig/20221110-082722-marostegui.json
  • 08:26 hashar@deploy1002: Finished deploy [gerrit/gerrit@84648b3]: Gerrit to 3.4.8 on gerrit2002 # T322724 (duration: 00m 10s)
  • 08:26 hashar@deploy1002: Started deploy [gerrit/gerrit@84648b3]: Gerrit to 3.4.8 on gerrit2002 # T322724
  • 08:17 moritzm: installing pixman security updates on buster
  • 08:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38865 and previous config saved to /var/cache/conftool/dbconfig/20221110-081216-marostegui.json
  • 08:09 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 136933
  • 08:08 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 136933
  • 08:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38864 and previous config saved to /var/cache/conftool/dbconfig/20221110-080823-marostegui.json
  • 08:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 08:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 08:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38863 and previous config saved to /var/cache/conftool/dbconfig/20221110-080746-marostegui.json
  • 08:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 08:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 07:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1123.eqiad.wmnet with reason: Maintenance
  • 07:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1122.eqiad.wmnet with reason: Maintenance
  • 07:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1123.eqiad.wmnet with reason: Maintenance
  • 07:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1122.eqiad.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2107.codfw.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 07:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2107.codfw.wmnet with reason: Maintenance
  • 07:00 ayounsi@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 06:45 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1011.eqiad.wmnet
  • 06:41 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-tool1011.eqiad.wmnet
  • 06:40 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1096.eqiad.wmnet
  • 06:32 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1096.eqiad.wmnet
  • 06:30 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1097.eqiad.wmnet
  • 06:22 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1097.eqiad.wmnet
  • 06:21 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1098.eqiad.wmnet
  • 06:13 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1098.eqiad.wmnet
  • 04:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T318605)', diff saved to https://phabricator.wikimedia.org/P38862 and previous config saved to /var/cache/conftool/dbconfig/20221110-044050-ladsgroup.json
  • 04:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 04:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 04:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T318605)', diff saved to https://phabricator.wikimedia.org/P38861 and previous config saved to /var/cache/conftool/dbconfig/20221110-044028-ladsgroup.json
  • 04:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38860 and previous config saved to /var/cache/conftool/dbconfig/20221110-042522-ladsgroup.json
  • 04:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38859 and previous config saved to /var/cache/conftool/dbconfig/20221110-041016-ladsgroup.json
  • 03:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T318605)', diff saved to https://phabricator.wikimedia.org/P38858 and previous config saved to /var/cache/conftool/dbconfig/20221110-035509-ladsgroup.json
  • 00:03 tzatziki: removing 1 file for legal compliance

2022-11-09

  • 23:57 tzatziki: removing 1 file for legal compliance
  • 23:44 tzatziki: removing 2 files for legal compliance
  • 23:22 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1034.eqiad.wmnet with OS bullseye
  • 23:17 tzatziki: removing 1 file for legal compliance
  • 23:07 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1034.eqiad.wmnet with reason: host reimage
  • 23:04 robh@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1034.eqiad.wmnet with reason: host reimage
  • 23:03 aikochou@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 23:00 aikochou@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 22:51 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1034.eqiad.wmnet with OS bullseye
  • 22:34 damilare: civicrm upgraded from f2017495 to 07fdeed5
  • 22:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T318605)', diff saved to https://phabricator.wikimedia.org/P38857 and previous config saved to /var/cache/conftool/dbconfig/20221109-221551-ladsgroup.json
  • 22:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 22:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 22:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T318605)', diff saved to https://phabricator.wikimedia.org/P38856 and previous config saved to /var/cache/conftool/dbconfig/20221109-221529-ladsgroup.json
  • 22:06 robh@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti1034.eqiad.wmnet with OS bullseye
  • 22:01 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1034.eqiad.wmnet with OS bullseye
  • 22:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38855 and previous config saved to /var/cache/conftool/dbconfig/20221109-220023-ladsgroup.json
  • 21:55 robh@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti1034.eqiad.wmnet with OS bullseye
  • 21:48 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1034.eqiad.wmnet with OS bullseye
  • 21:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38854 and previous config saved to /var/cache/conftool/dbconfig/20221109-214516-ladsgroup.json
  • 21:37 TheresNoTime: closing UTC late backport window
  • 21:35 samtar@deploy1002: Finished scap: Backport for Fix TOC misaligned when max width option is disable (T322162) (duration: 08m 48s)
  • 21:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T318605)', diff saved to https://phabricator.wikimedia.org/P38853 and previous config saved to /var/cache/conftool/dbconfig/20221109-213010-ladsgroup.json
  • 21:30 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:29 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:29 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:26 samtar@deploy1002: samtar and nray: Backport for Fix TOC misaligned when max width option is disable (T322162) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 21:26 samtar@deploy1002: Started scap: Backport for Fix TOC misaligned when max width option is disable (T322162)
  • 21:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2110 (T318605)', diff saved to https://phabricator.wikimedia.org/P38852 and previous config saved to /var/cache/conftool/dbconfig/20221109-211613-ladsgroup.json
  • 21:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 21:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 21:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T318605)', diff saved to https://phabricator.wikimedia.org/P38851 and previous config saved to /var/cache/conftool/dbconfig/20221109-211551-ladsgroup.json
  • 21:15 samtar@deploy1002: Finished scap: Backport for Only Enable LBFactory config callback in CLI in production (T298485) (duration: 05m 41s)
  • 21:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:10 samtar@deploy1002: samtar and dancy: Backport for Only Enable LBFactory config callback in CLI in production (T298485) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 21:09 samtar@deploy1002: Started scap: Backport for Only Enable LBFactory config callback in CLI in production (T298485)
  • 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:09 root@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 21:08 samtar@deploy1002: Finished scap: Backport for Add no=>nb to wgInterlanguageLinkCodeMap for some multilingual wikis (T322696) (duration: 06m 06s)
  • 21:02 samtar@deploy1002: samtar and jhsoby: Backport for Add no=>nb to wgInterlanguageLinkCodeMap for some multilingual wikis (T322696) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 21:02 samtar@deploy1002: Started scap: Backport for Add no=>nb to wgInterlanguageLinkCodeMap for some multilingual wikis (T322696)
  • 21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P38850 and previous config saved to /var/cache/conftool/dbconfig/20221109-210044-ladsgroup.json
  • 20:45 root@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1023.eqiad.wmnet with reason: host reimage
  • 20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P38849 and previous config saved to /var/cache/conftool/dbconfig/20221109-204538-ladsgroup.json
  • 20:42 root@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1023.eqiad.wmnet with reason: host reimage
  • 20:41 robh@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:35 mutante: gerrit1001 (gerrit) - restarting gerrit service to disable aggressive garbage collection. gerrit:854514 - T237807
  • 20:30 mutante: gerrit2002 (gerrit-replica) - restarting gerrit service
  • 20:30 root@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 20:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T318605)', diff saved to https://phabricator.wikimedia.org/P38848 and previous config saved to /var/cache/conftool/dbconfig/20221109-203031-ladsgroup.json
  • 20:26 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:26 robh@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1034.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:23 root@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 20:12 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1034.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:05 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:04 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:01 root@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 20:01 root@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 20:01 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1034.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:59 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1034.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:52 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:50 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:47 robh@cumin1001: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1034
  • 19:47 robh@cumin1001: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1034
  • 19:47 robh@cumin1001: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1033
  • 19:47 robh@cumin1001: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1033
  • 19:37 root@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:35 root@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:35 root@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1013.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 19:13 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1013.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 19:07 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:05 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:01 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:00 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:49 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:49 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:41 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:40 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:35 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:33 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:20 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:19 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 17:57 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wcqs1003.eqiad.wmnet with reason: data reload
  • 17:55 bking@cumin2002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on wcqs1003.eqiad.wmnet with reason: data reload
  • 17:45 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 17:40 volans@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Converted existing STAGED hosts to ACTIVE - volans@cumin1001 - T320696"
  • 17:37 volans@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Converted existing STAGED hosts to ACTIVE - volans@cumin1001 - T320696"
  • 17:29 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:28 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 17:17 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:16 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 17:09 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:09 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:55 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 8218
  • 16:52 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 8218
  • 16:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T322618)', diff saved to https://phabricator.wikimedia.org/P38843 and previous config saved to /var/cache/conftool/dbconfig/20221109-162849-ladsgroup.json
  • 16:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38842 and previous config saved to /var/cache/conftool/dbconfig/20221109-161343-ladsgroup.json
  • 15:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38841 and previous config saved to /var/cache/conftool/dbconfig/20221109-155836-ladsgroup.json
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T318605)', diff saved to https://phabricator.wikimedia.org/P38840 and previous config saved to /var/cache/conftool/dbconfig/20221109-154933-ladsgroup.json
  • 15:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 15:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 15:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38839 and previous config saved to /var/cache/conftool/dbconfig/20221109-154911-ladsgroup.json
  • 15:47 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet
  • 15:47 aikochou@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T322618)', diff saved to https://phabricator.wikimedia.org/P38838 and previous config saved to /var/cache/conftool/dbconfig/20221109-154330-ladsgroup.json
  • 15:42 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet
  • 15:42 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1099.eqiad.wmnet
  • 15:42 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-stretch2002.codfw.wmnet
  • 15:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T322618)', diff saved to https://phabricator.wikimedia.org/P38837 and previous config saved to /var/cache/conftool/dbconfig/20221109-153922-ladsgroup.json
  • 15:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 15:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 15:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T322618)', diff saved to https://phabricator.wikimedia.org/P38836 and previous config saved to /var/cache/conftool/dbconfig/20221109-153901-ladsgroup.json
  • 15:34 aikochou@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38835 and previous config saved to /var/cache/conftool/dbconfig/20221109-153405-ladsgroup.json
  • 15:33 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host kafka-stretch2002.codfw.wmnet
  • 15:32 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1099.eqiad.wmnet
  • 15:31 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1100.eqiad.wmnet
  • 15:24 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1100.eqiad.wmnet
  • 15:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38834 and previous config saved to /var/cache/conftool/dbconfig/20221109-152354-ladsgroup.json
  • 15:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38833 and previous config saved to /var/cache/conftool/dbconfig/20221109-151858-ladsgroup.json
  • 15:18 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-stretch2001.codfw.wmnet
  • 15:12 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1101.eqiad.wmnet
  • 15:11 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host kafka-stretch2001.codfw.wmnet
  • 15:10 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host kafka-stretch2001.codfw.wmnet
  • 15:10 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host kafka-stretch2001.codfw.wmnet
  • 15:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38832 and previous config saved to /var/cache/conftool/dbconfig/20221109-150848-ladsgroup.json
  • 15:04 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet
  • 15:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38831 and previous config saved to /var/cache/conftool/dbconfig/20221109-150351-ladsgroup.json
  • 15:03 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1101.eqiad.wmnet
  • 15:02 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host an-worker1101.eqiad.wmnet
  • 15:02 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1101.eqiad.wmnet
  • 15:02 moritzm: installing pixman security updates on buster
  • 14:57 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T322618)', diff saved to https://phabricator.wikimedia.org/P38830 and previous config saved to /var/cache/conftool/dbconfig/20221109-145341-ladsgroup.json
  • 14:50 moritzm: rolling restart of mw canaries to pick up libxml security update
  • 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T322618)', diff saved to https://phabricator.wikimedia.org/P38829 and previous config saved to /var/cache/conftool/dbconfig/20221109-144933-ladsgroup.json
  • 14:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 14:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T322618)', diff saved to https://phabricator.wikimedia.org/P38828 and previous config saved to /var/cache/conftool/dbconfig/20221109-144912-ladsgroup.json
  • 14:46 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=frwiki` in a tmux at mwmaint1002 (locally applied shorter MentorStore::INACTIVITY_THRESHOLD; T318457)
  • 14:43 sukhe: reprepro remove bullseye-wikimedia trafficserver: T321309
  • 14:40 moritzm: installing libxml2 security updates
  • 14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38827 and previous config saved to /var/cache/conftool/dbconfig/20221109-143404-ladsgroup.json
  • 14:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2106 (T318605)', diff saved to https://phabricator.wikimedia.org/P38826 and previous config saved to /var/cache/conftool/dbconfig/20221109-143050-ladsgroup.json
  • 14:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 14:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 14:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38825 and previous config saved to /var/cache/conftool/dbconfig/20221109-141858-ladsgroup.json
  • 14:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:12 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:12 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:11 Lucas_WMDE: UTC afternoon backport+config window done
  • 14:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:09 lucaswerkmeister-wmde@deploy1002: Finished scap: Backport for Enable show nearby feature for ruwiki (T321548) (duration: 05m 42s)
  • 14:04 lucaswerkmeister-wmde@deploy1002: lucaswerkmeister-wmde and lilients: Backport for Enable show nearby feature for ruwiki (T321548) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 14:04 lucaswerkmeister-wmde@deploy1002: Started scap: Backport for Enable show nearby feature for ruwiki (T321548)
  • 14:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T322618)', diff saved to https://phabricator.wikimedia.org/P38824 and previous config saved to /var/cache/conftool/dbconfig/20221109-140351-ladsgroup.json
  • 13:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T322618)', diff saved to https://phabricator.wikimedia.org/P38823 and previous config saved to /var/cache/conftool/dbconfig/20221109-135943-ladsgroup.json
  • 13:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 13:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 13:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T322618)', diff saved to https://phabricator.wikimedia.org/P38822 and previous config saved to /var/cache/conftool/dbconfig/20221109-135911-ladsgroup.json
  • 13:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38821 and previous config saved to /var/cache/conftool/dbconfig/20221109-134404-ladsgroup.json
  • 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38820 and previous config saved to /var/cache/conftool/dbconfig/20221109-132858-ladsgroup.json
  • 13:24 moritzm: drain ganeti1013 for eventual reimage to bullseye T311687
  • 13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T322618)', diff saved to https://phabricator.wikimedia.org/P38818 and previous config saved to /var/cache/conftool/dbconfig/20221109-131903-ladsgroup.json
  • 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf1004.eqiad.wmnet
  • 13:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T322618)', diff saved to https://phabricator.wikimedia.org/P38817 and previous config saved to /var/cache/conftool/dbconfig/20221109-131351-ladsgroup.json
  • 13:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf1004.eqiad.wmnet
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T322618)', diff saved to https://phabricator.wikimedia.org/P38816 and previous config saved to /var/cache/conftool/dbconfig/20221109-130944-ladsgroup.json
  • 13:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 13:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38815 and previous config saved to /var/cache/conftool/dbconfig/20221109-130923-ladsgroup.json
  • 13:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38814 and previous config saved to /var/cache/conftool/dbconfig/20221109-130357-ladsgroup.json
  • 12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38813 and previous config saved to /var/cache/conftool/dbconfig/20221109-125416-ladsgroup.json
  • 12:54 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 63199
  • 12:51 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply
  • 12:50 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 63199
  • 12:50 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-main: apply
  • 12:49 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 29169
  • 12:48 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 29169
  • 12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38812 and previous config saved to /var/cache/conftool/dbconfig/20221109-124850-ladsgroup.json
  • 12:43 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply
  • 12:42 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-main: apply
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38811 and previous config saved to /var/cache/conftool/dbconfig/20221109-123910-ladsgroup.json
  • 12:33 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
  • 12:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T322618)', diff saved to https://phabricator.wikimedia.org/P38810 and previous config saved to /var/cache/conftool/dbconfig/20221109-123344-ladsgroup.json
  • 12:33 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
  • 12:31 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:30 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:28 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:28 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:26 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
  • 12:26 hashar@deploy1002: Finished deploy [gerrit/gerrit@b83625a]: gerrit1001: Gerrit JavaScript plugins as standalone files # T319378 (duration: 00m 09s)
  • 12:26 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
  • 12:25 hashar@deploy1002: Started deploy [gerrit/gerrit@b83625a]: gerrit1001: Gerrit JavaScript plugins as standalone files # T319378
  • 12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T322618)', diff saved to https://phabricator.wikimedia.org/P38809 and previous config saved to /var/cache/conftool/dbconfig/20221109-122528-ladsgroup.json
  • 12:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 12:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T322618)', diff saved to https://phabricator.wikimedia.org/P38808 and previous config saved to /var/cache/conftool/dbconfig/20221109-122507-ladsgroup.json
  • 12:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38807 and previous config saved to /var/cache/conftool/dbconfig/20221109-122403-ladsgroup.json
  • 12:18 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply
  • 12:17 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply
  • 12:16 hashar@deploy1002: Finished deploy [gerrit/gerrit@b83625a]: gerrit2002: Gerrit JavaScript plugins as standalone files # T319378 (duration: 00m 10s)
  • 12:16 hashar@deploy1002: Started deploy [gerrit/gerrit@b83625a]: gerrit2002: Gerrit JavaScript plugins as standalone files # T319378
  • 12:11 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-logging-external: apply
  • 12:10 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-logging-external: apply
  • 12:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38806 and previous config saved to /var/cache/conftool/dbconfig/20221109-121001-ladsgroup.json
  • 12:03 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply
  • 12:03 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply
  • 11:56 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply
  • 11:56 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply
  • 11:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38805 and previous config saved to /var/cache/conftool/dbconfig/20221109-115454-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T322618)', diff saved to https://phabricator.wikimedia.org/P38804 and previous config saved to /var/cache/conftool/dbconfig/20221109-113948-ladsgroup.json
  • 11:38 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host datahubsearch1003.eqiad.wmnet
  • 11:36 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 11:34 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host datahubsearch1003.eqiad.wmnet
  • 11:33 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-eqiad
  • 11:32 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd1001.eqiad.wmnet
  • 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T322618)', diff saved to https://phabricator.wikimedia.org/P38803 and previous config saved to /var/cache/conftool/dbconfig/20221109-113144-ladsgroup.json
  • 11:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 11:31 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 11:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 11:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 11:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P38802 and previous config saved to /var/cache/conftool/dbconfig/20221109-113108-ladsgroup.json
  • 11:28 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1001.eqiad.wmnet
  • 11:28 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host dse-k8s-etcd1001.eqiad.wmnet
  • 11:28 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1001.eqiad.wmnet
  • 11:25 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host datahubsearch1002.eqiad.wmnet
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38801 and previous config saved to /var/cache/conftool/dbconfig/20221109-112347-ladsgroup.json
  • 11:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38800 and previous config saved to /var/cache/conftool/dbconfig/20221109-112326-ladsgroup.json
  • 11:21 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host datahubsearch1002.eqiad.wmnet
  • 11:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38799 and previous config saved to /var/cache/conftool/dbconfig/20221109-111601-ladsgroup.json
  • 11:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38798 and previous config saved to /var/cache/conftool/dbconfig/20221109-110819-ladsgroup.json
  • 11:05 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-codfw
  • 11:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38797 and previous config saved to /var/cache/conftool/dbconfig/20221109-110055-ladsgroup.json
  • 10:59 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1024.eqiad.wmnet to cluster eqiad and group C
  • 10:58 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1024.eqiad.wmnet to cluster eqiad and group C
  • 10:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38796 and previous config saved to /var/cache/conftool/dbconfig/20221109-105313-ladsgroup.json
  • 10:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P38794 and previous config saved to /var/cache/conftool/dbconfig/20221109-104548-ladsgroup.json
  • 10:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38793 and previous config saved to /var/cache/conftool/dbconfig/20221109-103806-ladsgroup.json
  • 10:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P38792 and previous config saved to /var/cache/conftool/dbconfig/20221109-103722-ladsgroup.json
  • 10:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 10:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 10:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 10:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 10:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T322618)', diff saved to https://phabricator.wikimedia.org/P38791 and previous config saved to /var/cache/conftool/dbconfig/20221109-103026-ladsgroup.json
  • 10:22 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host datahubsearch1001.eqiad.wmnet
  • 10:17 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1018.eqiad.wmnet to cluster eqiad and group B
  • 10:17 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host datahubsearch1001.eqiad.wmnet
  • 10:16 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1018.eqiad.wmnet to cluster eqiad and group B
  • 10:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38788 and previous config saved to /var/cache/conftool/dbconfig/20221109-101519-ladsgroup.json
  • 10:02 volans: set Netbox status to Active for 299 devices with role=server, tenant=none, status=staged - T320696
  • 10:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38787 and previous config saved to /var/cache/conftool/dbconfig/20221109-100013-ladsgroup.json
  • 09:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1024.eqiad.wmnet
  • 09:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1024.eqiad.wmnet
  • 09:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T322618)', diff saved to https://phabricator.wikimedia.org/P38786 and previous config saved to /var/cache/conftool/dbconfig/20221109-094506-ladsgroup.json
  • 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1018.eqiad.wmnet
  • 09:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38785 and previous config saved to /var/cache/conftool/dbconfig/20221109-093751-ladsgroup.json
  • 09:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 09:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T322618)', diff saved to https://phabricator.wikimedia.org/P38784 and previous config saved to /var/cache/conftool/dbconfig/20221109-093650-ladsgroup.json
  • 09:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T322618)', diff saved to https://phabricator.wikimedia.org/P38783 and previous config saved to /var/cache/conftool/dbconfig/20221109-093629-ladsgroup.json
  • 09:35 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1018.eqiad.wmnet
  • 09:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T322618)', diff saved to https://phabricator.wikimedia.org/P38782 and previous config saved to /var/cache/conftool/dbconfig/20221109-093454-ladsgroup.json
  • 09:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38781 and previous config saved to /var/cache/conftool/dbconfig/20221109-092122-ladsgroup.json
  • 09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38780 and previous config saved to /var/cache/conftool/dbconfig/20221109-091947-ladsgroup.json
  • 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1024.eqiad.wmnet with OS bullseye
  • 09:07 moritzm: installing nodejs security updates
  • 09:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38779 and previous config saved to /var/cache/conftool/dbconfig/20221109-090616-ladsgroup.json
  • 09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38778 and previous config saved to /var/cache/conftool/dbconfig/20221109-090441-ladsgroup.json
  • 09:03 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 54994
  • 08:59 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 54994
  • 08:57 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1024.eqiad.wmnet with reason: host reimage
  • 08:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38777 and previous config saved to /var/cache/conftool/dbconfig/20221109-085542-ladsgroup.json
  • 08:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 08:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 08:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 08:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 08:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 08:54 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1024.eqiad.wmnet with reason: host reimage
  • 08:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 08:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T322618)', diff saved to https://phabricator.wikimedia.org/P38776 and previous config saved to /var/cache/conftool/dbconfig/20221109-085109-ladsgroup.json
  • 08:51 kartik@deploy1002: Finished scap: Backport for Add channel for MessageBundle feature of Translate extension (T322430) (duration: 11m 19s)
  • 08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T322618)', diff saved to https://phabricator.wikimedia.org/P38775 and previous config saved to /var/cache/conftool/dbconfig/20221109-084934-ladsgroup.json
  • 08:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T322618)', diff saved to https://phabricator.wikimedia.org/P38774 and previous config saved to /var/cache/conftool/dbconfig/20221109-084525-ladsgroup.json
  • 08:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 08:45 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 08:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 08:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 08:44 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:44 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:43 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T322618)', diff saved to https://phabricator.wikimedia.org/P38773 and previous config saved to /var/cache/conftool/dbconfig/20221109-084254-ladsgroup.json
  • 08:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 08:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 08:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1024.eqiad.wmnet with OS bullseye
  • 08:40 kartik@deploy1002: kartik and abi: Backport for Add channel for MessageBundle feature of Translate extension (T322430) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
  • 08:39 kartik@deploy1002: Started scap: Backport for Add channel for MessageBundle feature of Translate extension (T322430)
  • 08:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1018.eqiad.wmnet with OS bullseye
  • 08:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:33 kartik@deploy1002: Finished scap: Backport for Update Metrics Platform streams (T322277) (duration: 08m 17s)
  • 08:32 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:32 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:31 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:30 ayounsi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1182.eqiad.wmnet with reason: paged then depooled
  • 08:30 ayounsi@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1182.eqiad.wmnet with reason: paged then depooled
  • 08:25 kartik@deploy1002: kartik and phuedx: Backport for Update Metrics Platform streams (T322277) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 08:24 kartik@deploy1002: Started scap: Backport for Update Metrics Platform streams (T322277)
  • 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1018.eqiad.wmnet with reason: host reimage
  • 08:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:21 kartik@deploy1002: Finished scap: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016) (duration: 08m 10s)
  • 08:20 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1018.eqiad.wmnet with reason: host reimage
  • 08:20 ayounsi@cumin1001: dbctl commit (dc=all): 'Depool db1182', diff saved to https://phabricator.wikimedia.org/P38772 and previous config saved to /var/cache/conftool/dbconfig/20221109-082045-ayounsi.json
  • 08:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:19 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:13 kartik@deploy1002: kartik and phuedx: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 08:12 kartik@deploy1002: Started scap: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016)
  • 08:06 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1018.eqiad.wmnet with OS bullseye
  • 07:24 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 6461
  • 07:22 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 6461
  • 07:22 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 8218
  • 07:22 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 8218
  • 07:21 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
  • 07:21 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 37271
  • 07:20 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37662
  • 07:20 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 37662
  • 07:20 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 29608
  • 07:19 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 29608
  • 07:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 8309
  • 07:18 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 8309
  • 07:17 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 54994
  • 07:16 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 54994
  • 07:16 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37693
  • 07:15 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 37693
  • 07:14 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 23889
  • 07:14 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 23889
  • 07:11 ayounsi@cumin1001: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 23889
  • 07:11 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 23889
  • 07:11 ayounsi@cumin1001: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 23889
  • 07:11 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 23889
  • 07:10 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 3225
  • 07:10 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 3225
  • 07:10 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 15412
  • 07:09 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 15412
  • 07:09 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 29169
  • 07:08 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 29169
  • 07:08 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37613
  • 07:08 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 37613
  • 07:08 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30990
  • 07:07 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 30990
  • 07:06 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 61955
  • 07:06 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 61955
  • 07:05 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 8220
  • 07:04 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 8220
  • 07:04 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 6774
  • 07:02 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 6774
  • 06:42 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 11404
  • 06:41 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 11404

2022-11-08

  • 22:00 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:59 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:59 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:59 urbanecm: UTC late evening B&C window done
  • 21:58 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:58 urbanecm@deploy1002: Finished scap: Backport for Revert "Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis" (duration: 05m 04s)
  • 21:53 urbanecm@deploy1002: urbanecm and urbanecm: Backport for Revert "Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis" synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 21:53 urbanecm@deploy1002: Started scap: Backport for Revert "Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis"
  • 21:43 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:42 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:42 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:41 urbanecm@deploy1002: Finished scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353) (duration: 06m 36s)
  • 21:41 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:35 urbanecm@deploy1002: urbanecm and matmarex: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 21:35 urbanecm@deploy1002: Started scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353)
  • 21:32 urbanecm@deploy1002: Finished scap: Backport for Bump sampling rate to 0.2 for various editing schemas on a/b test wikis (T321734), ThreadItemStore: Fix setting parent IDs when parent already existed (T322599) (duration: 05m 45s)
  • 21:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:30 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:30 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:26 urbanecm@deploy1002: urbanecm and kemayo and matmarex: Backport for Bump sampling rate to 0.2 for various editing schemas on a/b test wikis (T321734), ThreadItemStore: Fix setting parent IDs when parent already existed (T322599) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 21:26 urbanecm@deploy1002: Started scap: Backport for Bump sampling rate to 0.2 for various editing schemas on a/b test wikis (T321734), ThreadItemStore: Fix setting parent IDs when parent already existed (T322599)
  • 21:25 urbanecm@deploy1002: backport aborted: (duration: 01m 16s)
  • 21:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:23 urbanecm@deploy1002: Finished scap: Backport for Keep DiscussionTools "Share feedback..." links on WMF wikis for now (T322494) (duration: 04m 14s)
  • 21:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:19 urbanecm@deploy1002: Started scap: Backport for Keep DiscussionTools "Share feedback..." links on WMF wikis for now (T322494)
  • 21:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:17 urbanecm@deploy1002: Finished scap: Backport for ABtest for mobile, logged in users (T320993), ABtest for mobile, logged out users (T320993) (duration: 04m 10s)
  • 21:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:13 urbanecm@deploy1002: Started scap: Backport for ABtest for mobile, logged in users (T320993), ABtest for mobile, logged out users (T320993)
  • 21:13 urbanecm@deploy1002: backport aborted: (duration: 00m 01s)
  • 21:13 urbanecm@deploy1002: Finished scap: Backport for Enable history page visual diffs on beta cluster (T314588), Update wgSpecialContributeSkinsDisabled → wgSpecialContributeSkinsEnabled (T319327) (duration: 04m 33s)
  • 21:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:08 urbanecm@deploy1002: Started scap: Backport for Enable history page visual diffs on beta cluster (T314588), Update wgSpecialContributeSkinsDisabled → wgSpecialContributeSkinsEnabled (T319327)
  • 20:58 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:58 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:57 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:57 reedy@deploy1002: Synchronized wmf-config/LabsServices.php: T322667 (duration: 04m 02s)
  • 20:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321130)', diff saved to https://phabricator.wikimedia.org/P38770 and previous config saved to /var/cache/conftool/dbconfig/20221108-203111-marostegui.json
  • 20:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P38769 and previous config saved to /var/cache/conftool/dbconfig/20221108-201604-marostegui.json
  • 20:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P38768 and previous config saved to /var/cache/conftool/dbconfig/20221108-200058-marostegui.json
  • 19:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321130)', diff saved to https://phabricator.wikimedia.org/P38767 and previous config saved to /var/cache/conftool/dbconfig/20221108-194551-marostegui.json
  • 19:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 19:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 19:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T322618)', diff saved to https://phabricator.wikimedia.org/P38766 and previous config saved to /var/cache/conftool/dbconfig/20221108-194206-ladsgroup.json
  • 19:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T321130)', diff saved to https://phabricator.wikimedia.org/P38765 and previous config saved to /var/cache/conftool/dbconfig/20221108-193907-marostegui.json
  • 19:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 19:38 cstone: civicrm upgraded from b95f46bb to f2017495
  • 19:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 19:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38764 and previous config saved to /var/cache/conftool/dbconfig/20221108-193845-marostegui.json
  • 19:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P38763 and previous config saved to /var/cache/conftool/dbconfig/20221108-192659-ladsgroup.json
  • 19:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P38762 and previous config saved to /var/cache/conftool/dbconfig/20221108-192339-marostegui.json
  • 19:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P38761 and previous config saved to /var/cache/conftool/dbconfig/20221108-191152-ladsgroup.json
  • 19:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P38760 and previous config saved to /var/cache/conftool/dbconfig/20221108-190832-marostegui.json
  • 19:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T321123)', diff saved to https://phabricator.wikimedia.org/P38759 and previous config saved to /var/cache/conftool/dbconfig/20221108-190827-marostegui.json
  • 18:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1024.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 18:58 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1024.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T322618)', diff saved to https://phabricator.wikimedia.org/P38758 and previous config saved to /var/cache/conftool/dbconfig/20221108-185646-ladsgroup.json
  • 18:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1201 (T322618)', diff saved to https://phabricator.wikimedia.org/P38757 and previous config saved to /var/cache/conftool/dbconfig/20221108-185437-ladsgroup.json
  • 18:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 18:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 18:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T322618)', diff saved to https://phabricator.wikimedia.org/P38756 and previous config saved to /var/cache/conftool/dbconfig/20221108-185416-ladsgroup.json
  • 18:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38755 and previous config saved to /var/cache/conftool/dbconfig/20221108-185326-marostegui.json
  • 18:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38754 and previous config saved to /var/cache/conftool/dbconfig/20221108-185320-marostegui.json
  • 18:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38753 and previous config saved to /var/cache/conftool/dbconfig/20221108-184642-marostegui.json
  • 18:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 18:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 18:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T321130)', diff saved to https://phabricator.wikimedia.org/P38752 and previous config saved to /var/cache/conftool/dbconfig/20221108-184620-marostegui.json
  • 18:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P38751 and previous config saved to /var/cache/conftool/dbconfig/20221108-183909-ladsgroup.json
  • 18:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38750 and previous config saved to /var/cache/conftool/dbconfig/20221108-183814-marostegui.json
  • 18:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P38749 and previous config saved to /var/cache/conftool/dbconfig/20221108-183114-marostegui.json
  • 18:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P38748 and previous config saved to /var/cache/conftool/dbconfig/20221108-182403-ladsgroup.json
  • 18:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T321123)', diff saved to https://phabricator.wikimedia.org/P38747 and previous config saved to /var/cache/conftool/dbconfig/20221108-182307-marostegui.json
  • 18:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P38746 and previous config saved to /var/cache/conftool/dbconfig/20221108-181607-marostegui.json
  • 18:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T322618)', diff saved to https://phabricator.wikimedia.org/P38745 and previous config saved to /var/cache/conftool/dbconfig/20221108-180856-ladsgroup.json
  • 18:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T321123)', diff saved to https://phabricator.wikimedia.org/P38744 and previous config saved to /var/cache/conftool/dbconfig/20221108-180808-marostegui.json
  • 18:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 18:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 18:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T321123)', diff saved to https://phabricator.wikimedia.org/P38743 and previous config saved to /var/cache/conftool/dbconfig/20221108-180747-marostegui.json
  • 18:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T322618)', diff saved to https://phabricator.wikimedia.org/P38742 and previous config saved to /var/cache/conftool/dbconfig/20221108-180648-ladsgroup.json
  • 18:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 18:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 18:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38741 and previous config saved to /var/cache/conftool/dbconfig/20221108-180626-ladsgroup.json
  • 17:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T321130)', diff saved to https://phabricator.wikimedia.org/P38739 and previous config saved to /var/cache/conftool/dbconfig/20221108-175425-marostegui.json
  • 17:58 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 17:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38738 and previous config saved to /var/cache/conftool/dbconfig/20221108-175404-marostegui.json
  • 17:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38737 and previous config saved to /var/cache/conftool/dbconfig/20221108-175240-marostegui.json
  • 17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P38736 and previous config saved to /var/cache/conftool/dbconfig/20221108-175120-ladsgroup.json
  • 17:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1015.eqiad.wmnet
  • 17:40 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1015.eqiad.wmnet
  • 17:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1014.eqiad.wmnet
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P38735 and previous config saved to /var/cache/conftool/dbconfig/20221108-173857-marostegui.json
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38734 and previous config saved to /var/cache/conftool/dbconfig/20221108-173734-marostegui.json
  • 17:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P38733 and previous config saved to /var/cache/conftool/dbconfig/20221108-173613-ladsgroup.json
  • 17:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1014.eqiad.wmnet
  • 17:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1013.eqiad.wmnet
  • 17:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P38732 and previous config saved to /var/cache/conftool/dbconfig/20221108-172351-marostegui.json
  • 17:23 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1013.eqiad.wmnet
  • 17:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T321123)', diff saved to https://phabricator.wikimedia.org/P38731 and previous config saved to /var/cache/conftool/dbconfig/20221108-172227-marostegui.json
  • 17:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38730 and previous config saved to /var/cache/conftool/dbconfig/20221108-172107-ladsgroup.json
  • 17:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38729 and previous config saved to /var/cache/conftool/dbconfig/20221108-171857-ladsgroup.json
  • 17:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 17:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 17:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T322618)', diff saved to https://phabricator.wikimedia.org/P38728 and previous config saved to /var/cache/conftool/dbconfig/20221108-171835-ladsgroup.json
  • 17:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38727 and previous config saved to /var/cache/conftool/dbconfig/20221108-170844-marostegui.json
  • 17:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T321123)', diff saved to https://phabricator.wikimedia.org/P38726 and previous config saved to /var/cache/conftool/dbconfig/20221108-170752-marostegui.json
  • 17:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 17:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 17:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 17:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 17:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38725 and previous config saved to /var/cache/conftool/dbconfig/20221108-170715-marostegui.json
  • 17:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P38724 and previous config saved to /var/cache/conftool/dbconfig/20221108-170329-ladsgroup.json
  • 17:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38723 and previous config saved to /var/cache/conftool/dbconfig/20221108-170157-marostegui.json
  • 17:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T321130)', diff saved to https://phabricator.wikimedia.org/P38722 and previous config saved to /var/cache/conftool/dbconfig/20221108-170136-marostegui.json
  • 17:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet
  • 16:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet
  • 16:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2002.codfw.wmnet
  • 16:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38721 and previous config saved to /var/cache/conftool/dbconfig/20221108-165208-marostegui.json
  • 16:49 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2002.codfw.wmnet
  • 16:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P38720 and previous config saved to /var/cache/conftool/dbconfig/20221108-164822-ladsgroup.json
  • 16:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2002.codfw.wmnet
  • 16:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P38719 and previous config saved to /var/cache/conftool/dbconfig/20221108-164629-marostegui.json
  • 16:41 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow2002.codfw.wmnet
  • 16:39 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 16:38 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:37 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 16:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38718 and previous config saved to /var/cache/conftool/dbconfig/20221108-163702-marostegui.json
  • 16:36 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow5002.eqsin.wmnet
  • 16:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T322618)', diff saved to https://phabricator.wikimedia.org/P38717 and previous config saved to /var/cache/conftool/dbconfig/20221108-163316-ladsgroup.json
  • 16:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P38716 and previous config saved to /var/cache/conftool/dbconfig/20221108-163122-marostegui.json
  • 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T322618)', diff saved to https://phabricator.wikimedia.org/P38715 and previous config saved to /var/cache/conftool/dbconfig/20221108-163107-ladsgroup.json
  • 16:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 16:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 16:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T322618)', diff saved to https://phabricator.wikimedia.org/P38714 and previous config saved to /var/cache/conftool/dbconfig/20221108-163045-ladsgroup.json
  • 16:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow5002.eqsin.wmnet
  • 16:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow6001.drmrs.wmnet
  • 16:24 moritzm: drain ganeti1024 for eventual reimage to bullseye T311687
  • 16:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1018.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 16:22 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1018.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 16:22 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow6001.drmrs.wmnet
  • 16:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38713 and previous config saved to /var/cache/conftool/dbconfig/20221108-162155-marostegui.json
  • 16:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T321130)', diff saved to https://phabricator.wikimedia.org/P38712 and previous config saved to /var/cache/conftool/dbconfig/20221108-161616-marostegui.json
  • 16:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P38711 and previous config saved to /var/cache/conftool/dbconfig/20221108-161538-ladsgroup.json
  • 16:13 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T321130)', diff saved to https://phabricator.wikimedia.org/P38710 and previous config saved to /var/cache/conftool/dbconfig/20221108-161338-marostegui.json
  • 16:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T321130)', diff saved to https://phabricator.wikimedia.org/P38709 and previous config saved to /var/cache/conftool/dbconfig/20221108-161312-marostegui.json
  • 16:10 Emperor: upload wmf-beamer-style version 0.4 to apt
  • 16:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38708 and previous config saved to /var/cache/conftool/dbconfig/20221108-160632-marostegui.json
  • 16:06 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:06 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P38707 and previous config saved to /var/cache/conftool/dbconfig/20221108-160032-ladsgroup.json
  • 15:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P38706 and previous config saved to /var/cache/conftool/dbconfig/20221108-155805-marostegui.json
  • 15:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 15:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 15:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T321123)', diff saved to https://phabricator.wikimedia.org/P38705 and previous config saved to /var/cache/conftool/dbconfig/20221108-155229-marostegui.json
  • 15:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T322618)', diff saved to https://phabricator.wikimedia.org/P38704 and previous config saved to /var/cache/conftool/dbconfig/20221108-154525-ladsgroup.json
  • 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T322618)', diff saved to https://phabricator.wikimedia.org/P38703 and previous config saved to /var/cache/conftool/dbconfig/20221108-154317-ladsgroup.json
  • 15:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 15:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P38702 and previous config saved to /var/cache/conftool/dbconfig/20221108-154259-marostegui.json
  • 15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T322618)', diff saved to https://phabricator.wikimedia.org/P38701 and previous config saved to /var/cache/conftool/dbconfig/20221108-154221-ladsgroup.json
  • 15:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38700 and previous config saved to /var/cache/conftool/dbconfig/20221108-153722-marostegui.json
  • 15:35 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:35 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:34 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:34 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T321130)', diff saved to https://phabricator.wikimedia.org/P38699 and previous config saved to /var/cache/conftool/dbconfig/20221108-152752-marostegui.json
  • 15:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P38698 and previous config saved to /var/cache/conftool/dbconfig/20221108-152715-ladsgroup.json
  • 15:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38697 and previous config saved to /var/cache/conftool/dbconfig/20221108-152548-ladsgroup.json
  • 15:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38696 and previous config saved to /var/cache/conftool/dbconfig/20221108-152216-marostegui.json
  • 15:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T321130)', diff saved to https://phabricator.wikimedia.org/P38695 and previous config saved to /var/cache/conftool/dbconfig/20221108-152037-marostegui.json
  • 15:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 15:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 15:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T321130)', diff saved to https://phabricator.wikimedia.org/P38694 and previous config saved to /var/cache/conftool/dbconfig/20221108-152016-marostegui.json
  • 15:18 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:18 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:17 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:16 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P38693 and previous config saved to /var/cache/conftool/dbconfig/20221108-151208-ladsgroup.json
  • 15:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P38692 and previous config saved to /var/cache/conftool/dbconfig/20221108-151041-ladsgroup.json
  • 15:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T321123)', diff saved to https://phabricator.wikimedia.org/P38691 and previous config saved to /var/cache/conftool/dbconfig/20221108-150709-marostegui.json
  • 15:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P38690 and previous config saved to /var/cache/conftool/dbconfig/20221108-150509-marostegui.json
  • 15:04 daniel@deploy1002: Finished scap: Backport for Stash original wikitext when rendering unsaved content. (T321862) (duration: 07m 25s)
  • 15:03 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:03 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:01 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:01 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:01 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:01 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:01 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:57 daniel@deploy1002: daniel and daniel: Backport for Stash original wikitext when rendering unsaved content. (T321862) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 14:57 daniel@deploy1002: Started scap: Backport for Stash original wikitext when rendering unsaved content. (T321862)
  • 14:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T322618)', diff saved to https://phabricator.wikimedia.org/P38689 and previous config saved to /var/cache/conftool/dbconfig/20221108-145702-ladsgroup.json
  • 14:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P38688 and previous config saved to /var/cache/conftool/dbconfig/20221108-145535-ladsgroup.json
  • 14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T322618)', diff saved to https://phabricator.wikimedia.org/P38687 and previous config saved to /var/cache/conftool/dbconfig/20221108-145453-ladsgroup.json
  • 14:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 14:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38686 and previous config saved to /var/cache/conftool/dbconfig/20221108-145432-ladsgroup.json
  • 14:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T321123)', diff saved to https://phabricator.wikimedia.org/P38685 and previous config saved to /var/cache/conftool/dbconfig/20221108-145210-marostegui.json
  • 14:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 14:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 14:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T321123)', diff saved to https://phabricator.wikimedia.org/P38684 and previous config saved to /var/cache/conftool/dbconfig/20221108-145148-marostegui.json
  • 14:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P38683 and previous config saved to /var/cache/conftool/dbconfig/20221108-145003-marostegui.json
  • 14:43 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38682 and previous config saved to /var/cache/conftool/dbconfig/20221108-144028-ladsgroup.json
  • 14:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P38681 and previous config saved to /var/cache/conftool/dbconfig/20221108-143925-ladsgroup.json
  • 14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38680 and previous config saved to /var/cache/conftool/dbconfig/20221108-143815-ladsgroup.json
  • 14:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 14:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 14:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38679 and previous config saved to /var/cache/conftool/dbconfig/20221108-143743-ladsgroup.json
  • 14:37 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 14:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38678 and previous config saved to /var/cache/conftool/dbconfig/20221108-143642-marostegui.json
  • 14:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T321130)', diff saved to https://phabricator.wikimedia.org/P38677 and previous config saved to /var/cache/conftool/dbconfig/20221108-143457-marostegui.json
  • 14:33 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T321130)', diff saved to https://phabricator.wikimedia.org/P38676 and previous config saved to /var/cache/conftool/dbconfig/20221108-143220-marostegui.json
  • 14:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 14:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 14:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 14:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 14:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P38675 and previous config saved to /var/cache/conftool/dbconfig/20221108-142419-ladsgroup.json
  • 14:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 14:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 14:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T321130)', diff saved to https://phabricator.wikimedia.org/P38674 and previous config saved to /var/cache/conftool/dbconfig/20221108-142247-marostegui.json
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P38673 and previous config saved to /var/cache/conftool/dbconfig/20221108-142236-ladsgroup.json
  • 14:22 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:22 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38672 and previous config saved to /var/cache/conftool/dbconfig/20221108-142135-marostegui.json
  • 14:21 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:21 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:11 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 14:11 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 14:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38671 and previous config saved to /var/cache/conftool/dbconfig/20221108-140912-ladsgroup.json
  • 14:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38670 and previous config saved to /var/cache/conftool/dbconfig/20221108-140803-ladsgroup.json
  • 14:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 14:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 14:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P38669 and previous config saved to /var/cache/conftool/dbconfig/20221108-140741-marostegui.json
  • 14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P38668 and previous config saved to /var/cache/conftool/dbconfig/20221108-140730-ladsgroup.json
  • 14:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T321123)', diff saved to https://phabricator.wikimedia.org/P38667 and previous config saved to /var/cache/conftool/dbconfig/20221108-140628-marostegui.json
  • 13:56 dcausse@deploy1002: Finished deploy [wikimedia/discovery/analytics@248d897]: import_cirrus_indexes: increase driver mem (duration: 02m 23s)
  • 13:54 dcausse@deploy1002: Started deploy [wikimedia/discovery/analytics@248d897]: import_cirrus_indexes: increase driver mem
  • 13:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P38666 and previous config saved to /var/cache/conftool/dbconfig/20221108-135234-marostegui.json
  • 13:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38665 and previous config saved to /var/cache/conftool/dbconfig/20221108-135224-ladsgroup.json
  • 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T321123)', diff saved to https://phabricator.wikimedia.org/P38664 and previous config saved to /var/cache/conftool/dbconfig/20221108-135129-marostegui.json
  • 13:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 13:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38663 and previous config saved to /var/cache/conftool/dbconfig/20221108-135011-ladsgroup.json
  • 13:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 13:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 13:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38662 and previous config saved to /var/cache/conftool/dbconfig/20221108-134949-ladsgroup.json
  • 13:49 btullis@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 13:49 btullis@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 13:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 13:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T321123)', diff saved to https://phabricator.wikimedia.org/P38661 and previous config saved to /var/cache/conftool/dbconfig/20221108-134656-marostegui.json
  • 13:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P38660 and previous config saved to /var/cache/conftool/dbconfig/20221108-133730-ladsgroup.json
  • 13:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T321130)', diff saved to https://phabricator.wikimedia.org/P38659 and previous config saved to /var/cache/conftool/dbconfig/20221108-133721-marostegui.json
  • 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T321130)', diff saved to https://phabricator.wikimedia.org/P38658 and previous config saved to /var/cache/conftool/dbconfig/20221108-133505-marostegui.json
  • 13:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 13:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P38657 and previous config saved to /var/cache/conftool/dbconfig/20221108-133442-ladsgroup.json
  • 13:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 13:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T321130)', diff saved to https://phabricator.wikimedia.org/P38656 and previous config saved to /var/cache/conftool/dbconfig/20221108-133433-marostegui.json
  • 13:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38655 and previous config saved to /var/cache/conftool/dbconfig/20221108-133149-marostegui.json
  • 13:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow3002.esams.wmnet
  • 13:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38654 and previous config saved to /var/cache/conftool/dbconfig/20221108-132223-ladsgroup.json
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38653 and previous config saved to /var/cache/conftool/dbconfig/20221108-132014-ladsgroup.json
  • 13:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 13:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38652 and previous config saved to /var/cache/conftool/dbconfig/20221108-131952-ladsgroup.json
  • 13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P38651 and previous config saved to /var/cache/conftool/dbconfig/20221108-131936-ladsgroup.json
  • 13:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P38650 and previous config saved to /var/cache/conftool/dbconfig/20221108-131927-marostegui.json
  • 13:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow3002.esams.wmnet
  • 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4002.ulsfo.wmnet
  • 13:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38649 and previous config saved to /var/cache/conftool/dbconfig/20221108-131643-marostegui.json
  • 13:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow4002.ulsfo.wmnet
  • 13:07 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 15557
  • 13:05 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 15557
  • 13:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38648 and previous config saved to /var/cache/conftool/dbconfig/20221108-130446-ladsgroup.json
  • 13:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38647 and previous config saved to /var/cache/conftool/dbconfig/20221108-130429-ladsgroup.json
  • 13:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P38646 and previous config saved to /var/cache/conftool/dbconfig/20221108-130420-marostegui.json
  • 13:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38645 and previous config saved to /var/cache/conftool/dbconfig/20221108-130216-ladsgroup.json
  • 13:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 13:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 13:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38644 and previous config saved to /var/cache/conftool/dbconfig/20221108-130205-ladsgroup.json
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T321123)', diff saved to https://phabricator.wikimedia.org/P38643 and previous config saved to /var/cache/conftool/dbconfig/20221108-130136-marostegui.json
  • 12:55 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T321123)', diff saved to https://phabricator.wikimedia.org/P38642 and previous config saved to /var/cache/conftool/dbconfig/20221108-125529-marostegui.json
  • 12:55 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T321123)', diff saved to https://phabricator.wikimedia.org/P38641 and previous config saved to /var/cache/conftool/dbconfig/20221108-125508-marostegui.json
  • 12:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38640 and previous config saved to /var/cache/conftool/dbconfig/20221108-124939-ladsgroup.json
  • 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T321130)', diff saved to https://phabricator.wikimedia.org/P38639 and previous config saved to /var/cache/conftool/dbconfig/20221108-124914-marostegui.json
  • 12:47 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P38638 and previous config saved to /var/cache/conftool/dbconfig/20221108-124659-ladsgroup.json
  • 12:47 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T321130)', diff saved to https://phabricator.wikimedia.org/P38637 and previous config saved to /var/cache/conftool/dbconfig/20221108-124658-marostegui.json
  • 12:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 12:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 12:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T321130)', diff saved to https://phabricator.wikimedia.org/P38636 and previous config saved to /var/cache/conftool/dbconfig/20221108-124636-marostegui.json
  • 12:42 ladsgroup@deploy1002: Finished scap: Backport for Include core PSR-4 classes in the generated classmap (T274041) (duration: 05m 29s)
  • 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki2002.codfw.wmnet
  • 12:40 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:40 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38635 and previous config saved to /var/cache/conftool/dbconfig/20221108-124001-marostegui.json
  • 12:38 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:38 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 12:37 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for Include core PSR-4 classes in the generated classmap (T274041) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 12:37 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 12:37 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 12:37 ladsgroup@deploy1002: Started scap: Backport for Include core PSR-4 classes in the generated classmap (T274041)
  • 12:36 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 12:36 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host pki2002.codfw.wmnet
  • 12:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38634 and previous config saved to /var/cache/conftool/dbconfig/20221108-123433-ladsgroup.json
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P38633 and previous config saved to /var/cache/conftool/dbconfig/20221108-123152-ladsgroup.json
  • 12:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P38632 and previous config saved to /var/cache/conftool/dbconfig/20221108-123130-marostegui.json
  • 12:29 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp1002.wikimedia.org
  • 12:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38631 and previous config saved to /var/cache/conftool/dbconfig/20221108-122923-ladsgroup.json
  • 12:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 12:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 12:28 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:27 sukhe: reprepro -C main include bullseye-wikimedia libvmod-re2_1.5.3-3_amd64.changes: T321309
  • 12:25 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host idp1002.wikimedia.org
  • 12:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38630 and previous config saved to /var/cache/conftool/dbconfig/20221108-122455-marostegui.json
  • 12:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38629 and previous config saved to /var/cache/conftool/dbconfig/20221108-121646-ladsgroup.json
  • 12:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P38628 and previous config saved to /var/cache/conftool/dbconfig/20221108-121623-marostegui.json
  • 12:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38627 and previous config saved to /var/cache/conftool/dbconfig/20221108-121433-ladsgroup.json
  • 12:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 12:14 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 12:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 12:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 12:14 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 12:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 12:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T322618)', diff saved to https://phabricator.wikimedia.org/P38626 and previous config saved to /var/cache/conftool/dbconfig/20221108-121347-ladsgroup.json
  • 12:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T321123)', diff saved to https://phabricator.wikimedia.org/P38625 and previous config saved to /var/cache/conftool/dbconfig/20221108-120949-marostegui.json
  • 12:09 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:08 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:02 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T321123)', diff saved to https://phabricator.wikimedia.org/P38624 and previous config saved to /var/cache/conftool/dbconfig/20221108-120143-marostegui.json
  • 12:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 12:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38623 and previous config saved to /var/cache/conftool/dbconfig/20221108-120122-marostegui.json
  • 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T321130)', diff saved to https://phabricator.wikimedia.org/P38622 and previous config saved to /var/cache/conftool/dbconfig/20221108-120117-marostegui.json
  • 11:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P38621 and previous config saved to /var/cache/conftool/dbconfig/20221108-115840-ladsgroup.json
  • 11:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T321130)', diff saved to https://phabricator.wikimedia.org/P38620 and previous config saved to /var/cache/conftool/dbconfig/20221108-115452-marostegui.json
  • 11:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 11:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 11:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38619 and previous config saved to /var/cache/conftool/dbconfig/20221108-115430-marostegui.json
  • 11:52 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 11:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38618 and previous config saved to /var/cache/conftool/dbconfig/20221108-114615-marostegui.json
  • 11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P38617 and previous config saved to /var/cache/conftool/dbconfig/20221108-114333-ladsgroup.json
  • 11:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P38616 and previous config saved to /var/cache/conftool/dbconfig/20221108-113924-marostegui.json
  • 11:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38615 and previous config saved to /var/cache/conftool/dbconfig/20221108-113109-marostegui.json
  • 11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T322618)', diff saved to https://phabricator.wikimedia.org/P38614 and previous config saved to /var/cache/conftool/dbconfig/20221108-112825-ladsgroup.json
  • 11:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T322618)', diff saved to https://phabricator.wikimedia.org/P38613 and previous config saved to /var/cache/conftool/dbconfig/20221108-112612-ladsgroup.json
  • 11:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 11:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 11:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T322618)', diff saved to https://phabricator.wikimedia.org/P38612 and previous config saved to /var/cache/conftool/dbconfig/20221108-112551-ladsgroup.json
  • 11:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P38611 and previous config saved to /var/cache/conftool/dbconfig/20221108-112417-marostegui.json
  • 11:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38610 and previous config saved to /var/cache/conftool/dbconfig/20221108-111602-marostegui.json
  • 11:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P38609 and previous config saved to /var/cache/conftool/dbconfig/20221108-111044-ladsgroup.json
  • 11:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38608 and previous config saved to /var/cache/conftool/dbconfig/20221108-110956-marostegui.json
  • 11:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 11:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 11:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T321123)', diff saved to https://phabricator.wikimedia.org/P38607 and previous config saved to /var/cache/conftool/dbconfig/20221108-110934-marostegui.json
  • 11:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38606 and previous config saved to /var/cache/conftool/dbconfig/20221108-110911-marostegui.json
  • 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38605 and previous config saved to /var/cache/conftool/dbconfig/20221108-110243-marostegui.json
  • 11:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 11:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T321130)', diff saved to https://phabricator.wikimedia.org/P38604 and previous config saved to /var/cache/conftool/dbconfig/20221108-110232-marostegui.json
  • 11:00 moritzm: drain ganeti1024 for eventual reimage to bullseye T311687
  • 11:00 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 10:58 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 10:57 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 10:55 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P38603 and previous config saved to /var/cache/conftool/dbconfig/20221108-105538-ladsgroup.json
  • 10:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38602 and previous config saved to /var/cache/conftool/dbconfig/20221108-105428-marostegui.json
  • 10:52 btullis: added stevemunene to wmf and ops LDAP groups T322339
  • 10:51 moritzm: installing batik security updates
  • 10:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P38601 and previous config saved to /var/cache/conftool/dbconfig/20221108-104726-marostegui.json
  • 10:42 moritzm: installing ntfs-3g security updates
  • 10:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T322618)', diff saved to https://phabricator.wikimedia.org/P38600 and previous config saved to /var/cache/conftool/dbconfig/20221108-104031-ladsgroup.json
  • 10:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38599 and previous config saved to /var/cache/conftool/dbconfig/20221108-103921-marostegui.json
  • 10:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T322618)', diff saved to https://phabricator.wikimedia.org/P38598 and previous config saved to /var/cache/conftool/dbconfig/20221108-103817-ladsgroup.json
  • 10:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 10:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 10:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P38597 and previous config saved to /var/cache/conftool/dbconfig/20221108-103756-ladsgroup.json
  • 10:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P38596 and previous config saved to /var/cache/conftool/dbconfig/20221108-103219-marostegui.json
  • 10:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T321123)', diff saved to https://phabricator.wikimedia.org/P38595 and previous config saved to /var/cache/conftool/dbconfig/20221108-102415-marostegui.json
  • 10:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P38594 and previous config saved to /var/cache/conftool/dbconfig/20221108-102249-ladsgroup.json
  • 10:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T321123)', diff saved to https://phabricator.wikimedia.org/P38593 and previous config saved to /var/cache/conftool/dbconfig/20221108-101806-marostegui.json
  • 10:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 10:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 10:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38592 and previous config saved to /var/cache/conftool/dbconfig/20221108-101745-marostegui.json
  • 10:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T321130)', diff saved to https://phabricator.wikimedia.org/P38591 and previous config saved to /var/cache/conftool/dbconfig/20221108-101713-marostegui.json
  • 10:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T321130)', diff saved to https://phabricator.wikimedia.org/P38590 and previous config saved to /var/cache/conftool/dbconfig/20221108-101457-marostegui.json
  • 10:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 10:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 10:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T321130)', diff saved to https://phabricator.wikimedia.org/P38589 and previous config saved to /var/cache/conftool/dbconfig/20221108-101435-marostegui.json
  • 10:10 moritzm: installing glibc security updates on buster
  • 10:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P38588 and previous config saved to /var/cache/conftool/dbconfig/20221108-100743-ladsgroup.json
  • 10:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38587 and previous config saved to /var/cache/conftool/dbconfig/20221108-100239-marostegui.json
  • 09:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P38586 and previous config saved to /var/cache/conftool/dbconfig/20221108-095928-marostegui.json
  • 09:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P38585 and previous config saved to /var/cache/conftool/dbconfig/20221108-095236-ladsgroup.json
  • 09:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P38584 and previous config saved to /var/cache/conftool/dbconfig/20221108-095026-ladsgroup.json
  • 09:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 09:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 09:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Partially done', diff saved to https://phabricator.wikimedia.org/P38583 and previous config saved to /var/cache/conftool/dbconfig/20221108-094950-ladsgroup.json
  • 09:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38582 and previous config saved to /var/cache/conftool/dbconfig/20221108-094732-marostegui.json
  • 09:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P38581 and previous config saved to /var/cache/conftool/dbconfig/20221108-094655-ladsgroup.json
  • 09:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 09:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 09:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P38580 and previous config saved to /var/cache/conftool/dbconfig/20221108-094422-marostegui.json
  • 09:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38579 and previous config saved to /var/cache/conftool/dbconfig/20221108-093226-marostegui.json
  • 09:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T321130)', diff saved to https://phabricator.wikimedia.org/P38578 and previous config saved to /var/cache/conftool/dbconfig/20221108-092915-marostegui.json
  • 09:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T321130)', diff saved to https://phabricator.wikimedia.org/P38577 and previous config saved to /var/cache/conftool/dbconfig/20221108-092256-marostegui.json
  • 09:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 09:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 09:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38576 and previous config saved to /var/cache/conftool/dbconfig/20221108-092229-marostegui.json
  • 09:22 moritzm: installing ffmpeg security updates on buster
  • 09:17 moritzm: drain ganeti1018 for eventual reimage to bullseye T311687
  • 09:09 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1004.eqiad.wmnet to plain
  • 09:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1004.eqiad.wmnet to plain
  • 09:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P38575 and previous config saved to /var/cache/conftool/dbconfig/20221108-090722-marostegui.json
  • 08:54 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1004.eqiad.wmnet to drbd
  • 08:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P38574 and previous config saved to /var/cache/conftool/dbconfig/20221108-085216-marostegui.json
  • 08:48 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:47 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:47 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:44 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1004.eqiad.wmnet to drbd
  • 08:44 kartik@deploy1002: Finished scap: Backport for Revert "EditAttemptStep sampling rate to 1 for group1 wikis" (duration: 04m 26s)
  • 08:40 kartik@deploy1002: kartik and trainbranchbot: Backport for Revert "EditAttemptStep sampling rate to 1 for group1 wikis" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 08:39 kartik@deploy1002: Started scap: Backport for Revert "EditAttemptStep sampling rate to 1 for group1 wikis"
  • 08:37 kartik@deploy1002: Sync cancelled.
  • 08:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38573 and previous config saved to /var/cache/conftool/dbconfig/20221108-083709-marostegui.json
  • 08:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:35 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:35 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:34 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1006.eqiad.wmnet to plain
  • 08:33 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1006.eqiad.wmnet to plain
  • 08:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38572 and previous config saved to /var/cache/conftool/dbconfig/20221108-083210-marostegui.json
  • 08:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 08:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 08:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321123)', diff saved to https://phabricator.wikimedia.org/P38571 and previous config saved to /var/cache/conftool/dbconfig/20221108-083148-marostegui.json
  • 08:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38570 and previous config saved to /var/cache/conftool/dbconfig/20221108-083037-marostegui.json
  • 08:30 kartik@deploy1002: kartik and phuedx: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 08:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 08:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 08:30 kartik@deploy1002: Started scap: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016)
  • 08:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 08:25 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 08:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38569 and previous config saved to /var/cache/conftool/dbconfig/20221108-082546-marostegui.json
  • 08:24 kartik@deploy1002: Finished scap: Backport for Enable Content and Section translation on 6 Wikipedias (T319175) (duration: 08m 20s)
  • 08:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1006.eqiad.wmnet to drbd
  • 08:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:16 kartik@deploy1002: kartik and kartik: Backport for Enable Content and Section translation on 6 Wikipedias (T319175) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 08:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38568 and previous config saved to /var/cache/conftool/dbconfig/20221108-081641-marostegui.json
  • 08:16 kartik@deploy1002: Started scap: Backport for Enable Content and Section translation on 6 Wikipedias (T319175)
  • 08:14 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1006.eqiad.wmnet to drbd
  • 08:12 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P38567 and previous config saved to /var/cache/conftool/dbconfig/20221108-081040-marostegui.json
  • 08:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:09 kartik@deploy1002: Finished scap: Backport for Enable Content and Section translation in Bambara and Goan Konkani Wikipedias (T314557) (duration: 06m 44s)
  • 08:03 kartik@deploy1002: kartik and kartik: Backport for Enable Content and Section translation in Bambara and Goan Konkani Wikipedias (T314557) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 08:03 kartik@deploy1002: Started scap: Backport for Enable Content and Section translation in Bambara and Goan Konkani Wikipedias (T314557)
  • 08:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38566 and previous config saved to /var/cache/conftool/dbconfig/20221108-080135-marostegui.json
  • 07:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P38565 and previous config saved to /var/cache/conftool/dbconfig/20221108-075533-marostegui.json
  • 07:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321123)', diff saved to https://phabricator.wikimedia.org/P38564 and previous config saved to /var/cache/conftool/dbconfig/20221108-074628-marostegui.json
  • 07:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38563 and previous config saved to /var/cache/conftool/dbconfig/20221108-074027-marostegui.json
  • 07:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T321123)', diff saved to https://phabricator.wikimedia.org/P38562 and previous config saved to /var/cache/conftool/dbconfig/20221108-074022-marostegui.json
  • 07:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 07:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 07:37 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38561 and previous config saved to /var/cache/conftool/dbconfig/20221108-073711-marostegui.json
  • 07:37 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 07:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 07:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38560 and previous config saved to /var/cache/conftool/dbconfig/20221108-073649-marostegui.json
  • 07:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 07:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 07:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321123)', diff saved to https://phabricator.wikimedia.org/P38559 and previous config saved to /var/cache/conftool/dbconfig/20221108-073549-marostegui.json
  • 07:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T318605)', diff saved to https://phabricator.wikimedia.org/P38558 and previous config saved to /var/cache/conftool/dbconfig/20221108-072648-ladsgroup.json
  • 07:22 XioNoX: push pfw policies - T322613
  • 07:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P38557 and previous config saved to /var/cache/conftool/dbconfig/20221108-072143-marostegui.json
  • 07:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38556 and previous config saved to /var/cache/conftool/dbconfig/20221108-072042-marostegui.json
  • 07:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P38555 and previous config saved to /var/cache/conftool/dbconfig/20221108-071142-ladsgroup.json
  • 07:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P38554 and previous config saved to /var/cache/conftool/dbconfig/20221108-070636-marostegui.json
  • 07:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38553 and previous config saved to /var/cache/conftool/dbconfig/20221108-070536-marostegui.json
  • 06:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P38552 and previous config saved to /var/cache/conftool/dbconfig/20221108-065635-ladsgroup.json
  • 06:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38551 and previous config saved to /var/cache/conftool/dbconfig/20221108-065130-marostegui.json
  • 06:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321123)', diff saved to https://phabricator.wikimedia.org/P38550 and previous config saved to /var/cache/conftool/dbconfig/20221108-065029-marostegui.json
  • 06:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38549 and previous config saved to /var/cache/conftool/dbconfig/20221108-064447-marostegui.json
  • 06:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 06:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 06:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T321123)', diff saved to https://phabricator.wikimedia.org/P38548 and previous config saved to /var/cache/conftool/dbconfig/20221108-064422-marostegui.json
  • 06:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 06:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 06:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 06:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 06:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T318605)', diff saved to https://phabricator.wikimedia.org/P38547 and previous config saved to /var/cache/conftool/dbconfig/20221108-064129-ladsgroup.json
  • 06:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:38 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 63199
  • 06:36 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 63199
  • 06:35 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 22381
  • 06:35 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 22381
  • 06:34 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 4817
  • 06:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 06:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 06:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 06:33 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 4817
  • 06:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1138.eqiad.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1138.eqiad.wmnet with reason: Maintenance
  • 06:32 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 13150
  • 06:31 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 13150
  • 06:30 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 30058
  • 06:29 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 30058
  • 06:27 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 46416
  • 06:27 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 06:27 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 46416
  • 05:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2182 (T318605)', diff saved to https://phabricator.wikimedia.org/P38546 and previous config saved to /var/cache/conftool/dbconfig/20221108-052025-ladsgroup.json
  • 05:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 05:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 05:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38545 and previous config saved to /var/cache/conftool/dbconfig/20221108-052004-ladsgroup.json
  • 05:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P38544 and previous config saved to /var/cache/conftool/dbconfig/20221108-050457-ladsgroup.json
  • 04:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P38543 and previous config saved to /var/cache/conftool/dbconfig/20221108-044951-ladsgroup.json
  • 04:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38542 and previous config saved to /var/cache/conftool/dbconfig/20221108-043444-ladsgroup.json
  • 03:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 03:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 03:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318605)', diff saved to https://phabricator.wikimedia.org/P38541 and previous config saved to /var/cache/conftool/dbconfig/20221108-034607-ladsgroup.json
  • 03:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P38540 and previous config saved to /var/cache/conftool/dbconfig/20221108-033101-ladsgroup.json
  • 03:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20221108-031550-ladsgroup.json
  • 03:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38539 and previous config saved to /var/cache/conftool/dbconfig/20221108-031102-ladsgroup.json
  • 03:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 03:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 03:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38538 and previous config saved to /var/cache/conftool/dbconfig/20221108-031041-ladsgroup.json
  • 03:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:03 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318605)', diff saved to https://phabricator.wikimedia.org/P38537 and previous config saved to /var/cache/conftool/dbconfig/20221108-030043-ladsgroup.json
  • 02:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P38536 and previous config saved to /var/cache/conftool/dbconfig/20221108-025533-ladsgroup.json
  • 02:47 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host contint1002.mgmt.eqiad.wmnet with reboot policy FORCED
  • 02:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T318605)', diff saved to https://phabricator.wikimedia.org/P38535 and previous config saved to /var/cache/conftool/dbconfig/20221108-024600-ladsgroup.json
  • 02:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 02:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 02:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T318605)', diff saved to https://phabricator.wikimedia.org/P38534 and previous config saved to /var/cache/conftool/dbconfig/20221108-024539-ladsgroup.json
  • 02:44 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host contint1002.mgmt.eqiad.wmnet with reboot policy FORCED
  • 02:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P38533 and previous config saved to /var/cache/conftool/dbconfig/20221108-024027-ladsgroup.json
  • 02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P38532 and previous config saved to /var/cache/conftool/dbconfig/20221108-023032-ladsgroup.json
  • 02:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38531 and previous config saved to /var/cache/conftool/dbconfig/20221108-022520-ladsgroup.json
  • 02:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P38530 and previous config saved to /var/cache/conftool/dbconfig/20221108-021525-ladsgroup.json
  • 02:14 eileen: civicrm upgraded from 72fccce1 to b95f46bb
  • 02:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T318605)', diff saved to https://phabricator.wikimedia.org/P38529 and previous config saved to /var/cache/conftool/dbconfig/20221108-020019-ladsgroup.json
  • 01:47 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 01:47 tgr@deploy1002: Finished scap: Backport for Add UserRegistrationLookupHelper (duration: 04m 36s)
  • 01:46 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 01:46 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 01:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 01:43 tgr@deploy1002: tgr and tgr: Backport for Add UserRegistrationLookupHelper synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 01:42 tgr@deploy1002: Started scap: Backport for Add UserRegistrationLookupHelper
  • 01:39 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 01:38 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 01:37 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
  • 01:25 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 01:22 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
  • 01:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2168:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38528 and previous config saved to /var/cache/conftool/dbconfig/20221108-010245-ladsgroup.json
  • 01:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 01:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 01:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T318605)', diff saved to https://phabricator.wikimedia.org/P38527 and previous config saved to /var/cache/conftool/dbconfig/20221108-010224-ladsgroup.json
  • 00:55 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1194 (T318605)', diff saved to https://phabricator.wikimedia.org/P38526 and previous config saved to /var/cache/conftool/dbconfig/20221108-005338-ladsgroup.json
  • 00:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 00:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 00:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T318605)', diff saved to https://phabricator.wikimedia.org/P38525 and previous config saved to /var/cache/conftool/dbconfig/20221108-005317-ladsgroup.json
  • 00:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P38524 and previous config saved to /var/cache/conftool/dbconfig/20221108-004717-ladsgroup.json
  • 00:47 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:40 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 00:39 tgr@deploy1002: Finished scap: Backport for createExtensionTables.php: Remove closeConnection() (duration: 04m 43s)
  • 00:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38523 and previous config saved to /var/cache/conftool/dbconfig/20221108-003934-marostegui.json
  • 00:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 00:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 00:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 00:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P38522 and previous config saved to /var/cache/conftool/dbconfig/20221108-003810-ladsgroup.json
  • 00:36 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 00:35 tgr@deploy1002: tgr and tgr: Backport for createExtensionTables.php: Remove closeConnection() synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 00:35 tgr@deploy1002: Started scap: Backport for createExtensionTables.php: Remove closeConnection()
  • 00:34 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
  • 00:33 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 00:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P38521 and previous config saved to /var/cache/conftool/dbconfig/20221108-003210-ladsgroup.json
  • 00:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P38520 and previous config saved to /var/cache/conftool/dbconfig/20221108-002428-marostegui.json
  • 00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P38519 and previous config saved to /var/cache/conftool/dbconfig/20221108-002304-ladsgroup.json
  • 00:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T318605)', diff saved to https://phabricator.wikimedia.org/P38518 and previous config saved to /var/cache/conftool/dbconfig/20221108-001704-ladsgroup.json
  • 00:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P38517 and previous config saved to /var/cache/conftool/dbconfig/20221108-000922-marostegui.json
  • 00:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T318605)', diff saved to https://phabricator.wikimedia.org/P38516 and previous config saved to /var/cache/conftool/dbconfig/20221108-000757-ladsgroup.json
  • 00:02 tgr: running foreachwikiindblist growthexperiments.dblist extensions/WikimediaMaintenance/createExtensionTables.php growthexperiments

2022-11-07

  • 23:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T318605)', diff saved to https://phabricator.wikimedia.org/P38515 and previous config saved to /var/cache/conftool/dbconfig/20221107-235526-ladsgroup.json
  • 23:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 23:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 23:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T318605)', diff saved to https://phabricator.wikimedia.org/P38514 and previous config saved to /var/cache/conftool/dbconfig/20221107-235505-ladsgroup.json
  • 23:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38513 and previous config saved to /var/cache/conftool/dbconfig/20221107-235415-marostegui.json
  • 23:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38512 and previous config saved to /var/cache/conftool/dbconfig/20221107-235206-marostegui.json
  • 23:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 23:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 23:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321123)', diff saved to https://phabricator.wikimedia.org/P38511 and previous config saved to /var/cache/conftool/dbconfig/20221107-235144-marostegui.json
  • 23:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P38510 and previous config saved to /var/cache/conftool/dbconfig/20221107-233637-marostegui.json
  • 23:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P38509 and previous config saved to /var/cache/conftool/dbconfig/20221107-232447-ladsgroup.json
  • 23:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P38508 and previous config saved to /var/cache/conftool/dbconfig/20221107-232131-marostegui.json
  • 23:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T318605)', diff saved to https://phabricator.wikimedia.org/P38507 and previous config saved to /var/cache/conftool/dbconfig/20221107-230940-ladsgroup.json
  • 23:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321123)', diff saved to https://phabricator.wikimedia.org/P38506 and previous config saved to /var/cache/conftool/dbconfig/20221107-230624-marostegui.json
  • 23:04 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2172 (T321123)', diff saved to https://phabricator.wikimedia.org/P38505 and previous config saved to /var/cache/conftool/dbconfig/20221107-230414-marostegui.json
  • 23:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 23:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 23:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321123)', diff saved to https://phabricator.wikimedia.org/P38504 and previous config saved to /var/cache/conftool/dbconfig/20221107-230353-marostegui.json
  • 22:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38503 and previous config saved to /var/cache/conftool/dbconfig/20221107-225943-marostegui.json
  • 22:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2159 (T318605)', diff saved to https://phabricator.wikimedia.org/P38502 and previous config saved to /var/cache/conftool/dbconfig/20221107-225602-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T318605)', diff saved to https://phabricator.wikimedia.org/P38501 and previous config saved to /var/cache/conftool/dbconfig/20221107-225536-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T318605)', diff saved to https://phabricator.wikimedia.org/P38500 and previous config saved to /var/cache/conftool/dbconfig/20221107-225525-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 22:53 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on phab2001.codfw.wmnet with reason: T322250
  • 22:53 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on phab2001.codfw.wmnet with reason: T322250
  • 22:51 mutante: phab2001 - removing from production puppet role - removes ssh access, ferm rules, exim config and more T322250
  • 22:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P38499 and previous config saved to /var/cache/conftool/dbconfig/20221107-224847-marostegui.json
  • 22:44 maryum: Deployed patches for T316414 and T315123
  • 22:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P38498 and previous config saved to /var/cache/conftool/dbconfig/20221107-224437-marostegui.json
  • 22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P38497 and previous config saved to /var/cache/conftool/dbconfig/20221107-224029-ladsgroup.json
  • 22:36 ejegg: fundraising CiviCRM upgraded from c0db8f34 to 72fccce1
  • 22:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P38496 and previous config saved to /var/cache/conftool/dbconfig/20221107-223340-marostegui.json
  • 22:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P38495 and previous config saved to /var/cache/conftool/dbconfig/20221107-222930-marostegui.json
  • 22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P38494 and previous config saved to /var/cache/conftool/dbconfig/20221107-222523-ladsgroup.json
  • 22:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321123)', diff saved to https://phabricator.wikimedia.org/P38493 and previous config saved to /var/cache/conftool/dbconfig/20221107-221834-marostegui.json
  • 22:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 22:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 22:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 22:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2155 (T321123)', diff saved to https://phabricator.wikimedia.org/P38492 and previous config saved to /var/cache/conftool/dbconfig/20221107-221624-marostegui.json
  • 22:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 22:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 22:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 22:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 22:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38491 and previous config saved to /var/cache/conftool/dbconfig/20221107-221557-marostegui.json
  • 22:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38490 and previous config saved to /var/cache/conftool/dbconfig/20221107-221423-marostegui.json
  • 22:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 22:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38489 and previous config saved to /var/cache/conftool/dbconfig/20221107-221209-marostegui.json
  • 22:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 22:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 22:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38488 and previous config saved to /var/cache/conftool/dbconfig/20221107-221148-marostegui.json
  • 22:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T318605)', diff saved to https://phabricator.wikimedia.org/P38487 and previous config saved to /var/cache/conftool/dbconfig/20221107-221016-ladsgroup.json
  • 22:07 mutante: [apt1001:~] $ sudo -E reprepro --verbose --component thirdparty/terraform update bullseye-wikimedia - T322344
  • 22:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P38486 and previous config saved to /var/cache/conftool/dbconfig/20221107-220051-marostegui.json
  • 21:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P38485 and previous config saved to /var/cache/conftool/dbconfig/20221107-215641-marostegui.json
  • 21:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P38484 and previous config saved to /var/cache/conftool/dbconfig/20221107-214545-marostegui.json
  • 21:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 21:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 21:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38483 and previous config saved to /var/cache/conftool/dbconfig/20221107-214254-ladsgroup.json
  • 21:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P38482 and previous config saved to /var/cache/conftool/dbconfig/20221107-214135-marostegui.json
  • 21:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38481 and previous config saved to /var/cache/conftool/dbconfig/20221107-213038-marostegui.json
  • 21:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318955)', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20221107-212900-ladsgroup.json
  • 21:28 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38480 and previous config saved to /var/cache/conftool/dbconfig/20221107-212828-marostegui.json
  • 21:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38479 and previous config saved to /var/cache/conftool/dbconfig/20221107-212800-marostegui.json
  • 21:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P38478 and previous config saved to /var/cache/conftool/dbconfig/20221107-212748-ladsgroup.json
  • 21:26 mutante: DNS - removing phab1001-aphlict.eqiad.wmnet - should have no effect because we use aphlict.discovery.wmnet - but if it does, then it's Phabricator realtime notifications - T280597
  • 21:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38477 and previous config saved to /var/cache/conftool/dbconfig/20221107-212628-marostegui.json
  • 21:26 urbanecm: Start [urbanecm@mwmaint1002 /srv/mediawiki]$ foreachwikiindblist group0 extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all # T315510, running in mwmaint1002 at a tmux session under my name
  • 21:25 mutante: DNS - removing phab1001-aphlict.eqiad.wmnet - should have no effect because we use aphlict.discovery.wmnet - but if it does, then it's Phabricator realtime notifications
  • 21:23 urbanecm@deploy1002: Finished scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353) (duration: 05m 47s)
  • 21:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38476 and previous config saved to /var/cache/conftool/dbconfig/20221107-212156-marostegui.json
  • 21:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 21:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 21:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38475 and previous config saved to /var/cache/conftool/dbconfig/20221107-212135-marostegui.json
  • 21:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:19 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:17 urbanecm@deploy1002: urbanecm and matmarex: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 21:17 urbanecm@deploy1002: Started scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353)
  • 21:16 urbanecm@deploy1002: Finished scap: Backport for ThreadItemStore: Update existing rows if possible rather than insert+delete (T321121) (duration: 07m 30s)
  • 21:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38474 and previous config saved to /var/cache/conftool/dbconfig/20221107-211353-ladsgroup.json
  • 21:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P38473 and previous config saved to /var/cache/conftool/dbconfig/20221107-211253-marostegui.json
  • 21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P38472 and previous config saved to /var/cache/conftool/dbconfig/20221107-211241-ladsgroup.json
  • 21:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:09 urbanecm@deploy1002: urbanecm and matmarex: Backport for ThreadItemStore: Update existing rows if possible rather than insert+delete (T321121) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:09 urbanecm@deploy1002: Started scap: Backport for ThreadItemStore: Update existing rows if possible rather than insert+delete (T321121)
  • 21:08 urbanecm@deploy1002: Finished scap: Backport for Simplify some redundant settings, Clean up wgDiscussionToolsABTest config for beta cluster (duration: 04m 40s)
  • 21:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P38471 and previous config saved to /var/cache/conftool/dbconfig/20221107-210628-marostegui.json
  • 21:03 urbanecm@deploy1002: Started scap: Backport for Simplify some redundant settings, Clean up wgDiscussionToolsABTest config for beta cluster
  • 21:02 urbanecm@deploy1002: backport aborted: (duration: 00m 02s)
  • 20:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38470 and previous config saved to /var/cache/conftool/dbconfig/20221107-205847-ladsgroup.json
  • 20:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P38469 and previous config saved to /var/cache/conftool/dbconfig/20221107-205747-marostegui.json
  • 20:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38468 and previous config saved to /var/cache/conftool/dbconfig/20221107-205735-ladsgroup.json
  • 20:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P38467 and previous config saved to /var/cache/conftool/dbconfig/20221107-205122-marostegui.json
  • 20:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2150 (T318605)', diff saved to https://phabricator.wikimedia.org/P38466 and previous config saved to /var/cache/conftool/dbconfig/20221107-204827-ladsgroup.json
  • 20:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 20:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 20:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T318605)', diff saved to https://phabricator.wikimedia.org/P38465 and previous config saved to /var/cache/conftool/dbconfig/20221107-204805-ladsgroup.json
  • 20:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38464 and previous config saved to /var/cache/conftool/dbconfig/20221107-204340-ladsgroup.json
  • 20:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38463 and previous config saved to /var/cache/conftool/dbconfig/20221107-204240-marostegui.json
  • 20:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38462 and previous config saved to /var/cache/conftool/dbconfig/20221107-204131-marostegui.json
  • 20:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 20:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 20:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38461 and previous config saved to /var/cache/conftool/dbconfig/20221107-204110-marostegui.json
  • 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38460 and previous config saved to /var/cache/conftool/dbconfig/20221107-203626-ladsgroup.json
  • 20:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38459 and previous config saved to /var/cache/conftool/dbconfig/20221107-203615-marostegui.json
  • 20:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 20:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38458 and previous config saved to /var/cache/conftool/dbconfig/20221107-203609-ladsgroup.json
  • 20:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P38457 and previous config saved to /var/cache/conftool/dbconfig/20221107-203258-ladsgroup.json
  • 20:31 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38456 and previous config saved to /var/cache/conftool/dbconfig/20221107-203138-marostegui.json
  • 20:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 20:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 20:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T321130)', diff saved to https://phabricator.wikimedia.org/P38455 and previous config saved to /var/cache/conftool/dbconfig/20221107-203116-marostegui.json
  • 20:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P38454 and previous config saved to /var/cache/conftool/dbconfig/20221107-202603-marostegui.json
  • 20:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38453 and previous config saved to /var/cache/conftool/dbconfig/20221107-202102-ladsgroup.json
  • 20:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P38452 and previous config saved to /var/cache/conftool/dbconfig/20221107-201752-ladsgroup.json
  • 20:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P38451 and previous config saved to /var/cache/conftool/dbconfig/20221107-201610-marostegui.json
  • 20:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P38450 and previous config saved to /var/cache/conftool/dbconfig/20221107-201057-marostegui.json
  • 20:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38449 and previous config saved to /var/cache/conftool/dbconfig/20221107-200556-ladsgroup.json
  • 20:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T318605)', diff saved to https://phabricator.wikimedia.org/P38448 and previous config saved to /var/cache/conftool/dbconfig/20221107-200245-ladsgroup.json
  • 20:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P38447 and previous config saved to /var/cache/conftool/dbconfig/20221107-200103-marostegui.json
  • 19:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38446 and previous config saved to /var/cache/conftool/dbconfig/20221107-195550-marostegui.json
  • 19:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38445 and previous config saved to /var/cache/conftool/dbconfig/20221107-195340-marostegui.json
  • 19:53 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 19:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 19:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T321123)', diff saved to https://phabricator.wikimedia.org/P38444 and previous config saved to /var/cache/conftool/dbconfig/20221107-195319-marostegui.json
  • 19:51 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 19:51 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host sretest2002
  • 19:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38443 and previous config saved to /var/cache/conftool/dbconfig/20221107-195049-ladsgroup.json
  • 19:50 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2002
  • 19:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T321130)', diff saved to https://phabricator.wikimedia.org/P38442 and previous config saved to /var/cache/conftool/dbconfig/20221107-194557-marostegui.json
  • 19:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38441 and previous config saved to /var/cache/conftool/dbconfig/20221107-194335-ladsgroup.json
  • 19:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 19:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 19:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 19:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 19:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38440 and previous config saved to /var/cache/conftool/dbconfig/20221107-194319-ladsgroup.json
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T321130)', diff saved to https://phabricator.wikimedia.org/P38439 and previous config saved to /var/cache/conftool/dbconfig/20221107-194026-marostegui.json
  • 19:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 19:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 19:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P38438 and previous config saved to /var/cache/conftool/dbconfig/20221107-193813-marostegui.json
  • 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38437 and previous config saved to /var/cache/conftool/dbconfig/20221107-193646-ladsgroup.json
  • 19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T318605)', diff saved to https://phabricator.wikimedia.org/P38436 and previous config saved to /var/cache/conftool/dbconfig/20221107-193625-ladsgroup.json
  • 19:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 19:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 19:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38435 and previous config saved to /var/cache/conftool/dbconfig/20221107-193604-marostegui.json
  • 19:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38434 and previous config saved to /var/cache/conftool/dbconfig/20221107-192813-ladsgroup.json
  • 19:25 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P38433 and previous config saved to /var/cache/conftool/dbconfig/20221107-192306-marostegui.json
  • 19:24 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:24 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P38432 and previous config saved to /var/cache/conftool/dbconfig/20221107-192119-ladsgroup.json
  • 19:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P38431 and previous config saved to /var/cache/conftool/dbconfig/20221107-192058-marostegui.json
  • 19:16 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38430 and previous config saved to /var/cache/conftool/dbconfig/20221107-191306-ladsgroup.json
  • 19:10 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T321123)', diff saved to https://phabricator.wikimedia.org/P38429 and previous config saved to /var/cache/conftool/dbconfig/20221107-190800-marostegui.json
  • 19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P38428 and previous config saved to /var/cache/conftool/dbconfig/20221107-190612-ladsgroup.json
  • 19:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P38427 and previous config saved to /var/cache/conftool/dbconfig/20221107-190551-marostegui.json
  • 19:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2136 (T321123)', diff saved to https://phabricator.wikimedia.org/P38426 and previous config saved to /var/cache/conftool/dbconfig/20221107-190550-marostegui.json
  • 19:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 19:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 19:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38425 and previous config saved to /var/cache/conftool/dbconfig/20221107-190528-marostegui.json
  • 18:58 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38424 and previous config saved to /var/cache/conftool/dbconfig/20221107-185800-ladsgroup.json
  • 18:57 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T318605)', diff saved to https://phabricator.wikimedia.org/P38423 and previous config saved to /var/cache/conftool/dbconfig/20221107-185105-ladsgroup.json
  • 18:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38422 and previous config saved to /var/cache/conftool/dbconfig/20221107-185044-marostegui.json
  • 18:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38421 and previous config saved to /var/cache/conftool/dbconfig/20221107-185035-ladsgroup.json
  • 18:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 18:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 18:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P38420 and previous config saved to /var/cache/conftool/dbconfig/20221107-185022-marostegui.json
  • 18:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38419 and previous config saved to /var/cache/conftool/dbconfig/20221107-184510-marostegui.json
  • 18:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 18:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 18:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38418 and previous config saved to /var/cache/conftool/dbconfig/20221107-184502-ladsgroup.json
  • 18:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 18:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321130)', diff saved to https://phabricator.wikimedia.org/P38417 and previous config saved to /var/cache/conftool/dbconfig/20221107-184448-marostegui.json
  • 18:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T318605)', diff saved to https://phabricator.wikimedia.org/P38416 and previous config saved to /var/cache/conftool/dbconfig/20221107-183722-ladsgroup.json
  • 18:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 18:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 18:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 18:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 18:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T318605)', diff saved to https://phabricator.wikimedia.org/P38415 and previous config saved to /var/cache/conftool/dbconfig/20221107-183643-ladsgroup.json
  • 18:36 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P38414 and previous config saved to /var/cache/conftool/dbconfig/20221107-183515-marostegui.json
  • 18:33 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host arclamp2001.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38413 and previous config saved to /var/cache/conftool/dbconfig/20221107-182956-ladsgroup.json
  • 18:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P38412 and previous config saved to /var/cache/conftool/dbconfig/20221107-182941-marostegui.json
  • 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2122 (T318605)', diff saved to https://phabricator.wikimedia.org/P38411 and previous config saved to /var/cache/conftool/dbconfig/20221107-182704-ladsgroup.json
  • 18:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 18:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38410 and previous config saved to /var/cache/conftool/dbconfig/20221107-182642-ladsgroup.json
  • 18:25 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host arclamp2001.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:24 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host puppetdb2003.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P38409 and previous config saved to /var/cache/conftool/dbconfig/20221107-182137-ladsgroup.json
  • 18:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38408 and previous config saved to /var/cache/conftool/dbconfig/20221107-182009-marostegui.json
  • 18:18 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host puppetdb2003.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38407 and previous config saved to /var/cache/conftool/dbconfig/20221107-181759-marostegui.json
  • 18:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 18:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 18:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321123)', diff saved to https://phabricator.wikimedia.org/P38406 and previous config saved to /var/cache/conftool/dbconfig/20221107-181737-marostegui.json
  • 18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38405 and previous config saved to /var/cache/conftool/dbconfig/20221107-181449-ladsgroup.json
  • 18:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P38404 and previous config saved to /var/cache/conftool/dbconfig/20221107-181435-marostegui.json
  • 18:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P38403 and previous config saved to /var/cache/conftool/dbconfig/20221107-181135-ladsgroup.json
  • 18:11 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host arclamp2001
  • 18:10 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host arclamp2001
  • 18:09 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host puppetdb2003
  • 18:08 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host puppetdb2003
  • 18:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P38402 and previous config saved to /var/cache/conftool/dbconfig/20221107-180630-ladsgroup.json
  • 18:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38401 and previous config saved to /var/cache/conftool/dbconfig/20221107-180230-marostegui.json
  • 18:00 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38400 and previous config saved to /var/cache/conftool/dbconfig/20221107-175943-ladsgroup.json
  • 17:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321130)', diff saved to https://phabricator.wikimedia.org/P38399 and previous config saved to /var/cache/conftool/dbconfig/20221107-175928-marostegui.json
  • 17:58 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P38398 and previous config saved to /var/cache/conftool/dbconfig/20221107-175629-ladsgroup.json
  • 17:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T321130)', diff saved to https://phabricator.wikimedia.org/P38397 and previous config saved to /var/cache/conftool/dbconfig/20221107-175357-marostegui.json
  • 17:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 17:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 17:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321130)', diff saved to https://phabricator.wikimedia.org/P38396 and previous config saved to /var/cache/conftool/dbconfig/20221107-175335-marostegui.json
  • 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38395 and previous config saved to /var/cache/conftool/dbconfig/20221107-175228-ladsgroup.json
  • 17:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 17:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38394 and previous config saved to /var/cache/conftool/dbconfig/20221107-175217-ladsgroup.json
  • 17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T318605)', diff saved to https://phabricator.wikimedia.org/P38393 and previous config saved to /var/cache/conftool/dbconfig/20221107-175124-ladsgroup.json
  • 17:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38392 and previous config saved to /var/cache/conftool/dbconfig/20221107-174724-marostegui.json
  • 17:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38391 and previous config saved to /var/cache/conftool/dbconfig/20221107-174123-ladsgroup.json
  • 17:41 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 17:38 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P38390 and previous config saved to /var/cache/conftool/dbconfig/20221107-173829-marostegui.json
  • 17:37 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 17:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38389 and previous config saved to /var/cache/conftool/dbconfig/20221107-173711-ladsgroup.json
  • 17:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321123)', diff saved to https://phabricator.wikimedia.org/P38388 and previous config saved to /var/cache/conftool/dbconfig/20221107-173217-marostegui.json
  • 17:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2110 (T321123)', diff saved to https://phabricator.wikimedia.org/P38387 and previous config saved to /var/cache/conftool/dbconfig/20221107-173007-marostegui.json
  • 17:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 17:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 17:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38386 and previous config saved to /var/cache/conftool/dbconfig/20221107-172946-marostegui.json
  • 17:24 krinkle@deploy1002: Finished deploy [performance/arc-lamp@e1ac118]: https://gerrit.wikimedia.org/r/c/825870 - T322561, T315056 (duration: 00m 07s)
  • 17:24 krinkle@deploy1002: Started deploy [performance/arc-lamp@e1ac118]: https://gerrit.wikimedia.org/r/c/825870 - T322561, T315056
  • 17:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P38385 and previous config saved to /var/cache/conftool/dbconfig/20221107-172322-marostegui.json
  • 17:22 sukhe: reprepro -C main include bullseye-wikimedia purged_0.19_amd64.changes: T321309
  • 17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38384 and previous config saved to /var/cache/conftool/dbconfig/20221107-172204-ladsgroup.json
  • 17:22 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 17:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P38383 and previous config saved to /var/cache/conftool/dbconfig/20221107-171439-marostegui.json
  • 17:13 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 17:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321130)', diff saved to https://phabricator.wikimedia.org/P38382 and previous config saved to /var/cache/conftool/dbconfig/20221107-170816-marostegui.json
  • 17:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38381 and previous config saved to /var/cache/conftool/dbconfig/20221107-170658-ladsgroup.json
  • 17:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T321130)', diff saved to https://phabricator.wikimedia.org/P38380 and previous config saved to /var/cache/conftool/dbconfig/20221107-170247-marostegui.json
  • 17:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 17:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 16:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38379 and previous config saved to /var/cache/conftool/dbconfig/20221107-165943-ladsgroup.json
  • 16:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P38378 and previous config saved to /var/cache/conftool/dbconfig/20221107-165933-marostegui.json
  • 16:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 16:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 16:59 filippo@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host dispatch-be2001.codfw.wmnet
  • 16:59 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 16:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 16:58 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 16:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T321130)', diff saved to https://phabricator.wikimedia.org/P38377 and previous config saved to /var/cache/conftool/dbconfig/20221107-165847-marostegui.json
  • 16:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T318605)', diff saved to https://phabricator.wikimedia.org/P38376 and previous config saved to /var/cache/conftool/dbconfig/20221107-165108-ladsgroup.json
  • 16:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T318605)', diff saved to https://phabricator.wikimedia.org/P38375 and previous config saved to /var/cache/conftool/dbconfig/20221107-165046-ladsgroup.json
  • 16:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T318955)', diff saved to https://phabricator.wikimedia.org/P38374 and previous config saved to /var/cache/conftool/dbconfig/20221107-165036-ladsgroup.json
  • 16:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38373 and previous config saved to /var/cache/conftool/dbconfig/20221107-164427-marostegui.json
  • 16:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P38372 and previous config saved to /var/cache/conftool/dbconfig/20221107-164340-marostegui.json
  • 16:42 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38371 and previous config saved to /var/cache/conftool/dbconfig/20221107-164217-marostegui.json
  • 16:42 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321123)', diff saved to https://phabricator.wikimedia.org/P38370 and previous config saved to /var/cache/conftool/dbconfig/20221107-164122-marostegui.json
  • 16:38 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:35 filippo@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dispatch-be2001.codfw.wmnet on all recursors
  • 16:35 filippo@cumin1001: START - Cookbook sre.dns.wipe-cache dispatch-be2001.codfw.wmnet on all recursors
  • 16:35 filippo@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38369 and previous config saved to /var/cache/conftool/dbconfig/20221107-163540-ladsgroup.json
  • 16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38368 and previous config saved to /var/cache/conftool/dbconfig/20221107-163529-ladsgroup.json
  • 16:33 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P38367 and previous config saved to /var/cache/conftool/dbconfig/20221107-162834-marostegui.json
  • 16:26 filippo@cumin1001: START - Cookbook sre.dns.netbox
  • 16:26 filippo@cumin1001: START - Cookbook sre.ganeti.makevm for new host dispatch-be2001.codfw.wmnet
  • 16:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P38366 and previous config saved to /var/cache/conftool/dbconfig/20221107-162616-marostegui.json
  • 16:23 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:21 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:21 volans@cumin1001: START - Cookbook sre.dns.netbox
  • 16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38365 and previous config saved to /var/cache/conftool/dbconfig/20221107-162033-ladsgroup.json
  • 16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38364 and previous config saved to /var/cache/conftool/dbconfig/20221107-162023-ladsgroup.json
  • 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38363 and previous config saved to /var/cache/conftool/dbconfig/20221107-161837-ladsgroup.json
  • 16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T318605)', diff saved to https://phabricator.wikimedia.org/P38362 and previous config saved to /var/cache/conftool/dbconfig/20221107-161816-ladsgroup.json
  • 16:14 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T321130)', diff saved to https://phabricator.wikimedia.org/P38361 and previous config saved to /var/cache/conftool/dbconfig/20221107-161327-marostegui.json
  • 16:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1201 (T321130)', diff saved to https://phabricator.wikimedia.org/P38360 and previous config saved to /var/cache/conftool/dbconfig/20221107-161118-marostegui.json
  • 16:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P38359 and previous config saved to /var/cache/conftool/dbconfig/20221107-161109-marostegui.json
  • 16:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 16:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 16:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T321130)', diff saved to https://phabricator.wikimedia.org/P38358 and previous config saved to /var/cache/conftool/dbconfig/20221107-161050-marostegui.json
  • 16:06 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 16:06 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@e51ff67]: import_cirrus_indexes: set executor cores to 1 (duration: 02m 19s)
  • 16:05 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 16:05 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T318605)', diff saved to https://phabricator.wikimedia.org/P38357 and previous config saved to /var/cache/conftool/dbconfig/20221107-160527-ladsgroup.json
  • 16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T318955)', diff saved to https://phabricator.wikimedia.org/P38356 and previous config saved to /var/cache/conftool/dbconfig/20221107-160516-ladsgroup.json
  • 16:04 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 16:03 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1010.eqiad.wmnet to cluster eqiad and group C
  • 16:03 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@e51ff67]: import_cirrus_indexes: set executor cores to 1
  • 16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P38355 and previous config saved to /var/cache/conftool/dbconfig/20221107-160310-ladsgroup.json
  • 16:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1010.eqiad.wmnet to cluster eqiad and group C
  • 16:02 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:02 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 16:02 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T318955)', diff saved to https://phabricator.wikimedia.org/P38354 and previous config saved to /var/cache/conftool/dbconfig/20221107-160124-ladsgroup.json
  • 16:01 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 16:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 16:01 claime: cleaning up stale mwdebug kubernetes config
  • 16:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T318955)', diff saved to https://phabricator.wikimedia.org/P38353 and previous config saved to /var/cache/conftool/dbconfig/20221107-160102-ladsgroup.json
  • 16:00 cgoubert@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 15:59 cgoubert@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 15:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321123)', diff saved to https://phabricator.wikimedia.org/P38352 and previous config saved to /var/cache/conftool/dbconfig/20221107-155603-marostegui.json
  • 15:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P38351 and previous config saved to /var/cache/conftool/dbconfig/20221107-155544-marostegui.json
  • 15:55 elukey: upgrade istioctl to 1.15.3 on apt1001 for {buster,bullseye}-wikimedia - T322193
  • 15:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1199 (T321123)', diff saved to https://phabricator.wikimedia.org/P38350 and previous config saved to /var/cache/conftool/dbconfig/20221107-155455-marostegui.json
  • 15:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 15:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 15:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321123)', diff saved to https://phabricator.wikimedia.org/P38349 and previous config saved to /var/cache/conftool/dbconfig/20221107-155434-marostegui.json
  • 15:51 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:50 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:50 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:49 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P38348 and previous config saved to /var/cache/conftool/dbconfig/20221107-154803-ladsgroup.json
  • 15:46 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:46 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38347 and previous config saved to /var/cache/conftool/dbconfig/20221107-154556-ladsgroup.json
  • 15:44 sukhe: reprepro -C main include bullseye-wikimedia libvmod-querysort_0.3_amd64.changes: T321309
  • 15:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P38346 and previous config saved to /var/cache/conftool/dbconfig/20221107-154037-marostegui.json
  • 15:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P38345 and previous config saved to /var/cache/conftool/dbconfig/20221107-153927-marostegui.json
  • 15:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T318605)', diff saved to https://phabricator.wikimedia.org/P38344 and previous config saved to /var/cache/conftool/dbconfig/20221107-153257-ladsgroup.json
  • 15:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38343 and previous config saved to /var/cache/conftool/dbconfig/20221107-153049-ladsgroup.json
  • 15:27 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:27 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T321130)', diff saved to https://phabricator.wikimedia.org/P38342 and previous config saved to /var/cache/conftool/dbconfig/20221107-152531-marostegui.json
  • 15:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P38341 and previous config saved to /var/cache/conftool/dbconfig/20221107-152421-marostegui.json
  • 15:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T321130)', diff saved to https://phabricator.wikimedia.org/P38340 and previous config saved to /var/cache/conftool/dbconfig/20221107-152322-marostegui.json
  • 15:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 15:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 15:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38339 and previous config saved to /var/cache/conftool/dbconfig/20221107-152301-marostegui.json
  • 15:18 sukhe: reprepro -C main include bullseye-wikimedia libvmod-netmapper_1.9-2_amd64.changes: T321309
  • 15:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T318955)', diff saved to https://phabricator.wikimedia.org/P38338 and previous config saved to /var/cache/conftool/dbconfig/20221107-151543-ladsgroup.json
  • 15:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T318955)', diff saved to https://phabricator.wikimedia.org/P38337 and previous config saved to /var/cache/conftool/dbconfig/20221107-151151-ladsgroup.json
  • 15:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 15:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 15:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T318955)', diff saved to https://phabricator.wikimedia.org/P38336 and previous config saved to /var/cache/conftool/dbconfig/20221107-151130-ladsgroup.json
  • 15:09 sukhe: reprepro -C main include bullseye-wikimedia varnishkafka_1.1.0-2_amd64.changes: T321309
  • 15:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321123)', diff saved to https://phabricator.wikimedia.org/P38335 and previous config saved to /var/cache/conftool/dbconfig/20221107-150914-marostegui.json
  • 15:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1190 (T321123)', diff saved to https://phabricator.wikimedia.org/P38334 and previous config saved to /var/cache/conftool/dbconfig/20221107-150807-marostegui.json
  • 15:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 15:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P38333 and previous config saved to /var/cache/conftool/dbconfig/20221107-150754-marostegui.json
  • 15:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 15:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321123)', diff saved to https://phabricator.wikimedia.org/P38332 and previous config saved to /var/cache/conftool/dbconfig/20221107-150745-marostegui.json
  • 14:57 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1010.eqiad.wmnet
  • 14:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38331 and previous config saved to /var/cache/conftool/dbconfig/20221107-145623-ladsgroup.json
  • 14:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P38330 and previous config saved to /var/cache/conftool/dbconfig/20221107-145248-marostegui.json
  • 14:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P38329 and previous config saved to /var/cache/conftool/dbconfig/20221107-145239-marostegui.json
  • 14:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1010.eqiad.wmnet
  • 14:43 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38328 and previous config saved to /var/cache/conftool/dbconfig/20221107-144117-ladsgroup.json
  • 14:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38327 and previous config saved to /var/cache/conftool/dbconfig/20221107-143741-marostegui.json
  • 14:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P38326 and previous config saved to /var/cache/conftool/dbconfig/20221107-143732-marostegui.json
  • 14:37 urbanecm: UTC afternoon B&C window done
  • 14:36 urbanecm@deploy1002: Finished scap: Backport for MentorHooks: Add missing check for GEMentorshipUseIsActiveFlag (T322538), Rename QuitMentorship to ReassignMentees (T321382), ReassignMentees: Pass the actual performer to ChangeMentor (T321382), ManageMentorsRemoveMentor: Reassign mentees to a different mentor (T321382) (duration: 14m 56s)
  • 14:35 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38324 and previous config saved to /var/cache/conftool/dbconfig/20221107-143526-marostegui.json
  • 14:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 14:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 14:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321130)', diff saved to https://phabricator.wikimedia.org/P38323 and previous config saved to /var/cache/conftool/dbconfig/20221107-143504-marostegui.json
  • 14:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T318605)', diff saved to https://phabricator.wikimedia.org/P38322 and previous config saved to /var/cache/conftool/dbconfig/20221107-142904-ladsgroup.json
  • 14:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 14:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38321 and previous config saved to /var/cache/conftool/dbconfig/20221107-142832-ladsgroup.json
  • 14:26 urbanecm@deploy1002: urbanecm and urbanecm: Backport for MentorHooks: Add missing check for GEMentorshipUseIsActiveFlag (T322538), Rename QuitMentorship to ReassignMentees (T321382), ReassignMentees: Pass the actual performer to ChangeMentor (T321382), ManageMentorsRemoveMentor: Reassign mentees to a different mentor (T321382) synced to the testse
  • 14:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T318955)', diff saved to https://phabricator.wikimedia.org/P38320 and previous config saved to /var/cache/conftool/dbconfig/20221107-142610-ladsgroup.json
  • 14:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321123)', diff saved to https://phabricator.wikimedia.org/P38319 and previous config saved to /var/cache/conftool/dbconfig/20221107-142226-marostegui.json
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T318955)', diff saved to https://phabricator.wikimedia.org/P38318 and previous config saved to /var/cache/conftool/dbconfig/20221107-142219-ladsgroup.json
  • 14:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 14:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T318955)', diff saved to https://phabricator.wikimedia.org/P38317 and previous config saved to /var/cache/conftool/dbconfig/20221107-142157-ladsgroup.json
  • 14:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1010.eqiad.wmnet with OS bullseye
  • 14:21 urbanecm@deploy1002: Started scap: Backport for MentorHooks: Add missing check for GEMentorshipUseIsActiveFlag (T322538), Rename QuitMentorship to ReassignMentees (T321382), ReassignMentees: Pass the actual performer to ChangeMentor (T321382), ManageMentorsRemoveMentor: Reassign mentees to a different mentor (T321382)
  • 14:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T321123)', diff saved to https://phabricator.wikimedia.org/P38316 and previous config saved to /var/cache/conftool/dbconfig/20221107-142118-marostegui.json
  • 14:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 14:21 urbanecm@deploy1002: Backport cancelled.
  • 14:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 14:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 14:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 14:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38315 and previous config saved to /var/cache/conftool/dbconfig/20221107-142041-marostegui.json
  • 14:20 urbanecm@deploy1002: Finished scap: Backport for Set timezone for knwiki , knwiktionary , knwikiquote and knwikisource (T322471) (duration: 06m 01s)
  • 14:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P38314 and previous config saved to /var/cache/conftool/dbconfig/20221107-141958-marostegui.json
  • 14:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:18 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:14 urbanecm@deploy1002: urbanecm and anzx: Backport for Set timezone for knwiki , knwiktionary , knwikiquote and knwikisource (T322471) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 14:14 urbanecm@deploy1002: Started scap: Backport for Set timezone for knwiki , knwiktionary , knwikiquote and knwikisource (T322471)
  • 14:13 urbanecm@deploy1002: Finished scap: Backport for Enable flood flag on knwiki (T322472) (duration: 05m 10s)
  • 14:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38313 and previous config saved to /var/cache/conftool/dbconfig/20221107-141344-ladsgroup.json
  • 14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P38312 and previous config saved to /var/cache/conftool/dbconfig/20221107-141326-ladsgroup.json
  • 14:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2120 (T318605)', diff saved to https://phabricator.wikimedia.org/P38311 and previous config saved to /var/cache/conftool/dbconfig/20221107-141224-ladsgroup.json
  • 14:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 14:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T318605)', diff saved to https://phabricator.wikimedia.org/P38310 and previous config saved to /var/cache/conftool/dbconfig/20221107-141203-ladsgroup.json
  • 14:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:09 xcollazo@deploy1002: Finished deploy [airflow-dags/platform_eng@3bb99c2]: Deploying to Airflow platform_eng instance (duration: 00m 20s)
  • 14:09 urbanecm@deploy1002: urbanecm and anzx: Backport for Enable flood flag on knwiki (T322472) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 14:09 xcollazo@deploy1002: Started deploy [airflow-dags/platform_eng@3bb99c2]: Deploying to Airflow platform_eng instance
  • 14:08 urbanecm@deploy1002: Started scap: Backport for Enable flood flag on knwiki (T322472)
  • 14:08 urbanecm@deploy1002: Finished scap: Backport for testwiki: Add config for Visual Editor Feature Use instrument (T309602) (duration: 06m 43s)
  • 14:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38309 and previous config saved to /var/cache/conftool/dbconfig/20221107-140651-ladsgroup.json
  • 14:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P38308 and previous config saved to /var/cache/conftool/dbconfig/20221107-140535-marostegui.json
  • 14:05 btullis@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:05 btullis@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 14:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1010.eqiad.wmnet with reason: host reimage
  • 14:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P38307 and previous config saved to /var/cache/conftool/dbconfig/20221107-140451-marostegui.json
  • 14:02 urbanecm@deploy1002: urbanecm and cjming: Backport for testwiki: Add config for Visual Editor Feature Use instrument (T309602) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 14:01 urbanecm@deploy1002: Started scap: Backport for testwiki: Add config for Visual Editor Feature Use instrument (T309602)
  • 14:01 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1010.eqiad.wmnet with reason: host reimage
  • 13:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38306 and previous config saved to /var/cache/conftool/dbconfig/20221107-135837-ladsgroup.json
  • 13:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P38305 and previous config saved to /var/cache/conftool/dbconfig/20221107-135819-ladsgroup.json
  • 13:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P38304 and previous config saved to /var/cache/conftool/dbconfig/20221107-135656-ladsgroup.json
  • 13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38303 and previous config saved to /var/cache/conftool/dbconfig/20221107-135144-ladsgroup.json
  • 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P38302 and previous config saved to /var/cache/conftool/dbconfig/20221107-135028-marostegui.json
  • 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321130)', diff saved to https://phabricator.wikimedia.org/P38301 and previous config saved to /var/cache/conftool/dbconfig/20221107-134944-marostegui.json
  • 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T321130)', diff saved to https://phabricator.wikimedia.org/P38300 and previous config saved to /var/cache/conftool/dbconfig/20221107-134735-marostegui.json
  • 13:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 13:47 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1010.eqiad.wmnet with OS bullseye
  • 13:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321130)', diff saved to https://phabricator.wikimedia.org/P38299 and previous config saved to /var/cache/conftool/dbconfig/20221107-134714-marostegui.json
  • 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38298 and previous config saved to /var/cache/conftool/dbconfig/20221107-134331-ladsgroup.json
  • 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38297 and previous config saved to /var/cache/conftool/dbconfig/20221107-134313-ladsgroup.json
  • 13:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P38296 and previous config saved to /var/cache/conftool/dbconfig/20221107-134150-ladsgroup.json
  • 13:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T318955)', diff saved to https://phabricator.wikimedia.org/P38295 and previous config saved to /var/cache/conftool/dbconfig/20221107-133638-ladsgroup.json
  • 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38294 and previous config saved to /var/cache/conftool/dbconfig/20221107-133522-marostegui.json
  • 13:34 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38293 and previous config saved to /var/cache/conftool/dbconfig/20221107-133414-marostegui.json
  • 13:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 13:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321123)', diff saved to https://phabricator.wikimedia.org/P38292 and previous config saved to /var/cache/conftool/dbconfig/20221107-133353-marostegui.json
  • 13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T318955)', diff saved to https://phabricator.wikimedia.org/P38291 and previous config saved to /var/cache/conftool/dbconfig/20221107-133246-ladsgroup.json
  • 13:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 13:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T318955)', diff saved to https://phabricator.wikimedia.org/P38290 and previous config saved to /var/cache/conftool/dbconfig/20221107-133225-ladsgroup.json
  • 13:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P38289 and previous config saved to /var/cache/conftool/dbconfig/20221107-133208-marostegui.json
  • 13:29 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1010.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 13:29 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1010.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38288 and previous config saved to /var/cache/conftool/dbconfig/20221107-132824-ladsgroup.json
  • 13:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T318605)', diff saved to https://phabricator.wikimedia.org/P38287 and previous config saved to /var/cache/conftool/dbconfig/20221107-132643-ladsgroup.json
  • 13:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38286 and previous config saved to /var/cache/conftool/dbconfig/20221107-132109-ladsgroup.json
  • 13:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 13:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38285 and previous config saved to /var/cache/conftool/dbconfig/20221107-132048-ladsgroup.json
  • 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P38284 and previous config saved to /var/cache/conftool/dbconfig/20221107-131846-marostegui.json
  • 13:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38283 and previous config saved to /var/cache/conftool/dbconfig/20221107-131718-ladsgroup.json
  • 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P38282 and previous config saved to /var/cache/conftool/dbconfig/20221107-131701-marostegui.json
  • 13:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38281 and previous config saved to /var/cache/conftool/dbconfig/20221107-130541-ladsgroup.json
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P38280 and previous config saved to /var/cache/conftool/dbconfig/20221107-130340-marostegui.json
  • 13:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38279 and previous config saved to /var/cache/conftool/dbconfig/20221107-130212-ladsgroup.json
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321130)', diff saved to https://phabricator.wikimedia.org/P38278 and previous config saved to /var/cache/conftool/dbconfig/20221107-130155-marostegui.json
  • 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T321130)', diff saved to https://phabricator.wikimedia.org/P38277 and previous config saved to /var/cache/conftool/dbconfig/20221107-125946-marostegui.json
  • 12:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T321130)', diff saved to https://phabricator.wikimedia.org/P38276 and previous config saved to /var/cache/conftool/dbconfig/20221107-125529-marostegui.json
  • 12:51 XioNoX: Add NAT for frmon2001 - T321735
  • 12:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38275 and previous config saved to /var/cache/conftool/dbconfig/20221107-125035-ladsgroup.json
  • 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321123)', diff saved to https://phabricator.wikimedia.org/P38274 and previous config saved to /var/cache/conftool/dbconfig/20221107-124833-marostegui.json
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T318955)', diff saved to https://phabricator.wikimedia.org/P38273 and previous config saved to /var/cache/conftool/dbconfig/20221107-124706-ladsgroup.json
  • 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T321123)', diff saved to https://phabricator.wikimedia.org/P38272 and previous config saved to /var/cache/conftool/dbconfig/20221107-124526-marostegui.json
  • 12:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 12:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38271 and previous config saved to /var/cache/conftool/dbconfig/20221107-124504-marostegui.json
  • 12:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P38270 and previous config saved to /var/cache/conftool/dbconfig/20221107-124022-marostegui.json
  • 12:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38269 and previous config saved to /var/cache/conftool/dbconfig/20221107-123528-ladsgroup.json
  • 12:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P38268 and previous config saved to /var/cache/conftool/dbconfig/20221107-122957-marostegui.json
  • 12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38267 and previous config saved to /var/cache/conftool/dbconfig/20221107-122814-ladsgroup.json
  • 12:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38266 and previous config saved to /var/cache/conftool/dbconfig/20221107-122737-ladsgroup.json
  • 12:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P38265 and previous config saved to /var/cache/conftool/dbconfig/20221107-122516-marostegui.json
  • 12:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P38264 and previous config saved to /var/cache/conftool/dbconfig/20221107-121451-marostegui.json
  • 12:13 sukhe: reprepro -C main include bullseye-wikimedia prometheus-varnishkafka-exporter_0.1-2_amd64.changes: T321309
  • 12:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38263 and previous config saved to /var/cache/conftool/dbconfig/20221107-121230-ladsgroup.json
  • 12:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T321130)', diff saved to https://phabricator.wikimedia.org/P38262 and previous config saved to /var/cache/conftool/dbconfig/20221107-121009-marostegui.json
  • 12:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38261 and previous config saved to /var/cache/conftool/dbconfig/20221107-121004-ladsgroup.json
  • 12:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 12:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 12:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38260 and previous config saved to /var/cache/conftool/dbconfig/20221107-120942-ladsgroup.json
  • 12:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T321130)', diff saved to https://phabricator.wikimedia.org/P38259 and previous config saved to /var/cache/conftool/dbconfig/20221107-120800-marostegui.json
  • 12:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 12:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 12:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38258 and previous config saved to /var/cache/conftool/dbconfig/20221107-120739-marostegui.json
  • 12:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2108 (T318605)', diff saved to https://phabricator.wikimedia.org/P38257 and previous config saved to /var/cache/conftool/dbconfig/20221107-120614-ladsgroup.json
  • 12:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 12:05 vgutierrez: testing acme-chief 0.35 in acmechief-test1001
  • 12:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 12:05 volans@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts ganeti4003.ulsfo.wmnet
  • 12:05 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:04 volans@cumin1001: START - Cookbook sre.dns.netbox
  • 12:00 volans@cumin1001: START - Cookbook sre.hosts.decommission for hosts ganeti4003.ulsfo.wmnet
  • 11:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38256 and previous config saved to /var/cache/conftool/dbconfig/20221107-115944-marostegui.json
  • 11:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38255 and previous config saved to /var/cache/conftool/dbconfig/20221107-115737-marostegui.json
  • 11:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 11:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38254 and previous config saved to /var/cache/conftool/dbconfig/20221107-115723-ladsgroup.json
  • 11:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 11:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38253 and previous config saved to /var/cache/conftool/dbconfig/20221107-115715-marostegui.json
  • 11:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P38252 and previous config saved to /var/cache/conftool/dbconfig/20221107-115436-ladsgroup.json
  • 11:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P38251 and previous config saved to /var/cache/conftool/dbconfig/20221107-115232-marostegui.json
  • 11:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T318955)', diff saved to https://phabricator.wikimedia.org/P38250 and previous config saved to /var/cache/conftool/dbconfig/20221107-114649-ladsgroup.json
  • 11:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318955)', diff saved to https://phabricator.wikimedia.org/P38249 and previous config saved to /var/cache/conftool/dbconfig/20221107-114628-ladsgroup.json
  • 11:44 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 112
  • 11:44 ayounsi@cumin1001: START - Cookbook sre.network.debug for Netbox circuit ID 112
  • 11:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38248 and previous config saved to /var/cache/conftool/dbconfig/20221107-114217-ladsgroup.json
  • 11:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P38247 and previous config saved to /var/cache/conftool/dbconfig/20221107-114209-marostegui.json
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P38246 and previous config saved to /var/cache/conftool/dbconfig/20221107-113929-ladsgroup.json
  • 11:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P38245 and previous config saved to /var/cache/conftool/dbconfig/20221107-113726-marostegui.json
  • 11:36 arturo: running homer on cr-eqiad/cr-codfw for https://gerrit.wikimedia.org/r/853947 (T321220, T309407)
  • 11:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38244 and previous config saved to /var/cache/conftool/dbconfig/20221107-113452-ladsgroup.json
  • 11:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 11:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 11:32 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38243 and previous config saved to /var/cache/conftool/dbconfig/20221107-113224-root.json
  • 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38242 and previous config saved to /var/cache/conftool/dbconfig/20221107-113122-ladsgroup.json
  • 11:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 11:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38241 and previous config saved to /var/cache/conftool/dbconfig/20221107-112857-ladsgroup.json
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P38240 and previous config saved to /var/cache/conftool/dbconfig/20221107-112702-marostegui.json
  • 11:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38239 and previous config saved to /var/cache/conftool/dbconfig/20221107-112423-ladsgroup.json
  • 11:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38238 and previous config saved to /var/cache/conftool/dbconfig/20221107-112219-marostegui.json
  • 11:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd1002.eqiad.wmnet to plain
  • 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd1002.eqiad.wmnet to plain
  • 11:17 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38237 and previous config saved to /var/cache/conftool/dbconfig/20221107-111719-root.json
  • 11:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38236 and previous config saved to /var/cache/conftool/dbconfig/20221107-111637-marostegui.json
  • 11:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 11:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 11:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38235 and previous config saved to /var/cache/conftool/dbconfig/20221107-111616-marostegui.json
  • 11:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38234 and previous config saved to /var/cache/conftool/dbconfig/20221107-111615-ladsgroup.json
  • 11:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38233 and previous config saved to /var/cache/conftool/dbconfig/20221107-111351-ladsgroup.json
  • 11:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38232 and previous config saved to /var/cache/conftool/dbconfig/20221107-111156-marostegui.json
  • 11:06 arturo: running homer on cr-eqiad/cr-codfw for https://gerrit.wikimedia.org/r/c/operations/homer/public/+/853374 (T321220, T309407)
  • 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd1002.eqiad.wmnet to drbd
  • 11:04 _joe_: manually started dump_cloud_ip_ranges.service
  • 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38231 and previous config saved to /var/cache/conftool/dbconfig/20221107-110215-root.json
  • 11:01 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 11:01 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 11:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P38230 and previous config saved to /var/cache/conftool/dbconfig/20221107-110110-marostegui.json
  • 11:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318955)', diff saved to https://phabricator.wikimedia.org/P38229 and previous config saved to /var/cache/conftool/dbconfig/20221107-110109-ladsgroup.json
  • 10:59 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 10:59 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38228 and previous config saved to /var/cache/conftool/dbconfig/20221107-105844-ladsgroup.json
  • 10:57 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 10:57 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 10:56 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd1002.eqiad.wmnet to drbd
  • 10:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T318955)', diff saved to https://phabricator.wikimedia.org/P38227 and previous config saved to /var/cache/conftool/dbconfig/20221107-105015-ladsgroup.json
  • 10:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 10:47 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38226 and previous config saved to /var/cache/conftool/dbconfig/20221107-104710-root.json
  • 10:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P38225 and previous config saved to /var/cache/conftool/dbconfig/20221107-104603-marostegui.json
  • 10:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38224 and previous config saved to /var/cache/conftool/dbconfig/20221107-104338-ladsgroup.json
  • 10:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 10:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 10:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T318955)', diff saved to https://phabricator.wikimedia.org/P38223 and previous config saved to /var/cache/conftool/dbconfig/20221107-104101-ladsgroup.json
  • 10:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38222 and previous config saved to /var/cache/conftool/dbconfig/20221107-103622-ladsgroup.json
  • 10:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 10:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 10:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38221 and previous config saved to /var/cache/conftool/dbconfig/20221107-103549-ladsgroup.json
  • 10:32 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 5398
  • 10:32 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38220 and previous config saved to /var/cache/conftool/dbconfig/20221107-103205-root.json
  • 10:31 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 5398
  • 10:31 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30103
  • 10:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38219 and previous config saved to /var/cache/conftool/dbconfig/20221107-103056-marostegui.json
  • 10:30 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 30103
  • 10:29 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 399338
  • 10:29 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 399338
  • 10:28 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 59796
  • 10:28 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 59796
  • 10:27 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4817
  • 10:26 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 4817
  • 10:26 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 35598
  • 10:26 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 35598
  • 10:26 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30058
  • 10:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38218 and previous config saved to /var/cache/conftool/dbconfig/20221107-102555-ladsgroup.json
  • 10:25 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 30058
  • 10:25 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 46416
  • 10:25 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 46416
  • 10:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38217 and previous config saved to /var/cache/conftool/dbconfig/20221107-102509-marostegui.json
  • 10:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 10:25 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 10:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38216 and previous config saved to /var/cache/conftool/dbconfig/20221107-102458-marostegui.json
  • 10:24 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 3214
  • 10:24 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 3214
  • 10:21 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 17511
  • 10:21 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 17511
  • 10:21 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 12400
  • 10:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38215 and previous config saved to /var/cache/conftool/dbconfig/20221107-102043-ladsgroup.json
  • 10:20 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 12400
  • 10:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 7459
  • 10:19 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 7459
  • 10:17 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 6661
  • 10:17 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 6661
  • 10:17 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 5398
  • 10:17 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 5398
  • 10:17 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38214 and previous config saved to /var/cache/conftool/dbconfig/20221107-101700-root.json
  • 10:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38213 and previous config saved to /var/cache/conftool/dbconfig/20221107-101140-marostegui.json
  • 10:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38212 and previous config saved to /var/cache/conftool/dbconfig/20221107-101102-marostegui.json
  • 10:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38211 and previous config saved to /var/cache/conftool/dbconfig/20221107-101048-ladsgroup.json
  • 10:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38210 and previous config saved to /var/cache/conftool/dbconfig/20221107-100952-marostegui.json
  • 10:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38209 and previous config saved to /var/cache/conftool/dbconfig/20221107-100536-ladsgroup.json
  • 10:01 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38208 and previous config saved to /var/cache/conftool/dbconfig/20221107-100155-root.json
  • 09:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P38207 and previous config saved to /var/cache/conftool/dbconfig/20221107-095556-marostegui.json
  • 09:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T318955)', diff saved to https://phabricator.wikimedia.org/P38206 and previous config saved to /var/cache/conftool/dbconfig/20221107-095542-ladsgroup.json
  • 09:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38205 and previous config saved to /var/cache/conftool/dbconfig/20221107-095445-marostegui.json
  • 09:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T318955)', diff saved to https://phabricator.wikimedia.org/P38204 and previous config saved to /var/cache/conftool/dbconfig/20221107-095149-ladsgroup.json
  • 09:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 09:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 09:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38203 and previous config saved to /var/cache/conftool/dbconfig/20221107-095030-ladsgroup.json
  • 09:49 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 09:48 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 09:48 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudmetrics1004.eqiad.wmnet
  • 09:46 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38202 and previous config saved to /var/cache/conftool/dbconfig/20221107-094650-root.json
  • 09:46 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 09:46 cgoubert@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 09:45 cgoubert@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 09:44 cgoubert@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 09:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38201 and previous config saved to /var/cache/conftool/dbconfig/20221107-094315-ladsgroup.json
  • 09:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 09:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 09:42 cgoubert@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 09:42 ladsgroup@deploy1002: Finished scap: Backport for pruneRevData: Make it reload config (duration: 08m 10s)
  • 09:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 09:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 09:41 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 09:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 09:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P38200 and previous config saved to /var/cache/conftool/dbconfig/20221107-094050-marostegui.json
  • 09:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 09:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 09:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38199 and previous config saved to /var/cache/conftool/dbconfig/20221107-093939-marostegui.json
  • 09:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudmetrics1004.eqiad.wmnet
  • 09:38 elukey: restart rsyslog on ml-serve2001
  • 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudmetrics1003.eqiad.wmnet
  • 09:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38198 and previous config saved to /var/cache/conftool/dbconfig/20221107-093629-ladsgroup.json
  • 09:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for pruneRevData: Make it reload config synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 09:34 ladsgroup@deploy1002: Started scap: Backport for pruneRevData: Make it reload config
  • 09:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38197 and previous config saved to /var/cache/conftool/dbconfig/20221107-093352-marostegui.json
  • 09:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 09:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 09:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudmetrics1003.eqiad.wmnet
  • 09:29 moritzm: installing Django security updates
  • 09:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38196 and previous config saved to /var/cache/conftool/dbconfig/20221107-092543-marostegui.json
  • 09:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38195 and previous config saved to /var/cache/conftool/dbconfig/20221107-092436-marostegui.json
  • 09:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 09:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 09:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T321123)', diff saved to https://phabricator.wikimedia.org/P38194 and previous config saved to /var/cache/conftool/dbconfig/20221107-092414-marostegui.json
  • 09:18 moritzm: draining ganeti1010 for eventual reimage T311687
  • 09:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P38193 and previous config saved to /var/cache/conftool/dbconfig/20221107-090908-marostegui.json
  • 09:08 Emperor: set thanos ring replicas to 3.30 T311690
  • 09:06 urbanecm@deploy1002: Finished scap: Backport for Set wmgVisualEditorAccessRestbaseDirectly = false for testwiki and dewiki.beta (T320531) (duration: 09m 07s)
  • 09:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 09:03 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 09:03 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 09:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:57 urbanecm@deploy1002: urbanecm and daniel: Backport for Set wmgVisualEditorAccessRestbaseDirectly = false for testwiki and dewiki.beta (T320531) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 08:57 urbanecm@deploy1002: Started scap: Backport for Set wmgVisualEditorAccessRestbaseDirectly = false for testwiki and dewiki.beta (T320531)
  • 08:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P38192 and previous config saved to /var/cache/conftool/dbconfig/20221107-085402-marostegui.json
  • 08:52 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:50 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:49 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc3 master" (duration: 04m 22s)
  • 08:45 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc2014 to pc3 master" synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 08:45 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc3 master"
  • 08:40 urbanecm: UTC morning B&C window done
  • 08:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T321123)', diff saved to https://phabricator.wikimedia.org/P38191 and previous config saved to /var/cache/conftool/dbconfig/20221107-083855-marostegui.json
  • 08:38 urbanecm@deploy1002: Finished scap: Backport for ContentTranslation: Move haw, ps and xh Wikipedias out of Beta (duration: 06m 56s)
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T321123)', diff saved to https://phabricator.wikimedia.org/P38190 and previous config saved to /var/cache/conftool/dbconfig/20221107-083648-marostegui.json
  • 08:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 08:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321123)', diff saved to https://phabricator.wikimedia.org/P38189 and previous config saved to /var/cache/conftool/dbconfig/20221107-083626-marostegui.json
  • 08:35 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:33 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:31 urbanecm@deploy1002: urbanecm and kartik: Backport for ContentTranslation: Move haw, ps and xh Wikipedias out of Beta synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 08:31 urbanecm@deploy1002: Started scap: Backport for ContentTranslation: Move haw, ps and xh Wikipedias out of Beta
  • 08:30 urbanecm@deploy1002: Finished scap: Backport for Set ContentTranslation MT threshold to 75 in Japanese WP (T321819) (duration: 06m 24s)
  • 08:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:24 urbanecm@deploy1002: urbanecm and kartik: Backport for Set ContentTranslation MT threshold to 75 in Japanese WP (T321819) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 08:24 urbanecm@deploy1002: Started scap: Backport for Set ContentTranslation MT threshold to 75 in Japanese WP (T321819)
  • 08:23 urbanecm@deploy1002: Finished scap: Backport for Set VisualEditorDefaultParsoidClient for dewiki-beta mad testwiki (T320531) (duration: 18m 54s)
  • 08:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38188 and previous config saved to /var/cache/conftool/dbconfig/20221107-082120-marostegui.json
  • 08:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38185 and previous config saved to /var/cache/conftool/dbconfig/20221107-080613-marostegui.json
  • 08:05 urbanecm@deploy1002: urbanecm and daniel: Backport for Set VisualEditorDefaultParsoidClient for dewiki-beta mad testwiki (T320531) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 08:04 urbanecm@deploy1002: Started scap: Backport for Set VisualEditorDefaultParsoidClient for dewiki-beta mad testwiki (T320531)
  • 08:03 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:01 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:00 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc2014 to pc3 master (duration: 04m 18s)
  • 07:56 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc2014 to pc3 master synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 07:55 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc2014 to pc3 master
  • 07:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321123)', diff saved to https://phabricator.wikimedia.org/P38184 and previous config saved to /var/cache/conftool/dbconfig/20221107-075106-marostegui.json
  • 07:50 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:50 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T321123)', diff saved to https://phabricator.wikimedia.org/P38183 and previous config saved to /var/cache/conftool/dbconfig/20221107-074959-marostegui.json
  • 07:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 07:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 07:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38182 and previous config saved to /var/cache/conftool/dbconfig/20221107-074938-marostegui.json
  • 07:47 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc1014 to pc1 master" (duration: 04m 53s)
  • 07:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:44 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=frwiki` in a tmux at mwmaint1002 (T318457)
  • 07:42 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc1014 to pc1 master" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 07:42 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc1014 to pc1 master"
  • 07:37 elukey: `elukey@aux-k8s-worker1002:~$ sudo systemctl reset-failed ifup@ens13.service`
  • 07:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38181 and previous config saved to /var/cache/conftool/dbconfig/20221107-073431-marostegui.json
  • 07:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38180 and previous config saved to /var/cache/conftool/dbconfig/20221107-071925-marostegui.json
  • 07:17 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc1014 to pc1 master (T322295) (duration: 04m 29s)
  • 07:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:13 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc1014 to pc1 master (T322295) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
  • 07:13 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc1014 to pc1 master (T322295)
  • 07:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2011.codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover
  • 07:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on pc2011.codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover
  • 07:05 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=bnwiki` in a tmux at mwmaint1002 (T318457)
  • 07:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38179 and previous config saved to /var/cache/conftool/dbconfig/20221107-070418-marostegui.json
  • 07:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38178 and previous config saved to /var/cache/conftool/dbconfig/20221107-070311-marostegui.json
  • 07:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 07:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 07:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38177 and previous config saved to /var/cache/conftool/dbconfig/20221107-070249-marostegui.json
  • 07:02 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=cswiki` in a tmux at mwmaint1002 (T318457)
  • 07:01 urbanecm@deploy1002: Finished scap: Backport for Add support for gemm_mentee_is_active (T318457), Calculate mentorship-related metrics (T318684) (duration: 06m 27s)
  • 06:55 urbanecm@deploy1002: urbanecm and urbanecm: Backport for Add support for gemm_mentee_is_active (T318457), Calculate mentorship-related metrics (T318684) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 06:55 urbanecm@deploy1002: Started scap: Backport for Add support for gemm_mentee_is_active (T318457), Calculate mentorship-related metrics (T318684)
  • 06:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es2024 T322406', diff saved to https://phabricator.wikimedia.org/P38176 and previous config saved to /var/cache/conftool/dbconfig/20221107-065251-root.json
  • 06:50 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es2023 to es5 primary and set section read-write T322406', diff saved to https://phabricator.wikimedia.org/P38175 and previous config saved to /var/cache/conftool/dbconfig/20221107-065048-root.json
  • 06:49 marostegui: Starting es5 codfw failover from es2024 to es2023 - T322406
  • 06:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38174 and previous config saved to /var/cache/conftool/dbconfig/20221107-064743-marostegui.json
  • 06:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322406
  • 06:46 marostegui@cumin1001: dbctl commit (dc=all): 'Set es2023 with weight 0 T322406', diff saved to https://phabricator.wikimedia.org/P38173 and previous config saved to /var/cache/conftool/dbconfig/20221107-064608-root.json
  • 06:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322406
  • 06:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38172 and previous config saved to /var/cache/conftool/dbconfig/20221107-063236-marostegui.json
  • 06:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38171 and previous config saved to /var/cache/conftool/dbconfig/20221107-061730-marostegui.json
  • 06:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38170 and previous config saved to /var/cache/conftool/dbconfig/20221107-061019-marostegui.json
  • 06:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 06:10 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 3292
  • 06:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 06:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 06:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 06:09 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 3292
  • 06:06 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 61461
  • 06:05 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 61461
  • 06:01 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 25091
  • 05:59 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 25091
  • 05:54 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 20115
  • 05:54 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 20115
  • 05:53 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 7843
  • 05:53 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 7843

2022-11-06

  • 08:23 elukey: restart rsyslog on centralog2002
  • 08:19 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 08:19 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 08:17 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 08:17 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 07:50 elukey: restart kube-apiserver on ml-serve-ctrl1001
  • 07:48 elukey: restart kube-apiserver on ml-serve-ctrl1002 - high HTTP 409 registered since days ago

2022-11-05

  • 12:56 mfossati@deploy1002: Finished deploy [airflow-dags/platform_eng@c849762]: (no justification provided) (duration: 00m 49s)
  • 12:55 mfossati@deploy1002: Started deploy [airflow-dags/platform_eng@c849762]: (no justification provided)
  • 09:39 elukey: reinstall kubernetes-node on ml-staging200[12] to allow puppet to run (cleanup after yesterday issue, worker nodes had master role applied)
  • 09:32 elukey: restart kube-apiserver on ml-staging-ctrl2001
  • 09:31 elukey: restart kube-apiserver on ml-staging-ctrl2002

2022-11-04

  • 18:31 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS buster
  • 18:12 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 18:09 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 17:44 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS buster
  • 17:25 fnegri@cumin1001: conftool action : set/pooled=yes; selector: name=dbproxy1019.eqiad.wmnet,service=wikireplicas-a
  • 17:19 fnegri@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host dbproxy1018.eqiad.wmnet
  • 17:08 fnegri@cumin1001: START - Cookbook sre.hosts.reboot-single for host dbproxy1018.eqiad.wmnet
  • 17:06 fnegri@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host dbproxy1018.eqiad.wmnet
  • 17:06 fnegri@cumin1001: START - Cookbook sre.hosts.reboot-single for host dbproxy1018.eqiad.wmnet
  • 17:04 fnegri@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host dbproxy1018.eqiad.wmnet
  • 17:04 fnegri@cumin1001: START - Cookbook sre.hosts.reboot-single for host dbproxy1018.eqiad.wmnet
  • 17:01 mvernon@cumin2002: conftool action : set/weight=40; selector: service=nginx,name=moss-fe2001.codfw.wmnet
  • 17:01 mvernon@cumin2002: conftool action : set/weight=40; selector: service=swift-fe,name=moss-fe2001.codfw.wmnet
  • 17:00 mvernon@cumin2002: conftool action : set/weight=40; selector: service=nginx,name=moss-fe1001.eqiad.wmnet
  • 17:00 mvernon@cumin2002: conftool action : set/weight=40; selector: service=swift-fe,name=moss-fe1001.eqiad.wmnet
  • 16:58 Emperor: rolling restart of swift-proxies to bring moss-fe{1,2}001 into service T322424
  • 16:55 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe2001.codfw.wmnet
  • 16:53 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe1001.eqiad.wmnet
  • 16:48 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host moss-fe2001.codfw.wmnet
  • 16:48 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host moss-fe1001.eqiad.wmnet
  • 16:41 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe2001.codfw.wmnet
  • 16:41 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe1001.eqiad.wmnet
  • 16:35 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host moss-fe2001.codfw.wmnet
  • 16:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp4052']
  • 16:34 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host moss-fe1001.eqiad.wmnet
  • 16:34 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 16:33 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cp4052']
  • 16:29 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe2001.codfw.wmnet with OS bullseye
  • 16:26 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe1001.eqiad.wmnet with OS bullseye
  • 16:13 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe2001.codfw.wmnet with reason: host reimage
  • 16:11 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: host reimage
  • 16:10 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe2001.codfw.wmnet with reason: host reimage
  • 16:07 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: host reimage
  • 16:06 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 16:06 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 15:57 Emperor: repool ms-fe{1,2}009
  • 15:55 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host moss-fe2001.codfw.wmnet with OS bullseye
  • 15:54 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host moss-fe1001.eqiad.wmnet with OS bullseye
  • 15:48 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 15:43 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:41 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 15:00 elukey: `elukey@cumin1001:~$ sudo cumin 'ms-fe2*' 'systemctl restart swift-proxy' -b 1 -s 20`
  • 14:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 14:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 14:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T318955)', diff saved to https://phabricator.wikimedia.org/P38159 and previous config saved to /var/cache/conftool/dbconfig/20221104-145225-ladsgroup.json
  • 14:52 vgutierrez@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=swift,name=eqiad
  • 14:51 Emperor: restart swift-proxy on ms-fe1012
  • 14:48 elukey: restart swift-proxy on ms-fe1011
  • 14:44 Emperor: restart swift-proxy on ms-fe1010
  • 14:41 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 14:37 vgutierrez@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=swift,name=eqiad
  • 14:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P38158 and previous config saved to /var/cache/conftool/dbconfig/20221104-143718-ladsgroup.json
  • 14:28 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 14:26 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp4052']
  • 14:25 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 14:24 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 14:23 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp4052']
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P38157 and previous config saved to /var/cache/conftool/dbconfig/20221104-142212-ladsgroup.json
  • 14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T318955)', diff saved to https://phabricator.wikimedia.org/P38156 and previous config saved to /var/cache/conftool/dbconfig/20221104-140705-ladsgroup.json
  • 14:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1196 (T318955)', diff saved to https://phabricator.wikimedia.org/P38155 and previous config saved to /var/cache/conftool/dbconfig/20221104-140427-ladsgroup.json
  • 14:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 14:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 14:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T318955)', diff saved to https://phabricator.wikimedia.org/P38154 and previous config saved to /var/cache/conftool/dbconfig/20221104-140405-ladsgroup.json
  • 13:58 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 13:58 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dbprov2004
  • 13:57 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dbprov2004
  • 13:56 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:54 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 13:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P38153 and previous config saved to /var/cache/conftool/dbconfig/20221104-134859-ladsgroup.json
  • 13:45 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 13:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P38152 and previous config saved to /var/cache/conftool/dbconfig/20221104-133353-ladsgroup.json
  • 13:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T318955)', diff saved to https://phabricator.wikimedia.org/P38151 and previous config saved to /var/cache/conftool/dbconfig/20221104-131846-ladsgroup.json
  • 13:17 sukhe: reprepro -C main include bullseye-wikimedia python-logstash_0.4.6-3_amd64.changes: T321309
  • 13:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1186 (T318955)', diff saved to https://phabricator.wikimedia.org/P38150 and previous config saved to /var/cache/conftool/dbconfig/20221104-131607-ladsgroup.json
  • 13:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 13:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 13:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T318955)', diff saved to https://phabricator.wikimedia.org/P38149 and previous config saved to /var/cache/conftool/dbconfig/20221104-131546-ladsgroup.json
  • 13:11 sukhe: reprepro -C main include bullseye-wikimedia prometheus-rdkafka-exporter_0.3_amd64.changes: T321309
  • 13:10 sukhe: reprepro -C main include bullseye-wikimedia file-read-backwards_2.0.0-3_amd64.changes: T321309
  • 13:09 sukhe: reprepro -C main include bullseye-wikimedia fifo-log-demux_0.6.3_amd64.changes: T321309
  • 13:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P38148 and previous config saved to /var/cache/conftool/dbconfig/20221104-130039-ladsgroup.json
  • 12:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P38147 and previous config saved to /var/cache/conftool/dbconfig/20221104-124533-ladsgroup.json
  • 12:36 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 100%: After reboot', diff saved to https://phabricator.wikimedia.org/P38146 and previous config saved to /var/cache/conftool/dbconfig/20221104-123606-root.json
  • 12:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T318955)', diff saved to https://phabricator.wikimedia.org/P38145 and previous config saved to /var/cache/conftool/dbconfig/20221104-123026-ladsgroup.json
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1184 (T318955)', diff saved to https://phabricator.wikimedia.org/P38144 and previous config saved to /var/cache/conftool/dbconfig/20221104-122747-ladsgroup.json
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T318955)', diff saved to https://phabricator.wikimedia.org/P38143 and previous config saved to /var/cache/conftool/dbconfig/20221104-122726-ladsgroup.json
  • 12:21 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 75%: After reboot', diff saved to https://phabricator.wikimedia.org/P38142 and previous config saved to /var/cache/conftool/dbconfig/20221104-122101-root.json
  • 12:19 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:18 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T318955)', diff saved to https://phabricator.wikimedia.org/P38141 and previous config saved to /var/cache/conftool/dbconfig/20221104-121848-ladsgroup.json
  • 12:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P38140 and previous config saved to /var/cache/conftool/dbconfig/20221104-121219-ladsgroup.json
  • 12:05 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 50%: After reboot', diff saved to https://phabricator.wikimedia.org/P38139 and previous config saved to /var/cache/conftool/dbconfig/20221104-120556-root.json
  • 12:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P38138 and previous config saved to /var/cache/conftool/dbconfig/20221104-120342-ladsgroup.json
  • 11:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P38137 and previous config saved to /var/cache/conftool/dbconfig/20221104-115713-ladsgroup.json
  • 11:50 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 25%: After reboot', diff saved to https://phabricator.wikimedia.org/P38136 and previous config saved to /var/cache/conftool/dbconfig/20221104-115051-root.json
  • 11:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P38135 and previous config saved to /var/cache/conftool/dbconfig/20221104-114835-ladsgroup.json
  • 11:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T318955)', diff saved to https://phabricator.wikimedia.org/P38134 and previous config saved to /var/cache/conftool/dbconfig/20221104-114207-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1169 (T318955)', diff saved to https://phabricator.wikimedia.org/P38133 and previous config saved to /var/cache/conftool/dbconfig/20221104-113929-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 11:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T318955)', diff saved to https://phabricator.wikimedia.org/P38132 and previous config saved to /var/cache/conftool/dbconfig/20221104-113725-ladsgroup.json
  • 11:35 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 10%: After reboot', diff saved to https://phabricator.wikimedia.org/P38131 and previous config saved to /var/cache/conftool/dbconfig/20221104-113546-root.json
  • 11:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T318955)', diff saved to https://phabricator.wikimedia.org/P38130 and previous config saved to /var/cache/conftool/dbconfig/20221104-113329-ladsgroup.json
  • 11:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2176 (T318955)', diff saved to https://phabricator.wikimedia.org/P38129 and previous config saved to /var/cache/conftool/dbconfig/20221104-113048-ladsgroup.json
  • 11:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 11:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 11:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T318955)', diff saved to https://phabricator.wikimedia.org/P38128 and previous config saved to /var/cache/conftool/dbconfig/20221104-113027-ladsgroup.json
  • 11:27 elukey: restart kube-apiserver on ml-serve-ctrl2002 - high latencies for LIST (knative resources)
  • 11:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P38127 and previous config saved to /var/cache/conftool/dbconfig/20221104-112218-ladsgroup.json
  • 11:20 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 5%: After reboot', diff saved to https://phabricator.wikimedia.org/P38125 and previous config saved to /var/cache/conftool/dbconfig/20221104-112041-root.json
  • 11:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P38124 and previous config saved to /var/cache/conftool/dbconfig/20221104-111521-ladsgroup.json
  • 11:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P38123 and previous config saved to /var/cache/conftool/dbconfig/20221104-110712-ladsgroup.json
  • 11:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P38122 and previous config saved to /var/cache/conftool/dbconfig/20221104-110014-ladsgroup.json
  • 10:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T318955)', diff saved to https://phabricator.wikimedia.org/P38121 and previous config saved to /var/cache/conftool/dbconfig/20221104-105205-ladsgroup.json
  • 10:50 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 1%: After reboot', diff saved to https://phabricator.wikimedia.org/P38120 and previous config saved to /var/cache/conftool/dbconfig/20221104-105031-root.json
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1135 (T318955)', diff saved to https://phabricator.wikimedia.org/P38119 and previous config saved to /var/cache/conftool/dbconfig/20221104-104927-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 10:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T318955)', diff saved to https://phabricator.wikimedia.org/P38117 and previous config saved to /var/cache/conftool/dbconfig/20221104-104508-ladsgroup.json
  • 10:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2174 (T318955)', diff saved to https://phabricator.wikimedia.org/P38116 and previous config saved to /var/cache/conftool/dbconfig/20221104-104227-ladsgroup.json
  • 10:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 10:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 07:27 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P38115 and previous config saved to /var/cache/conftool/dbconfig/20221104-072722-root.json
  • 07:12 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P38114 and previous config saved to /var/cache/conftool/dbconfig/20221104-071217-root.json
  • 06:57 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P38113 and previous config saved to /var/cache/conftool/dbconfig/20221104-065712-root.json
  • 06:42 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P38112 and previous config saved to /var/cache/conftool/dbconfig/20221104-064207-root.json
  • 06:32 marostegui@cumin1001: dbctl commit (dc=all): 'Give weight to es2021', diff saved to https://phabricator.wikimedia.org/P38111 and previous config saved to /var/cache/conftool/dbconfig/20221104-063250-root.json
  • 06:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es2020 T322389', diff saved to https://phabricator.wikimedia.org/P38110 and previous config saved to /var/cache/conftool/dbconfig/20221104-063224-root.json
  • 06:31 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es2021 to es4 primary and set section read-write T322389', diff saved to https://phabricator.wikimedia.org/P38109 and previous config saved to /var/cache/conftool/dbconfig/20221104-063128-root.json
  • 06:30 marostegui: Starting es4 codfw failover from es2020 to es2021 - T322389
  • 06:27 marostegui@cumin1001: dbctl commit (dc=all): 'Set es2021 with weight 0 T322389', diff saved to https://phabricator.wikimedia.org/P38108 and previous config saved to /var/cache/conftool/dbconfig/20221104-062740-root.json
  • 06:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es4 T322389
  • 06:27 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 10%: After schema change', diff saved to https://phabricator.wikimedia.org/P38107 and previous config saved to /var/cache/conftool/dbconfig/20221104-062702-root.json
  • 06:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es4 T322389
  • 06:11 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 5%: After schema change', diff saved to https://phabricator.wikimedia.org/P38106 and previous config saved to /var/cache/conftool/dbconfig/20221104-061157-root.json
  • 05:56 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 3%: After schema change', diff saved to https://phabricator.wikimedia.org/P38105 and previous config saved to /var/cache/conftool/dbconfig/20221104-055652-root.json
  • 05:41 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 1%: After schema change', diff saved to https://phabricator.wikimedia.org/P38104 and previous config saved to /var/cache/conftool/dbconfig/20221104-054147-root.json
  • 03:24 ejegg: payments-wiki upgraded from 1c8f522f to 8baa6bb5
  • 01:29 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 46s)
  • 01:28 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)

2022-11-03

  • 22:45 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 01m 00s)
  • 22:44 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 22:43 krinkle@deploy1002: Finished deploy [integration/docroot@44f1640]: (no justification provided) (duration: 00m 29s)
  • 22:42 krinkle@deploy1002: Started deploy [integration/docroot@44f1640]: (no justification provided)
  • 22:40 cwhite: logstash eqiad - opensearch 2.2.0 upgrade complete T304440
  • 22:13 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 02m 07s)
  • 22:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 22:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 22:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T318605)', diff saved to https://phabricator.wikimedia.org/P38102 and previous config saved to /var/cache/conftool/dbconfig/20221103-221329-ladsgroup.json
  • 22:11 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 22:11 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 08s)
  • 22:11 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P38101 and previous config saved to /var/cache/conftool/dbconfig/20221103-215823-ladsgroup.json
  • 21:51 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 16s)
  • 21:51 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 21:48 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 50s)
  • 21:47 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 21:47 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 05s)
  • 21:47 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 21:46 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 50s)
  • 21:45 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P38100 and previous config saved to /var/cache/conftool/dbconfig/20221103-214317-ladsgroup.json
  • 21:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T318605)', diff saved to https://phabricator.wikimedia.org/P38099 and previous config saved to /var/cache/conftool/dbconfig/20221103-212810-ladsgroup.json
  • 21:26 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 02m 31s)
  • 21:24 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 21:22 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:22 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:22 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:22 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:19 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:19 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:19 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:19 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:08 ryankemper: [WCQS] Pooled `wcqs100[1,2]`
  • 21:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:03 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:03 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1200 (T318605)', diff saved to https://phabricator.wikimedia.org/P38098 and previous config saved to /var/cache/conftool/dbconfig/20221103-205855-ladsgroup.json
  • 20:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 20:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 20:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38097 and previous config saved to /var/cache/conftool/dbconfig/20221103-205823-ladsgroup.json
  • 20:57 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:56 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:56 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:56 urbanecm@deploy1002: Finished scap: Backport for ApiSetMenteeStatus: Check GEMentorshipEnabled in wiki config (T321805), SpecialManageMentors: Do not include explanatory text on transclusion (T321773) (duration: 06m 34s)
  • 20:55 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:52 bking@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
  • 20:50 urbanecm@deploy1002: urbanecm and urbanecm: Backport for ApiSetMenteeStatus: Check GEMentorshipEnabled in wiki config (T321805), SpecialManageMentors: Do not include explanatory text on transclusion (T321773) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 20:50 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 20:50 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:49 urbanecm@deploy1002: Started scap: Backport for ApiSetMenteeStatus: Check GEMentorshipEnabled in wiki config (T321805), SpecialManageMentors: Do not include explanatory text on transclusion (T321773)
  • 20:47 bking@cumin1001: START - Cookbook sre.wdqs.data-transfer
  • 20:46 samtar@deploy1002: Finished scap: Backport for Update lv and bn wordmarks (T319223) (duration: 06m 36s)
  • 20:45 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:44 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:44 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:43 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38096 and previous config saved to /var/cache/conftool/dbconfig/20221103-204316-ladsgroup.json
  • 20:39 samtar@deploy1002: samtar and jdlrobson: Backport for Update lv and bn wordmarks (T319223) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 20:39 samtar@deploy1002: Started scap: Backport for Update lv and bn wordmarks (T319223)
  • 20:35 ryankemper: T322037 Rolling changes in https://gerrit.wikimedia.org/r/c/operations/puppet/+/852885 and https://gerrit.wikimedia.org/r/853006 out to query service fleet, 4 hosts at a time: `ryankemper@cumin1001:~$ sudo -E cumin -b 4 'A:wcqs-public or A:wdqs-all' 'run-puppet-agent --force'`
  • 20:35 samtar@deploy1002: Finished scap: Backport for Finish moving to Page Tools naming convention (duration: 07m 50s)
  • 20:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38094 and previous config saved to /var/cache/conftool/dbconfig/20221103-202810-ladsgroup.json
  • 20:27 samtar@deploy1002: samtar and jdlrobson: Backport for Finish moving to Page Tools naming convention synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 20:27 samtar@deploy1002: Started scap: Backport for Finish moving to Page Tools naming convention
  • 20:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:20 thcipriani@deploy1002: Finished scap: Backport for Enable parsoid cache warming on testwiki. (T320535) (duration: 10m 30s)
  • 20:16 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:15 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:15 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:14 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38093 and previous config saved to /var/cache/conftool/dbconfig/20221103-201303-ladsgroup.json
  • 20:10 thcipriani@deploy1002: thcipriani and daniel: Backport for Enable parsoid cache warming on testwiki. (T320535) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 20:10 thcipriani@deploy1002: Started scap: Backport for Enable parsoid cache warming on testwiki. (T320535)
  • 20:06 samtar@deploy1002: backport aborted: (duration: 05m 23s)
  • 19:56 ryankemper: Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/852885; disabled puppet on query service fleet via `ryankemper@cumin1001:~$ sudo -E cumin 'A:wcqs-public or A:wdqs-all' 'sudo disable-puppet "T322037"'`; testing change on `wdqs1009`
  • 19:49 cstone: civicrm upgraded from d1f286f0 to c0db8f34
  • 19:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38092 and previous config saved to /var/cache/conftool/dbconfig/20221103-194839-ladsgroup.json
  • 19:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 19:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 19:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38091 and previous config saved to /var/cache/conftool/dbconfig/20221103-194818-ladsgroup.json
  • 19:40 brennen@deploy1002: Finished deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 (duration: 02m 24s)
  • 19:37 brennen@deploy1002: Started deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004
  • 19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38090 and previous config saved to /var/cache/conftool/dbconfig/20221103-193311-ladsgroup.json
  • 19:29 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 48s)
  • 19:28 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 19:26 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 48s)
  • 19:25 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 19:22 brennen@deploy1002: Finished deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 (duration: 00m 25s)
  • 19:22 brennen@deploy1002: Started deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004
  • 19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38089 and previous config saved to /var/cache/conftool/dbconfig/20221103-191805-ladsgroup.json
  • 19:06 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 01m 10s)
  • 19:05 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 19:03 ladsgroup@deploy1002: Finished scap: Backport for WikiExporter: Avoid calling reload in processing every row (T298485 T322360) (duration: 04m 24s)
  • 19:03 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 19:03 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 19:03 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38088 and previous config saved to /var/cache/conftool/dbconfig/20221103-190258-ladsgroup.json
  • 19:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:59 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for WikiExporter: Avoid calling reload in processing every row (T298485 T322360) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 18:59 ladsgroup@deploy1002: Started scap: Backport for WikiExporter: Avoid calling reload in processing every row (T298485 T322360)
  • 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38087 and previous config saved to /var/cache/conftool/dbconfig/20221103-182756-ladsgroup.json
  • 18:27 jynus@cumin1001: dbctl commit (dc=all): 'increase db1144:3315 load', diff saved to https://phabricator.wikimedia.org/P38086 and previous config saved to /var/cache/conftool/dbconfig/20221103-182750-jynus.json
  • 18:23 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 02m 56s)
  • 18:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 18:20 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 18:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 18:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 18:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 18:16 bblack: lvs1017: restart pybal to clear etcd error states
  • 18:15 bblack: lvs1018: restart pybal to clear etcd error states
  • 18:14 bblack: lvs1019: restart pybal to clear etcd error states
  • 18:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:09 bblack: lvs1020: restart pybal to hopefully clear etcd error states
  • 18:07 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.40.0-wmf.8 refs T320513
  • 17:46 vgutierrez: vgutierrez@conf1007:~$ sudo -i systemctl start etcd
  • 17:46 fnegri@cumin1001: conftool action : set/pooled=inactive; selector: name=dbproxy1018.eqiad.wmnet,service=wikireplicas-b
  • 17:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 17:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 17:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P38084 and previous config saved to /var/cache/conftool/dbconfig/20221103-174306-marostegui.json
  • 17:39 elukey: `sudo truncate -s 20G /var/log/nginx/etcd_access.log.1` on conf100[7-9], root partition full
  • 17:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P38083 and previous config saved to /var/cache/conftool/dbconfig/20221103-173843-ladsgroup.json
  • 17:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P38082 and previous config saved to /var/cache/conftool/dbconfig/20221103-173022-ladsgroup.json
  • 17:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P38081 and previous config saved to /var/cache/conftool/dbconfig/20221103-172759-marostegui.json
  • 17:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T318955)', diff saved to https://phabricator.wikimedia.org/P38080 and previous config saved to /var/cache/conftool/dbconfig/20221103-172338-ladsgroup.json
  • 17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T318955)', diff saved to https://phabricator.wikimedia.org/P38079 and previous config saved to /var/cache/conftool/dbconfig/20221103-172235-ladsgroup.json
  • 17:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P38078 and previous config saved to /var/cache/conftool/dbconfig/20221103-172100-ladsgroup.json
  • 17:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1134 (T318955)', diff saved to https://phabricator.wikimedia.org/P38077 and previous config saved to /var/cache/conftool/dbconfig/20221103-171959-ladsgroup.json
  • 17:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2173 (T318955)', diff saved to https://phabricator.wikimedia.org/P38076 and previous config saved to /var/cache/conftool/dbconfig/20221103-171952-ladsgroup.json
  • 17:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1134.eqiad.wmnet with reason: Maintenance
  • 17:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 17:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 17:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 17:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1134.eqiad.wmnet with reason: Maintenance
  • 17:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 17:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P38075 and previous config saved to /var/cache/conftool/dbconfig/20221103-171925-ladsgroup.json
  • 17:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1133.eqiad.wmnet with reason: Maintenance
  • 17:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1133.eqiad.wmnet with reason: Maintenance
  • 17:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 (T318955)', diff saved to https://phabricator.wikimedia.org/P38074 and previous config saved to /var/cache/conftool/dbconfig/20221103-171850-ladsgroup.json
  • 17:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P38073 and previous config saved to /var/cache/conftool/dbconfig/20221103-171512-ladsgroup.json
  • 17:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38072 and previous config saved to /var/cache/conftool/dbconfig/20221103-171250-marostegui.json
  • 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38071 and previous config saved to /var/cache/conftool/dbconfig/20221103-171028-marostegui.json
  • 17:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 17:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T321123)', diff saved to https://phabricator.wikimedia.org/P38070 and previous config saved to /var/cache/conftool/dbconfig/20221103-171004-marostegui.json
  • 17:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T318605)', diff saved to https://phabricator.wikimedia.org/P38069 and previous config saved to /var/cache/conftool/dbconfig/20221103-170553-ladsgroup.json
  • 17:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P38068 and previous config saved to /var/cache/conftool/dbconfig/20221103-170417-ladsgroup.json
  • 17:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P38067 and previous config saved to /var/cache/conftool/dbconfig/20221103-170341-ladsgroup.json
  • 17:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38066 and previous config saved to /var/cache/conftool/dbconfig/20221103-170003-ladsgroup.json
  • 16:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P38065 and previous config saved to /var/cache/conftool/dbconfig/20221103-165456-marostegui.json
  • 16:53 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 16:52 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 16:52 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 16:51 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 16:49 fnegri@cumin1001: conftool action : set/pooled=no; selector: name=dbproxy1018.eqiad.wmnet,service=wikireplicas-b
  • 16:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P38063 and previous config saved to /var/cache/conftool/dbconfig/20221103-164909-ladsgroup.json
  • 16:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P38062 and previous config saved to /var/cache/conftool/dbconfig/20221103-164833-ladsgroup.json
  • 16:48 fnegri@cumin1001: conftool action : set/pooled=yes; selector: name=dbproxy1019.eqiad.wmnet,service=wikireplicas-b
  • 16:42 fnegri@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1019.eqiad.wmnet with OS bullseye
  • 16:41 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 16:40 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P38056 and previous config saved to /var/cache/conftool/dbconfig/20221103-163947-marostegui.json
  • 16:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P38055 and previous config saved to /var/cache/conftool/dbconfig/20221103-163402-ladsgroup.json
  • 16:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 (T318955)', diff saved to https://phabricator.wikimedia.org/P38054 and previous config saved to /var/cache/conftool/dbconfig/20221103-163324-ladsgroup.json
  • 16:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P38053 and previous config saved to /var/cache/conftool/dbconfig/20221103-163219-ladsgroup.json
  • 16:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 16:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P38052 and previous config saved to /var/cache/conftool/dbconfig/20221103-163156-ladsgroup.json
  • 16:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1132 (T318955)', diff saved to https://phabricator.wikimedia.org/P38051 and previous config saved to /var/cache/conftool/dbconfig/20221103-163041-ladsgroup.json
  • 16:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1132.eqiad.wmnet with reason: Maintenance
  • 16:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1132.eqiad.wmnet with reason: Maintenance
  • 16:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128 (T318955)', diff saved to https://phabricator.wikimedia.org/P38050 and previous config saved to /var/cache/conftool/dbconfig/20221103-163016-ladsgroup.json
  • 16:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2157 (T318605)', diff saved to https://phabricator.wikimedia.org/P38049 and previous config saved to /var/cache/conftool/dbconfig/20221103-162927-ladsgroup.json
  • 16:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 16:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 16:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38048 and previous config saved to /var/cache/conftool/dbconfig/20221103-162904-ladsgroup.json
  • 16:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T321123)', diff saved to https://phabricator.wikimedia.org/P38047 and previous config saved to /var/cache/conftool/dbconfig/20221103-162437-marostegui.json
  • 16:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2120 (T321123)', diff saved to https://phabricator.wikimedia.org/P38046 and previous config saved to /var/cache/conftool/dbconfig/20221103-162214-marostegui.json
  • 16:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 16:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 16:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T321123)', diff saved to https://phabricator.wikimedia.org/P38045 and previous config saved to /var/cache/conftool/dbconfig/20221103-162152-marostegui.json
  • 16:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38044 and previous config saved to /var/cache/conftool/dbconfig/20221103-162141-ladsgroup.json
  • 16:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 16:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 16:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38043 and previous config saved to /var/cache/conftool/dbconfig/20221103-162118-ladsgroup.json
  • 16:20 fnegri@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1019.eqiad.wmnet with reason: host reimage
  • 16:18 fnegri@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy1019.eqiad.wmnet with reason: host reimage
  • 16:18 sukhe: reprepro -C main include bullseye-wikimedia trafficserver_9.1.3-1wm3_amd64.changes: T321309
  • 16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P38042 and previous config saved to /var/cache/conftool/dbconfig/20221103-161648-ladsgroup.json
  • 16:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P38041 and previous config saved to /var/cache/conftool/dbconfig/20221103-161507-ladsgroup.json
  • 16:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P38039 and previous config saved to /var/cache/conftool/dbconfig/20221103-161356-ladsgroup.json
  • 16:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 16:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 16:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 16:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 16:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P38037 and previous config saved to /var/cache/conftool/dbconfig/20221103-160645-marostegui.json
  • 16:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P38036 and previous config saved to /var/cache/conftool/dbconfig/20221103-160611-ladsgroup.json
  • 16:02 fnegri@cumin1001: START - Cookbook sre.hosts.reimage for host dbproxy1019.eqiad.wmnet with OS bullseye
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P38035 and previous config saved to /var/cache/conftool/dbconfig/20221103-160141-ladsgroup.json
  • 15:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P38034 and previous config saved to /var/cache/conftool/dbconfig/20221103-155958-ladsgroup.json
  • 15:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P38033 and previous config saved to /var/cache/conftool/dbconfig/20221103-155847-ladsgroup.json
  • 15:54 sukhe: sudo -i reprepro -C main include bullseye-wikimedia varnish_6.0.10-1wm2_amd64.changes: T321309
  • 15:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P38032 and previous config saved to /var/cache/conftool/dbconfig/20221103-155136-marostegui.json
  • 15:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P38031 and previous config saved to /var/cache/conftool/dbconfig/20221103-155101-ladsgroup.json
  • 15:48 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:46 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 15:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P38030 and previous config saved to /var/cache/conftool/dbconfig/20221103-154631-ladsgroup.json
  • 15:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128 (T318955)', diff saved to https://phabricator.wikimedia.org/P38029 and previous config saved to /var/cache/conftool/dbconfig/20221103-154449-ladsgroup.json
  • 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2167:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P38028 and previous config saved to /var/cache/conftool/dbconfig/20221103-154347-ladsgroup.json
  • 15:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38027 and previous config saved to /var/cache/conftool/dbconfig/20221103-154339-ladsgroup.json
  • 15:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T318955)', diff saved to https://phabricator.wikimedia.org/P38026 and previous config saved to /var/cache/conftool/dbconfig/20221103-154325-ladsgroup.json
  • 15:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1128 (T318955)', diff saved to https://phabricator.wikimedia.org/P38025 and previous config saved to /var/cache/conftool/dbconfig/20221103-154209-ladsgroup.json
  • 15:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1128.eqiad.wmnet with reason: Maintenance
  • 15:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1128.eqiad.wmnet with reason: Maintenance
  • 15:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T318955)', diff saved to https://phabricator.wikimedia.org/P38024 and previous config saved to /var/cache/conftool/dbconfig/20221103-154145-ladsgroup.json
  • 15:38 fnegri@cumin1001: conftool action : set/pooled=inactive; selector: name=dbproxy1019.eqiad.wmnet
  • 15:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T321123)', diff saved to https://phabricator.wikimedia.org/P38023 and previous config saved to /var/cache/conftool/dbconfig/20221103-153628-marostegui.json
  • 15:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38022 and previous config saved to /var/cache/conftool/dbconfig/20221103-153553-ladsgroup.json
  • 15:34 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2108 (T321123)', diff saved to https://phabricator.wikimedia.org/P38021 and previous config saved to /var/cache/conftool/dbconfig/20221103-153404-marostegui.json
  • 15:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 15:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 15:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 15:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 15:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 15:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 15:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 15:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 15:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T321123)', diff saved to https://phabricator.wikimedia.org/P38020 and previous config saved to /var/cache/conftool/dbconfig/20221103-153224-marostegui.json
  • 15:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P38019 and previous config saved to /var/cache/conftool/dbconfig/20221103-152817-ladsgroup.json
  • 15:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P38018 and previous config saved to /var/cache/conftool/dbconfig/20221103-152638-ladsgroup.json
  • 15:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:17 Emperor: comment out www-data crontab on cloudmetrics100{1,2} T297712
  • 15:17 otto@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Set destination_event_service for rc0.mediawiki.page_content_change to fix canary producer job (duration: 03m 36s)
  • 15:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P38017 and previous config saved to /var/cache/conftool/dbconfig/20221103-151716-marostegui.json
  • 15:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 15:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P38016 and previous config saved to /var/cache/conftool/dbconfig/20221103-151307-ladsgroup.json
  • 15:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P38015 and previous config saved to /var/cache/conftool/dbconfig/20221103-151129-ladsgroup.json
  • 15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38014 and previous config saved to /var/cache/conftool/dbconfig/20221103-150633-ladsgroup.json
  • 15:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 15:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T318605)', diff saved to https://phabricator.wikimedia.org/P38013 and previous config saved to /var/cache/conftool/dbconfig/20221103-150610-ladsgroup.json
  • 15:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P38012 and previous config saved to /var/cache/conftool/dbconfig/20221103-150208-marostegui.json
  • 14:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T318955)', diff saved to https://phabricator.wikimedia.org/P38011 and previous config saved to /var/cache/conftool/dbconfig/20221103-145759-ladsgroup.json
  • 14:56 fnegri@cumin1001: conftool action : set/pooled=yes; selector: name=dbproxy1018.eqiad.wmnet
  • 14:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T318955)', diff saved to https://phabricator.wikimedia.org/P38010 and previous config saved to /var/cache/conftool/dbconfig/20221103-145620-ladsgroup.json
  • 14:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2153 (T318955)', diff saved to https://phabricator.wikimedia.org/P38009 and previous config saved to /var/cache/conftool/dbconfig/20221103-145516-ladsgroup.json
  • 14:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 14:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T318955)', diff saved to https://phabricator.wikimedia.org/P38008 and previous config saved to /var/cache/conftool/dbconfig/20221103-145453-ladsgroup.json
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1119 (T318955)', diff saved to https://phabricator.wikimedia.org/P38007 and previous config saved to /var/cache/conftool/dbconfig/20221103-145339-ladsgroup.json
  • 14:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1119.eqiad.wmnet with reason: Maintenance
  • 14:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1119.eqiad.wmnet with reason: Maintenance
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 (T318955)', diff saved to https://phabricator.wikimedia.org/P38006 and previous config saved to /var/cache/conftool/dbconfig/20221103-145316-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38005 and previous config saved to /var/cache/conftool/dbconfig/20221103-145133-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 14:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T318605)', diff saved to https://phabricator.wikimedia.org/P38004 and previous config saved to /var/cache/conftool/dbconfig/20221103-145110-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P38003 and previous config saved to /var/cache/conftool/dbconfig/20221103-145101-ladsgroup.json
  • 14:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T321123)', diff saved to https://phabricator.wikimedia.org/P38002 and previous config saved to /var/cache/conftool/dbconfig/20221103-144658-marostegui.json
  • 14:45 fnegri@cumin1001: conftool action : set/pooled=no; selector: name=dbproxy1019.eqiad.wmnet
  • 14:41 fnegri@cumin1001: conftool action : set/pooled=no; selector: service=wikireplicas-b,name=dbproxy1019.eqiad.wmnet
  • 14:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P38001 and previous config saved to /var/cache/conftool/dbconfig/20221103-143943-ladsgroup.json
  • 14:38 fnegri@cumin1001: conftool action : set/pooled=no; selector: service=wikireplicas-b,name=dbproxy1019
  • 14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P38000 and previous config saved to /var/cache/conftool/dbconfig/20221103-143809-ladsgroup.json
  • 14:37 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T321123)', diff saved to https://phabricator.wikimedia.org/P37999 and previous config saved to /var/cache/conftool/dbconfig/20221103-143745-marostegui.json
  • 14:37 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 14:37 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 14:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T321123)', diff saved to https://phabricator.wikimedia.org/P37998 and previous config saved to /var/cache/conftool/dbconfig/20221103-143722-marostegui.json
  • 14:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P37997 and previous config saved to /var/cache/conftool/dbconfig/20221103-143603-ladsgroup.json
  • 14:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P37996 and previous config saved to /var/cache/conftool/dbconfig/20221103-143552-ladsgroup.json
  • 14:26 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:26 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:26 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P37995 and previous config saved to /var/cache/conftool/dbconfig/20221103-142434-ladsgroup.json
  • 14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P37994 and previous config saved to /var/cache/conftool/dbconfig/20221103-142301-ladsgroup.json
  • 14:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P37993 and previous config saved to /var/cache/conftool/dbconfig/20221103-142215-marostegui.json
  • 14:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P37992 and previous config saved to /var/cache/conftool/dbconfig/20221103-142055-ladsgroup.json
  • 14:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T318605)', diff saved to https://phabricator.wikimedia.org/P37991 and previous config saved to /var/cache/conftool/dbconfig/20221103-142044-ladsgroup.json
  • 14:16 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:11 claime: Sunsetting search.wikimedia.org, starting a 2 week grace period before decommission - T316296
  • 14:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T318955)', diff saved to https://phabricator.wikimedia.org/P37990 and previous config saved to /var/cache/conftool/dbconfig/20221103-140926-ladsgroup.json
  • 14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 (T318955)', diff saved to https://phabricator.wikimedia.org/P37989 and previous config saved to /var/cache/conftool/dbconfig/20221103-140753-ladsgroup.json
  • 14:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P37988 and previous config saved to /var/cache/conftool/dbconfig/20221103-140703-marostegui.json
  • 14:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2146 (T318955)', diff saved to https://phabricator.wikimedia.org/P37987 and previous config saved to /var/cache/conftool/dbconfig/20221103-140643-ladsgroup.json
  • 14:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 14:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 14:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T318955)', diff saved to https://phabricator.wikimedia.org/P37986 and previous config saved to /var/cache/conftool/dbconfig/20221103-140621-ladsgroup.json
  • 14:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T318605)', diff saved to https://phabricator.wikimedia.org/P37985 and previous config saved to /var/cache/conftool/dbconfig/20221103-140541-ladsgroup.json
  • 14:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1118 (T318955)', diff saved to https://phabricator.wikimedia.org/P37984 and previous config saved to /var/cache/conftool/dbconfig/20221103-140509-ladsgroup.json
  • 14:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 14:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 14:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T318955)', diff saved to https://phabricator.wikimedia.org/P37983 and previous config saved to /var/cache/conftool/dbconfig/20221103-140447-ladsgroup.json
  • 13:58 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:57 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:57 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:56 Lucas_WMDE: UTC afternoon backport+config window done
  • 13:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:56 lucaswerkmeister-wmde@deploy1002: Finished scap: Backport for Media border option applies to the media element, not the wrapper (T318300) (duration: 06m 31s)
  • 13:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2128 (T318605)', diff saved to https://phabricator.wikimedia.org/P37982 and previous config saved to /var/cache/conftool/dbconfig/20221103-135522-ladsgroup.json
  • 13:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 13:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 13:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 13:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 13:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T318605)', diff saved to https://phabricator.wikimedia.org/P37981 and previous config saved to /var/cache/conftool/dbconfig/20221103-135454-ladsgroup.json
  • 13:52 fnegri@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on dbproxy1019.eqiad.wmnet with reason: T313445
  • 13:52 fnegri@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on dbproxy1019.eqiad.wmnet with reason: T313445
  • 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T321123)', diff saved to https://phabricator.wikimedia.org/P37980 and previous config saved to /var/cache/conftool/dbconfig/20221103-135155-marostegui.json
  • 13:51 Lucas_WMDE: Finished scap: Backport for Enable the CampaignEvents extension on test(2)wiki and officewiki (T318592) (duration: 20m 46s) (originally at 13:38:40 UTC; logmsgbot dropped the message)
  • 13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P37979 and previous config saved to /var/cache/conftool/dbconfig/20221103-135113-ladsgroup.json
  • 13:50 lucaswerkmeister-wmde@deploy1002: lucaswerkmeister-wmde and arlolra: Backport for Media border option applies to the media element, not the wrapper (T318300) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 13:50 lucaswerkmeister-wmde@deploy1002: Started scap: Backport for Media border option applies to the media element, not the wrapper (T318300)
  • 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1194 (T321123)', diff saved to https://phabricator.wikimedia.org/P37978 and previous config saved to /var/cache/conftool/dbconfig/20221103-134943-marostegui.json
  • 13:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P37977 and previous config saved to /var/cache/conftool/dbconfig/20221103-134935-ladsgroup.json
  • 13:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 13:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321123)', diff saved to https://phabricator.wikimedia.org/P37976 and previous config saved to /var/cache/conftool/dbconfig/20221103-134920-marostegui.json
  • 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1107 (T318955)', diff saved to https://phabricator.wikimedia.org/P37962 and previous config saved to /var/cache/conftool/dbconfig/20221103-131638-ladsgroup.json
  • 13:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: labstore1006.wikimedia.org
  • 13:28 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: labstore1006.wikimedia.org
  • 13:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T318955)', diff saved to https://phabricator.wikimedia.org/P37961 and previous config saved to /var/cache/conftool/dbconfig/20221103-131614-ladsgroup.json
  • 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: labstore1007.wikimedia.org
  • 13:28 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: labstore1007.wikimedia.org
  • 13:28 lucaswerkmeister-wmde@deploy1002: Finished scap: Backport for Remove $wgCampaignEventsDatabaseName (T318592) (duration: 08m 44s)
  • 13:20 Lucas_WMDE: lucaswerkmeister-wmde and daimona: Backport for Enable the CampaignEvents extension on test(2)wiki and officewiki (T318592) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet (on behalf of scap – log message got lost?)
  • 13:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P37960 and previous config saved to /var/cache/conftool/dbconfig/20221103-131103-ladsgroup.json
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T318605)', diff saved to https://phabricator.wikimedia.org/P37959 and previous config saved to /var/cache/conftool/dbconfig/20221103-130931-ladsgroup.json
  • 13:07 lucaswerkmeister-wmde@deploy1002: lucaswerkmeister-wmde and daimona: Backport for Remove $wgCampaignEventsDatabaseName (T318592) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 13:06 lucaswerkmeister-wmde@deploy1002: Started scap: Backport for Remove $wgCampaignEventsDatabaseName (T318592)
  • 13:06 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1025.eqiad.wmnet to cluster eqiad and group A
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321123)', diff saved to https://phabricator.wikimedia.org/P37958 and previous config saved to /var/cache/conftool/dbconfig/20221103-130352-marostegui.json
  • 13:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1025.eqiad.wmnet to cluster eqiad and group A
  • 13:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P37957 and previous config saved to /var/cache/conftool/dbconfig/20221103-130153-ladsgroup.json
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T321123)', diff saved to https://phabricator.wikimedia.org/P37956 and previous config saved to /var/cache/conftool/dbconfig/20221103-130140-marostegui.json
  • 13:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 13:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T321123)', diff saved to https://phabricator.wikimedia.org/P37955 and previous config saved to /var/cache/conftool/dbconfig/20221103-130117-marostegui.json
  • 13:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P37954 and previous config saved to /var/cache/conftool/dbconfig/20221103-130106-ladsgroup.json
  • 12:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P37953 and previous config saved to /var/cache/conftool/dbconfig/20221103-125555-ladsgroup.json
  • 12:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P37952 and previous config saved to /var/cache/conftool/dbconfig/20221103-124646-ladsgroup.json
  • 12:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P37951 and previous config saved to /var/cache/conftool/dbconfig/20221103-124607-marostegui.json
  • 12:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P37950 and previous config saved to /var/cache/conftool/dbconfig/20221103-124557-ladsgroup.json
  • 12:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2123 (T318605)', diff saved to https://phabricator.wikimedia.org/P37949 and previous config saved to /var/cache/conftool/dbconfig/20221103-124516-ladsgroup.json
  • 12:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 12:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 12:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T318605)', diff saved to https://phabricator.wikimedia.org/P37948 and previous config saved to /var/cache/conftool/dbconfig/20221103-124454-ladsgroup.json
  • 12:44 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry
  • 12:43 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on gitlab1004.wikimedia.org with reason: upgrade gitlab1004 to new version
  • 12:43 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 0:20:00 on gitlab1004.wikimedia.org with reason: upgrade gitlab1004 to new version
  • 12:41 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry
  • 12:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100 (T318605)', diff saved to https://phabricator.wikimedia.org/P37947 and previous config saved to /var/cache/conftool/dbconfig/20221103-124047-ladsgroup.json
  • 12:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1025.eqiad.wmnet
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T318955)', diff saved to https://phabricator.wikimedia.org/P37946 and previous config saved to /var/cache/conftool/dbconfig/20221103-123137-ladsgroup.json
  • 12:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P37945 and previous config saved to /var/cache/conftool/dbconfig/20221103-123101-marostegui.json
  • 12:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T318955)', diff saved to https://phabricator.wikimedia.org/P37944 and previous config saved to /var/cache/conftool/dbconfig/20221103-123048-ladsgroup.json
  • 12:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P37943 and previous config saved to /var/cache/conftool/dbconfig/20221103-122944-ladsgroup.json
  • 12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2130 (T318955)', diff saved to https://phabricator.wikimedia.org/P37942 and previous config saved to /var/cache/conftool/dbconfig/20221103-122854-ladsgroup.json
  • 12:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 12:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T318955)', diff saved to https://phabricator.wikimedia.org/P37941 and previous config saved to /var/cache/conftool/dbconfig/20221103-122831-ladsgroup.json
  • 12:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1025.eqiad.wmnet
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T318955)', diff saved to https://phabricator.wikimedia.org/P37940 and previous config saved to /var/cache/conftool/dbconfig/20221103-122709-ladsgroup.json
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 12:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 12:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P37939 and previous config saved to /var/cache/conftool/dbconfig/20221103-122640-ladsgroup.json
  • 12:16 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:16 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T321123)', diff saved to https://phabricator.wikimedia.org/P37938 and previous config saved to /var/cache/conftool/dbconfig/20221103-121553-marostegui.json
  • 12:15 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:15 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1100 (T318605)', diff saved to https://phabricator.wikimedia.org/P37937 and previous config saved to /var/cache/conftool/dbconfig/20221103-121458-ladsgroup.json
  • 12:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 12:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P37936 and previous config saved to /var/cache/conftool/dbconfig/20221103-121436-ladsgroup.json
  • 12:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 12:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P37935 and previous config saved to /var/cache/conftool/dbconfig/20221103-121423-ladsgroup.json
  • 12:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P37934 and previous config saved to /var/cache/conftool/dbconfig/20221103-121320-ladsgroup.json
  • 12:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P37932 and previous config saved to /var/cache/conftool/dbconfig/20221103-121133-ladsgroup.json
  • 12:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1025.eqiad.wmnet with OS bullseye
  • 12:08 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:08 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:07 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:06 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:05 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T318605)', diff saved to https://phabricator.wikimedia.org/P37931 and previous config saved to /var/cache/conftool/dbconfig/20221103-115928-ladsgroup.json
  • 11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P37930 and previous config saved to /var/cache/conftool/dbconfig/20221103-115916-ladsgroup.json
  • 11:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P37929 and previous config saved to /var/cache/conftool/dbconfig/20221103-115813-ladsgroup.json
  • 11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P37928 and previous config saved to /var/cache/conftool/dbconfig/20221103-115624-ladsgroup.json
  • 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1025.eqiad.wmnet with reason: host reimage
  • 11:55 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 11:52 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1025.eqiad.wmnet with reason: host reimage
  • 11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P37924 and previous config saved to /var/cache/conftool/dbconfig/20221103-114408-ladsgroup.json
  • 11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T318955)', diff saved to https://phabricator.wikimedia.org/P37923 and previous config saved to /var/cache/conftool/dbconfig/20221103-114304-ladsgroup.json
  • 11:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T321123)', diff saved to https://phabricator.wikimedia.org/P37922 and previous config saved to /var/cache/conftool/dbconfig/20221103-114135-marostegui.json
  • 11:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P37921 and previous config saved to /var/cache/conftool/dbconfig/20221103-114116-ladsgroup.json
  • 11:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 11:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 11:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 11:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37920 and previous config saved to /var/cache/conftool/dbconfig/20221103-114054-marostegui.json
  • 11:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2116 (T318955)', diff saved to https://phabricator.wikimedia.org/P37919 and previous config saved to /var/cache/conftool/dbconfig/20221103-114021-ladsgroup.json
  • 11:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T318955)', diff saved to https://phabricator.wikimedia.org/P37918 and previous config saved to /var/cache/conftool/dbconfig/20221103-113956-ladsgroup.json
  • 11:39 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye
  • 11:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P37917 and previous config saved to /var/cache/conftool/dbconfig/20221103-113833-ladsgroup.json
  • 11:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P37916 and previous config saved to /var/cache/conftool/dbconfig/20221103-113809-ladsgroup.json
  • 11:37 volans@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1025.mgmt.eqiad.wmnet with reboot policy GRACEFUL
  • 11:35 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1025.eqiad.wmnet with OS bullseye
  • 11:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P37914 and previous config saved to /var/cache/conftool/dbconfig/20221103-112900-ladsgroup.json
  • 11:28 volans@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1025.mgmt.eqiad.wmnet with reboot policy GRACEFUL
  • 11:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P37913 and previous config saved to /var/cache/conftool/dbconfig/20221103-112546-marostegui.json
  • 11:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P37912 and previous config saved to /var/cache/conftool/dbconfig/20221103-112448-ladsgroup.json
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2111 (T318605)', diff saved to https://phabricator.wikimedia.org/P37911 and previous config saved to /var/cache/conftool/dbconfig/20221103-112343-ladsgroup.json
  • 11:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P37910 and previous config saved to /var/cache/conftool/dbconfig/20221103-112300-ladsgroup.json
  • 11:16 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 11:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P37909 and previous config saved to /var/cache/conftool/dbconfig/20221103-111037-marostegui.json
  • 11:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P37908 and previous config saved to /var/cache/conftool/dbconfig/20221103-110939-ladsgroup.json
  • 11:09 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P37907 and previous config saved to /var/cache/conftool/dbconfig/20221103-110751-ladsgroup.json
  • 11:06 volans@cumin1001: START - Cookbook sre.dns.netbox
  • 11:06 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye
  • 11:04 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1025.eqiad.wmnet with OS bullseye
  • 10:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37906 and previous config saved to /var/cache/conftool/dbconfig/20221103-105527-marostegui.json
  • 10:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T318955)', diff saved to https://phabricator.wikimedia.org/P37905 and previous config saved to /var/cache/conftool/dbconfig/20221103-105429-ladsgroup.json
  • 10:53 jmm@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti1025.eqiad.wmnet']
  • 10:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P37904 and previous config saved to /var/cache/conftool/dbconfig/20221103-105243-ladsgroup.json
  • 10:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2103 (T318955)', diff saved to https://phabricator.wikimedia.org/P37903 and previous config saved to /var/cache/conftool/dbconfig/20221103-105148-ladsgroup.json
  • 10:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 10:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 10:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 10:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 10:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P37902 and previous config saved to /var/cache/conftool/dbconfig/20221103-104957-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P37901 and previous config saved to /var/cache/conftool/dbconfig/20221103-104942-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 10:43 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37900 and previous config saved to /var/cache/conftool/dbconfig/20221103-104313-marostegui.json
  • 10:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 10:42 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 10:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T321123)', diff saved to https://phabricator.wikimedia.org/P37899 and previous config saved to /var/cache/conftool/dbconfig/20221103-104239-marostegui.json
  • 10:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P37898 and previous config saved to /var/cache/conftool/dbconfig/20221103-102730-marostegui.json
  • 10:19 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1025.eqiad.wmnet']
  • 10:12 jmm@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ganeti1025.eqiad.wmnet']
  • 10:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P37897 and previous config saved to /var/cache/conftool/dbconfig/20221103-101222-marostegui.json
  • 09:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T321123)', diff saved to https://phabricator.wikimedia.org/P37896 and previous config saved to /var/cache/conftool/dbconfig/20221103-095715-marostegui.json
  • 09:55 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T321123)', diff saved to https://phabricator.wikimedia.org/P37895 and previous config saved to /var/cache/conftool/dbconfig/20221103-095501-marostegui.json
  • 09:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 09:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 09:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T321123)', diff saved to https://phabricator.wikimedia.org/P37894 and previous config saved to /var/cache/conftool/dbconfig/20221103-095409-marostegui.json
  • 09:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P37893 and previous config saved to /var/cache/conftool/dbconfig/20221103-093901-marostegui.json
  • 09:36 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1025.eqiad.wmnet']
  • 09:26 elukey@cumin1001: END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES codfw cluster: Roll restart of ORES's daemons.
  • 09:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P37892 and previous config saved to /var/cache/conftool/dbconfig/20221103-092353-marostegui.json
  • 09:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T321123)', diff saved to https://phabricator.wikimedia.org/P37891 and previous config saved to /var/cache/conftool/dbconfig/20221103-090844-marostegui.json
  • 09:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T321123)', diff saved to https://phabricator.wikimedia.org/P37890 and previous config saved to /var/cache/conftool/dbconfig/20221103-090631-marostegui.json
  • 09:06 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 09:06 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 09:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T321123)', diff saved to https://phabricator.wikimedia.org/P37889 and previous config saved to /var/cache/conftool/dbconfig/20221103-090607-marostegui.json
  • 09:05 elukey@cumin1001: START - Cookbook sre.ores.roll-restart-workers for ORES codfw cluster: Roll restart of ORES's daemons.
  • 09:02 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye
  • 09:02 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1025.eqiad.wmnet with OS bullseye
  • 08:56 elukey@cumin1001: END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES eqiad cluster: Roll restart of ORES's daemons.
  • 08:53 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye
  • 08:53 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1025.eqiad.wmnet with OS bullseye
  • 08:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P37888 and previous config saved to /var/cache/conftool/dbconfig/20221103-085059-marostegui.json
  • 08:44 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye
  • 08:43 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti1025.eqiad.wmnet with OS bullseye
  • 08:39 moritzm: installing ruby-nokogiri security updates
  • 08:37 elukey@cumin1001: START - Cookbook sre.ores.roll-restart-workers for ORES eqiad cluster: Roll restart of ORES's daemons.
  • 08:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P37887 and previous config saved to /var/cache/conftool/dbconfig/20221103-083549-marostegui.json
  • 08:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T321123)', diff saved to https://phabricator.wikimedia.org/P37886 and previous config saved to /var/cache/conftool/dbconfig/20221103-082040-marostegui.json
  • 08:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T321123)', diff saved to https://phabricator.wikimedia.org/P37885 and previous config saved to /var/cache/conftool/dbconfig/20221103-081827-marostegui.json
  • 08:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 08:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 08:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37884 and previous config saved to /var/cache/conftool/dbconfig/20221103-081805-marostegui.json
  • 08:17 moritzm: installing glibc security updates on buster
  • 08:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P37883 and previous config saved to /var/cache/conftool/dbconfig/20221103-080257-marostegui.json
  • 08:01 moritzm: installing exim4 security updates
  • 07:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye
  • 07:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1025.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 07:55 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1025.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 07:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P37882 and previous config saved to /var/cache/conftool/dbconfig/20221103-074748-marostegui.json
  • 07:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37881 and previous config saved to /var/cache/conftool/dbconfig/20221103-073240-marostegui.json
  • 07:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37880 and previous config saved to /var/cache/conftool/dbconfig/20221103-073028-marostegui.json
  • 07:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 07:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 07:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37879 and previous config saved to /var/cache/conftool/dbconfig/20221103-073004-marostegui.json
  • 07:15 marostegui: Create idm and idm_staging databases on m5 T320426
  • 07:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P37878 and previous config saved to /var/cache/conftool/dbconfig/20221103-071455-marostegui.json
  • 06:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P37877 and previous config saved to /var/cache/conftool/dbconfig/20221103-065946-marostegui.json
  • 06:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37876 and previous config saved to /var/cache/conftool/dbconfig/20221103-064438-marostegui.json
  • 06:42 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37875 and previous config saved to /var/cache/conftool/dbconfig/20221103-064225-marostegui.json
  • 06:42 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 06:42 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 06:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1130.eqiad.wmnet with reason: Maintenance
  • 06:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1130.eqiad.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2113.codfw.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2113.codfw.wmnet with reason: Maintenance

2022-11-02

  • 23:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37874 and previous config saved to /var/cache/conftool/dbconfig/20221102-232540-ladsgroup.json
  • 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P37873 and previous config saved to /var/cache/conftool/dbconfig/20221102-231031-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P37872 and previous config saved to /var/cache/conftool/dbconfig/20221102-225523-ladsgroup.json
  • 22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37871 and previous config saved to /var/cache/conftool/dbconfig/20221102-224014-ladsgroup.json
  • 21:58 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp4052']
  • 21:57 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 21:53 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp4052']
  • 21:53 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 21:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp4052.mgmt.ulsfo.wmnet with reboot policy FORCED
  • 21:31 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cp4052.mgmt.ulsfo.wmnet with reboot policy FORCED
  • 21:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T318605)', diff saved to https://phabricator.wikimedia.org/P37869 and previous config saved to /var/cache/conftool/dbconfig/20221102-211342-ladsgroup.json
  • 20:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P37868 and previous config saved to /var/cache/conftool/dbconfig/20221102-205833-ladsgroup.json
  • 20:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P37867 and previous config saved to /var/cache/conftool/dbconfig/20221102-204325-ladsgroup.json
  • 20:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37866 and previous config saved to /var/cache/conftool/dbconfig/20221102-203621-ladsgroup.json
  • 20:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 20:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 20:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318605)', diff saved to https://phabricator.wikimedia.org/P37865 and previous config saved to /var/cache/conftool/dbconfig/20221102-203547-ladsgroup.json
  • 20:33 TheresNoTime: UTC late backport window
  • 20:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T318605)', diff saved to https://phabricator.wikimedia.org/P37864 and previous config saved to /var/cache/conftool/dbconfig/20221102-202815-ladsgroup.json
  • 20:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P37863 and previous config saved to /var/cache/conftool/dbconfig/20221102-202037-ladsgroup.json
  • 20:18 samtar@deploy1002: Finished scap: Backport for Fix remaining Wikipedia logos (T319223) (duration: 05m 46s)
  • 20:16 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:15 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:14 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:13 samtar@deploy1002: samtar and jdlrobson: Backport for Fix remaining Wikipedia logos (T319223) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 20:12 samtar@deploy1002: Started scap: Backport for Fix remaining Wikipedia logos (T319223)
  • 20:10 samtar@deploy1002: Finished scap: Backport for Remove Research Incentive survey from enwiki (T318333) (duration: 05m 45s)
  • 20:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T318605)', diff saved to https://phabricator.wikimedia.org/P37862 and previous config saved to /var/cache/conftool/dbconfig/20221102-200610-ladsgroup.json
  • 20:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 20:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 20:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T318605)', diff saved to https://phabricator.wikimedia.org/P37861 and previous config saved to /var/cache/conftool/dbconfig/20221102-200537-ladsgroup.json
  • 20:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P37860 and previous config saved to /var/cache/conftool/dbconfig/20221102-200528-ladsgroup.json
  • 20:05 samtar@deploy1002: samtar and dani: Backport for Remove Research Incentive survey from enwiki (T318333) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 20:05 samtar@deploy1002: Started scap: Backport for Remove Research Incentive survey from enwiki (T318333)
  • 20:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:01 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 19:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P37859 and previous config saved to /var/cache/conftool/dbconfig/20221102-195029-ladsgroup.json
  • 19:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318605)', diff saved to https://phabricator.wikimedia.org/P37858 and previous config saved to /var/cache/conftool/dbconfig/20221102-195019-ladsgroup.json
  • 19:45 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp4052
  • 19:44 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp4052
  • 19:39 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:37 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 19:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P37857 and previous config saved to /var/cache/conftool/dbconfig/20221102-193522-ladsgroup.json
  • 19:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T321123)', diff saved to https://phabricator.wikimedia.org/P37856 and previous config saved to /var/cache/conftool/dbconfig/20221102-192039-marostegui.json
  • 19:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T318605)', diff saved to https://phabricator.wikimedia.org/P37855 and previous config saved to /var/cache/conftool/dbconfig/20221102-192014-ladsgroup.json
  • 19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T318605)', diff saved to https://phabricator.wikimedia.org/P37854 and previous config saved to /var/cache/conftool/dbconfig/20221102-191623-ladsgroup.json
  • 19:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 19:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 19:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 19:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 19:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318605)', diff saved to https://phabricator.wikimedia.org/P37853 and previous config saved to /var/cache/conftool/dbconfig/20221102-191557-ladsgroup.json
  • 19:11 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=codfw,name=phab2001-vcs.codfw.wmnet
  • 19:11 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,name=phab2001-vcs.codfw.wmnet
  • 19:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P37852 and previous config saved to /var/cache/conftool/dbconfig/20221102-190531-marostegui.json
  • 19:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P37851 and previous config saved to /var/cache/conftool/dbconfig/20221102-190048-ladsgroup.json
  • 18:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T318605)', diff saved to https://phabricator.wikimedia.org/P37850 and previous config saved to /var/cache/conftool/dbconfig/20221102-185627-ladsgroup.json
  • 18:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 18:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T318605)', diff saved to https://phabricator.wikimedia.org/P37849 and previous config saved to /var/cache/conftool/dbconfig/20221102-185604-ladsgroup.json
  • 18:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P37848 and previous config saved to /var/cache/conftool/dbconfig/20221102-185023-marostegui.json
  • 18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P37847 and previous config saved to /var/cache/conftool/dbconfig/20221102-184538-ladsgroup.json
  • 18:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P37846 and previous config saved to /var/cache/conftool/dbconfig/20221102-184056-ladsgroup.json
  • 18:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T321123)', diff saved to https://phabricator.wikimedia.org/P37845 and previous config saved to /var/cache/conftool/dbconfig/20221102-183514-marostegui.json
  • 18:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2178 (T321123)', diff saved to https://phabricator.wikimedia.org/P37844 and previous config saved to /var/cache/conftool/dbconfig/20221102-183327-marostegui.json
  • 18:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 18:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 18:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37843 and previous config saved to /var/cache/conftool/dbconfig/20221102-183305-marostegui.json
  • 18:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318605)', diff saved to https://phabricator.wikimedia.org/P37842 and previous config saved to /var/cache/conftool/dbconfig/20221102-183031-ladsgroup.json
  • 18:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P37841 and previous config saved to /var/cache/conftool/dbconfig/20221102-182548-ladsgroup.json
  • 18:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P37840 and previous config saved to /var/cache/conftool/dbconfig/20221102-181753-marostegui.json
  • 18:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T318605)', diff saved to https://phabricator.wikimedia.org/P37839 and previous config saved to /var/cache/conftool/dbconfig/20221102-181039-ladsgroup.json
  • 18:10 jhuneidi@deploy1002: Synchronized php: group1 wikis to 1.40.0-wmf.8 refs T320513 (duration: 03m 43s)
  • 18:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:07 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.8 refs T320513
  • 18:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P37838 and previous config saved to /var/cache/conftool/dbconfig/20221102-180245-marostegui.json
  • 18:01 ejegg: updated fundraising python tools from 4c143d97 to 72570bdd
  • 17:52 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37837 and previous config saved to /var/cache/conftool/dbconfig/20221102-174737-marostegui.json
  • 17:46 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cp4052.ulsfo.wmnet
  • 17:46 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37836 and previous config saved to /var/cache/conftool/dbconfig/20221102-174504-marostegui.json
  • 17:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 17:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 17:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321123)', diff saved to https://phabricator.wikimedia.org/P37835 and previous config saved to /var/cache/conftool/dbconfig/20221102-174451-marostegui.json
  • 17:43 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 17:41 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 17:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 17:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 17:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T318955)', diff saved to https://phabricator.wikimedia.org/P37834 and previous config saved to /var/cache/conftool/dbconfig/20221102-174138-ladsgroup.json
  • 17:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T318605)', diff saved to https://phabricator.wikimedia.org/P37833 and previous config saved to /var/cache/conftool/dbconfig/20221102-174110-ladsgroup.json
  • 17:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 17:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 17:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T318605)', diff saved to https://phabricator.wikimedia.org/P37832 and previous config saved to /var/cache/conftool/dbconfig/20221102-174048-ladsgroup.json
  • 17:40 mutante: clouddumps1002 - /usr/local/bin/dump-fetch-phabdumps.sh T322221
  • 17:39 pt1979@cumin2002: START - Cookbook sre.hosts.decommission for hosts cp4052.ulsfo.wmnet
  • 17:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P37831 and previous config saved to /var/cache/conftool/dbconfig/20221102-172944-marostegui.json
  • 17:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P37830 and previous config saved to /var/cache/conftool/dbconfig/20221102-172630-ladsgroup.json
  • 17:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P37829 and previous config saved to /var/cache/conftool/dbconfig/20221102-172540-ladsgroup.json
  • 17:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P37828 and previous config saved to /var/cache/conftool/dbconfig/20221102-171436-marostegui.json
  • 17:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P37827 and previous config saved to /var/cache/conftool/dbconfig/20221102-171122-ladsgroup.json
  • 17:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P37826 and previous config saved to /var/cache/conftool/dbconfig/20221102-171032-ladsgroup.json
  • 17:06 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:04 hashar@deploy1002: Finished deploy [integration/docroot@8d2f4a0]: Remove .zuul-change font-weight - T322168 (duration: 00m 10s)
  • 17:04 hashar@deploy1002: Started deploy [integration/docroot@8d2f4a0]: Remove .zuul-change font-weight - T322168
  • 16:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321123)', diff saved to https://phabricator.wikimedia.org/P37825 and previous config saved to /var/cache/conftool/dbconfig/20221102-165927-marostegui.json
  • 16:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2157 (T321123)', diff saved to https://phabricator.wikimedia.org/P37824 and previous config saved to /var/cache/conftool/dbconfig/20221102-165656-marostegui.json
  • 16:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 16:56 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:56 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 16:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37823 and previous config saved to /var/cache/conftool/dbconfig/20221102-165631-marostegui.json
  • 16:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T318955)', diff saved to https://phabricator.wikimedia.org/P37822 and previous config saved to /var/cache/conftool/dbconfig/20221102-165614-ladsgroup.json
  • 16:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T318605)', diff saved to https://phabricator.wikimedia.org/P37821 and previous config saved to /var/cache/conftool/dbconfig/20221102-165523-ladsgroup.json
  • 16:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T318955)', diff saved to https://phabricator.wikimedia.org/P37820 and previous config saved to /var/cache/conftool/dbconfig/20221102-165400-ladsgroup.json
  • 16:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 16:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T318955)', diff saved to https://phabricator.wikimedia.org/P37819 and previous config saved to /var/cache/conftool/dbconfig/20221102-165337-ladsgroup.json
  • 16:52 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 16:46 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 16:45 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 16:44 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 16:43 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
  • 16:42 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:41 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
  • 16:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P37818 and previous config saved to /var/cache/conftool/dbconfig/20221102-164123-marostegui.json
  • 16:40 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
  • 16:40 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 16:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P37817 and previous config saved to /var/cache/conftool/dbconfig/20221102-163829-ladsgroup.json
  • 16:37 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 16:36 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 16:35 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 16:35 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
  • 16:34 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
  • 16:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T318605)', diff saved to https://phabricator.wikimedia.org/P37816 and previous config saved to /var/cache/conftool/dbconfig/20221102-163334-ladsgroup.json
  • 16:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 16:33 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
  • 16:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 16:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T318605)', diff saved to https://phabricator.wikimedia.org/P37815 and previous config saved to /var/cache/conftool/dbconfig/20221102-163300-ladsgroup.json
  • 16:32 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 16:31 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 16:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T318605)', diff saved to https://phabricator.wikimedia.org/P37814 and previous config saved to /var/cache/conftool/dbconfig/20221102-162629-ladsgroup.json
  • 16:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P37813 and previous config saved to /var/cache/conftool/dbconfig/20221102-162614-marostegui.json
  • 16:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P37812 and previous config saved to /var/cache/conftool/dbconfig/20221102-162320-ladsgroup.json
  • 16:22 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 16:22 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:21 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 16:21 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 16:20 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
  • 16:19 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
  • 16:19 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
  • 16:19 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 16:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P37811 and previous config saved to /var/cache/conftool/dbconfig/20221102-161753-ladsgroup.json
  • 16:12 Daimona: Creating schema for the CampaignEvents extension on testwiki, test2wiki and officewiki # T318595
  • 16:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37810 and previous config saved to /var/cache/conftool/dbconfig/20221102-161104-marostegui.json
  • 16:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37809 and previous config saved to /var/cache/conftool/dbconfig/20221102-160834-marostegui.json
  • 16:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 16:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 16:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T321123)', diff saved to https://phabricator.wikimedia.org/P37808 and previous config saved to /var/cache/conftool/dbconfig/20221102-160809-marostegui.json
  • 16:06 moritzm: installing glibc security updates on buster
  • 16:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T318955)', diff saved to https://phabricator.wikimedia.org/P37807 and previous config saved to /var/cache/conftool/dbconfig/20221102-160600-ladsgroup.json
  • 16:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T318955)', diff saved to https://phabricator.wikimedia.org/P37806 and previous config saved to /var/cache/conftool/dbconfig/20221102-160537-ladsgroup.json
  • 16:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P37805 and previous config saved to /var/cache/conftool/dbconfig/20221102-160243-ladsgroup.json
  • 16:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 16:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T318950)', diff saved to https://phabricator.wikimedia.org/P37804 and previous config saved to /var/cache/conftool/dbconfig/20221102-160136-ladsgroup.json
  • 15:53 dcausse: restarting blazegraph on wdqs1007 (BlazegraphFreeAllocatorsDecreasingRapidly)
  • 15:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P37803 and previous config saved to /var/cache/conftool/dbconfig/20221102-155302-marostegui.json
  • 15:52 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 15:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P37802 and previous config saved to /var/cache/conftool/dbconfig/20221102-155026-ladsgroup.json
  • 15:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T318605)', diff saved to https://phabricator.wikimedia.org/P37801 and previous config saved to /var/cache/conftool/dbconfig/20221102-154736-ladsgroup.json
  • 15:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P37800 and previous config saved to /var/cache/conftool/dbconfig/20221102-154628-ladsgroup.json
  • 15:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P37799 and previous config saved to /var/cache/conftool/dbconfig/20221102-153754-marostegui.json
  • 15:35 otto@cumin1001: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
  • 15:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P37798 and previous config saved to /var/cache/conftool/dbconfig/20221102-153519-ladsgroup.json
  • 15:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P37797 and previous config saved to /var/cache/conftool/dbconfig/20221102-153121-ladsgroup.json
  • 15:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T321123)', diff saved to https://phabricator.wikimedia.org/P37796 and previous config saved to /var/cache/conftool/dbconfig/20221102-152244-marostegui.json
  • 15:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2128 (T321123)', diff saved to https://phabricator.wikimedia.org/P37795 and previous config saved to /var/cache/conftool/dbconfig/20221102-152113-marostegui.json
  • 15:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 15:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 15:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 15:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 15:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T321123)', diff saved to https://phabricator.wikimedia.org/P37794 and previous config saved to /var/cache/conftool/dbconfig/20221102-152045-marostegui.json
  • 15:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T318955)', diff saved to https://phabricator.wikimedia.org/P37793 and previous config saved to /var/cache/conftool/dbconfig/20221102-152012-ladsgroup.json
  • 15:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T318950)', diff saved to https://phabricator.wikimedia.org/P37792 and previous config saved to /var/cache/conftool/dbconfig/20221102-151613-ladsgroup.json
  • 15:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1201 (T318950)', diff saved to https://phabricator.wikimedia.org/P37791 and previous config saved to /var/cache/conftool/dbconfig/20221102-151403-ladsgroup.json
  • 15:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 15:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 15:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T318950)', diff saved to https://phabricator.wikimedia.org/P37790 and previous config saved to /var/cache/conftool/dbconfig/20221102-151341-ladsgroup.json
  • 15:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P37789 and previous config saved to /var/cache/conftool/dbconfig/20221102-150538-marostegui.json
  • 15:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T318955)', diff saved to https://phabricator.wikimedia.org/P37788 and previous config saved to /var/cache/conftool/dbconfig/20221102-150508-ladsgroup.json
  • 15:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37787 and previous config saved to /var/cache/conftool/dbconfig/20221102-150444-ladsgroup.json
  • 14:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P37786 and previous config saved to /var/cache/conftool/dbconfig/20221102-145833-ladsgroup.json
  • 14:53 otto@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
  • 14:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P37785 and previous config saved to /var/cache/conftool/dbconfig/20221102-145030-marostegui.json
  • 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P37784 and previous config saved to /var/cache/conftool/dbconfig/20221102-144937-ladsgroup.json
  • 14:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T318605)', diff saved to https://phabricator.wikimedia.org/P37783 and previous config saved to /var/cache/conftool/dbconfig/20221102-144719-ladsgroup.json
  • 14:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 14:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 14:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318605)', diff saved to https://phabricator.wikimedia.org/P37782 and previous config saved to /var/cache/conftool/dbconfig/20221102-144657-ladsgroup.json
  • 14:45 moritzm: installing ffmpeg security updates on bullseye
  • 14:43 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 14:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P37781 and previous config saved to /var/cache/conftool/dbconfig/20221102-144325-ladsgroup.json
  • 14:41 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 14:38 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:37 jhathaway@cumin1001: START - Cookbook sre.dns.netbox
  • 14:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T321123)', diff saved to https://phabricator.wikimedia.org/P37780 and previous config saved to /var/cache/conftool/dbconfig/20221102-143522-marostegui.json
  • 14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P37779 and previous config saved to /var/cache/conftool/dbconfig/20221102-143430-ladsgroup.json
  • 14:34 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 14:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2123 (T321123)', diff saved to https://phabricator.wikimedia.org/P37778 and previous config saved to /var/cache/conftool/dbconfig/20221102-143350-marostegui.json
  • 14:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 14:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 14:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T321123)', diff saved to https://phabricator.wikimedia.org/P37777 and previous config saved to /var/cache/conftool/dbconfig/20221102-143324-marostegui.json
  • 14:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P37776 and previous config saved to /var/cache/conftool/dbconfig/20221102-143150-ladsgroup.json
  • 14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T318950)', diff saved to https://phabricator.wikimedia.org/P37775 and previous config saved to /var/cache/conftool/dbconfig/20221102-142818-ladsgroup.json
  • 14:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T318950)', diff saved to https://phabricator.wikimedia.org/P37774 and previous config saved to /var/cache/conftool/dbconfig/20221102-142605-ladsgroup.json
  • 14:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 14:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 14:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T318950)', diff saved to https://phabricator.wikimedia.org/P37773 and previous config saved to /var/cache/conftool/dbconfig/20221102-142540-ladsgroup.json
  • 14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T318950)', diff saved to https://phabricator.wikimedia.org/P37772 and previous config saved to /var/cache/conftool/dbconfig/20221102-142345-ladsgroup.json
  • 14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T318955)', diff saved to https://phabricator.wikimedia.org/P37771 and previous config saved to /var/cache/conftool/dbconfig/20221102-142331-ladsgroup.json
  • 14:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 14:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318605)', diff saved to https://phabricator.wikimedia.org/P37770 and previous config saved to /var/cache/conftool/dbconfig/20221102-142258-ladsgroup.json
  • 14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37769 and previous config saved to /var/cache/conftool/dbconfig/20221102-141922-ladsgroup.json
  • 14:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P37768 and previous config saved to /var/cache/conftool/dbconfig/20221102-141815-marostegui.json
  • 14:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P37767 and previous config saved to /var/cache/conftool/dbconfig/20221102-141640-ladsgroup.json
  • 14:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P37766 and previous config saved to /var/cache/conftool/dbconfig/20221102-141029-ladsgroup.json
  • 14:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P37765 and previous config saved to /var/cache/conftool/dbconfig/20221102-140837-ladsgroup.json
  • 14:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P37764 and previous config saved to /var/cache/conftool/dbconfig/20221102-140822-ladsgroup.json
  • 14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P37763 and previous config saved to /var/cache/conftool/dbconfig/20221102-140749-ladsgroup.json
  • 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 100%: After reboot', diff saved to https://phabricator.wikimedia.org/P37762 and previous config saved to /var/cache/conftool/dbconfig/20221102-140355-root.json
  • 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P37761 and previous config saved to /var/cache/conftool/dbconfig/20221102-140307-marostegui.json
  • 14:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318605)', diff saved to https://phabricator.wikimedia.org/P37760 and previous config saved to /var/cache/conftool/dbconfig/20221102-140133-ladsgroup.json
  • 14:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37759 and previous config saved to /var/cache/conftool/dbconfig/20221102-135746-ladsgroup.json
  • 13:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318955)', diff saved to https://phabricator.wikimedia.org/P37758 and previous config saved to /var/cache/conftool/dbconfig/20221102-135723-ladsgroup.json
  • 13:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P37757 and previous config saved to /var/cache/conftool/dbconfig/20221102-135521-ladsgroup.json
  • 13:54 vgutierrez: re-enabled puppet in A:cp - T321776
  • 13:54 moritzm: import puppetdb 7.11.2-1 to component/puppetdb7 for bookworm-wikimedia T321783
  • 13:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P37756 and previous config saved to /var/cache/conftool/dbconfig/20221102-135328-ladsgroup.json
  • 13:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P37755 and previous config saved to /var/cache/conftool/dbconfig/20221102-135315-ladsgroup.json
  • 13:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P37754 and previous config saved to /var/cache/conftool/dbconfig/20221102-135240-ladsgroup.json
  • 13:48 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 75%: After reboot', diff saved to https://phabricator.wikimedia.org/P37753 and previous config saved to /var/cache/conftool/dbconfig/20221102-134849-root.json
  • 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T321123)', diff saved to https://phabricator.wikimedia.org/P37752 and previous config saved to /var/cache/conftool/dbconfig/20221102-134758-marostegui.json
  • 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2111 (T321123)', diff saved to https://phabricator.wikimedia.org/P37751 and previous config saved to /var/cache/conftool/dbconfig/20221102-134527-marostegui.json
  • 13:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 13:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T321123)', diff saved to https://phabricator.wikimedia.org/P37750 and previous config saved to /var/cache/conftool/dbconfig/20221102-134404-marostegui.json
  • 13:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37749 and previous config saved to /var/cache/conftool/dbconfig/20221102-134216-ladsgroup.json
  • 13:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T318950)', diff saved to https://phabricator.wikimedia.org/P37748 and previous config saved to /var/cache/conftool/dbconfig/20221102-134012-ladsgroup.json
  • 13:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T318950)', diff saved to https://phabricator.wikimedia.org/P37747 and previous config saved to /var/cache/conftool/dbconfig/20221102-133819-ladsgroup.json
  • 13:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T318955)', diff saved to https://phabricator.wikimedia.org/P37746 and previous config saved to /var/cache/conftool/dbconfig/20221102-133807-ladsgroup.json
  • 13:38 vgutierrez: uploaded trafficserver 9.1.3-1wm2 to apt.wm.o (buster-wikimedia) - T321776
  • 13:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318605)', diff saved to https://phabricator.wikimedia.org/P37745 and previous config saved to /var/cache/conftool/dbconfig/20221102-133733-ladsgroup.json
  • 13:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T318605)', diff saved to https://phabricator.wikimedia.org/P37744 and previous config saved to /var/cache/conftool/dbconfig/20221102-133637-ladsgroup.json
  • 13:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 13:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 13:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T318950)', diff saved to https://phabricator.wikimedia.org/P37743 and previous config saved to /var/cache/conftool/dbconfig/20221102-133559-ladsgroup.json
  • 13:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T318955)', diff saved to https://phabricator.wikimedia.org/P37742 and previous config saved to /var/cache/conftool/dbconfig/20221102-133549-ladsgroup.json
  • 13:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 13:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 13:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 13:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37741 and previous config saved to /var/cache/conftool/dbconfig/20221102-133533-ladsgroup.json
  • 13:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 13:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37740 and previous config saved to /var/cache/conftool/dbconfig/20221102-133526-ladsgroup.json
  • 13:35 vgutierrez: vgutierrez@apt1001:~$ sudo -i reprepro --delete clearvanished - T321776
  • 13:35 vgutierrez: vgutierrez@apt1001:~$ sudo -i reprepro clearvanished - T321776
  • 13:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T318950)', diff saved to https://phabricator.wikimedia.org/P37739 and previous config saved to /var/cache/conftool/dbconfig/20221102-133402-ladsgroup.json
  • 13:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 50%: After reboot', diff saved to https://phabricator.wikimedia.org/P37738 and previous config saved to /var/cache/conftool/dbconfig/20221102-133343-root.json
  • 13:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 13:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T318950)', diff saved to https://phabricator.wikimedia.org/P37737 and previous config saved to /var/cache/conftool/dbconfig/20221102-133338-ladsgroup.json
  • 13:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P37736 and previous config saved to /var/cache/conftool/dbconfig/20221102-132855-marostegui.json
  • 13:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37735 and previous config saved to /var/cache/conftool/dbconfig/20221102-132707-ladsgroup.json
  • 13:22 vgutierrez: disable puppet on A:cp before merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/850087 - T321776
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P37734 and previous config saved to /var/cache/conftool/dbconfig/20221102-132025-ladsgroup.json
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P37733 and previous config saved to /var/cache/conftool/dbconfig/20221102-132017-ladsgroup.json
  • 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 25%: After reboot', diff saved to https://phabricator.wikimedia.org/P37732 and previous config saved to /var/cache/conftool/dbconfig/20221102-131837-root.json
  • 13:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P37731 and previous config saved to /var/cache/conftool/dbconfig/20221102-131830-ladsgroup.json
  • 13:16 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:15 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:15 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:14 Lucas_WMDE: UTC afternoon backport+config window done
  • 13:14 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P37730 and previous config saved to /var/cache/conftool/dbconfig/20221102-131348-marostegui.json
  • 13:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318955)', diff saved to https://phabricator.wikimedia.org/P37729 and previous config saved to /var/cache/conftool/dbconfig/20221102-131159-ladsgroup.json
  • 13:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T318955)', diff saved to https://phabricator.wikimedia.org/P37728 and previous config saved to /var/cache/conftool/dbconfig/20221102-130948-ladsgroup.json
  • 13:10 cjming@deploy1002: Finished scap: Backport for testwiki: Add mediawiki.visual_editor_feature_use stream (T309602) (duration: 04m 56s)
  • 13:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 13:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T318955)', diff saved to https://phabricator.wikimedia.org/P37727 and previous config saved to /var/cache/conftool/dbconfig/20221102-130923-ladsgroup.json
  • 13:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P37726 and previous config saved to /var/cache/conftool/dbconfig/20221102-130518-ladsgroup.json
  • 13:05 cjming@deploy1002: cjming and cjming: Backport for testwiki: Add mediawiki.visual_editor_feature_use stream (T309602) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 13:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P37725 and previous config saved to /var/cache/conftool/dbconfig/20221102-130509-ladsgroup.json
  • 13:04 cjming@deploy1002: Started scap: Backport for testwiki: Add mediawiki.visual_editor_feature_use stream (T309602)
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 10%: After reboot', diff saved to https://phabricator.wikimedia.org/P37724 and previous config saved to /var/cache/conftool/dbconfig/20221102-130331-root.json
  • 13:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P37723 and previous config saved to /var/cache/conftool/dbconfig/20221102-130322-ladsgroup.json
  • 12:59 moritzm: draining ganeti1025 for eventual reimage T311687
  • 12:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T321123)', diff saved to https://phabricator.wikimedia.org/P37722 and previous config saved to /var/cache/conftool/dbconfig/20221102-125840-marostegui.json
  • 12:56 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1200 (T321123)', diff saved to https://phabricator.wikimedia.org/P37721 and previous config saved to /var/cache/conftool/dbconfig/20221102-125607-marostegui.json
  • 12:56 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T321123)', diff saved to https://phabricator.wikimedia.org/P37720 and previous config saved to /var/cache/conftool/dbconfig/20221102-125544-marostegui.json
  • 12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P37719 and previous config saved to /var/cache/conftool/dbconfig/20221102-125415-ladsgroup.json
  • 12:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37718 and previous config saved to /var/cache/conftool/dbconfig/20221102-125009-ladsgroup.json
  • 12:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37717 and previous config saved to /var/cache/conftool/dbconfig/20221102-125001-ladsgroup.json
  • 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 5%: After reboot', diff saved to https://phabricator.wikimedia.org/P37716 and previous config saved to /var/cache/conftool/dbconfig/20221102-124824-root.json
  • 12:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T318950)', diff saved to https://phabricator.wikimedia.org/P37715 and previous config saved to /var/cache/conftool/dbconfig/20221102-124812-ladsgroup.json
  • 12:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37714 and previous config saved to /var/cache/conftool/dbconfig/20221102-124754-ladsgroup.json
  • 12:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37713 and previous config saved to /var/cache/conftool/dbconfig/20221102-124743-ladsgroup.json
  • 12:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 12:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37712 and previous config saved to /var/cache/conftool/dbconfig/20221102-124732-ladsgroup.json
  • 12:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T318955)', diff saved to https://phabricator.wikimedia.org/P37711 and previous config saved to /var/cache/conftool/dbconfig/20221102-124720-ladsgroup.json
  • 12:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T318950)', diff saved to https://phabricator.wikimedia.org/P37710 and previous config saved to /var/cache/conftool/dbconfig/20221102-124602-ladsgroup.json
  • 12:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 12:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 12:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T318950)', diff saved to https://phabricator.wikimedia.org/P37709 and previous config saved to /var/cache/conftool/dbconfig/20221102-124537-ladsgroup.json
  • 12:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P37708 and previous config saved to /var/cache/conftool/dbconfig/20221102-124037-marostegui.json
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P37707 and previous config saved to /var/cache/conftool/dbconfig/20221102-123906-ladsgroup.json
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 3%: After reboot', diff saved to https://phabricator.wikimedia.org/P37706 and previous config saved to /var/cache/conftool/dbconfig/20221102-123319-root.json
  • 12:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P37705 and previous config saved to /var/cache/conftool/dbconfig/20221102-123224-ladsgroup.json
  • 12:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P37704 and previous config saved to /var/cache/conftool/dbconfig/20221102-123213-ladsgroup.json
  • 12:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P37701 and previous config saved to /var/cache/conftool/dbconfig/20221102-123029-ladsgroup.json
  • 12:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P37700 and previous config saved to /var/cache/conftool/dbconfig/20221102-122529-marostegui.json
  • 12:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T318955)', diff saved to https://phabricator.wikimedia.org/P37699 and previous config saved to /var/cache/conftool/dbconfig/20221102-122356-ladsgroup.json
  • 12:18 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 1%: After reboot', diff saved to https://phabricator.wikimedia.org/P37698 and previous config saved to /var/cache/conftool/dbconfig/20221102-121812-root.json
  • 12:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P37697 and previous config saved to /var/cache/conftool/dbconfig/20221102-121716-ladsgroup.json
  • 12:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P37696 and previous config saved to /var/cache/conftool/dbconfig/20221102-121704-ladsgroup.json
  • 12:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P37695 and previous config saved to /var/cache/conftool/dbconfig/20221102-121521-ladsgroup.json
  • 12:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 12:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 12:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 12:10 marostegui@deploy1002: Finished scap: Backport for Revert "db-production: Disable writes one es4" (duration: 04m 37s)
  • 12:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T321123)', diff saved to https://phabricator.wikimedia.org/P37694 and previous config saved to /var/cache/conftool/dbconfig/20221102-121020-marostegui.json
  • 12:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 12:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1185 (T321123)', diff saved to https://phabricator.wikimedia.org/P37693 and previous config saved to /var/cache/conftool/dbconfig/20221102-120805-marostegui.json
  • 12:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 12:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 12:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T321123)', diff saved to https://phabricator.wikimedia.org/P37692 and previous config saved to /var/cache/conftool/dbconfig/20221102-120742-marostegui.json
  • 12:08 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "db-production: Disable writes one es4" synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 12:06 marostegui@deploy1002: Started scap: Backport for Revert "db-production: Disable writes one es4"
  • 12:06 marostegui@deploy1002: Backport cancelled.
  • 12:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T318955)', diff saved to https://phabricator.wikimedia.org/P37691 and previous config saved to /var/cache/conftool/dbconfig/20221102-120505-ladsgroup.json
  • 12:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 12:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 12:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37690 and previous config saved to /var/cache/conftool/dbconfig/20221102-120436-ladsgroup.json
  • 12:04 marostegui@cumin1001: dbctl commit (dc=all): 'Add some weight to es4 master', diff saved to https://phabricator.wikimedia.org/P37689 and previous config saved to /var/cache/conftool/dbconfig/20221102-120233-marostegui.json
  • 12:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37688 and previous config saved to /var/cache/conftool/dbconfig/20221102-120209-ladsgroup.json
  • 12:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T318955)', diff saved to https://phabricator.wikimedia.org/P37687 and previous config saved to /var/cache/conftool/dbconfig/20221102-120157-ladsgroup.json
  • 12:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T318950)', diff saved to https://phabricator.wikimedia.org/P37686 and previous config saved to /var/cache/conftool/dbconfig/20221102-120013-ladsgroup.json
  • 12:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37685 and previous config saved to /var/cache/conftool/dbconfig/20221102-115948-ladsgroup.json
  • 12:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 12:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T318955)', diff saved to https://phabricator.wikimedia.org/P37684 and previous config saved to /var/cache/conftool/dbconfig/20221102-115940-ladsgroup.json
  • 12:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 11:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T318950)', diff saved to https://phabricator.wikimedia.org/P37683 and previous config saved to /var/cache/conftool/dbconfig/20221102-115925-ladsgroup.json
  • 11:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37682 and previous config saved to /var/cache/conftool/dbconfig/20221102-115855-ladsgroup.json
  • 11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T318950)', diff saved to https://phabricator.wikimedia.org/P37681 and previous config saved to /var/cache/conftool/dbconfig/20221102-115802-ladsgroup.json
  • 11:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 11:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 11:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 11:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 11:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 11:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 11:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T318950)', diff saved to https://phabricator.wikimedia.org/P37680 and previous config saved to /var/cache/conftool/dbconfig/20221102-115705-ladsgroup.json
  • 11:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es1020', diff saved to https://phabricator.wikimedia.org/P37679 and previous config saved to /var/cache/conftool/dbconfig/20221102-115448-root.json
  • 11:53 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es1021 to es4 primary T322181', diff saved to https://phabricator.wikimedia.org/P37678 and previous config saved to /var/cache/conftool/dbconfig/20221102-115313-root.json
  • 11:52 marostegui: Starting es4 eqiad failover from es1020 to es1021 - T322181
  • 11:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P37677 and previous config saved to /var/cache/conftool/dbconfig/20221102-115233-marostegui.json
  • 11:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 11:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T318605)', diff saved to https://phabricator.wikimedia.org/P37676 and previous config saved to /var/cache/conftool/dbconfig/20221102-115023-ladsgroup.json
  • 11:49 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 11:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P37675 and previous config saved to /var/cache/conftool/dbconfig/20221102-114927-ladsgroup.json
  • 11:48 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 11:48 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 11:47 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 11:47 marostegui@deploy1002: Finished scap: Backport for db-production: Disable writes one es4 (T322181) (duration: 04m 43s)
  • 11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P37674 and previous config saved to /var/cache/conftool/dbconfig/20221102-114416-ladsgroup.json
  • 11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P37673 and previous config saved to /var/cache/conftool/dbconfig/20221102-114347-ladsgroup.json
  • 11:43 marostegui@deploy1002: marostegui and marostegui: Backport for db-production: Disable writes one es4 (T322181) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 11:42 marostegui@deploy1002: Started scap: Backport for db-production: Disable writes one es4 (T322181)
  • 11:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P37672 and previous config saved to /var/cache/conftool/dbconfig/20221102-114157-ladsgroup.json
  • 11:42 marostegui@cumin1001: dbctl commit (dc=all): 'Set es1021 with weight 0 T322181', diff saved to https://phabricator.wikimedia.org/P37671 and previous config saved to /var/cache/conftool/dbconfig/20221102-114107-root.json
  • 11:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es4 T322181
  • 11:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es4 T322181
  • 11:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P37670 and previous config saved to /var/cache/conftool/dbconfig/20221102-113726-marostegui.json
  • 11:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T318605)', diff saved to https://phabricator.wikimedia.org/P37669 and previous config saved to /var/cache/conftool/dbconfig/20221102-113542-ladsgroup.json
  • 11:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 11:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P37668 and previous config saved to /var/cache/conftool/dbconfig/20221102-113515-ladsgroup.json
  • 11:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 11:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318605)', diff saved to https://phabricator.wikimedia.org/P37667 and previous config saved to /var/cache/conftool/dbconfig/20221102-113506-ladsgroup.json
  • 11:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P37666 and previous config saved to /var/cache/conftool/dbconfig/20221102-113419-ladsgroup.json
  • 11:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P37665 and previous config saved to /var/cache/conftool/dbconfig/20221102-112909-ladsgroup.json
  • 11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P37664 and previous config saved to /var/cache/conftool/dbconfig/20221102-112839-ladsgroup.json
  • 11:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P37663 and previous config saved to /var/cache/conftool/dbconfig/20221102-112648-ladsgroup.json
  • 11:24 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 11:24 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 11:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T321123)', diff saved to https://phabricator.wikimedia.org/P37662 and previous config saved to /var/cache/conftool/dbconfig/20221102-112217-marostegui.json
  • 11:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P37661 and previous config saved to /var/cache/conftool/dbconfig/20221102-112008-ladsgroup.json
  • 11:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P37660 and previous config saved to /var/cache/conftool/dbconfig/20221102-111958-ladsgroup.json
  • 11:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37659 and previous config saved to /var/cache/conftool/dbconfig/20221102-111911-ladsgroup.json
  • 11:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T318950)', diff saved to https://phabricator.wikimedia.org/P37658 and previous config saved to /var/cache/conftool/dbconfig/20221102-111400-ladsgroup.json
  • 11:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37657 and previous config saved to /var/cache/conftool/dbconfig/20221102-111331-ladsgroup.json
  • 11:13 vgutierrez: pool cp1075, cp2027 and cp3050 running HAProxy 2.6.6 - T321775
  • 11:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T318950)', diff saved to https://phabricator.wikimedia.org/P37656 and previous config saved to /var/cache/conftool/dbconfig/20221102-111147-ladsgroup.json
  • 11:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T318950)', diff saved to https://phabricator.wikimedia.org/P37655 and previous config saved to /var/cache/conftool/dbconfig/20221102-111141-ladsgroup.json
  • 11:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 11:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 11:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 11:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 11:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 11:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37654 and previous config saved to /var/cache/conftool/dbconfig/20221102-111113-ladsgroup.json
  • 11:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 11:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 11:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T318950)', diff saved to https://phabricator.wikimedia.org/P37653 and previous config saved to /var/cache/conftool/dbconfig/20221102-111059-ladsgroup.json
  • 11:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 11:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T318955)', diff saved to https://phabricator.wikimedia.org/P37652 and previous config saved to /var/cache/conftool/dbconfig/20221102-111051-ladsgroup.json
  • 11:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T318950)', diff saved to https://phabricator.wikimedia.org/P37651 and previous config saved to /var/cache/conftool/dbconfig/20221102-110931-ladsgroup.json
  • 11:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 11:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 11:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37650 and previous config saved to /var/cache/conftool/dbconfig/20221102-110909-ladsgroup.json
  • 11:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P37649 and previous config saved to /var/cache/conftool/dbconfig/20221102-110451-ladsgroup.json
  • 11:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37648 and previous config saved to /var/cache/conftool/dbconfig/20221102-110314-ladsgroup.json
  • 11:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 11:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:57 vgutierrez: depool cp1075, cp2027 and cp3050 prior to HAProxy 2.6 upgrade - T321775
  • 10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P37647 and previous config saved to /var/cache/conftool/dbconfig/20221102-105551-ladsgroup.json
  • 10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P37646 and previous config saved to /var/cache/conftool/dbconfig/20221102-105544-ladsgroup.json
  • 10:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P37645 and previous config saved to /var/cache/conftool/dbconfig/20221102-105400-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318605)', diff saved to https://phabricator.wikimedia.org/P37644 and previous config saved to /var/cache/conftool/dbconfig/20221102-104942-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T318955)', diff saved to https://phabricator.wikimedia.org/P37643 and previous config saved to /var/cache/conftool/dbconfig/20221102-104932-ladsgroup.json
  • 10:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P37642 and previous config saved to /var/cache/conftool/dbconfig/20221102-104042-ladsgroup.json
  • 10:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P37641 and previous config saved to /var/cache/conftool/dbconfig/20221102-104034-ladsgroup.json
  • 10:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P37640 and previous config saved to /var/cache/conftool/dbconfig/20221102-103851-ladsgroup.json
  • 10:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T321123)', diff saved to https://phabricator.wikimedia.org/P37639 and previous config saved to /var/cache/conftool/dbconfig/20221102-103555-marostegui.json
  • 10:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 10:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 10:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37638 and previous config saved to /var/cache/conftool/dbconfig/20221102-103453-marostegui.json
  • 10:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P37637 and previous config saved to /var/cache/conftool/dbconfig/20221102-103424-ladsgroup.json
  • 10:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T318605)', diff saved to https://phabricator.wikimedia.org/P37636 and previous config saved to /var/cache/conftool/dbconfig/20221102-103400-ladsgroup.json
  • 10:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 10:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 10:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 10:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 10:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T318950)', diff saved to https://phabricator.wikimedia.org/P37635 and previous config saved to /var/cache/conftool/dbconfig/20221102-102533-ladsgroup.json
  • 10:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T318955)', diff saved to https://phabricator.wikimedia.org/P37634 and previous config saved to /var/cache/conftool/dbconfig/20221102-102527-ladsgroup.json
  • 10:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37633 and previous config saved to /var/cache/conftool/dbconfig/20221102-102342-ladsgroup.json
  • 10:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T318950)', diff saved to https://phabricator.wikimedia.org/P37632 and previous config saved to /var/cache/conftool/dbconfig/20221102-102320-ladsgroup.json
  • 10:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 10:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T318955)', diff saved to https://phabricator.wikimedia.org/P37631 and previous config saved to /var/cache/conftool/dbconfig/20221102-102310-ladsgroup.json
  • 10:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 10:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 10:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 10:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 10:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T318950)', diff saved to https://phabricator.wikimedia.org/P37630 and previous config saved to /var/cache/conftool/dbconfig/20221102-102256-ladsgroup.json
  • 10:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 10:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T318955)', diff saved to https://phabricator.wikimedia.org/P37629 and previous config saved to /var/cache/conftool/dbconfig/20221102-102243-ladsgroup.json
  • 10:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37628 and previous config saved to /var/cache/conftool/dbconfig/20221102-102233-ladsgroup.json
  • 10:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 10:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 10:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37627 and previous config saved to /var/cache/conftool/dbconfig/20221102-102221-ladsgroup.json
  • 10:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P37626 and previous config saved to /var/cache/conftool/dbconfig/20221102-101946-marostegui.json
  • 10:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P37625 and previous config saved to /var/cache/conftool/dbconfig/20221102-101916-ladsgroup.json
  • 10:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P37624 and previous config saved to /var/cache/conftool/dbconfig/20221102-100746-ladsgroup.json
  • 10:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P37623 and previous config saved to /var/cache/conftool/dbconfig/20221102-100736-ladsgroup.json
  • 10:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P37622 and previous config saved to /var/cache/conftool/dbconfig/20221102-100713-ladsgroup.json
  • 10:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P37621 and previous config saved to /var/cache/conftool/dbconfig/20221102-100438-marostegui.json
  • 10:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T318955)', diff saved to https://phabricator.wikimedia.org/P37620 and previous config saved to /var/cache/conftool/dbconfig/20221102-100408-ladsgroup.json
  • 10:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T318955)', diff saved to https://phabricator.wikimedia.org/P37619 and previous config saved to /var/cache/conftool/dbconfig/20221102-100156-ladsgroup.json
  • 10:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 10:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 10:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37618 and previous config saved to /var/cache/conftool/dbconfig/20221102-100133-ladsgroup.json
  • 09:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P37617 and previous config saved to /var/cache/conftool/dbconfig/20221102-095237-ladsgroup.json
  • 09:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P37616 and previous config saved to /var/cache/conftool/dbconfig/20221102-095225-ladsgroup.json
  • 09:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P37615 and previous config saved to /var/cache/conftool/dbconfig/20221102-095205-ladsgroup.json
  • 09:51 moritzm: installing exim4 security updates
  • 09:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37614 and previous config saved to /var/cache/conftool/dbconfig/20221102-094928-marostegui.json
  • 09:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37613 and previous config saved to /var/cache/conftool/dbconfig/20221102-094709-marostegui.json
  • 09:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 09:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 09:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37612 and previous config saved to /var/cache/conftool/dbconfig/20221102-094644-marostegui.json
  • 09:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P37611 and previous config saved to /var/cache/conftool/dbconfig/20221102-094622-ladsgroup.json
  • 09:41 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1016.eqiad.wmnet to cluster eqiad and group B
  • 09:40 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1016.eqiad.wmnet to cluster eqiad and group B
  • 09:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T318950)', diff saved to https://phabricator.wikimedia.org/P37610 and previous config saved to /var/cache/conftool/dbconfig/20221102-093730-ladsgroup.json
  • 09:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 09:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T318955)', diff saved to https://phabricator.wikimedia.org/P37609 and previous config saved to /var/cache/conftool/dbconfig/20221102-093717-ladsgroup.json
  • 09:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 09:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37608 and previous config saved to /var/cache/conftool/dbconfig/20221102-093657-ladsgroup.json
  • 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1016.eqiad.wmnet
  • 09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T318950)', diff saved to https://phabricator.wikimedia.org/P37607 and previous config saved to /var/cache/conftool/dbconfig/20221102-093517-ladsgroup.json
  • 09:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance
  • 09:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 09:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance
  • 09:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T318955)', diff saved to https://phabricator.wikimedia.org/P37606 and previous config saved to /var/cache/conftool/dbconfig/20221102-093459-ladsgroup.json
  • 09:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 09:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T318950)', diff saved to https://phabricator.wikimedia.org/P37605 and previous config saved to /var/cache/conftool/dbconfig/20221102-093453-ladsgroup.json
  • 09:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 09:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37604 and previous config saved to /var/cache/conftool/dbconfig/20221102-093443-ladsgroup.json
  • 09:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 09:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T318955)', diff saved to https://phabricator.wikimedia.org/P37603 and previous config saved to /var/cache/conftool/dbconfig/20221102-093436-ladsgroup.json
  • 09:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37602 and previous config saved to /var/cache/conftool/dbconfig/20221102-093420-ladsgroup.json
  • 09:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P37601 and previous config saved to /var/cache/conftool/dbconfig/20221102-093135-marostegui.json
  • 09:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P37600 and previous config saved to /var/cache/conftool/dbconfig/20221102-093115-ladsgroup.json
  • 09:28 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1016.eqiad.wmnet
  • 09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P37599 and previous config saved to /var/cache/conftool/dbconfig/20221102-091942-ladsgroup.json
  • 09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P37598 and previous config saved to /var/cache/conftool/dbconfig/20221102-091928-ladsgroup.json
  • 09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P37597 and previous config saved to /var/cache/conftool/dbconfig/20221102-091912-ladsgroup.json
  • 09:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P37596 and previous config saved to /var/cache/conftool/dbconfig/20221102-091628-marostegui.json
  • 09:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37595 and previous config saved to /var/cache/conftool/dbconfig/20221102-091606-ladsgroup.json
  • 09:06 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1028.eqiad.wmnet to cluster eqiad and group C
  • 09:05 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1028.eqiad.wmnet to cluster eqiad and group C
  • 09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P37594 and previous config saved to /var/cache/conftool/dbconfig/20221102-090435-ladsgroup.json
  • 09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P37593 and previous config saved to /var/cache/conftool/dbconfig/20221102-090418-ladsgroup.json
  • 09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P37592 and previous config saved to /var/cache/conftool/dbconfig/20221102-090404-ladsgroup.json
  • 09:03 mvernon@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=swift,name=eqiad
  • 09:01 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet
  • 09:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37591 and previous config saved to /var/cache/conftool/dbconfig/20221102-090119-marostegui.json
  • 09:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37590 and previous config saved to /var/cache/conftool/dbconfig/20221102-090007-ladsgroup.json
  • 08:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 08:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 08:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37589 and previous config saved to /var/cache/conftool/dbconfig/20221102-085903-marostegui.json
  • 08:58 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 08:58 Emperor: repool ms-fe10-12
  • 08:58 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 08:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T321123)', diff saved to https://phabricator.wikimedia.org/P37588 and previous config saved to /var/cache/conftool/dbconfig/20221102-085838-marostegui.json
  • 08:55 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet
  • 08:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:54 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:54 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:53 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1016.eqiad.wmnet with OS bullseye
  • 08:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T318950)', diff saved to https://phabricator.wikimedia.org/P37587 and previous config saved to /var/cache/conftool/dbconfig/20221102-084927-ladsgroup.json
  • 08:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T318955)', diff saved to https://phabricator.wikimedia.org/P37586 and previous config saved to /var/cache/conftool/dbconfig/20221102-084910-ladsgroup.json
  • 08:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37585 and previous config saved to /var/cache/conftool/dbconfig/20221102-084853-ladsgroup.json
  • 08:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T318950)', diff saved to https://phabricator.wikimedia.org/P37584 and previous config saved to /var/cache/conftool/dbconfig/20221102-084713-ladsgroup.json
  • 08:51 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc1014 to pc2 master" (duration: 04m 16s)
  • 08:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 08:50 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T318955)', diff saved to https://phabricator.wikimedia.org/P37583 and previous config saved to /var/cache/conftool/dbconfig/20221102-084653-ladsgroup.json
  • 08:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 08:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37582 and previous config saved to /var/cache/conftool/dbconfig/20221102-084643-ladsgroup.json
  • 08:49 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti1028.eqiad.wmnet
  • 08:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 08:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 08:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 08:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 08:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 08:47 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T318605)', diff saved to https://phabricator.wikimedia.org/P37581 and previous config saved to /var/cache/conftool/dbconfig/20221102-084540-ladsgroup.json
  • 08:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 08:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 08:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P37580 and previous config saved to /var/cache/conftool/dbconfig/20221102-084330-marostegui.json
  • 08:43 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc1014 to pc2 master" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 08:42 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc1014 to pc2 master"
  • 08:36 moritzm: draining ganeti1020 for eventual reimage T311687
  • 08:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet
  • 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1016.eqiad.wmnet with reason: host reimage
  • 08:33 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti1028.eqiad.wmnet
  • 08:30 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1016.eqiad.wmnet with reason: host reimage
  • 08:29 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 100%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37579 and previous config saved to /var/cache/conftool/dbconfig/20221102-082942-root.json
  • 08:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P37578 and previous config saved to /var/cache/conftool/dbconfig/20221102-082822-marostegui.json
  • 08:25 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:24 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:24 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:23 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet
  • 08:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:18 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc1014 to pc2 master (duration: 04m 43s)
  • 08:18 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:15 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1016.eqiad.wmnet with OS bullseye
  • 08:14 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 75%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37577 and previous config saved to /var/cache/conftool/dbconfig/20221102-081437-root.json
  • 08:14 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc1014 to pc2 master synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 08:13 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc1014 to pc2 master
  • 08:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T321123)', diff saved to https://phabricator.wikimedia.org/P37576 and previous config saved to /var/cache/conftool/dbconfig/20221102-081313-marostegui.json
  • 08:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T321123)', diff saved to https://phabricator.wikimedia.org/P37575 and previous config saved to /var/cache/conftool/dbconfig/20221102-081059-marostegui.json
  • 08:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1110.eqiad.wmnet with reason: Maintenance
  • 08:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1110.eqiad.wmnet with reason: Maintenance
  • 08:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100 (T321123)', diff saved to https://phabricator.wikimedia.org/P37574 and previous config saved to /var/cache/conftool/dbconfig/20221102-081034-marostegui.json
  • 08:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:05 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc2 master" (duration: 05m 01s)
  • 08:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:03 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:01 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc2014 to pc2 master" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 08:00 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc2 master"
  • 07:59 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 50%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37573 and previous config saved to /var/cache/conftool/dbconfig/20221102-075930-root.json
  • 07:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P37572 and previous config saved to /var/cache/conftool/dbconfig/20221102-075527-marostegui.json
  • 07:48 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:47 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:47 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:44 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc2014 to pc2 master (duration: 04m 13s)
  • 07:44 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 25%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37571 and previous config saved to /var/cache/conftool/dbconfig/20221102-074422-root.json
  • 07:40 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc2014 to pc2 master synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 07:40 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc2014 to pc2 master
  • 07:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P37570 and previous config saved to /var/cache/conftool/dbconfig/20221102-074016-marostegui.json
  • 07:29 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 10%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37569 and previous config saved to /var/cache/conftool/dbconfig/20221102-072916-root.json
  • 07:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100 (T321123)', diff saved to https://phabricator.wikimedia.org/P37568 and previous config saved to /var/cache/conftool/dbconfig/20221102-072508-marostegui.json
  • 07:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1100 (T321123)', diff saved to https://phabricator.wikimedia.org/P37567 and previous config saved to /var/cache/conftool/dbconfig/20221102-072254-marostegui.json
  • 07:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 07:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37566 and previous config saved to /var/cache/conftool/dbconfig/20221102-072220-marostegui.json
  • 07:14 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 5%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37565 and previous config saved to /var/cache/conftool/dbconfig/20221102-071410-root.json
  • 07:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P37564 and previous config saved to /var/cache/conftool/dbconfig/20221102-070712-marostegui.json
  • 07:06 urbanecm@deploy1002: Finished scap: Backport for Deploy GrowthExperiments to 100% users at all wikis but dewiki (T320876) (duration: 04m 38s)
  • 07:02 urbanecm@deploy1002: urbanecm and urbanecm: Backport for Deploy GrowthExperiments to 100% users at all wikis but dewiki (T320876) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 07:01 urbanecm@deploy1002: Started scap: Backport for Deploy GrowthExperiments to 100% users at all wikis but dewiki (T320876)
  • 06:59 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 3%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37563 and previous config saved to /var/cache/conftool/dbconfig/20221102-065903-root.json
  • 06:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P37562 and previous config saved to /var/cache/conftool/dbconfig/20221102-065203-marostegui.json
  • 06:43 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 1%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37561 and previous config saved to /var/cache/conftool/dbconfig/20221102-064357-root.json
  • 06:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37560 and previous config saved to /var/cache/conftool/dbconfig/20221102-063653-marostegui.json
  • 06:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37559 and previous config saved to /var/cache/conftool/dbconfig/20221102-063038-marostegui.json
  • 06:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 06:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 05:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 05:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 05:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T318605)', diff saved to https://phabricator.wikimedia.org/P37558 and previous config saved to /var/cache/conftool/dbconfig/20221102-054747-ladsgroup.json
  • 05:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P37557 and previous config saved to /var/cache/conftool/dbconfig/20221102-053238-ladsgroup.json
  • 05:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P37556 and previous config saved to /var/cache/conftool/dbconfig/20221102-051730-ladsgroup.json
  • 05:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T318605)', diff saved to https://phabricator.wikimedia.org/P37555 and previous config saved to /var/cache/conftool/dbconfig/20221102-050222-ladsgroup.json
  • 04:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1203 (T318605)', diff saved to https://phabricator.wikimedia.org/P37554 and previous config saved to /var/cache/conftool/dbconfig/20221102-042904-ladsgroup.json
  • 04:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 04:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 04:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T318605)', diff saved to https://phabricator.wikimedia.org/P37553 and previous config saved to /var/cache/conftool/dbconfig/20221102-042838-ladsgroup.json
  • 04:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P37552 and previous config saved to /var/cache/conftool/dbconfig/20221102-041330-ladsgroup.json
  • 03:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P37551 and previous config saved to /var/cache/conftool/dbconfig/20221102-035821-ladsgroup.json
  • 03:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T318605)', diff saved to https://phabricator.wikimedia.org/P37550 and previous config saved to /var/cache/conftool/dbconfig/20221102-034312-ladsgroup.json
  • 03:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1193 (T318605)', diff saved to https://phabricator.wikimedia.org/P37549 and previous config saved to /var/cache/conftool/dbconfig/20221102-030959-ladsgroup.json
  • 03:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 03:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 03:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T318605)', diff saved to https://phabricator.wikimedia.org/P37548 and previous config saved to /var/cache/conftool/dbconfig/20221102-030934-ladsgroup.json
  • 02:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P37547 and previous config saved to /var/cache/conftool/dbconfig/20221102-025427-ladsgroup.json
  • 02:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P37546 and previous config saved to /var/cache/conftool/dbconfig/20221102-023919-ladsgroup.json
  • 02:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T318605)', diff saved to https://phabricator.wikimedia.org/P37545 and previous config saved to /var/cache/conftool/dbconfig/20221102-022408-ladsgroup.json
  • 01:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1192 (T318605)', diff saved to https://phabricator.wikimedia.org/P37544 and previous config saved to /var/cache/conftool/dbconfig/20221102-015019-ladsgroup.json
  • 01:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 01:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 01:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T318605)', diff saved to https://phabricator.wikimedia.org/P37543 and previous config saved to /var/cache/conftool/dbconfig/20221102-014955-ladsgroup.json
  • 01:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P37542 and previous config saved to /var/cache/conftool/dbconfig/20221102-013447-ladsgroup.json
  • 01:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P37541 and previous config saved to /var/cache/conftool/dbconfig/20221102-011937-ladsgroup.json
  • 01:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T318605)', diff saved to https://phabricator.wikimedia.org/P37540 and previous config saved to /var/cache/conftool/dbconfig/20221102-010958-ladsgroup.json
  • 01:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T318605)', diff saved to https://phabricator.wikimedia.org/P37539 and previous config saved to /var/cache/conftool/dbconfig/20221102-010430-ladsgroup.json
  • 00:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P37538 and previous config saved to /var/cache/conftool/dbconfig/20221102-005451-ladsgroup.json
  • 00:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P37537 and previous config saved to /var/cache/conftool/dbconfig/20221102-003941-ladsgroup.json
  • 00:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1178 (T318605)', diff saved to https://phabricator.wikimedia.org/P37536 and previous config saved to /var/cache/conftool/dbconfig/20221102-002936-ladsgroup.json
  • 00:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 00:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 00:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37535 and previous config saved to /var/cache/conftool/dbconfig/20221102-002913-ladsgroup.json
  • 00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T318605)', diff saved to https://phabricator.wikimedia.org/P37534 and previous config saved to /var/cache/conftool/dbconfig/20221102-002433-ladsgroup.json
  • 00:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P37533 and previous config saved to /var/cache/conftool/dbconfig/20221102-001401-ladsgroup.json

2022-11-01

  • 23:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P37532 and previous config saved to /var/cache/conftool/dbconfig/20221101-235853-ladsgroup.json
  • 23:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2181 (T318605)', diff saved to https://phabricator.wikimedia.org/P37531 and previous config saved to /var/cache/conftool/dbconfig/20221101-234957-ladsgroup.json
  • 23:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
  • 23:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
  • 23:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37530 and previous config saved to /var/cache/conftool/dbconfig/20221101-234935-ladsgroup.json
  • 23:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37529 and previous config saved to /var/cache/conftool/dbconfig/20221101-234346-ladsgroup.json
  • 23:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3318', diff saved to https://phabricator.wikimedia.org/P37528 and previous config saved to /var/cache/conftool/dbconfig/20221101-233427-ladsgroup.json
  • 23:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3318', diff saved to https://phabricator.wikimedia.org/P37527 and previous config saved to /var/cache/conftool/dbconfig/20221101-231919-ladsgroup.json
  • 23:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37526 and previous config saved to /var/cache/conftool/dbconfig/20221101-230833-ladsgroup.json
  • 23:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 23:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 23:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T318605)', diff saved to https://phabricator.wikimedia.org/P37525 and previous config saved to /var/cache/conftool/dbconfig/20221101-230811-ladsgroup.json
  • 23:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37524 and previous config saved to /var/cache/conftool/dbconfig/20221101-230411-ladsgroup.json
  • 22:55 Emperor: depool ms-fe2009
  • 22:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P37523 and previous config saved to /var/cache/conftool/dbconfig/20221101-225303-ladsgroup.json
  • 22:43 krinkle@deploy1002: Finished deploy [integration/docroot@2ddd7d9]: (no justification provided) (duration: 00m 33s)
  • 22:43 krinkle@deploy1002: Started deploy [integration/docroot@2ddd7d9]: (no justification provided)
  • 22:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P37522 and previous config saved to /var/cache/conftool/dbconfig/20221101-223754-ladsgroup.json
  • 22:32 jhathaway@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aux-k8s-worker1002.eqiad.wmnet
  • 22:32 Emperor: rolling restart of eqiad swift front-ends
  • 22:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2168:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37521 and previous config saved to /var/cache/conftool/dbconfig/20221101-222858-ladsgroup.json
  • 22:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 22:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 22:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37520 and previous config saved to /var/cache/conftool/dbconfig/20221101-222835-ladsgroup.json
  • 22:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T318605)', diff saved to https://phabricator.wikimedia.org/P37519 and previous config saved to /var/cache/conftool/dbconfig/20221101-222247-ladsgroup.json
  • 22:20 jhathaway@cumin1001: conftool action : set/pooled=false; selector: dnsdisc=swift,name=eqiad
  • 22:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3318', diff saved to https://phabricator.wikimedia.org/P37518 and previous config saved to /var/cache/conftool/dbconfig/20221101-221328-ladsgroup.json
  • 22:09 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-worker1002.eqiad.wmnet on all recursors
  • 22:09 jhathaway@cumin1001: START - Cookbook sre.dns.wipe-cache aux-k8s-worker1002.eqiad.wmnet on all recursors
  • 22:08 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3318', diff saved to https://phabricator.wikimedia.org/P37517 and previous config saved to /var/cache/conftool/dbconfig/20221101-215820-ladsgroup.json
  • 21:53 jhathaway@cumin1001: START - Cookbook sre.dns.netbox
  • 21:53 jhathaway@cumin1001: START - Cookbook sre.ganeti.makevm for new host aux-k8s-worker1002.eqiad.wmnet
  • 21:52 jhathaway@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aux-k8s-worker1001.eqiad.wmnet
  • 21:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1172 (T318605)', diff saved to https://phabricator.wikimedia.org/P37516 and previous config saved to /var/cache/conftool/dbconfig/20221101-214659-ladsgroup.json
  • 21:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 21:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37515 and previous config saved to /var/cache/conftool/dbconfig/20221101-214311-ladsgroup.json
  • 21:29 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-worker1001.eqiad.wmnet on all recursors
  • 21:29 jhathaway@cumin1001: START - Cookbook sre.dns.wipe-cache aux-k8s-worker1001.eqiad.wmnet on all recursors
  • 21:28 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:21 jhathaway@cumin1001: START - Cookbook sre.dns.netbox
  • 21:20 jhathaway@cumin1001: START - Cookbook sre.ganeti.makevm for new host aux-k8s-worker1001.eqiad.wmnet
  • 21:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 21:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 21:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T318605)', diff saved to https://phabricator.wikimedia.org/P37514 and previous config saved to /var/cache/conftool/dbconfig/20221101-211013-ladsgroup.json
  • 21:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2167:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37513 and previous config saved to /var/cache/conftool/dbconfig/20221101-210658-ladsgroup.json
  • 21:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 21:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 21:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T318605)', diff saved to https://phabricator.wikimedia.org/P37512 and previous config saved to /var/cache/conftool/dbconfig/20221101-210622-ladsgroup.json
  • 20:56 ryankemper: T322037 Re-enabled puppet across `A:wdqs-all` and `A:wcqs-public`
  • 20:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P37511 and previous config saved to /var/cache/conftool/dbconfig/20221101-205505-ladsgroup.json
  • 20:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P37510 and previous config saved to /var/cache/conftool/dbconfig/20221101-205115-ladsgroup.json
  • 20:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P37509 and previous config saved to /var/cache/conftool/dbconfig/20221101-203957-ladsgroup.json
  • 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P37508 and previous config saved to /var/cache/conftool/dbconfig/20221101-203607-ladsgroup.json
  • 20:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:34 cjming: end of UTC late backport window
  • 20:33 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:33 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:33 cjming@deploy1002: Finished scap: Backport for Update Edit Attempt Step sampling rate to 1 for group 0 wikis (T312016) (duration: 04m 29s)
  • 20:32 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:29 cjming@deploy1002: cjming and cjming: Backport for Update Edit Attempt Step sampling rate to 1 for group 0 wikis (T312016) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 20:29 cjming@deploy1002: Started scap: Backport for Update Edit Attempt Step sampling rate to 1 for group 0 wikis (T312016)
  • 20:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:26 cjming@deploy1002: Finished scap: Backport for Add MP stream for VisualEditorFeatureUse instrument (T309602) (duration: 04m 36s)
  • 20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T318605)', diff saved to https://phabricator.wikimedia.org/P37507 and previous config saved to /var/cache/conftool/dbconfig/20221101-202449-ladsgroup.json
  • 20:21 cjming@deploy1002: cjming and cjming: Backport for Add MP stream for VisualEditorFeatureUse instrument (T309602) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 20:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T318605)', diff saved to https://phabricator.wikimedia.org/P37506 and previous config saved to /var/cache/conftool/dbconfig/20221101-202059-ladsgroup.json
  • 20:21 cjming@deploy1002: Started scap: Backport for Add MP stream for VisualEditorFeatureUse instrument (T309602)
  • 20:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:18 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:17 cjming@deploy1002: Finished scap: Backport for Enable DiscussionTools visual enhancements beta feature at jawiki (T318127) (duration: 04m 55s)
  • 20:16 jhathaway: restarting pybal on lvs1019.eqiad.net
  • 20:12 cjming@deploy1002: cjming and matmarex: Backport for Enable DiscussionTools visual enhancements beta feature at jawiki (T318127) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 20:12 cjming@deploy1002: Started scap: Backport for Enable DiscussionTools visual enhancements beta feature at jawiki (T318127)
  • 20:10 jhathaway: restarting pybal on lvs1020.eqiad.net
  • 20:10 ejegg: civicrm upgraded from 6f511710 to 97b9d830
  • 20:09 cjming@deploy1002: Finished scap: Backport for Enable DiscussionTools mobile visual enhancements at jawiki (T318870) (duration: 05m 31s)
  • 20:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:06 ryankemper: T322037 Disabled puppet across `A:wdqs-all` and `A:wcqs-public`
  • 19:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2166 (T318605)', diff saved to https://phabricator.wikimedia.org/P37505 and previous config saved to /var/cache/conftool/dbconfig/20221101-194718-ladsgroup.json
  • 19:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
  • 19:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
  • 19:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T318605)', diff saved to https://phabricator.wikimedia.org/P37504 and previous config saved to /var/cache/conftool/dbconfig/20221101-194655-ladsgroup.json
  • 19:41 jhathaway@puppetmaster1001: conftool action : set/pooled=yes:weight=1; selector: cluster=aux-k8s,service=kubemaster
  • 19:36 otto@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Enable rc0.mediawiki.page_change on group0 wikis - T311129 (duration: 03m 38s)
  • 19:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1167 (T318605)', diff saved to https://phabricator.wikimedia.org/P37503 and previous config saved to /var/cache/conftool/dbconfig/20221101-193404-ladsgroup.json
  • 19:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 19:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 19:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 19:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1126 (T318605)', diff saved to https://phabricator.wikimedia.org/P37502 and previous config saved to /var/cache/conftool/dbconfig/20221101-193323-ladsgroup.json
  • 19:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P37501 and previous config saved to /var/cache/conftool/dbconfig/20221101-193148-ladsgroup.json
  • 19:26 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 19:25 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 19:25 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 19:24 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 19:19 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 19:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 19:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1126', diff saved to https://phabricator.wikimedia.org/P37500 and previous config saved to /var/cache/conftool/dbconfig/20221101-191815-ladsgroup.json
  • 19:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P37499 and previous config saved to /var/cache/conftool/dbconfig/20221101-191639-ladsgroup.json
  • 19:15 otto@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Declare rc0.mediawiki.page_content_change stream - T307959 T308017 (duration: 03m 42s)
  • 19:14 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1126', diff saved to https://phabricator.wikimedia.org/P37498 and previous config saved to /var/cache/conftool/dbconfig/20221101-190307-ladsgroup.json
  • 19:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T318605)', diff saved to https://phabricator.wikimedia.org/P37497 and previous config saved to /var/cache/conftool/dbconfig/20221101-190132-ladsgroup.json
  • 18:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1126 (T318605)', diff saved to https://phabricator.wikimedia.org/P37496 and previous config saved to /var/cache/conftool/dbconfig/20221101-184758-ladsgroup.json
  • 18:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 18:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 18:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T318955)', diff saved to https://phabricator.wikimedia.org/P37495 and previous config saved to /var/cache/conftool/dbconfig/20221101-183920-ladsgroup.json
  • 18:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2164 (T318605)', diff saved to https://phabricator.wikimedia.org/P37494 and previous config saved to /var/cache/conftool/dbconfig/20221101-182734-ladsgroup.json
  • 18:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 18:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 18:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
  • 18:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
  • 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T318605)', diff saved to https://phabricator.wikimedia.org/P37493 and previous config saved to /var/cache/conftool/dbconfig/20221101-182655-ladsgroup.json
  • 18:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 18:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 18:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T318950)', diff saved to https://phabricator.wikimedia.org/P37492 and previous config saved to /var/cache/conftool/dbconfig/20221101-182421-ladsgroup.json
  • 18:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P37491 and previous config saved to /var/cache/conftool/dbconfig/20221101-182412-ladsgroup.json
  • 18:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:24 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:22 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=varnish-fe
  • 18:22 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-be
  • 18:22 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-tls
  • 18:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:18 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.8 refs T320513
  • 18:14 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4052.ulsfo.wmnet
  • 18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1126 (T318605)', diff saved to https://phabricator.wikimedia.org/P37490 and previous config saved to /var/cache/conftool/dbconfig/20221101-181310-ladsgroup.json
  • 18:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1126.eqiad.wmnet with reason: Maintenance
  • 18:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1126.eqiad.wmnet with reason: Maintenance
  • 18:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P37489 and previous config saved to /var/cache/conftool/dbconfig/20221101-181148-ladsgroup.json
  • 18:11 jhuneidi@deploy1002: Finished scap: testwikis wikis to 1.40.0-wmf.8 refs T320513 (duration: 04m 18s)
  • 18:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P37488 and previous config saved to /var/cache/conftool/dbconfig/20221101-180913-ladsgroup.json
  • 18:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P37487 and previous config saved to /var/cache/conftool/dbconfig/20221101-180902-ladsgroup.json
  • 18:07 jhuneidi@deploy1002: Started scap: testwikis wikis to 1.40.0-wmf.8 refs T320513
  • 18:05 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4052.ulsfo.wmnet
  • 18:03 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4051.ulsfo.wmnet
  • 17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P37486 and previous config saved to /var/cache/conftool/dbconfig/20221101-175639-ladsgroup.json
  • 17:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P37479 and previous config saved to /var/cache/conftool/dbconfig/20221101-173712-ladsgroup.json
  • 17:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T318950)', diff saved to https://phabricator.wikimedia.org/P37478 and previous config saved to /var/cache/conftool/dbconfig/20221101-173636-ladsgroup.json
  • 17:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 17:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 17:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T318950)', diff saved to https://phabricator.wikimedia.org/P37477 and previous config saved to /var/cache/conftool/dbconfig/20221101-173624-ladsgroup.json
  • 17:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T318955)', diff saved to https://phabricator.wikimedia.org/P37476 and previous config saved to /var/cache/conftool/dbconfig/20221101-173607-ladsgroup.json
  • 17:35 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4049.ulsfo.wmnet
  • 17:34 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4048.ulsfo.wmnet
  • 17:24 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4048.ulsfo.wmnet
  • 17:24 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4047.ulsfo.wmnet
  • 17:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1114', diff saved to https://phabricator.wikimedia.org/P37475 and previous config saved to /var/cache/conftool/dbconfig/20221101-172341-ladsgroup.json
  • 17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P37474 and previous config saved to /var/cache/conftool/dbconfig/20221101-172204-ladsgroup.json
  • 17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P37473 and previous config saved to /var/cache/conftool/dbconfig/20221101-172116-ladsgroup.json
  • 17:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P37472 and previous config saved to /var/cache/conftool/dbconfig/20221101-172058-ladsgroup.json
  • 17:14 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4047.ulsfo.wmnet
  • 17:14 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4046.ulsfo.wmnet
  • 17:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1114', diff saved to https://phabricator.wikimedia.org/P37471 and previous config saved to /var/cache/conftool/dbconfig/20221101-170832-ladsgroup.json
  • 17:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2163 (T318605)', diff saved to https://phabricator.wikimedia.org/P37470 and previous config saved to /var/cache/conftool/dbconfig/20221101-170752-ladsgroup.json
  • 17:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
  • 17:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
  • 17:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37469 and previous config saved to /var/cache/conftool/dbconfig/20221101-170730-ladsgroup.json
  • 17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T318955)', diff saved to https://phabricator.wikimedia.org/P37468 and previous config saved to /var/cache/conftool/dbconfig/20221101-170656-ladsgroup.json
  • 17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P37467 and previous config saved to /var/cache/conftool/dbconfig/20221101-170608-ladsgroup.json
  • 17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P37466 and previous config saved to /var/cache/conftool/dbconfig/20221101-170550-ladsgroup.json
  • 17:06 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4046.ulsfo.wmnet
  • 17:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T318955)', diff saved to https://phabricator.wikimedia.org/P37465 and previous config saved to /var/cache/conftool/dbconfig/20221101-170447-ladsgroup.json
  • 17:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 17:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 17:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T318955)', diff saved to https://phabricator.wikimedia.org/P37464 and previous config saved to /var/cache/conftool/dbconfig/20221101-170424-ladsgroup.json
  • 16:58 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4045.ulsfo.wmnet
  • 16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1114 (T318605)', diff saved to https://phabricator.wikimedia.org/P37463 and previous config saved to /var/cache/conftool/dbconfig/20221101-165323-ladsgroup.json
  • 16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P37462 and previous config saved to /var/cache/conftool/dbconfig/20221101-165221-ladsgroup.json
  • 16:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T318950)', diff saved to https://phabricator.wikimedia.org/P37461 and previous config saved to /var/cache/conftool/dbconfig/20221101-165100-ladsgroup.json
  • 16:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T318955)', diff saved to https://phabricator.wikimedia.org/P37460 and previous config saved to /var/cache/conftool/dbconfig/20221101-165042-ladsgroup.json
  • 16:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T318955)', diff saved to https://phabricator.wikimedia.org/P37459 and previous config saved to /var/cache/conftool/dbconfig/20221101-164930-ladsgroup.json
  • 16:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P37458 and previous config saved to /var/cache/conftool/dbconfig/20221101-164914-ladsgroup.json
  • 16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37457 and previous config saved to /var/cache/conftool/dbconfig/20221101-164907-ladsgroup.json
  • 16:50 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4045.ulsfo.wmnet
  • 16:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T318950)', diff saved to https://phabricator.wikimedia.org/P37456 and previous config saved to /var/cache/conftool/dbconfig/20221101-164845-ladsgroup.json
  • 16:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T318950)', diff saved to https://phabricator.wikimedia.org/P37455 and previous config saved to /var/cache/conftool/dbconfig/20221101-164832-ladsgroup.json
  • 16:42 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:41 jhathaway@cumin1001: START - Cookbook sre.dns.netbox
  • 16:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P37454 and previous config saved to /var/cache/conftool/dbconfig/20221101-163713-ladsgroup.json
  • 16:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T318950)', diff saved to https://phabricator.wikimedia.org/P37453 and previous config saved to /var/cache/conftool/dbconfig/20221101-163706-ladsgroup.json
  • 16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P37452 and previous config saved to /var/cache/conftool/dbconfig/20221101-163407-ladsgroup.json
  • 16:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P37451 and previous config saved to /var/cache/conftool/dbconfig/20221101-163358-ladsgroup.json
  • 16:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P37450 and previous config saved to /var/cache/conftool/dbconfig/20221101-163324-ladsgroup.json
  • 16:27 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:25 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 16:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37449 and previous config saved to /var/cache/conftool/dbconfig/20221101-162206-ladsgroup.json
  • 16:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P37448 and previous config saved to /var/cache/conftool/dbconfig/20221101-162158-ladsgroup.json
  • 16:21 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:19 jhathaway@cumin1001: START - Cookbook sre.dns.netbox
  • 16:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T318955)', diff saved to https://phabricator.wikimedia.org/P37447 and previous config saved to /var/cache/conftool/dbconfig/20221101-161859-ladsgroup.json
  • 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P37446 and previous config saved to /var/cache/conftool/dbconfig/20221101-161851-ladsgroup.json
  • 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P37445 and previous config saved to /var/cache/conftool/dbconfig/20221101-161816-ladsgroup.json
  • 16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T318955)', diff saved to https://phabricator.wikimedia.org/P37444 and previous config saved to /var/cache/conftool/dbconfig/20221101-161648-ladsgroup.json
  • 16:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1114 (T318605)', diff saved to https://phabricator.wikimedia.org/P37443 and previous config saved to /var/cache/conftool/dbconfig/20221101-161636-ladsgroup.json
  • 16:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1114.eqiad.wmnet with reason: Maintenance
  • 16:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T318955)', diff saved to https://phabricator.wikimedia.org/P37442 and previous config saved to /var/cache/conftool/dbconfig/20221101-161625-ladsgroup.json
  • 16:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1114.eqiad.wmnet with reason: Maintenance
  • 16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1111 (T318605)', diff saved to https://phabricator.wikimedia.org/P37441 and previous config saved to /var/cache/conftool/dbconfig/20221101-161614-ladsgroup.json
  • 16:10 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4044.ulsfo.wmnet
  • 16:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P37440 and previous config saved to /var/cache/conftool/dbconfig/20221101-160649-ladsgroup.json
  • 16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37439 and previous config saved to /var/cache/conftool/dbconfig/20221101-160344-ladsgroup.json
  • 16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T318950)', diff saved to https://phabricator.wikimedia.org/P37438 and previous config saved to /var/cache/conftool/dbconfig/20221101-160308-ladsgroup.json
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P37437 and previous config saved to /var/cache/conftool/dbconfig/20221101-160116-ladsgroup.json
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1111', diff saved to https://phabricator.wikimedia.org/P37436 and previous config saved to /var/cache/conftool/dbconfig/20221101-160106-ladsgroup.json
  • 16:00 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4044.ulsfo.wmnet
  • 15:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T318950)', diff saved to https://phabricator.wikimedia.org/P37435 and previous config saved to /var/cache/conftool/dbconfig/20221101-155458-ladsgroup.json
  • 15:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37434 and previous config saved to /var/cache/conftool/dbconfig/20221101-155446-ladsgroup.json
  • 15:53 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4043.ulsfo.wmnet
  • 15:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T318950)', diff saved to https://phabricator.wikimedia.org/P37433 and previous config saved to /var/cache/conftool/dbconfig/20221101-155142-ladsgroup.json
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37432 and previous config saved to /var/cache/conftool/dbconfig/20221101-155002-ladsgroup.json
  • 15:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 15:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 15:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37431 and previous config saved to /var/cache/conftool/dbconfig/20221101-154938-ladsgroup.json
  • 15:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T318950)', diff saved to https://phabricator.wikimedia.org/P37430 and previous config saved to /var/cache/conftool/dbconfig/20221101-154919-ladsgroup.json
  • 15:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 15:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 15:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37429 and previous config saved to /var/cache/conftool/dbconfig/20221101-154907-ladsgroup.json
  • 15:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37428 and previous config saved to /var/cache/conftool/dbconfig/20221101-154844-ladsgroup.json
  • 15:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance
  • 15:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance
  • 15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T318605)', diff saved to https://phabricator.wikimedia.org/P37427 and previous config saved to /var/cache/conftool/dbconfig/20221101-154819-ladsgroup.json
  • 15:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P37426 and previous config saved to /var/cache/conftool/dbconfig/20221101-154607-ladsgroup.json
  • 15:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1111', diff saved to https://phabricator.wikimedia.org/P37425 and previous config saved to /var/cache/conftool/dbconfig/20221101-154557-ladsgroup.json
  • 15:42 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4043.ulsfo.wmnet
  • 15:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P37424 and previous config saved to /var/cache/conftool/dbconfig/20221101-153938-ladsgroup.json
  • 15:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P37423 and previous config saved to /var/cache/conftool/dbconfig/20221101-153430-ladsgroup.json
  • 15:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P37422 and previous config saved to /var/cache/conftool/dbconfig/20221101-153400-ladsgroup.json
  • 15:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P37421 and previous config saved to /var/cache/conftool/dbconfig/20221101-153311-ladsgroup.json
  • 15:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T318955)', diff saved to https://phabricator.wikimedia.org/P37420 and previous config saved to /var/cache/conftool/dbconfig/20221101-153059-ladsgroup.json
  • 15:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1111 (T318605)', diff saved to https://phabricator.wikimedia.org/P37419 and previous config saved to /var/cache/conftool/dbconfig/20221101-153049-ladsgroup.json
  • 15:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T318955)', diff saved to https://phabricator.wikimedia.org/P37418 and previous config saved to /var/cache/conftool/dbconfig/20221101-152850-ladsgroup.json
  • 15:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 15:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 15:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T318955)', diff saved to https://phabricator.wikimedia.org/P37417 and previous config saved to /var/cache/conftool/dbconfig/20221101-152827-ladsgroup.json
  • 15:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P37416 and previous config saved to /var/cache/conftool/dbconfig/20221101-152430-ladsgroup.json
  • 15:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P37415 and previous config saved to /var/cache/conftool/dbconfig/20221101-151923-ladsgroup.json
  • 15:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P37414 and previous config saved to /var/cache/conftool/dbconfig/20221101-151853-ladsgroup.json
  • 15:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P37413 and previous config saved to /var/cache/conftool/dbconfig/20221101-151803-ladsgroup.json
  • 15:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P37412 and previous config saved to /var/cache/conftool/dbconfig/20221101-151320-ladsgroup.json
  • 15:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 15:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37411 and previous config saved to /var/cache/conftool/dbconfig/20221101-150922-ladsgroup.json
  • 15:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37410 and previous config saved to /var/cache/conftool/dbconfig/20221101-150711-ladsgroup.json
  • 15:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318950)', diff saved to https://phabricator.wikimedia.org/P37409 and previous config saved to /var/cache/conftool/dbconfig/20221101-150659-ladsgroup.json
  • 15:06 dancy@deploy1002: Pruned MediaWiki: 1.40.0-wmf.6 (duration: 01m 47s)
  • 15:04 dancy@deploy1002: Finished scap: testwikis wikis to 1.40.0-wmf.8 refs T320513 (duration: 05m 05s)
  • 15:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37408 and previous config saved to /var/cache/conftool/dbconfig/20221101-150415-ladsgroup.json
  • 15:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37407 and previous config saved to /var/cache/conftool/dbconfig/20221101-150345-ladsgroup.json
  • 15:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T318605)', diff saved to https://phabricator.wikimedia.org/P37406 and previous config saved to /var/cache/conftool/dbconfig/20221101-150255-ladsgroup.json
  • 15:02 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4042.ulsfo.wmnet
  • 15:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37405 and previous config saved to /var/cache/conftool/dbconfig/20221101-150122-ladsgroup.json
  • 15:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 15:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 15:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T318950)', diff saved to https://phabricator.wikimedia.org/P37404 and previous config saved to /var/cache/conftool/dbconfig/20221101-150107-ladsgroup.json
  • 14:59 dancy@deploy1002: Started scap: testwikis wikis to 1.40.0-wmf.8 refs T320513
  • 14:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P37403 and previous config saved to /var/cache/conftool/dbconfig/20221101-145813-ladsgroup.json
  • 14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37402 and previous config saved to /var/cache/conftool/dbconfig/20221101-145152-ladsgroup.json
  • 14:54 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4042.ulsfo.wmnet
  • 14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1111 (T318605)', diff saved to https://phabricator.wikimedia.org/P37401 and previous config saved to /var/cache/conftool/dbconfig/20221101-145026-ladsgroup.json
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37400 and previous config saved to /var/cache/conftool/dbconfig/20221101-145019-ladsgroup.json
  • 14:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1111.eqiad.wmnet with reason: Maintenance
  • 14:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 14:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1111.eqiad.wmnet with reason: Maintenance
  • 14:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1104 (T318605)', diff saved to https://phabricator.wikimedia.org/P37399 and previous config saved to /var/cache/conftool/dbconfig/20221101-145004-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T318955)', diff saved to https://phabricator.wikimedia.org/P37398 and previous config saved to /var/cache/conftool/dbconfig/20221101-144954-ladsgroup.json
  • 14:48 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4041.ulsfo.wmnet
  • 14:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P37397 and previous config saved to /var/cache/conftool/dbconfig/20221101-144559-ladsgroup.json
  • 14:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T318955)', diff saved to https://phabricator.wikimedia.org/P37396 and previous config saved to /var/cache/conftool/dbconfig/20221101-144302-ladsgroup.json
  • 14:40 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4041.ulsfo.wmnet
  • 14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37394 and previous config saved to /var/cache/conftool/dbconfig/20221101-143645-ladsgroup.json
  • 14:38 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging EJoseph out of all services on: 803 hosts
  • 14:38 jmm@cumin2002: START - Cookbook sre.idm.logout Logging EJoseph out of all services on: 803 hosts
  • 14:38 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging EJoseph out of all services on: 1202 hosts
  • 14:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1104', diff saved to https://phabricator.wikimedia.org/P37393 and previous config saved to /var/cache/conftool/dbconfig/20221101-143453-ladsgroup.json
  • 14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P37392 and previous config saved to /var/cache/conftool/dbconfig/20221101-143445-ladsgroup.json
  • 14:34 jmm@cumin2002: START - Cookbook sre.idm.logout Logging EJoseph out of all services on: 1202 hosts
  • 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1028.eqiad.wmnet with OS bullseye
  • 14:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P37391 and previous config saved to /var/cache/conftool/dbconfig/20221101-143051-ladsgroup.json
  • 14:31 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4040.ulsfo.wmnet
  • 14:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 14:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2161 (T318605)', diff saved to https://phabricator.wikimedia.org/P37390 and previous config saved to /var/cache/conftool/dbconfig/20221101-142854-ladsgroup.json
  • 14:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance
  • 14:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T318955)', diff saved to https://phabricator.wikimedia.org/P37389 and previous config saved to /var/cache/conftool/dbconfig/20221101-142842-ladsgroup.json
  • 14:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance
  • 14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T318605)', diff saved to https://phabricator.wikimedia.org/P37388 and previous config saved to /var/cache/conftool/dbconfig/20221101-142832-ladsgroup.json
  • 14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318950)', diff saved to https://phabricator.wikimedia.org/P37387 and previous config saved to /var/cache/conftool/dbconfig/20221101-142136-ladsgroup.json
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1104', diff saved to https://phabricator.wikimedia.org/P37386 and previous config saved to /var/cache/conftool/dbconfig/20221101-141945-ladsgroup.json
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P37385 and previous config saved to /var/cache/conftool/dbconfig/20221101-141936-ladsgroup.json
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T318950)', diff saved to https://phabricator.wikimedia.org/P37384 and previous config saved to /var/cache/conftool/dbconfig/20221101-141924-ladsgroup.json
  • 14:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 14:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T318950)', diff saved to https://phabricator.wikimedia.org/P37383 and previous config saved to /var/cache/conftool/dbconfig/20221101-141913-ladsgroup.json
  • 14:19 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4040.ulsfo.wmnet
  • 14:19 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:18 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4039.ulsfo.wmnet
  • 14:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T318950)', diff saved to https://phabricator.wikimedia.org/P37382 and previous config saved to /var/cache/conftool/dbconfig/20221101-141544-ladsgroup.json
  • 14:14 otto@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Use eventgate-analytics-external for rc0.mediawiki.page_change stream - T311129 (duration: 03m 42s)
  • 14:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P37381 and previous config saved to /var/cache/conftool/dbconfig/20221101-141335-ladsgroup.json
  • 14:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T318950)', diff saved to https://phabricator.wikimedia.org/P37380 and previous config saved to /var/cache/conftool/dbconfig/20221101-141322-ladsgroup.json
  • 14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P37379 and previous config saved to /var/cache/conftool/dbconfig/20221101-141321-ladsgroup.json
  • 14:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 14:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37378 and previous config saved to /var/cache/conftool/dbconfig/20221101-141308-ladsgroup.json
  • 14:12 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:10 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti1028.eqiad.wmnet with reason: host reimage
  • 14:10 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1028.eqiad.wmnet with reason: host reimage
  • 14:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P37375 and previous config saved to /var/cache/conftool/dbconfig/20221101-140402-ladsgroup.json
  • 14:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:02 urbanecm@deploy1002: Finished scap: Backport for [GrowthExperiments] Remove wmgGEFeaturesMayBeAvailableToNewcomers (duration: 04m 32s)
  • 14:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P37374 and previous config saved to /var/cache/conftool/dbconfig/20221101-135827-ladsgroup.json
  • 13:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P37373 and previous config saved to /var/cache/conftool/dbconfig/20221101-135811-ladsgroup.json
  • 13:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P37372 and previous config saved to /var/cache/conftool/dbconfig/20221101-135800-ladsgroup.json
  • 13:59 moritzm: draining ganeti1016 for eventual reimage T311687
  • 13:59 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1028.eqiad.wmnet with OS bullseye
  • 13:59 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
  • 13:59 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4038.ulsfo.wmnet
  • 13:59 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
  • 13:57 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4037.ulsfo.wmnet
  • 13:56 moritzm: installing exim4 security updates on buster
  • 13:55 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:54 urbanecm@deploy1002: Started scap: Backport for [GrowthExperiments] Remove wmgGEFeaturesMayBeAvailableToNewcomers
  • 13:53 urbanecm@deploy1002: Finished scap: Backport for Copy reverse-proxy-staging.php to reverse-proxy-labs.php, "reverse-proxy-staging.php" -> "reverse-staging-labs.php", Delete "reverse-proxy-staging.php" (duration: 04m 30s)
  • 13:53 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:53 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:53 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T318955)', diff saved to https://phabricator.wikimedia.org/P37371 and previous config saved to /var/cache/conftool/dbconfig/20221101-135120-ladsgroup.json
  • 13:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 13:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 13:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 13:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 13:49 urbanecm@deploy1002: urbanecm and zabe: Backport for Copy reverse-proxy-staging.php to reverse-proxy-labs.php, "reverse-proxy-staging.php" -> "reverse-staging-labs.php", Delete "reverse-proxy-staging.php" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 13:49 urbanecm@deploy1002: Started scap: Backport for Copy reverse-proxy-staging.php to reverse-proxy-labs.php, "reverse-proxy-staging.php" -> "reverse-staging-labs.php", Delete "reverse-proxy-staging.php"
  • 13:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P37370 and previous config saved to /var/cache/conftool/dbconfig/20221101-134854-ladsgroup.json
  • 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T318955)', diff saved to https://phabricator.wikimedia.org/P37369 and previous config saved to /var/cache/conftool/dbconfig/20221101-134318-ladsgroup.json
  • 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T318605)', diff saved to https://phabricator.wikimedia.org/P37368 and previous config saved to /var/cache/conftool/dbconfig/20221101-134302-ladsgroup.json
  • 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P37367 and previous config saved to /var/cache/conftool/dbconfig/20221101-134252-ladsgroup.json
  • 13:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T318955)', diff saved to https://phabricator.wikimedia.org/P37366 and previous config saved to /var/cache/conftool/dbconfig/20221101-134108-ladsgroup.json
  • 13:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 13:42 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4037.ulsfo.wmnet
  • 13:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 13:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37365 and previous config saved to /var/cache/conftool/dbconfig/20221101-134045-ladsgroup.json
  • 13:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 13:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 13:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T318955)', diff saved to https://phabricator.wikimedia.org/P37364 and previous config saved to /var/cache/conftool/dbconfig/20221101-133857-ladsgroup.json
  • 13:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T318950)', diff saved to https://phabricator.wikimedia.org/P37363 and previous config saved to /var/cache/conftool/dbconfig/20221101-133346-ladsgroup.json
  • 13:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T318950)', diff saved to https://phabricator.wikimedia.org/P37362 and previous config saved to /var/cache/conftool/dbconfig/20221101-133132-ladsgroup.json
  • 13:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 13:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 13:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 13:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 13:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37361 and previous config saved to /var/cache/conftool/dbconfig/20221101-133113-ladsgroup.json
  • 13:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1104 (T318605)', diff saved to https://phabricator.wikimedia.org/P37360 and previous config saved to /var/cache/conftool/dbconfig/20221101-132841-ladsgroup.json
  • 13:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1104.eqiad.wmnet with reason: Maintenance
  • 13:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1104.eqiad.wmnet with reason: Maintenance
  • 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37359 and previous config saved to /var/cache/conftool/dbconfig/20221101-132817-ladsgroup.json
  • 13:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37358 and previous config saved to /var/cache/conftool/dbconfig/20221101-132745-ladsgroup.json
  • 13:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P37357 and previous config saved to /var/cache/conftool/dbconfig/20221101-132537-ladsgroup.json
  • 13:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37356 and previous config saved to /var/cache/conftool/dbconfig/20221101-132523-ladsgroup.json
  • 13:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 13:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 13:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T318950)', diff saved to https://phabricator.wikimedia.org/P37355 and previous config saved to /var/cache/conftool/dbconfig/20221101-132500-ladsgroup.json
  • 13:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P37354 and previous config saved to /var/cache/conftool/dbconfig/20221101-132348-ladsgroup.json
  • 13:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:22 urbanecm: UTC afternoon B&C window done
  • 13:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:20 urbanecm@deploy1002: Finished scap: Backport for zhwikivoyage: Add wordmark (T322133) (duration: 06m 36s)
  • 13:20 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1028.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 13:17 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1028.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 13:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P37353 and previous config saved to /var/cache/conftool/dbconfig/20221101-131605-ladsgroup.json
  • 13:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:14 urbanecm@deploy1002: urbanecm and stang: Backport for zhwikivoyage: Add wordmark (T322133) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 13:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1012.eqiad.wmnet
  • 13:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:14 urbanecm@deploy1002: Started scap: Backport for zhwikivoyage: Add wordmark (T322133)
  • 13:13 urbanecm@deploy1002: Finished scap: Backport for viwiki: Increase autoconfirmed edit count to 10 (T322105) (duration: 10m 35s)
  • 13:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P37352 and previous config saved to /var/cache/conftool/dbconfig/20221101-131309-ladsgroup.json
  • 13:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P37351 and previous config saved to /var/cache/conftool/dbconfig/20221101-131026-ladsgroup.json
  • 13:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P37350 and previous config saved to /var/cache/conftool/dbconfig/20221101-130952-ladsgroup.json
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2154 (T318605)', diff saved to https://phabricator.wikimedia.org/P37349 and previous config saved to /var/cache/conftool/dbconfig/20221101-130919-ladsgroup.json
  • 13:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
  • 13:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
  • 13:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T318605)', diff saved to https://phabricator.wikimedia.org/P37348 and previous config saved to /var/cache/conftool/dbconfig/20221101-130856-ladsgroup.json
  • 13:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P37347 and previous config saved to /var/cache/conftool/dbconfig/20221101-130839-ladsgroup.json
  • 13:06 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1012.eqiad.wmnet
  • 13:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1010.eqiad.wmnet
  • 13:03 urbanecm@deploy1002: urbanecm and stang: Backport for viwiki: Increase autoconfirmed edit count to 10 (T322105) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 13:02 urbanecm@deploy1002: Started scap: Backport for viwiki: Increase autoconfirmed edit count to 10 (T322105)
  • 13:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P37346 and previous config saved to /var/cache/conftool/dbconfig/20221101-130056-ladsgroup.json
  • 12:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P37345 and previous config saved to /var/cache/conftool/dbconfig/20221101-125801-ladsgroup.json
  • 12:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1010.eqiad.wmnet
  • 12:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1008.eqiad.wmnet
  • 12:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37344 and previous config saved to /var/cache/conftool/dbconfig/20221101-125516-ladsgroup.json
  • 12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P37343 and previous config saved to /var/cache/conftool/dbconfig/20221101-125443-ladsgroup.json
  • 12:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P37342 and previous config saved to /var/cache/conftool/dbconfig/20221101-125348-ladsgroup.json
  • 12:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T318955)', diff saved to https://phabricator.wikimedia.org/P37341 and previous config saved to /var/cache/conftool/dbconfig/20221101-125331-ladsgroup.json
  • 12:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1008.eqiad.wmnet
  • 12:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37340 and previous config saved to /var/cache/conftool/dbconfig/20221101-124548-ladsgroup.json
  • 12:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37339 and previous config saved to /var/cache/conftool/dbconfig/20221101-124334-ladsgroup.json
  • 12:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 12:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 12:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 12:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 12:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T318950)', diff saved to https://phabricator.wikimedia.org/P37338 and previous config saved to /var/cache/conftool/dbconfig/20221101-124301-ladsgroup.json
  • 12:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37337 and previous config saved to /var/cache/conftool/dbconfig/20221101-124253-ladsgroup.json
  • 12:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37336 and previous config saved to /var/cache/conftool/dbconfig/20221101-124225-ladsgroup.json
  • 12:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 12:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 12:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37335 and previous config saved to /var/cache/conftool/dbconfig/20221101-124202-ladsgroup.json
  • 12:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T318955)', diff saved to https://phabricator.wikimedia.org/P37334 and previous config saved to /var/cache/conftool/dbconfig/20221101-124012-ladsgroup.json
  • 12:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 12:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T318955)', diff saved to https://phabricator.wikimedia.org/P37333 and previous config saved to /var/cache/conftool/dbconfig/20221101-123949-ladsgroup.json
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T318950)', diff saved to https://phabricator.wikimedia.org/P37332 and previous config saved to /var/cache/conftool/dbconfig/20221101-123936-ladsgroup.json
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P37331 and previous config saved to /var/cache/conftool/dbconfig/20221101-123839-ladsgroup.json
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T318950)', diff saved to https://phabricator.wikimedia.org/P37330 and previous config saved to /var/cache/conftool/dbconfig/20221101-123714-ladsgroup.json
  • 12:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 12:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 12:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 12:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 12:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T318950)', diff saved to https://phabricator.wikimedia.org/P37329 and previous config saved to /var/cache/conftool/dbconfig/20221101-123646-ladsgroup.json
  • 12:28 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1006.eqiad.wmnet
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P37328 and previous config saved to /var/cache/conftool/dbconfig/20221101-122750-ladsgroup.json
  • 12:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P37327 and previous config saved to /var/cache/conftool/dbconfig/20221101-122654-ladsgroup.json
  • 12:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P37326 and previous config saved to /var/cache/conftool/dbconfig/20221101-122442-ladsgroup.json
  • 12:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T318605)', diff saved to https://phabricator.wikimedia.org/P37325 and previous config saved to /var/cache/conftool/dbconfig/20221101-122329-ladsgroup.json
  • 12:21 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1006.eqiad.wmnet
  • 12:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P37324 and previous config saved to /var/cache/conftool/dbconfig/20221101-122138-ladsgroup.json
  • 12:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P37323 and previous config saved to /var/cache/conftool/dbconfig/20221101-121242-ladsgroup.json
  • 12:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P37322 and previous config saved to /var/cache/conftool/dbconfig/20221101-121147-ladsgroup.json
  • 12:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P37321 and previous config saved to /var/cache/conftool/dbconfig/20221101-120934-ladsgroup.json
  • 12:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P37320 and previous config saved to /var/cache/conftool/dbconfig/20221101-120630-ladsgroup.json
  • 12:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37319 and previous config saved to /var/cache/conftool/dbconfig/20221101-120341-ladsgroup.json
  • 12:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 12:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 12:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37318 and previous config saved to /var/cache/conftool/dbconfig/20221101-120318-ladsgroup.json
  • 11:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T318950)', diff saved to https://phabricator.wikimedia.org/P37317 and previous config saved to /var/cache/conftool/dbconfig/20221101-115734-ladsgroup.json
  • 11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37316 and previous config saved to /var/cache/conftool/dbconfig/20221101-115638-ladsgroup.json
  • 11:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T318955)', diff saved to https://phabricator.wikimedia.org/P37315 and previous config saved to /var/cache/conftool/dbconfig/20221101-115426-ladsgroup.json
  • 11:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T318950)', diff saved to https://phabricator.wikimedia.org/P37314 and previous config saved to /var/cache/conftool/dbconfig/20221101-115122-ladsgroup.json
  • 11:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2152 (T318605)', diff saved to https://phabricator.wikimedia.org/P37313 and previous config saved to /var/cache/conftool/dbconfig/20221101-114943-ladsgroup.json
  • 11:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance
  • 11:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance
  • 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T318950)', diff saved to https://phabricator.wikimedia.org/P37312 and previous config saved to /var/cache/conftool/dbconfig/20221101-114858-ladsgroup.json
  • 11:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 11:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 11:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T318950)', diff saved to https://phabricator.wikimedia.org/P37311 and previous config saved to /var/cache/conftool/dbconfig/20221101-114835-ladsgroup.json
  • 11:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T318950)', diff saved to https://phabricator.wikimedia.org/P37310 and previous config saved to /var/cache/conftool/dbconfig/20221101-114820-ladsgroup.json
  • 11:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318', diff saved to https://phabricator.wikimedia.org/P37309 and previous config saved to /var/cache/conftool/dbconfig/20221101-114811-ladsgroup.json
  • 11:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 11:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 11:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37308 and previous config saved to /var/cache/conftool/dbconfig/20221101-114755-ladsgroup.json
  • 11:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37307 and previous config saved to /var/cache/conftool/dbconfig/20221101-114145-ladsgroup.json
  • 11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 11:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37306 and previous config saved to /var/cache/conftool/dbconfig/20221101-114123-ladsgroup.json
  • 11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T318955)', diff saved to https://phabricator.wikimedia.org/P37305 and previous config saved to /var/cache/conftool/dbconfig/20221101-114121-ladsgroup.json
  • 11:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 11:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 11:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T318955)', diff saved to https://phabricator.wikimedia.org/P37304 and previous config saved to /var/cache/conftool/dbconfig/20221101-114057-ladsgroup.json
  • 11:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P37303 and previous config saved to /var/cache/conftool/dbconfig/20221101-113327-ladsgroup.json
  • 11:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318', diff saved to https://phabricator.wikimedia.org/P37302 and previous config saved to /var/cache/conftool/dbconfig/20221101-113301-ladsgroup.json
  • 11:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P37301 and previous config saved to /var/cache/conftool/dbconfig/20221101-113248-ladsgroup.json
  • 11:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P37300 and previous config saved to /var/cache/conftool/dbconfig/20221101-112612-ladsgroup.json
  • 11:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P37299 and previous config saved to /var/cache/conftool/dbconfig/20221101-112549-ladsgroup.json
  • 11:19 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host puppetdb-test2001.codfw.wmnet
  • 11:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P37298 and previous config saved to /var/cache/conftool/dbconfig/20221101-111819-ladsgroup.json
  • 11:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37297 and previous config saved to /var/cache/conftool/dbconfig/20221101-111753-ladsgroup.json
  • 11:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P37296 and previous config saved to /var/cache/conftool/dbconfig/20221101-111739-ladsgroup.json
  • 11:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 11:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 11:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P37295 and previous config saved to /var/cache/conftool/dbconfig/20221101-111106-ladsgroup.json
  • 11:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P37294 and previous config saved to /var/cache/conftool/dbconfig/20221101-111042-ladsgroup.json
  • 11:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetdb-test2001.codfw.wmnet
  • 11:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T318950)', diff saved to https://phabricator.wikimedia.org/P37293 and previous config saved to /var/cache/conftool/dbconfig/20221101-110311-ladsgroup.json
  • 11:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37292 and previous config saved to /var/cache/conftool/dbconfig/20221101-110232-ladsgroup.json
  • 11:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T318950)', diff saved to https://phabricator.wikimedia.org/P37291 and previous config saved to /var/cache/conftool/dbconfig/20221101-110045-ladsgroup.json
  • 11:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 11:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 11:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37290 and previous config saved to /var/cache/conftool/dbconfig/20221101-110019-ladsgroup.json
  • 11:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 11:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 11:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 11:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 10:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 10:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 10:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37289 and previous config saved to /var/cache/conftool/dbconfig/20221101-105557-ladsgroup.json
  • 10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T318955)', diff saved to https://phabricator.wikimedia.org/P37288 and previous config saved to /var/cache/conftool/dbconfig/20221101-105534-ladsgroup.json
  • 10:48 moritzm: updating libdatetime-timezone-perl from latest Debian SUA update
  • 10:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T318955)', diff saved to https://phabricator.wikimedia.org/P37287 and previous config saved to /var/cache/conftool/dbconfig/20221101-104215-ladsgroup.json
  • 10:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 10:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37286 and previous config saved to /var/cache/conftool/dbconfig/20221101-104154-ladsgroup.json
  • 10:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 10:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 10:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37285 and previous config saved to /var/cache/conftool/dbconfig/20221101-103934-ladsgroup.json
  • 10:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 10:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 10:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 09:43 moritzm: imported quickstack 20161026-1+deb12u1 to apt.wikimedia.org/bookworm-wikimedia T321783
  • 08:32 moritzm: draining ganeti1028 for eventual reimage T311687
  • 08:29 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Faidon Liambotis out of all services on: 1203 hosts
  • 08:29 jmm@cumin2002: START - Cookbook sre.idm.logout Logging Faidon Liambotis out of all services on: 1203 hosts
  • 08:29 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Faidon Liambotis out of all services on: 802 hosts
  • 08:27 jmm@cumin2002: START - Cookbook sre.idm.logout Logging Faidon Liambotis out of all services on: 802 hosts
  • 03:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:45 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:45 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:36 mwpresync@deploy1002: Finished scap: testwikis wikis to 1.40.0-wmf.8 refs T320513 (duration: 33m 56s)
  • 03:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:02 mwpresync@deploy1002: Started scap: testwikis wikis to 1.40.0-wmf.8 refs T320513
  • 02:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 02:30 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 02:30 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 02:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 02:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 02:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 02:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 02:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debu

Other archives

2000s

2010s

2020s

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Server_Admin_Log/Archive_59&oldid=2037377"