The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
2021-12-29
10:30 elukey: kill tcpdump process on kubestagemaster1001 (kept a big pcap file opened that kept growing)
2021-12-28
11:27 godog: powercycle ms-be1059 -- was powered down
02:39 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
02:35 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
02:35 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
02:31 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
02:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
02:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
02:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
02:14 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
15:37 mforns@deploy1002: Started deploy [airflow-dags/analytics@f1522be]: (no justification provided)
15:24 mforns@deploy1002: Finished deploy [analytics/refinery@fcf104e] (hadoop-test): Adhoc train for anomaly detection queries TEST [analytics/refinery@fcf104e] (duration: 06m 44s)
15:17 mforns@deploy1002: Started deploy [analytics/refinery@fcf104e] (hadoop-test): Adhoc train for anomaly detection queries TEST [analytics/refinery@fcf104e]
22:52 mbsantos@deploy1002: Started deploy [kartotherian/deploy@2dc8b8b] (eqiad): Update kartotherian-package to e843e8f
22:13 ejegg: updated fundraising CiviCRM from d4cea6a9 to 07efd9fb
22:01 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on ganeti2007.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
22:01 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on ganeti2007.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
21:29 sbassett: Reverted previous mitigation for T297416
20:26 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:24 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:19 hashar@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.13 refs T293954
19:58 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:58 urbanecm: UTC evening B&C window done
19:58 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.13/extensions/GrowthExperiments/includes/Mentorship/MentorPageMentorManager.php: b8e64fe: MentorManager: Only invalidate cache when mentor list exists (T297827) (duration: 01m 06s)
19:57 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:51 legoktm: depooling mw1456 for benchmarking (T297259)
19:31 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:31 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.13/extensions/Wikibase/: 7799383: bridge: Reenable scrolling by mounting into parent (duration: 01m 12s)
19:30 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:24 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
{{safesubst:SAL entry|1=15:35 ladsgroup@deploy1002: Synchronized php-1.38.0-wmf.12/includes/: Backport: [[gerrit:747696|Allow setting max execution time to several special pages (T297708)], Part II (duration: 01m 11s)}}
{{safesubst:SAL entry|1=15:34 ladsgroup@deploy1002: Synchronized php-1.38.0-wmf.12/includes/DefaultSettings.php: Backport: [[gerrit:747696|Allow setting max execution time to several special pages (T297708)], Part I (duration: 01m 05s)}}
{{safesubst:SAL entry|1=15:32 ladsgroup@deploy1002: Synchronized php-1.38.0-wmf.13/includes/: Backport: [[gerrit:747695|Allow setting max execution time to several special pages (T297708)], Part II (duration: 01m 12s)}}
{{safesubst:SAL entry|1=15:30 ladsgroup@deploy1002: Synchronized php-1.38.0-wmf.13/includes/DefaultSettings.php: Backport: [[gerrit:747695|Allow setting max execution time to several special pages (T297708)], Part I (duration: 01m 06s)}}
15:29 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:28 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:26 volans@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic1084.mgmt.eqiad.wmnet with reboot policy FORCED
15:25 volans@cumin1001: START - Cookbook sre.hosts.provision for host elastic1084.mgmt.eqiad.wmnet with reboot policy FORCED
15:24 volans@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic1084.mgmt.eqiad.wmnet with reboot policy FORCED
15:24 volans@cumin1001: START - Cookbook sre.hosts.provision for host elastic1084.mgmt.eqiad.wmnet with reboot policy FORCED
15:22 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:21 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:19 volans@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic1084.mgmt.eqiad.wmnet with reboot policy FORCED
15:19 volans@cumin1001: START - Cookbook sre.hosts.provision for host elastic1084.mgmt.eqiad.wmnet with reboot policy FORCED
15:15 volans@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic1084.mgmt.eqiad.wmnet with reboot policy FORCED
15:07 volans@cumin1001: START - Cookbook sre.hosts.provision for host elastic1084.mgmt.eqiad.wmnet with reboot policy FORCED
15:05 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:04 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
14:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host idp-test1001.wikimedia.org
14:58 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
14:57 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
14:55 elukey: shutdown kafka-main2001 for BIOS+NIC firmware upgrades
14:51 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
14:50 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
14:44 moritzm: drain primary/secondary instances off ganeti2007 T296622
13:43 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
13:38 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
13:33 volans: upgraded spicerack to v1.1.0 on cumin[1001,2001]
13:27 mbsantos@deploy1002: Finished deploy [kartotherian/deploy@e36c241] (eqiad): Change osm-intl and osm source to get MVT from Tegola (Full production for Tegola) (duration: 01m 39s)
13:25 mbsantos@deploy1002: Started deploy [kartotherian/deploy@e36c241] (eqiad): Change osm-intl and osm source to get MVT from Tegola (Full production for Tegola)
11:08 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2003.codfw.wmnet with OS buster
10:59 btullis: pushed new packages for druid version 0.19.0-2 on buster using reprepro
10:31 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host kafka-main2003.codfw.wmnet with OS buster
10:28 elukey: second attempt to reimage kafka-main2003 to buster
10:09 moritzm: drain primary/secondary instances off ganeti2007 T296622
10:04 moritzm: switched kubetcd2004 to DRBD-based storage to allow migration for reimages
09:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kubetcd2004.codfw.wmnet with reason: switch to drbd storage
09:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on kubetcd2004.codfw.wmnet with reason: switch to drbd storage
09:46 moritzm: added ganeti2028 to ganeti codfw cluster T294139
09:38 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on ganeti2015.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
09:38 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on ganeti2015.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
09:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2024.codfw.wmnet with OS buster
08:56 moritzm: drain primary/secondary instances off ganeti2015 T296622
08:43 moritzm: switch ml-etcd2003 to DRBD-based storage to allow migration for reimages
08:39 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:32 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:31 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.13/extensions/GrowthExperiments/: 35c055c: MentorPageMentorManager: Do not fail hard with no mentor list configured (T297827) (duration: 01m 09s)
08:20 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2024.codfw.wmnet with OS buster
08:07 dcausse: restart blazegraph on wdqs1013 (jvm stuck for 4hours)
03:27 hoo: Stopped rebuildItemsPerSite on mwmaint1002 (was slightly beyond item Q72056756), as it has a memory leak (and would OOM in a few days)
01:53 mutante: miscweb1002 / miscweb2002 - both backends 'PASS: 26 requests sent to miscweb1002.eqiad.wmnet. All assertions passed.' again after fixing httpbb tests and T297605
01:50 mutante: miscweb1002 - re-enabling puppet after deployment for T297605
01:03 legoktm: removing current dump from static-codereview to replace it with a new one
00:37 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
00:30 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
00:22 legoktm: upgraded php7.2 on mw1414 for mysqlnd memory leak fix part 2 (T297667)
00:19 legoktm: uploaded 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf5 to buster-wikimedia for T297667
19:22 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2024.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
19:22 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti2024.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
19:21 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:51 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-test-coord1002.eqiad.wmnet with OS buster
18:26 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-test-coord1002.eqiad.wmnet with OS buster
18:24 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet
13:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet
13:29 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
13:28 kormat: uploaded wmfdb 0.1 to apt.wm.o for buster+bullseye T297618
13:24 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:50 moritzm: drain primary/secondary instances off ganeti2024 T296622
12:43 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on cloudvirt-wdqs1001.eqiad.wmnet with reason: T297454
12:43 dcaro@cumin1001: START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on cloudvirt-wdqs1001.eqiad.wmnet with reason: T297454
12:43 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt-wdqs1001.eqiad.wmnet with reason: T297454
12:43 dcaro@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on cloudvirt-wdqs1001.eqiad.wmnet with reason: T297454
12:40 Lucas_WMDE: UTC morning backport+config window done
00:29 catrope@deploy1002: Synchronized php-1.38.0-wmf.13/extensions/MediaSearch/resources/components/SearchResults.vue: Backport: Don't attempt to scroll to a non-existing result (duration: 01m 05s)
20:02 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:01 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:58 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: e127f4c: zhwiki: Promote Growth features out of dark mode (T287884) (duration: 00m 57s)
19:54 urbanecm: UTC evening B&C window done
19:53 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.12/skins/Vector/resources/skins.vector.es6/AB.js: 62e84e7: Prevent A/B test enrollment hook from firing for unsampled (T297662) (duration: 00m 56s)
19:51 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 40f0cff: VE on zh.wiki: Enable single-edit-tab mode, and other config like en.wiki (T296269) (duration: 00m 57s)
19:29 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:28 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:19 ryankemper@cumin1001: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic rolling restart - ryankemper@cumin1001 - T297468
19:18 bblack: lvs1020 - rebooting on new config
19:17 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:16 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:10 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:09 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:47 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1014.eqiad.wmnet
17:41 topranks: Temporarily deactivated BGP peering to AS8932 at AMS-IX (cr2-esams) as peer is constantly tripping max-prefix configuration for a few days, and according to peeringdb they should be within limit.
17:41 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1014.eqiad.wmnet
17:39 bblack@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:38 mutante: aphlict1001 - (Phabricator realtime notifications) - out of disk, attempting to gzip a large log
17:35 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:53 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1010.eqiad.wmnet
15:51 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:50 moritzm: drain primary/secondary instances off ganeti2023 T296622
15:50 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:49 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host aqs1010.eqiad.wmnet
15:49 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1010.eqiad.wmnet
15:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2018.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
15:47 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti2018.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
15:42 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:21 hashar@deploy1002: Finished scap: Push wmf.13 without promoting any wikis (duration: 29m 31s)
15:15 bblack@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs1019.eqiad.wmnet with OS buster
15:13 bblack@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs1018.eqiad.wmnet with OS buster
15:09 vgutierrez@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host cp4025.ulsfo.wmnet with OS buster
14:52 hashar@deploy1002: Started scap: Push wmf.13 without promoting any wikis
14:49 bblack@cumin1001: START - Cookbook sre.hosts.reimage for host lvs1019.eqiad.wmnet with OS buster
14:47 bblack@cumin1001: START - Cookbook sre.hosts.reimage for host lvs1018.eqiad.wmnet with OS buster
14:13 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
14:12 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
14:02 vgutierrez@cumin1001: START - Cookbook sre.hosts.reimage for host cp4025.ulsfo.wmnet with OS buster
14:01 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
13:59 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
13:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T277354)', diff saved to https://phabricator.wikimedia.org/P18229 and previous config saved to /var/cache/conftool/dbconfig/20211214-135601-marostegui.json
13:51 vgutierrez: depool cp4025 to be reimaged as cache::upload_envoy - T271421
13:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P18228 and previous config saved to /var/cache/conftool/dbconfig/20211214-134056-marostegui.json
13:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet
13:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet
13:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P18227 and previous config saved to /var/cache/conftool/dbconfig/20211214-132551-marostegui.json
13:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T277354)', diff saved to https://phabricator.wikimedia.org/P18226 and previous config saved to /var/cache/conftool/dbconfig/20211214-131047-marostegui.json
13:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 (T277354)', diff saved to https://phabricator.wikimedia.org/P18225 and previous config saved to /var/cache/conftool/dbconfig/20211214-130853-marostegui.json
13:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1099.eqiad.wmnet with reason: Maintenance
13:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1099.eqiad.wmnet with reason: Maintenance
13:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T277354)', diff saved to https://phabricator.wikimedia.org/P18224 and previous config saved to /var/cache/conftool/dbconfig/20211214-130845-marostegui.json
12:57 ayounsi@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netflow5001.eqsin.wmnet
12:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P18223 and previous config saved to /var/cache/conftool/dbconfig/20211214-125340-marostegui.json
12:45 ayounsi@cumin1001: START - Cookbook sre.hosts.decommission for hosts netflow5001.eqsin.wmnet
12:44 ayounsi@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netflow3001.esams.wmnet
12:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P18222 and previous config saved to /var/cache/conftool/dbconfig/20211214-123836-marostegui.json
12:27 ayounsi@cumin1001: START - Cookbook sre.hosts.decommission for hosts netflow3001.esams.wmnet
12:24 ayounsi@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netflow1001.eqiad.wmnet
12:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T277354)', diff saved to https://phabricator.wikimedia.org/P18221 and previous config saved to /var/cache/conftool/dbconfig/20211214-122331-marostegui.json
12:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 (T277354)', diff saved to https://phabricator.wikimedia.org/P18220 and previous config saved to /var/cache/conftool/dbconfig/20211214-122137-marostegui.json
12:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1105.eqiad.wmnet with reason: Maintenance
12:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1105.eqiad.wmnet with reason: Maintenance
12:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T277354)', diff saved to https://phabricator.wikimedia.org/P18219 and previous config saved to /var/cache/conftool/dbconfig/20211214-122129-marostegui.json
12:19 moritzm: drain primary/secondary instances off ganeti2018 T296622
12:14 ayounsi@cumin1001: START - Cookbook sre.hosts.decommission for hosts netflow1001.eqiad.wmnet
12:06 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P18218 and previous config saved to /var/cache/conftool/dbconfig/20211214-120624-marostegui.json
12:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2017.codfw.wmnet
12:01 ayounsi@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netflow2001.codfw.wmnet
11:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2017.codfw.wmnet
11:51 ayounsi@cumin1001: START - Cookbook sre.hosts.decommission for hosts netflow2001.codfw.wmnet
11:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P18217 and previous config saved to /var/cache/conftool/dbconfig/20211214-115119-marostegui.json
11:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T277354)', diff saved to https://phabricator.wikimedia.org/P18216 and previous config saved to /var/cache/conftool/dbconfig/20211214-113615-marostegui.json
11:35 hnowlan: joining restbase2026-b to cassandra
11:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2008.codfw.wmnet
11:34 jbond: move idp service to eqiad
11:34 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T277354)', diff saved to https://phabricator.wikimedia.org/P18215 and previous config saved to /var/cache/conftool/dbconfig/20211214-113428-marostegui.json
11:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db[1106,1154].eqiad.wmnet with reason: Maintenance
11:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db[1106,1154].eqiad.wmnet with reason: Maintenance
11:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T277354)', diff saved to https://phabricator.wikimedia.org/P18214 and previous config saved to /var/cache/conftool/dbconfig/20211214-113417-marostegui.json
11:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2008.codfw.wmnet
11:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P18213 and previous config saved to /var/cache/conftool/dbconfig/20211214-111912-marostegui.json
11:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2021.codfw.wmnet
11:09 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2021.codfw.wmnet
11:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P18212 and previous config saved to /var/cache/conftool/dbconfig/20211214-110407-marostegui.json
10:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T277354)', diff saved to https://phabricator.wikimedia.org/P18211 and previous config saved to /var/cache/conftool/dbconfig/20211214-104903-marostegui.json
10:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1119 (T277354)', diff saved to https://phabricator.wikimedia.org/P18210 and previous config saved to /var/cache/conftool/dbconfig/20211214-104717-marostegui.json
10:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1119.eqiad.wmnet with reason: Maintenance
10:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1119.eqiad.wmnet with reason: Maintenance
10:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1128.eqiad.wmnet with reason: Maintenance
10:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1128.eqiad.wmnet with reason: Maintenance
10:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1133.eqiad.wmnet with reason: Maintenance
10:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1133.eqiad.wmnet with reason: Maintenance
10:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T277354)', diff saved to https://phabricator.wikimedia.org/P18209 and previous config saved to /var/cache/conftool/dbconfig/20211214-104538-marostegui.json
10:36 ayounsi@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netflow4001.ulsfo.wmnet
10:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P18208 and previous config saved to /var/cache/conftool/dbconfig/20211214-103033-marostegui.json
10:29 ayounsi@cumin1001: START - Cookbook sre.hosts.decommission for hosts netflow4001.ulsfo.wmnet
10:22 ayounsi@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts netflow2001.codfw.wmnet
10:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2021.codfw.wmnet with OS buster
10:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P18207 and previous config saved to /var/cache/conftool/dbconfig/20211214-101528-marostegui.json
10:10 ayounsi@cumin1001: START - Cookbook sre.hosts.decommission for hosts netflow2001.codfw.wmnet
10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T277354)', diff saved to https://phabricator.wikimedia.org/P18206 and previous config saved to /var/cache/conftool/dbconfig/20211214-100023-marostegui.json
09:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1134 (T277354)', diff saved to https://phabricator.wikimedia.org/P18205 and previous config saved to /var/cache/conftool/dbconfig/20211214-095837-marostegui.json
09:58 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1134.eqiad.wmnet with reason: Maintenance
09:58 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1134.eqiad.wmnet with reason: Maintenance
09:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T277354)', diff saved to https://phabricator.wikimedia.org/P18204 and previous config saved to /var/cache/conftool/dbconfig/20211214-095829-marostegui.json
09:46 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2021.codfw.wmnet with OS buster
09:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2017.codfw.wmnet with OS buster
09:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P18203 and previous config saved to /var/cache/conftool/dbconfig/20211214-094324-marostegui.json
09:39 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
09:38 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
09:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P18202 and previous config saved to /var/cache/conftool/dbconfig/20211214-092820-marostegui.json
09:15 ryankemper@cumin2001: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw rolling restart - ryankemper@cumin2001 - T297468
09:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T277354)', diff saved to https://phabricator.wikimedia.org/P18201 and previous config saved to /var/cache/conftool/dbconfig/20211214-091315-marostegui.json
09:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1135 (T277354)', diff saved to https://phabricator.wikimedia.org/P18200 and previous config saved to /var/cache/conftool/dbconfig/20211214-091130-marostegui.json
09:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1135.eqiad.wmnet with reason: Maintenance
09:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1135.eqiad.wmnet with reason: Maintenance
09:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1139.eqiad.wmnet with reason: Maintenance
09:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1139.eqiad.wmnet with reason: Maintenance
09:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1140.eqiad.wmnet with reason: Maintenance
09:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1140.eqiad.wmnet with reason: Maintenance
09:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T277354)', diff saved to https://phabricator.wikimedia.org/P18199 and previous config saved to /var/cache/conftool/dbconfig/20211214-090948-marostegui.json
09:04 ayounsi@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host netflow1002.eqiad.wmnet
08:57 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2017.codfw.wmnet with OS buster
08:56 ayounsi@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host netflow5002.eqsin.wmnet
08:54 moritzm: failover Ganeti master to ganeti2016 T296622
08:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P18198 and previous config saved to /var/cache/conftool/dbconfig/20211214-085443-marostegui.json
08:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2008.codfw.wmnet with OS buster
08:49 ayounsi@cumin1001: START - Cookbook sre.ganeti.makevm for new host netflow1002.eqiad.wmnet
08:49 ayounsi@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host netflow4002.ulsfo.wmnet
08:43 ayounsi@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host netflow3002.esams.wmnet
08:43 ayounsi@cumin1001: START - Cookbook sre.ganeti.makevm for new host netflow5002.eqsin.wmnet
08:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P18197 and previous config saved to /var/cache/conftool/dbconfig/20211214-083938-marostegui.json
08:35 ayounsi@cumin1001: START - Cookbook sre.ganeti.makevm for new host netflow4002.ulsfo.wmnet
08:34 dcausse: restart blazegraph on wdqs1013 (jvm stuck for 5h)
08:33 ayounsi@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host netflow5002.eqsin.wmnet
08:33 ayounsi@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host netflow4002.ulsfo.wmnet
08:30 ayounsi@cumin1001: START - Cookbook sre.ganeti.makevm for new host netflow5002.eqsin.wmnet
08:30 ayounsi@cumin1001: START - Cookbook sre.ganeti.makevm for new host netflow4002.ulsfo.wmnet
08:29 ayounsi@cumin1001: START - Cookbook sre.ganeti.makevm for new host netflow3002.esams.wmnet
08:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T277354)', diff saved to https://phabricator.wikimedia.org/P18196 and previous config saved to /var/cache/conftool/dbconfig/20211214-082433-marostegui.json
08:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1163 (T277354)', diff saved to https://phabricator.wikimedia.org/P18195 and previous config saved to /var/cache/conftool/dbconfig/20211214-082249-marostegui.json
08:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1163.eqiad.wmnet with reason: Maintenance
08:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1163.eqiad.wmnet with reason: Maintenance
08:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 (T277354)', diff saved to https://phabricator.wikimedia.org/P18194 and previous config saved to /var/cache/conftool/dbconfig/20211214-082241-marostegui.json
08:22 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2008.codfw.wmnet with OS buster
08:09 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P18193 and previous config saved to /var/cache/conftool/dbconfig/20211214-080736-marostegui.json
08:03 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
07:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P18192 and previous config saved to /var/cache/conftool/dbconfig/20211214-075232-marostegui.json
07:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 (T277354)', diff saved to https://phabricator.wikimedia.org/P18191 and previous config saved to /var/cache/conftool/dbconfig/20211214-073727-marostegui.json
07:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1164 (T277354)', diff saved to https://phabricator.wikimedia.org/P18190 and previous config saved to /var/cache/conftool/dbconfig/20211214-073541-marostegui.json
07:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1164.eqiad.wmnet with reason: Maintenance
07:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1164.eqiad.wmnet with reason: Maintenance
07:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T277354)', diff saved to https://phabricator.wikimedia.org/P18189 and previous config saved to /var/cache/conftool/dbconfig/20211214-073534-marostegui.json
07:24 ryankemper: T297468 `sudo cookbook sre.elasticsearch.rolling-operation search_codfw "codfw rolling restart" --nodes-per-run 3 --start-datetime 2021-12-14T01:27:58 --task-id T297468` on `ryankemper@cumin2001` tmux `elastic_restarts`
07:24 ryankemper@cumin2001: START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw rolling restart - ryankemper@cumin2001 - T297468
07:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P18188 and previous config saved to /var/cache/conftool/dbconfig/20211214-072029-marostegui.json
07:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P18187 and previous config saved to /var/cache/conftool/dbconfig/20211214-070524-marostegui.json
06:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T277354)', diff saved to https://phabricator.wikimedia.org/P18186 and previous config saved to /var/cache/conftool/dbconfig/20211214-065019-marostegui.json
06:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1169 (T277354)', diff saved to https://phabricator.wikimedia.org/P18185 and previous config saved to /var/cache/conftool/dbconfig/20211214-064833-marostegui.json
06:48 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1169.eqiad.wmnet with reason: Maintenance
06:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1169.eqiad.wmnet with reason: Maintenance
06:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T277354)', diff saved to https://phabricator.wikimedia.org/P18184 and previous config saved to /var/cache/conftool/dbconfig/20211214-064825-marostegui.json
06:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P18183 and previous config saved to /var/cache/conftool/dbconfig/20211214-063321-marostegui.json
06:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P18182 and previous config saved to /var/cache/conftool/dbconfig/20211214-061816-marostegui.json
06:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T277354)', diff saved to https://phabricator.wikimedia.org/P18181 and previous config saved to /var/cache/conftool/dbconfig/20211214-060311-marostegui.json
06:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1184 (T277354)', diff saved to https://phabricator.wikimedia.org/P18180 and previous config saved to /var/cache/conftool/dbconfig/20211214-060125-marostegui.json
06:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1184.eqiad.wmnet with reason: Maintenance
06:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1184.eqiad.wmnet with reason: Maintenance
06:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
06:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
05:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 15 hosts with reason: Maintenance
05:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on 15 hosts with reason: Maintenance
05:35 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
05:34 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
05:12 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
05:11 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
04:42 ryankemper@cumin1001: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad rolling restart - ryankemper@cumin1001 - T297468
02:34 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
02:33 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
02:06 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
02:05 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
01:42 ryankemper: T297468 `sudo cookbook sre.elasticsearch.rolling-operation search_eqiad "eqiad rolling restart" --nodes-per-run 3 --start-datetime 2021-12-14T01:27:58 --task-id T297468` on `ryankemper@cumin1001` tmux `elastic_restarts`
01:41 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad rolling restart - ryankemper@cumin1001 - T297468
01:41 tgr: UTC late deploys done
01:39 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
01:37 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:47 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:45 lucaswerkmeister-wmde@deploy1002: Synchronized php-1.38.0-wmf.9/extensions/WikibaseLexeme: Backport: Remove most of mw.wikibase.lexeme Lua module (T297404) (no-op because Lexeme Lua is not yet enabled in prod) (duration: 00m 58s)
17:43 accraze@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' .
17:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2021.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
17:41 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti2021.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
17:40 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:39 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
16:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1102.eqiad.wmnet with reason: Maintenance
16:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1102.eqiad.wmnet with reason: Maintenance
16:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T277354)', diff saved to https://phabricator.wikimedia.org/P18177 and previous config saved to /var/cache/conftool/dbconfig/20211213-161414-marostegui.json
15:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P18176 and previous config saved to /var/cache/conftool/dbconfig/20211213-155909-marostegui.json
15:57 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:56 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:53 lucaswerkmeister-wmde@deploy1002: Synchronized php-1.38.0-wmf.12/extensions/WikibaseLexeme: Backport: Add form:hasGrammaticalFeature() method (T297478) (no-op because Lexeme Lua is not yet enabled in prod) (duration: 00m 57s)
15:50 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:49 lucaswerkmeister-wmde@deploy1002: Synchronized php-1.38.0-wmf.12/extensions/WikibaseLexeme: Backport: Remove most of mw.wikibase.lexeme Lua module (T297404) (no-op because Lexeme Lua is not yet enabled in prod) (duration: 00m 58s)
15:49 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P18175 and previous config saved to /var/cache/conftool/dbconfig/20211213-154404-marostegui.json
15:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T277354)', diff saved to https://phabricator.wikimedia.org/P18174 and previous config saved to /var/cache/conftool/dbconfig/20211213-152859-marostegui.json
15:25 robh: dns6001 returned to service (icinga checks going green) via T286507
15:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T277354)', diff saved to https://phabricator.wikimedia.org/P18173 and previous config saved to /var/cache/conftool/dbconfig/20211213-151657-marostegui.json
15:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db[1112,1154].eqiad.wmnet with reason: Maintenance
15:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db[1112,1154].eqiad.wmnet with reason: Maintenance
15:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1123 (T277354)', diff saved to https://phabricator.wikimedia.org/P18172 and previous config saved to /var/cache/conftool/dbconfig/20211213-151645-marostegui.json
15:15 hnowlan: joining restbase2025-b to cassandra cluster
15:15 robh: dns6002 bios update done, returned to green in icinga, dns6001 coming down next for firmware update via T286507
15:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1123', diff saved to https://phabricator.wikimedia.org/P18171 and previous config saved to /var/cache/conftool/dbconfig/20211213-150141-marostegui.json
14:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1123', diff saved to https://phabricator.wikimedia.org/P18170 and previous config saved to /var/cache/conftool/dbconfig/20211213-144636-marostegui.json
14:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1123 (T277354)', diff saved to https://phabricator.wikimedia.org/P18169 and previous config saved to /var/cache/conftool/dbconfig/20211213-143131-marostegui.json
14:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1123 (T277354)', diff saved to https://phabricator.wikimedia.org/P18168 and previous config saved to /var/cache/conftool/dbconfig/20211213-142141-marostegui.json
14:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1123.eqiad.wmnet with reason: Maintenance
14:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1123.eqiad.wmnet with reason: Maintenance
14:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1145.eqiad.wmnet with reason: Maintenance
14:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1145.eqiad.wmnet with reason: Maintenance
14:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T277354)', diff saved to https://phabricator.wikimedia.org/P18167 and previous config saved to /var/cache/conftool/dbconfig/20211213-141052-marostegui.json
13:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P18166 and previous config saved to /var/cache/conftool/dbconfig/20211213-135547-marostegui.json
13:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P18165 and previous config saved to /var/cache/conftool/dbconfig/20211213-134042-marostegui.json
13:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T277354)', diff saved to https://phabricator.wikimedia.org/P18164 and previous config saved to /var/cache/conftool/dbconfig/20211213-132537-marostegui.json
13:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T277354)', diff saved to https://phabricator.wikimedia.org/P18163 and previous config saved to /var/cache/conftool/dbconfig/20211213-131538-marostegui.json
13:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1166.eqiad.wmnet with reason: Maintenance
13:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1166.eqiad.wmnet with reason: Maintenance
13:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T277354)', diff saved to https://phabricator.wikimedia.org/P18162 and previous config saved to /var/cache/conftool/dbconfig/20211213-131529-marostegui.json
13:04 ayounsi@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host netflow2002.codfw.wmnet
13:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P18161 and previous config saved to /var/cache/conftool/dbconfig/20211213-130024-marostegui.json
12:48 ayounsi@cumin1001: START - Cookbook sre.ganeti.makevm for new host netflow2002.codfw.wmnet
12:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P18160 and previous config saved to /var/cache/conftool/dbconfig/20211213-124519-marostegui.json
12:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T277354)', diff saved to https://phabricator.wikimedia.org/P18159 and previous config saved to /var/cache/conftool/dbconfig/20211213-123014-marostegui.json
12:28 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:27 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T277354)', diff saved to https://phabricator.wikimedia.org/P18158 and previous config saved to /var/cache/conftool/dbconfig/20211213-121811-marostegui.json
12:18 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1175.eqiad.wmnet with reason: Maintenance
12:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1175.eqiad.wmnet with reason: Maintenance
12:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T277354)', diff saved to https://phabricator.wikimedia.org/P18157 and previous config saved to /var/cache/conftool/dbconfig/20211213-121803-marostegui.json
12:11 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 361214b: Enable Disambiguator notifications on more wikis (T297175) (duration: 00m 56s)
12:11 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:07 hnowlan: joining restbase2025-a to cassandra cluster
12:05 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on restbase[2025-2026].codfw.wmnet with reason: New cassandra hosts awaiting syncing
12:05 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on restbase[2025-2026].codfw.wmnet with reason: New cassandra hosts awaiting syncing
12:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P18156 and previous config saved to /var/cache/conftool/dbconfig/20211213-120259-marostegui.json
11:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P18155 and previous config saved to /var/cache/conftool/dbconfig/20211213-114754-marostegui.json
11:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T277354)', diff saved to https://phabricator.wikimedia.org/P18154 and previous config saved to /var/cache/conftool/dbconfig/20211213-113249-marostegui.json
11:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T277354)', diff saved to https://phabricator.wikimedia.org/P18153 and previous config saved to /var/cache/conftool/dbconfig/20211213-112245-marostegui.json
11:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1179.eqiad.wmnet with reason: Maintenance
11:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1179.eqiad.wmnet with reason: Maintenance
11:22 urbanecm: Run namespaceDupes.php --wiki=$WIKI --fix --add-prefix=BROKEN for wikis in P18152
11:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
11:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
10:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 7 hosts with reason: Maintenance
10:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on 7 hosts with reason: Maintenance
10:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1172', diff saved to https://phabricator.wikimedia.org/P18151 and previous config saved to /var/cache/conftool/dbconfig/20211213-101707-marostegui.json
10:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1172 (T277354)', diff saved to https://phabricator.wikimedia.org/P18150 and previous config saved to /var/cache/conftool/dbconfig/20211213-101427-marostegui.json
10:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1172.eqiad.wmnet with reason: Maintenance
10:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1172.eqiad.wmnet with reason: Maintenance
10:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance
10:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance
10:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1126', diff saved to https://phabricator.wikimedia.org/P18149 and previous config saved to /var/cache/conftool/dbconfig/20211213-101013-marostegui.json
10:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1126 (T277354)', diff saved to https://phabricator.wikimedia.org/P18148 and previous config saved to /var/cache/conftool/dbconfig/20211213-100238-marostegui.json
10:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1126.eqiad.wmnet with reason: Maintenance
10:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1126.eqiad.wmnet with reason: Maintenance
10:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1116.eqiad.wmnet with reason: Maintenance
10:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1116.eqiad.wmnet with reason: Maintenance
10:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1114', diff saved to https://phabricator.wikimedia.org/P18147 and previous config saved to /var/cache/conftool/dbconfig/20211213-100143-marostegui.json
09:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2130 and db2074', diff saved to https://phabricator.wikimedia.org/P18146 and previous config saved to /var/cache/conftool/dbconfig/20211213-095949-marostegui.json
09:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1114 (T277354)', diff saved to https://phabricator.wikimedia.org/P18145 and previous config saved to /var/cache/conftool/dbconfig/20211213-095851-marostegui.json
09:58 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1114.eqiad.wmnet with reason: Maintenance
09:58 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1114.eqiad.wmnet with reason: Maintenance
09:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1111', diff saved to https://phabricator.wikimedia.org/P18144 and previous config saved to /var/cache/conftool/dbconfig/20211213-095728-marostegui.json
09:49 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1111 (T277354)', diff saved to https://phabricator.wikimedia.org/P18143 and previous config saved to /var/cache/conftool/dbconfig/20211213-094900-marostegui.json
09:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1111.eqiad.wmnet with reason: Maintenance
09:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1111.eqiad.wmnet with reason: Maintenance
09:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1104 (T277354)', diff saved to https://phabricator.wikimedia.org/P18142 and previous config saved to /var/cache/conftool/dbconfig/20211213-094853-marostegui.json
09:45 urbanecm: Staging at mwdebug1001 ended
09:43 urbanecm: pwnwiki: Create DB tables for GrowthExperiments
09:43 dcausse@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' .
09:42 urbanecm: Stagging at mwdebug1001
09:38 Amir1: revoking DROP from centralauth grant of wikiadmin (T249683)
09:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1104', diff saved to https://phabricator.wikimedia.org/P18141 and previous config saved to /var/cache/conftool/dbconfig/20211213-093348-marostegui.json
09:32 dcausse@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' .
09:24 dcausse@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' .
09:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1104', diff saved to https://phabricator.wikimedia.org/P18140 and previous config saved to /var/cache/conftool/dbconfig/20211213-091844-marostegui.json
09:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1104 (T277354)', diff saved to https://phabricator.wikimedia.org/P18139 and previous config saved to /var/cache/conftool/dbconfig/20211213-090339-marostegui.json
08:48 Amir1: removing grant of '%a%' on db1123 (s3) T296537
08:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1104 (T277354)', diff saved to https://phabricator.wikimedia.org/P18138 and previous config saved to /var/cache/conftool/dbconfig/20211213-084729-marostegui.json
08:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1104.eqiad.wmnet with reason: Maintenance
08:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1104.eqiad.wmnet with reason: Maintenance
08:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T277354)', diff saved to https://phabricator.wikimedia.org/P18137 and previous config saved to /var/cache/conftool/dbconfig/20211213-084721-marostegui.json
08:45 Amir1: fixing centralauth grants of wikiuser on all of s7 T296537
08:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P18136 and previous config saved to /var/cache/conftool/dbconfig/20211213-083217-marostegui.json
08:27 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:20 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P18135 and previous config saved to /var/cache/conftool/dbconfig/20211213-081712-marostegui.json
08:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T277354)', diff saved to https://phabricator.wikimedia.org/P18133 and previous config saved to /var/cache/conftool/dbconfig/20211213-080207-marostegui.json
08:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3318 (T277354)', diff saved to https://phabricator.wikimedia.org/P18132 and previous config saved to /var/cache/conftool/dbconfig/20211213-080101-marostegui.json
08:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1101.eqiad.wmnet with reason: Maintenance
08:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1101.eqiad.wmnet with reason: Maintenance
08:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T277354)', diff saved to https://phabricator.wikimedia.org/P18131 and previous config saved to /var/cache/conftool/dbconfig/20211213-080054-marostegui.json
07:59 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
07:52 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
07:46 moritzm: drain primary/secondary instances off ganeti2021 T296622
07:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318', diff saved to https://phabricator.wikimedia.org/P18130 and previous config saved to /var/cache/conftool/dbconfig/20211213-074549-marostegui.json
07:40 Amir1: start of clean up of flaggedtemplates on all flaggedrevs wikis: T296380
07:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318', diff saved to https://phabricator.wikimedia.org/P18129 and previous config saved to /var/cache/conftool/dbconfig/20211213-073044-marostegui.json
07:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T277354)', diff saved to https://phabricator.wikimedia.org/P18128 and previous config saved to /var/cache/conftool/dbconfig/20211213-071539-marostegui.json
07:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3318 (T277354)', diff saved to https://phabricator.wikimedia.org/P18127 and previous config saved to /var/cache/conftool/dbconfig/20211213-071433-marostegui.json
07:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1099.eqiad.wmnet with reason: Maintenance
07:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1099.eqiad.wmnet with reason: Maintenance
07:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 13 hosts with reason: Maintenance
07:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on 13 hosts with reason: Maintenance
07:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T277354)', diff saved to https://phabricator.wikimedia.org/P18126 and previous config saved to /var/cache/conftool/dbconfig/20211213-070430-marostegui.json
07:02 marostegui@cumin1001: dbctl commit (dc=all): 'db1123 (re)pooling @ 100%: After mysql restart', diff saved to https://phabricator.wikimedia.org/P18125 and previous config saved to /var/cache/conftool/dbconfig/20211213-070204-root.json
06:51 elukey: run `apt-get clean` on aphlict1001 to free some space
06:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P18124 and previous config saved to /var/cache/conftool/dbconfig/20211213-064926-marostegui.json
06:47 marostegui@cumin1001: dbctl commit (dc=all): 'db1123 (re)pooling @ 75%: After mysql restart', diff saved to https://phabricator.wikimedia.org/P18123 and previous config saved to /var/cache/conftool/dbconfig/20211213-064700-root.json
06:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P18122 and previous config saved to /var/cache/conftool/dbconfig/20211213-063421-marostegui.json
06:31 marostegui@cumin1001: dbctl commit (dc=all): 'db1123 (re)pooling @ 50%: After mysql restart', diff saved to https://phabricator.wikimedia.org/P18121 and previous config saved to /var/cache/conftool/dbconfig/20211213-063156-root.json
06:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T277354)', diff saved to https://phabricator.wikimedia.org/P18120 and previous config saved to /var/cache/conftool/dbconfig/20211213-061916-marostegui.json
06:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1181 (T277354)', diff saved to https://phabricator.wikimedia.org/P18119 and previous config saved to /var/cache/conftool/dbconfig/20211213-061756-marostegui.json
06:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance
06:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance
06:16 marostegui@cumin1001: dbctl commit (dc=all): 'db1123 (re)pooling @ 25%: After mysql restart', diff saved to https://phabricator.wikimedia.org/P18118 and previous config saved to /var/cache/conftool/dbconfig/20211213-061652-root.json
06:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1123 for a restart', diff saved to https://phabricator.wikimedia.org/P18117 and previous config saved to /var/cache/conftool/dbconfig/20211213-060343-marostegui.json
2021-12-12
14:35 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host graphite1004.eqiad.wmnet
14:30 filippo@cumin1001: START - Cookbook sre.hosts.reboot-single for host graphite1004.eqiad.wmnet
14:23 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on graphite1004.eqiad.wmnet with reason: powercycle
14:23 filippo@cumin1001: START - Cookbook sre.hosts.downtime for 0:30:00 on graphite1004.eqiad.wmnet with reason: powercycle
14:08 filippo@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host graphite1004.eqiad.wmnet
14:08 filippo@cumin1001: START - Cookbook sre.hosts.reboot-single for host graphite1004.eqiad.wmnet
04:17 ejegg: updated SmashPig standalone (IPN listener) from 211f8e65 to 9e885819
2021-12-11
19:03 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster
00:04 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
00:00 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
2021-12-10
22:39 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
22:33 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
22:12 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
22:11 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
22:09 dancy@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.9 refs T293953
21:10 rzl: sudo cumin -b7 -s10 -p0 'A:mw-eqiad and not P{mw1414.eqiad.wmnet}' restart-php7.2-fpm - T297517
21:09 rzl: rzl@mw1414:~$ sudo depool - preserving for investigation, T297517
20:43 rzl: sudo cumin -b2 -s10 -p0 'A:parsoid and not P{wtp1025.eqiad.wmnet}' restart-php7.2-fpm - T297517
20:38 rzl: rzl@wtp1025:~$ sudo restart-php7.2-fpm - T297517 - rolling restart to follow
18:50 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts copernicium.wikimedia.org
18:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on ganeti2017.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
18:11 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on ganeti2017.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
18:04 jhathaway@cumin1001: START - Cookbook sre.hosts.decommission for hosts copernicium.wikimedia.org
16:06 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
16:05 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:54 jynus: increase backup2006's allocated disk space
15:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T277354)', diff saved to https://phabricator.wikimedia.org/P18112 and previous config saved to /var/cache/conftool/dbconfig/20211210-152410-marostegui.json
15:19 jelto@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
15:15 jelto: remove tiller from eqiad Kubernetes cluster
15:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P18111 and previous config saved to /var/cache/conftool/dbconfig/20211210-150906-marostegui.json
15:08 jelto@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
15:06 jynus: increase backup2005's allocated disk space
14:55 moritzm: drain primary/secondary instances off ganeti2017 T296622
14:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P18110 and previous config saved to /var/cache/conftool/dbconfig/20211210-145401-marostegui.json
14:50 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
14:49 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on restbase2026.codfw.wmnet with reason: New cassandra hosts awaiting syncing
14:49 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on restbase2026.codfw.wmnet with reason: New cassandra hosts awaiting syncing
14:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on ganeti2008.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
14:48 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on ganeti2008.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
14:48 jelto: remove tiller from staging-eqiad Kubernetes cluster
14:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T277354)', diff saved to https://phabricator.wikimedia.org/P18109 and previous config saved to /var/cache/conftool/dbconfig/20211210-143856-marostegui.json
14:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T277354)', diff saved to https://phabricator.wikimedia.org/P18108 and previous config saved to /var/cache/conftool/dbconfig/20211210-143636-marostegui.json
14:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1174.eqiad.wmnet with reason: Maintenance
14:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1174.eqiad.wmnet with reason: Maintenance
14:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T277354)', diff saved to https://phabricator.wikimedia.org/P18107 and previous config saved to /var/cache/conftool/dbconfig/20211210-143628-marostegui.json
14:36 hnowlan@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2026.codfw.wmnet with OS buster
14:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P18106 and previous config saved to /var/cache/conftool/dbconfig/20211210-142123-marostegui.json
14:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P18105 and previous config saved to /var/cache/conftool/dbconfig/20211210-140618-marostegui.json
14:01 jynus: increase backup2004's allocated disk space
13:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T277354)', diff saved to https://phabricator.wikimedia.org/P18104 and previous config saved to /var/cache/conftool/dbconfig/20211210-135114-marostegui.json
13:49 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T277354)', diff saved to https://phabricator.wikimedia.org/P18103 and previous config saved to /var/cache/conftool/dbconfig/20211210-134953-marostegui.json
13:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db[1155,1158].eqiad.wmnet with reason: Maintenance
13:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db[1155,1158].eqiad.wmnet with reason: Maintenance
13:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T277354)', diff saved to https://phabricator.wikimedia.org/P18102 and previous config saved to /var/cache/conftool/dbconfig/20211210-134941-marostegui.json
13:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P18101 and previous config saved to /var/cache/conftool/dbconfig/20211210-133437-marostegui.json
13:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P18100 and previous config saved to /var/cache/conftool/dbconfig/20211210-131932-marostegui.json
13:17 hnowlan@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2025.codfw.wmnet with OS buster
13:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T277354)', diff saved to https://phabricator.wikimedia.org/P18099 and previous config saved to /var/cache/conftool/dbconfig/20211210-130427-marostegui.json
13:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T277354)', diff saved to https://phabricator.wikimedia.org/P18098 and previous config saved to /var/cache/conftool/dbconfig/20211210-130051-marostegui.json
13:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1127.eqiad.wmnet with reason: Maintenance
13:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1127.eqiad.wmnet with reason: Maintenance
12:56 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on restbase[2024-2025].codfw.wmnet with reason: New cassandra hosts awaiting syncing
12:56 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on restbase[2024-2025].codfw.wmnet with reason: New cassandra hosts awaiting syncing
12:53 hnowlan@cumin2002: START - Cookbook sre.hosts.reimage for host restbase2026.codfw.wmnet with OS buster
12:51 hnowlan@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2024.codfw.wmnet with OS buster
12:37 hnowlan: including cassandra-tools in cassandra311 component of buster-wikimedia
12:31 _joe_: manually modifying configmaps for rsyslog in mwdebug for live troubleshooting.
12:28 hnowlan@cumin2002: START - Cookbook sre.hosts.reimage for host restbase2025.codfw.wmnet with OS buster
12:08 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:03 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:02 hnowlan@cumin2002: START - Cookbook sre.hosts.reimage for host restbase2024.codfw.wmnet with OS buster
10:03 jayme: published docker-registry.discovery.wmnet/cert-manager/cainjector:1.5.4-2 docker-registry.discovery.wmnet/cert-manager/webhook:1.5.4-2 docker-registry.discovery.wmnet/cert-manager/controller:1.5.4-2
10:00 vgutierrez: repool cp5006
09:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T277354)', diff saved to https://phabricator.wikimedia.org/P18097 and previous config saved to /var/cache/conftool/dbconfig/20211210-095833-marostegui.json
09:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P18096 and previous config saved to /var/cache/conftool/dbconfig/20211210-094328-marostegui.json
09:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P18095 and previous config saved to /var/cache/conftool/dbconfig/20211210-092823-marostegui.json
09:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T277354)', diff saved to https://phabricator.wikimedia.org/P18094 and previous config saved to /var/cache/conftool/dbconfig/20211210-091319-marostegui.json
09:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T277354)', diff saved to https://phabricator.wikimedia.org/P18093 and previous config saved to /var/cache/conftool/dbconfig/20211210-091041-marostegui.json
09:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1098.eqiad.wmnet with reason: Maintenance
09:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1098.eqiad.wmnet with reason: Maintenance
09:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T277354)', diff saved to https://phabricator.wikimedia.org/P18092 and previous config saved to /var/cache/conftool/dbconfig/20211210-091034-marostegui.json
08:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P18091 and previous config saved to /var/cache/conftool/dbconfig/20211210-085529-marostegui.json
08:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P18090 and previous config saved to /var/cache/conftool/dbconfig/20211210-084024-marostegui.json
08:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T277354)', diff saved to https://phabricator.wikimedia.org/P18089 and previous config saved to /var/cache/conftool/dbconfig/20211210-082520-marostegui.json
08:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T277354)', diff saved to https://phabricator.wikimedia.org/P18088 and previous config saved to /var/cache/conftool/dbconfig/20211210-082041-marostegui.json
08:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1101.eqiad.wmnet with reason: Maintenance
08:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1101.eqiad.wmnet with reason: Maintenance
08:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T277354)', diff saved to https://phabricator.wikimedia.org/P18087 and previous config saved to /var/cache/conftool/dbconfig/20211210-082034-marostegui.json
08:13 moritzm: drain primary/secondary instance off ganeti2008 T296622
08:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P18086 and previous config saved to /var/cache/conftool/dbconfig/20211210-080529-marostegui.json
07:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P18085 and previous config saved to /var/cache/conftool/dbconfig/20211210-075024-marostegui.json
07:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T277354)', diff saved to https://phabricator.wikimedia.org/P18084 and previous config saved to /var/cache/conftool/dbconfig/20211210-073520-marostegui.json
07:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T277354)', diff saved to https://phabricator.wikimedia.org/P18083 and previous config saved to /var/cache/conftool/dbconfig/20211210-073342-marostegui.json
07:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1170.eqiad.wmnet with reason: Maintenance
07:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1170.eqiad.wmnet with reason: Maintenance
07:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance
07:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance
07:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
07:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
06:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 11 hosts with reason: Maintenance
06:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on 11 hosts with reason: Maintenance
05:55 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1139.eqiad.wmnet with reason: Maintenance
05:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1139.eqiad.wmnet with reason: Maintenance
05:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1102.eqiad.wmnet with reason: Maintenance
05:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1102.eqiad.wmnet with reason: Maintenance
19:13 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:12 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:09 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: c1e9551: Deploy sticky header and A/B test enrollment to office, test wikis (T295972) (duration: 01m 06s)
11:20 btullis@cumin1001: END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid analytics cluster: Roll restart of Druid jvm daemons.
11:20 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
11:19 topranks: Changing export policy applied on eqiad CRs for local confed to not rewrite next-hop for routes learnt from other WMF POPs (T295672)
11:19 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
11:13 elukey: reboot ores2001 (lost connectivity, we suspect some weird problem with the NIC, but no traces in the kernel logs)
11:10 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
11:09 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
11:07 hnowlan@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
10:58 btullis@cumin1001: END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid analytics cluster: Roll restart of Druid jvm daemons.
10:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2027.codfw.wmnet
10:52 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
10:48 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
10:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2027.codfw.wmnet
10:45 btullis@cumin1001: START - Cookbook sre.druid.roll-restart-workers for Druid analytics cluster: Roll restart of Druid jvm daemons.
10:02 vgutierrez: pool durum2002
10:00 vgutierrez: depool durum2002
09:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2020.codfw.wmnet
09:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2020.codfw.wmnet
09:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2020.codfw.wmnet with OS buster
08:19 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2020.codfw.wmnet with OS buster
08:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2020.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
08:12 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti2020.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
03:37 cwhite: bounce superset on an-tool1010 and 1005 to pick up statsd changes T247963
03:34 cwhite: bounce navtiming on webperf1001 to pick up statsd changes T297265
03:32 cwhite@deploy1002: Synchronized wmf-config/ProductionServices.php: fail over statsd to graphite2003 T297265 (duration: 01m 05s)
03:28 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
03:27 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
02:54 cwhite: failover statsd ingest host to graphite2003 T297265
21:41 dancy@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.12 refs T293953
21:30 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
21:30 ladsgroup@deploy1002: Synchronized php-1.38.0-wmf.12/extensions/WikidataPageBanner/includes/WikidataPageBanner.php: Backport: Make sure 'enable-toc' key is set (T297318) (duration: 01m 05s)
21:29 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:53 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:51 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:49 taavi@deploy1002: Synchronized php-1.38.0-wmf.12/extensions/3D/src/Hooks.php: Backport: Remove use of $wgUseAjax (duration: 01m 07s)
20:30 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:29 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:28 jhathaway: enable exim on mx2001
20:27 jgiannelos@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
20:27 jgiannelos@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
20:26 jgiannelos@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
20:23 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ldap-replica2006.wikimedia.org
15:33 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 10 hosts with reason: debugging bird/anycast-hc issues
15:33 sukhe@cumin1001: START - Cookbook sre.hosts.downtime for 0:30:00 on 10 hosts with reason: debugging bird/anycast-hc issues
15:25 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2026.codfw.wmnet with OS buster
15:21 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on durum2002.codfw.wmnet with reason: debugging bird/anycast-hc issues
15:21 sukhe@cumin1001: START - Cookbook sre.hosts.downtime for 0:30:00 on durum2002.codfw.wmnet with reason: debugging bird/anycast-hc issues
15:14 sukhe: running authdns-update for Gerrit:744094
15:09 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host restbase2026.codfw.wmnet with OS buster
14:38 jbond: renable puppet fleet wide post monitoring refactor 744787
14:15 Amir1: fixing heartbeat grants for wikiuser across the cluster (T296537)
14:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti[2013-2014].codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
14:11 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti[2013-2014].codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
14:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kubetcd2006.codfw.wmnet with reason: switch to drbd storage
14:07 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on kubetcd2006.codfw.wmnet with reason: switch to drbd storage
13:52 Amir1: removing wikiuser@localhost on s6 (T296537)
13:45 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase2026.codfw.wmnet with OS buster
13:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kubetcd2004.codfw.wmnet with reason: switch to drbd storage
13:42 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on kubetcd2004.codfw.wmnet with reason: switch to drbd storage
13:39 jbond: disable puppet fleet wide to rollout 744787
13:26 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host restbase2026.codfw.wmnet with OS buster
13:16 jelto: update GitLab to 14.4.4-ce.0
13:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ganeti2014.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
13:07 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on ganeti2014.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
12:46 Lucas_WMDE: UTC morning backport+config window done
12:44 lucaswerkmeister-wmde@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'termbox' for release 'production' .
12:42 lucaswerkmeister-wmde@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'termbox' for release 'production' .
12:39 lucaswerkmeister-wmde@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' .
12:39 lucaswerkmeister-wmde@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' .
12:24 jbond: merge refactor of monitoring classes 725045
12:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T277354)', diff saved to https://phabricator.wikimedia.org/P18071 and previous config saved to /var/cache/conftool/dbconfig/20211207-121655-marostegui.json
12:10 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:09 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P18070 and previous config saved to /var/cache/conftool/dbconfig/20211207-120150-marostegui.json
11:51 moritzm: draining primary/secondary instances off ganeti2014 T296622
11:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P18069 and previous config saved to /var/cache/conftool/dbconfig/20211207-114645-marostegui.json
11:38 cmooney@cumin1001: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudvirt1028.eqiad.wmnet
11:32 cmooney@cumin1001: START - Cookbook sre.hosts.dhcp for host cloudvirt1028.eqiad.wmnet
11:31 topranks: removing IP addressing on cloudvirt1028 manually and forcing DHCP to debug reimage failure (T296906)
11:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T277354)', diff saved to https://phabricator.wikimedia.org/P18068 and previous config saved to /var/cache/conftool/dbconfig/20211207-113140-marostegui.json
11:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T277354)', diff saved to https://phabricator.wikimedia.org/P18067 and previous config saved to /var/cache/conftool/dbconfig/20211207-113005-marostegui.json
11:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db[1155-1156].eqiad.wmnet with reason: Maintenance T277354
11:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db[1155-1156].eqiad.wmnet with reason: Maintenance T277354
11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T277354)', diff saved to https://phabricator.wikimedia.org/P18066 and previous config saved to /var/cache/conftool/dbconfig/20211207-112707-marostegui.json
11:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kubetcd2004.codfw.wmnet with reason: switch to drbd storage
11:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on kubetcd2004.codfw.wmnet with reason: switch to drbd storage
11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P18065 and previous config saved to /var/cache/conftool/dbconfig/20211207-111203-marostegui.json
11:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2012.codfw.wmnet
11:06 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2012.codfw.wmnet
10:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P18064 and previous config saved to /var/cache/conftool/dbconfig/20211207-105658-marostegui.json
10:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T277354)', diff saved to https://phabricator.wikimedia.org/P18063 and previous config saved to /var/cache/conftool/dbconfig/20211207-104153-marostegui.json
10:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T277354)', diff saved to https://phabricator.wikimedia.org/P18062 and previous config saved to /var/cache/conftool/dbconfig/20211207-104018-marostegui.json
10:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1182.eqiad.wmnet with reason: Maintenance T277354
10:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1182.eqiad.wmnet with reason: Maintenance T277354
10:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T277354)', diff saved to https://phabricator.wikimedia.org/P18061 and previous config saved to /var/cache/conftool/dbconfig/20211207-104010-marostegui.json
10:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2016.codfw.wmnet
10:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2016.codfw.wmnet
10:25 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
10:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P18060 and previous config saved to /var/cache/conftool/dbconfig/20211207-102505-marostegui.json
10:24 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
10:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ganeti2013.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
10:13 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on ganeti2013.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
10:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2012.codfw.wmnet with OS buster
10:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P18059 and previous config saved to /var/cache/conftool/dbconfig/20211207-101001-marostegui.json
10:01 marostegui: Deploy schema change on mailman (m5) T286552
09:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T277354)', diff saved to https://phabricator.wikimedia.org/P18058 and previous config saved to /var/cache/conftool/dbconfig/20211207-095456-marostegui.json
09:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T277354)', diff saved to https://phabricator.wikimedia.org/P18057 and previous config saved to /var/cache/conftool/dbconfig/20211207-095319-marostegui.json
09:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1170.eqiad.wmnet with reason: Maintenance T277354
09:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1170.eqiad.wmnet with reason: Maintenance T277354
09:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T277354)', diff saved to https://phabricator.wikimedia.org/P18056 and previous config saved to /var/cache/conftool/dbconfig/20211207-095312-marostegui.json
09:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P18055 and previous config saved to /var/cache/conftool/dbconfig/20211207-093807-marostegui.json
09:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2016.codfw.wmnet with OS buster
09:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P18054 and previous config saved to /var/cache/conftool/dbconfig/20211207-092302-marostegui.json
09:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T277354)', diff saved to https://phabricator.wikimedia.org/P18053 and previous config saved to /var/cache/conftool/dbconfig/20211207-090758-marostegui.json
09:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T277354)', diff saved to https://phabricator.wikimedia.org/P18052 and previous config saved to /var/cache/conftool/dbconfig/20211207-090620-marostegui.json
09:06 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T277354
09:06 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T277354
09:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T277354)', diff saved to https://phabricator.wikimedia.org/P18051 and previous config saved to /var/cache/conftool/dbconfig/20211207-090613-marostegui.json
08:55 moritzm: draining primary/secondary instances off ganeti2013 T296622
08:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P18050 and previous config saved to /var/cache/conftool/dbconfig/20211207-085108-marostegui.json
08:47 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2016.codfw.wmnet with OS buster
08:45 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti2016.codfw.wmnet with OS buster
08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P18049 and previous config saved to /var/cache/conftool/dbconfig/20211207-083604-marostegui.json
08:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T277354)', diff saved to https://phabricator.wikimedia.org/P18048 and previous config saved to /var/cache/conftool/dbconfig/20211207-082059-marostegui.json
08:19 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T277354)', diff saved to https://phabricator.wikimedia.org/P18047 and previous config saved to /var/cache/conftool/dbconfig/20211207-081936-marostegui.json
08:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1129.eqiad.wmnet with reason: Maintenance T277354
08:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1129.eqiad.wmnet with reason: Maintenance T277354
08:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T277354)', diff saved to https://phabricator.wikimedia.org/P18046 and previous config saved to /var/cache/conftool/dbconfig/20211207-081928-marostegui.json
08:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P18045 and previous config saved to /var/cache/conftool/dbconfig/20211207-080424-marostegui.json
07:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P18044 and previous config saved to /var/cache/conftool/dbconfig/20211207-074919-marostegui.json
07:46 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
07:43 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
07:39 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 2178202: Deploy Growth mentor dashboard to all wikis (T278920) (duration: 00m 58s)
07:37 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
07:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T277354)', diff saved to https://phabricator.wikimedia.org/P18043 and previous config saved to /var/cache/conftool/dbconfig/20211207-073413-marostegui.json
07:33 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
07:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T277354)', diff saved to https://phabricator.wikimedia.org/P18042 and previous config saved to /var/cache/conftool/dbconfig/20211207-073252-marostegui.json
07:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1162.eqiad.wmnet with reason: Maintenance T277354
07:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1162.eqiad.wmnet with reason: Maintenance T277354
07:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 8 hosts with reason: Maintenance T277354
07:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on 8 hosts with reason: Maintenance T277354
07:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T277354)', diff saved to https://phabricator.wikimedia.org/P18041 and previous config saved to /var/cache/conftool/dbconfig/20211207-072311-marostegui.json
07:16 marostegui: power off db2074, db2078, db2101, db2130, dbproxy2004 T296930
07:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P18040 and previous config saved to /var/cache/conftool/dbconfig/20211207-070806-marostegui.json
06:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P18039 and previous config saved to /var/cache/conftool/dbconfig/20211207-065301-marostegui.json
06:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T277354)', diff saved to https://phabricator.wikimedia.org/P18038 and previous config saved to /var/cache/conftool/dbconfig/20211207-063756-marostegui.json
06:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T277354)', diff saved to https://phabricator.wikimedia.org/P18037 and previous config saved to /var/cache/conftool/dbconfig/20211207-063621-marostegui.json
06:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1105.eqiad.wmnet with reason: Maintenance T277354
06:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1105.eqiad.wmnet with reason: Maintenance T277354
06:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance T277354
06:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance T277354
06:31 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1100 (T277354)', diff saved to https://phabricator.wikimedia.org/P18036 and previous config saved to /var/cache/conftool/dbconfig/20211207-063140-marostegui.json
06:16 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1100', diff saved to https://phabricator.wikimedia.org/P18035 and previous config saved to /var/cache/conftool/dbconfig/20211207-061635-marostegui.json
06:14 marostegui: Apply SET GLOBAL innodb_checksum_algorithm=full_crc32; on db1107 T287244
06:01 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1100', diff saved to https://phabricator.wikimedia.org/P18034 and previous config saved to /var/cache/conftool/dbconfig/20211207-060130-marostegui.json
05:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2074 and db2130 T296930', diff saved to https://phabricator.wikimedia.org/P18033 and previous config saved to /var/cache/conftool/dbconfig/20211207-055808-marostegui.json
05:46 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1100 (T277354)', diff saved to https://phabricator.wikimedia.org/P18032 and previous config saved to /var/cache/conftool/dbconfig/20211207-054625-marostegui.json
05:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1100 (T277354)', diff saved to https://phabricator.wikimedia.org/P18031 and previous config saved to /var/cache/conftool/dbconfig/20211207-054506-marostegui.json
05:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1100.eqiad.wmnet with reason: Maintenance T277354
05:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db1100.eqiad.wmnet with reason: Maintenance T277354
00:10 cwhite: end codfw opensearch upgrade T288621
16:24 majavah: starting "foreachwiki namespaceDupes.php --fix | tee namespaceDupes-T293839-fix.txt" in mwmaint1002 screen session, T293839
15:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ganeti2012.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
15:55 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on ganeti2012.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
14:45 elukey: roll restart of nfacctd on netflow* nodes to pick up the new CA bundle for librdkafka
14:19 moritzm: draining primary/secondary instances off ganeti2012 T296622
14:06 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2016.codfw.wmnet with OS buster
14:00 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 4d8a75d: Deploy Growth features on zhwiki in dark mode (T287884) (duration: 00m 56s)
13:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ganeti2016.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
13:31 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on ganeti2016.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
13:30 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
13:25 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2011.codfw.wmnet with reason: readding to cluster after reimage
12:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti2011.codfw.wmnet with reason: readding to cluster after reimage
11:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2011.codfw.wmnet
11:41 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2011.codfw.wmnet
11:28 Amir1: dropping wikiadmin@localhost from all of s3 (T296511)
11:21 Amir1: dropping wikiadmin@localhost from all of s2 (T296511)
11:12 moritzm: draining primary/secondary instances off ganeti2016 T296622
10:38 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ml-etcd2003.codfw.wmnet with reason: switch to drbd storage
10:38 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on ml-etcd2003.codfw.wmnet with reason: switch to drbd storage
10:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2011.codfw.wmnet with OS buster
10:31 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
10:28 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
10:23 moritzm: draining primary/secondary instances off ganeti2015 T296622
09:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2011.codfw.wmnet with OS buster
09:09 elukey: move kafka main codfw to fixed uid/gid for the kafka user (requires a stop/start of all daemons) - T296982
08:13 moritzm: installing remaining icu security updates on buster
2021-12-04
01:14 mutante: mx2001 - did not come back from reboot, did not get IP on interface, could not start ferm, logged in via console with root password, in /etc/network/interfaces replaced all "ens5" with "ens13", rebooted again, selected previous kernel version
00:54 mutante: rebooting mx2001
00:31 jynus: manually restarting clamav on otrs1001 after being killed
11:01 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on ganeti2011.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
11:01 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on ganeti2011.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
09:38 moritzm: draining primary/secondary instances off ganeti2011 T296622
09:25 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2009.codfw.wmnet to ganeti01.svc.codfw.wmnet
09:24 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2009.codfw.wmnet to ganeti01.svc.codfw.wmnet
09:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2009.codfw.wmnet
09:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2009.codfw.wmnet
09:15 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1161 (T277354)', diff saved to https://phabricator.wikimedia.org/P18019 and previous config saved to /var/cache/conftool/dbconfig/20211203-091537-marostegui.json
09:00 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1161', diff saved to https://phabricator.wikimedia.org/P18018 and previous config saved to /var/cache/conftool/dbconfig/20211203-090033-marostegui.json
08:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2009.codfw.wmnet with OS buster
08:45 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1161', diff saved to https://phabricator.wikimedia.org/P18017 and previous config saved to /var/cache/conftool/dbconfig/20211203-084528-marostegui.json
08:44 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:43 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:30 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1161 (T277354)', diff saved to https://phabricator.wikimedia.org/P18016 and previous config saved to /var/cache/conftool/dbconfig/20211203-083023-marostegui.json
08:30 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2009.codfw.wmnet with OS buster
08:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T277354)', diff saved to https://phabricator.wikimedia.org/P18015 and previous config saved to /var/cache/conftool/dbconfig/20211203-082859-marostegui.json
08:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db[1154,1161].eqiad.wmnet with reason: Maintenance T277354
08:28 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db[1154,1161].eqiad.wmnet with reason: Maintenance T277354
08:28 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1110 (T277354)', diff saved to https://phabricator.wikimedia.org/P18014 and previous config saved to /var/cache/conftool/dbconfig/20211203-082848-marostegui.json
08:13 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1110', diff saved to https://phabricator.wikimedia.org/P18013 and previous config saved to /var/cache/conftool/dbconfig/20211203-081343-marostegui.json
07:58 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1110', diff saved to https://phabricator.wikimedia.org/P18012 and previous config saved to /var/cache/conftool/dbconfig/20211203-075839-marostegui.json
07:43 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1110 (T277354)', diff saved to https://phabricator.wikimedia.org/P18011 and previous config saved to /var/cache/conftool/dbconfig/20211203-074334-marostegui.json
07:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T277354)', diff saved to https://phabricator.wikimedia.org/P18010 and previous config saved to /var/cache/conftool/dbconfig/20211203-073910-marostegui.json
07:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1110.eqiad.wmnet with reason: Maintenance T277354
07:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db1110.eqiad.wmnet with reason: Maintenance T277354
07:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1150.eqiad.wmnet with reason: Maintenance T277354
07:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db1150.eqiad.wmnet with reason: Maintenance T277354
07:34 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1144:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P18009 and previous config saved to /var/cache/conftool/dbconfig/20211203-073404-marostegui.json
07:19 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P18008 and previous config saved to /var/cache/conftool/dbconfig/20211203-071900-marostegui.json
07:03 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P18007 and previous config saved to /var/cache/conftool/dbconfig/20211203-070355-marostegui.json
06:48 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1144:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P18006 and previous config saved to /var/cache/conftool/dbconfig/20211203-064850-marostegui.json
06:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1125.eqiad.wmnet with OS bullseye
06:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P18005 and previous config saved to /var/cache/conftool/dbconfig/20211203-062019-marostegui.json
06:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1144.eqiad.wmnet with reason: Maintenance T277354
06:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db1144.eqiad.wmnet with reason: Maintenance T277354
06:20 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1113:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P18004 and previous config saved to /var/cache/conftool/dbconfig/20211203-062011-marostegui.json
06:05 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P18003 and previous config saved to /var/cache/conftool/dbconfig/20211203-060506-marostegui.json
06:02 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1125.eqiad.wmnet with OS bullseye
05:50 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P18002 and previous config saved to /var/cache/conftool/dbconfig/20211203-055001-marostegui.json
05:34 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1113:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P18001 and previous config saved to /var/cache/conftool/dbconfig/20211203-053457-marostegui.json
05:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P18000 and previous config saved to /var/cache/conftool/dbconfig/20211203-053032-marostegui.json
05:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1113.eqiad.wmnet with reason: Maintenance T277354
05:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db1113.eqiad.wmnet with reason: Maintenance T277354
01:39 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2025.codfw.wmnet with OS buster
01:06 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host restbase2025.codfw.wmnet with OS buster
01:05 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2024.codfw.wmnet with OS buster
18:56 legoktm: upgraded scap to 4.1.0 on A:mw-canary, A:parsoid-canary, A:mw-jobrunner-canary (T296867)
18:45 legoktm: uploaded scap 4.1.0 to apt.wm.o (T296867)
18:22 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster
18:19 vgutierrez: re-enable puppet on cp3064 - T296874
18:14 hoo: Started Wikibase rebuildItemsPerSite on mwmaint1002 for wikidatawiki. Can be killed at any time, if necessary.
17:51 vgutierrez: puppet disabled on cp3064 to manually increase number of maxconns in HAProxy - T296874
17:38 ryankemper: [WDQS] Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/743216/; as a result of the fix `'-Dwdqs.throttling-filter.time-bucket-capacity-in-seconds=240', '-Dwdqs.throttling-filter.time-bucket-refill-amount-in-seconds=120', '-Dwdqs.throttling-filter.ban-duration-in-minutes=60'` will now be in the `extra_jvm_opts` for `wdqs-internal` hosts
15:38 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2022.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
15:38 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti2022.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage
14:51 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17997 and previous config saved to /var/cache/conftool/dbconfig/20211202-145151-marostegui.json
14:36 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17996 and previous config saved to /var/cache/conftool/dbconfig/20211202-143646-marostegui.json
14:21 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17995 and previous config saved to /var/cache/conftool/dbconfig/20211202-142141-marostegui.json
14:06 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17994 and previous config saved to /var/cache/conftool/dbconfig/20211202-140636-marostegui.json
14:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17993 and previous config saved to /var/cache/conftool/dbconfig/20211202-140557-marostegui.json
14:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2137.codfw.wmnet with reason: Maintenance T277354
14:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db2137.codfw.wmnet with reason: Maintenance T277354
14:05 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17992 and previous config saved to /var/cache/conftool/dbconfig/20211202-140548-marostegui.json
13:50 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17990 and previous config saved to /var/cache/conftool/dbconfig/20211202-135043-marostegui.json
13:49 hnowlan: roll-restarting tilerator,tileratorui,kartotherian in eqiad
13:37 hnowlan: roll-restarting tilerator,tileratorui,kartotherian in codfw
13:35 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17989 and previous config saved to /var/cache/conftool/dbconfig/20211202-133538-marostegui.json
13:20 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17988 and previous config saved to /var/cache/conftool/dbconfig/20211202-132034-marostegui.json
13:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17987 and previous config saved to /var/cache/conftool/dbconfig/20211202-131959-marostegui.json
13:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2094,2128].codfw.wmnet with reason: Maintenance T277354
13:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db[2094,2128].codfw.wmnet with reason: Maintenance T277354
13:19 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2113 (T277354)', diff saved to https://phabricator.wikimedia.org/P17986 and previous config saved to /var/cache/conftool/dbconfig/20211202-131949-marostegui.json
13:04 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2113 (T277354)', diff saved to https://phabricator.wikimedia.org/P17985 and previous config saved to /var/cache/conftool/dbconfig/20211202-130444-marostegui.json
12:49 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2113 (T277354)', diff saved to https://phabricator.wikimedia.org/P17983 and previous config saved to /var/cache/conftool/dbconfig/20211202-124940-marostegui.json
12:34 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2113 (T277354)', diff saved to https://phabricator.wikimedia.org/P17982 and previous config saved to /var/cache/conftool/dbconfig/20211202-123435-marostegui.json
12:34 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2113 (T277354)', diff saved to https://phabricator.wikimedia.org/P17981 and previous config saved to /var/cache/conftool/dbconfig/20211202-123356-marostegui.json
12:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2113.codfw.wmnet with reason: Maintenance T277354
12:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db2113.codfw.wmnet with reason: Maintenance T277354
12:33 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2111 (T277354)', diff saved to https://phabricator.wikimedia.org/P17980 and previous config saved to /var/cache/conftool/dbconfig/20211202-123348-marostegui.json
12:31 moritzm: installing NSS security updates
12:27 Lucas_WMDE: UTC morning backport+config window done
12:18 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2111 (T277354)', diff saved to https://phabricator.wikimedia.org/P17979 and previous config saved to /var/cache/conftool/dbconfig/20211202-121843-marostegui.json
12:12 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti2009.codfw.wmnet with OS buster
12:03 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2111 (T277354)', diff saved to https://phabricator.wikimedia.org/P17978 and previous config saved to /var/cache/conftool/dbconfig/20211202-120338-marostegui.json
12:03 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2009.codfw.wmnet with OS buster
11:48 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2111 (T277354)', diff saved to https://phabricator.wikimedia.org/P17977 and previous config saved to /var/cache/conftool/dbconfig/20211202-114833-marostegui.json
11:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2111 (T277354)', diff saved to https://phabricator.wikimedia.org/P17976 and previous config saved to /var/cache/conftool/dbconfig/20211202-114755-marostegui.json
11:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2111.codfw.wmnet with reason: Maintenance T277354
11:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db2111.codfw.wmnet with reason: Maintenance T277354
11:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2101.codfw.wmnet with reason: Maintenance T277354
11:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db2101.codfw.wmnet with reason: Maintenance T277354
11:47 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
11:47 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2089:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17975 and previous config saved to /var/cache/conftool/dbconfig/20211202-114711-marostegui.json
11:32 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2089:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17974 and previous config saved to /var/cache/conftool/dbconfig/20211202-113206-marostegui.json
11:28 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
11:21 moritzm: draining primary/secondary instances off ganeti2022 T296622
11:17 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2089:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17973 and previous config saved to /var/cache/conftool/dbconfig/20211202-111702-marostegui.json
11:01 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2089:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17972 and previous config saved to /var/cache/conftool/dbconfig/20211202-110157-marostegui.json
11:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2089:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17971 and previous config saved to /var/cache/conftool/dbconfig/20211202-110120-marostegui.json
11:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2089.codfw.wmnet with reason: Maintenance T277354
11:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db2089.codfw.wmnet with reason: Maintenance T277354
11:01 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2075 (T277354)', diff saved to https://phabricator.wikimedia.org/P17970 and previous config saved to /var/cache/conftool/dbconfig/20211202-110110-marostegui.json
10:46 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2075 (T277354)', diff saved to https://phabricator.wikimedia.org/P17969 and previous config saved to /var/cache/conftool/dbconfig/20211202-104606-marostegui.json
10:31 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2075 (T277354)', diff saved to https://phabricator.wikimedia.org/P17968 and previous config saved to /var/cache/conftool/dbconfig/20211202-103100-marostegui.json
10:15 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2075 (T277354)', diff saved to https://phabricator.wikimedia.org/P17967 and previous config saved to /var/cache/conftool/dbconfig/20211202-101555-marostegui.json
10:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2075 (T277354)', diff saved to https://phabricator.wikimedia.org/P17966 and previous config saved to /var/cache/conftool/dbconfig/20211202-101522-marostegui.json
10:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2075.codfw.wmnet with reason: Maintenance T277354
10:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db2075.codfw.wmnet with reason: Maintenance T277354
10:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 9 hosts with reason: Maintenance T277354
10:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 9 hosts with reason: Maintenance T277354
10:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance T277354
10:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance T277354
10:03 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1096:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17964 and previous config saved to /var/cache/conftool/dbconfig/20211202-100307-marostegui.json
09:52 moritzm: draining primary/secondary instances off ganeti2009 T296622
09:48 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1096:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17963 and previous config saved to /var/cache/conftool/dbconfig/20211202-094802-marostegui.json
09:32 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1096:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17962 and previous config saved to /var/cache/conftool/dbconfig/20211202-093257-marostegui.json
09:27 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2010.codfw.wmnet to ganeti01.svc.codfw.wmnet
09:27 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2010.codfw.wmnet to ganeti01.svc.codfw.wmnet
09:17 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1096:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17961 and previous config saved to /var/cache/conftool/dbconfig/20211202-091753-marostegui.json
09:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17960 and previous config saved to /var/cache/conftool/dbconfig/20211202-091629-marostegui.json
09:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1096.eqiad.wmnet with reason: Maintenance T277354
09:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db1096.eqiad.wmnet with reason: Maintenance T277354
08:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2010.codfw.wmnet
08:45 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2010.codfw.wmnet
08:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2010.codfw.wmnet with OS buster
08:29 dcausse: restarting blazegraph on wdqs1007 (jvm stuck for 4h)
08:03 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2010.codfw.wmnet with OS buster
02:50 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster
02:43 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster
02:40 andrew@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1028.eqiad.wmnet with OS buster
02:15 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster
02:14 andrew@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1028.eqiad.wmnet with OS buster
01:52 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster
01:21 ryankemper: T280001 Rolling restart of low-traffic pybal hosts complete. All of `wcqs` is pooled and the pybal / ipvs related alerts have cleared
01:16 ryankemper: T280001 Pooled `wcqs200[1-3]` (had been left unpooled from when we last removed wcqs from production)
01:12 ryankemper: T280001 Restarting pybal on low-traffic primaries `lvs2009` and `lvs1015`: `ryankemper@cumin1001:~$ sudo cumin 'P{lvs2009*,lvs1015*}' 'sudo systemctl restart pybal'`
01:08 ryankemper: T280001 Sanity check of `sudo ipvsadm -L -n` on backup `lvs2010` and `lvs1016` looks good (for ex `lvs1016` has `TCP 10.2.2.67:443 wrr`)
12:05 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 2bd14e8: Add templateeditor group and protection level at viwiki (T296154) (duration: 00m 56s)
12:05 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1165 (T277354)', diff saved to https://phabricator.wikimedia.org/P17935 and previous config saved to /var/cache/conftool/dbconfig/20211201-120515-marostegui.json
11:50 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1165 (T277354)', diff saved to https://phabricator.wikimedia.org/P17934 and previous config saved to /var/cache/conftool/dbconfig/20211201-115011-marostegui.json
11:35 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1165 (T277354)', diff saved to https://phabricator.wikimedia.org/P17933 and previous config saved to /var/cache/conftool/dbconfig/20211201-113506-marostegui.json
11:34 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T277354)', diff saved to https://phabricator.wikimedia.org/P17932 and previous config saved to /var/cache/conftool/dbconfig/20211201-113354-marostegui.json
11:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db[1155,1165].eqiad.wmnet with reason: Maintenance T277354
11:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db[1155,1165].eqiad.wmnet with reason: Maintenance T277354
11:31 vgutierrez: test HAProxy 2.4.9 on cp3064 - T290005
11:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1140.eqiad.wmnet with reason: Maintenance T277354
11:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1140.eqiad.wmnet with reason: Maintenance T277354
11:29 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1113:3316 (T277354)', diff saved to https://phabricator.wikimedia.org/P17931 and previous config saved to /var/cache/conftool/dbconfig/20211201-112952-marostegui.json
11:14 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1113:3316 (T277354)', diff saved to https://phabricator.wikimedia.org/P17930 and previous config saved to /var/cache/conftool/dbconfig/20211201-111448-marostegui.json
10:59 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1113:3316 (T277354)', diff saved to https://phabricator.wikimedia.org/P17929 and previous config saved to /var/cache/conftool/dbconfig/20211201-105943-marostegui.json
10:44 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1113:3316 (T277354)', diff saved to https://phabricator.wikimedia.org/P17928 and previous config saved to /var/cache/conftool/dbconfig/20211201-104438-marostegui.json
10:43 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T277354)', diff saved to https://phabricator.wikimedia.org/P17927 and previous config saved to /var/cache/conftool/dbconfig/20211201-104316-marostegui.json
10:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1113.eqiad.wmnet with reason: Maintenance T277354
10:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1113.eqiad.wmnet with reason: Maintenance T277354
10:43 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1098:3316 (T277354)', diff saved to https://phabricator.wikimedia.org/P17926 and previous config saved to /var/cache/conftool/dbconfig/20211201-104308-marostegui.json
09:41 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1096:3316 (T277354)', diff saved to https://phabricator.wikimedia.org/P17920 and previous config saved to /var/cache/conftool/dbconfig/20211201-094120-marostegui.json
09:39 ladsgroup@deploy1002: Synchronized php-1.38.0-wmf.9/extensions/FlaggedRevs/backend/FlaggedRevision.php: Backport: Drop using ft_title and ft_namespace (T296380) (duration: 00m 56s)
09:32 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
09:31 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
09:26 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1096:3316 (T277354)', diff saved to https://phabricator.wikimedia.org/P17919 and previous config saved to /var/cache/conftool/dbconfig/20211201-092615-marostegui.json
09:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubetcd2005.codfw.wmnet with reason: Switch to DRBD for migration
09:16 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kubetcd2005.codfw.wmnet with reason: Switch to DRBD for migration
09:11 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db1096:3316 (T277354)', diff saved to https://phabricator.wikimedia.org/P17918 and previous config saved to /var/cache/conftool/dbconfig/20211201-091110-marostegui.json
09:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T277354)', diff saved to https://phabricator.wikimedia.org/P17917 and previous config saved to /var/cache/conftool/dbconfig/20211201-090948-marostegui.json
09:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1096.eqiad.wmnet with reason: Maintenance T277354
09:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1096.eqiad.wmnet with reason: Maintenance T277354
09:03 vgutierrez: rolling restart of haproxy and varnish on O:cache::text_haproxy and O:cache::upload_haproxy - T290005
08:56 moritzm: draining primary/secondary instance off ganeti2010 T296622
08:51 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:41 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:32 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
06:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2141.codfw.wmnet with reason: Maintenance T277354
06:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db2141.codfw.wmnet with reason: Maintenance T277354
06:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2124.codfw.wmnet with reason: Maintenance T277354
06:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db2124.codfw.wmnet with reason: Maintenance T277354
06:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2117.codfw.wmnet with reason: Maintenance T277354
06:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db2117.codfw.wmnet with reason: Maintenance T277354
00:35 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
00:35 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
22:30 krinkle@deploy1002: Started deploy [integration/docroot@2af7007]: Ia89b6591639e5
22:21 mutante: welcome Majavah to MediaWiki deployers (T296777)
20:09 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 5443b78: uzwiki: Deploy Growth features to newcomers (T294245) (duration: 00m 57s)
18:09 legoktm: uploaded php-yaml for component/php72 (T296331)
18:08 vgutierrez: restart haproxy on cp3064 - T290005
17:44 jynus@cumin1001: dbctl commit (dc=all): 'Repool db1163 fully', diff saved to https://phabricator.wikimedia.org/P17912 and previous config saved to /var/cache/conftool/dbconfig/20211130-174434-jynus.json
17:39 jynus@cumin1001: dbctl commit (dc=all): 'Repool db1163 at 50%', diff saved to https://phabricator.wikimedia.org/P17911 and previous config saved to /var/cache/conftool/dbconfig/20211130-173935-jynus.json
17:35 jynus@cumin1001: dbctl commit (dc=all): 'Repool db1163 at 25%', diff saved to https://phabricator.wikimedia.org/P17910 and previous config saved to /var/cache/conftool/dbconfig/20211130-173517-jynus.json
17:15 jynus@cumin1001: dbctl commit (dc=all): 'Repool db1163 at 5%', diff saved to https://phabricator.wikimedia.org/P17908 and previous config saved to /var/cache/conftool/dbconfig/20211130-171550-jynus.json
17:00 jynus: move db1139:s1 under db1118
16:57 jynus@cumin1001: dbctl commit (dc=all): 'Depool db1163 fully', diff saved to https://phabricator.wikimedia.org/P17907 and previous config saved to /var/cache/conftool/dbconfig/20211130-165718-jynus.json
16:29 XioNoX: Move cr2-codfw lumen transit link to BO cable - T289241
16:26 XioNoX: Move cr2-codfw eqord link to BO cable - T289241
16:23 XioNoX: Move cr2-codfw pfw3 link to BO cable - T289241
16:20 Emperor: reboot ms-be2059 to fix device enumeration order re T295563
16:14 jynus@cumin1001: dbctl commit (dc=all): 'Depool db1163 at 25%', diff saved to https://phabricator.wikimedia.org/P17906 and previous config saved to /var/cache/conftool/dbconfig/20211130-161457-jynus.json
16:07 jynus@cumin1001: dbctl commit (dc=all): 'Depool db1163 at 50%', diff saved to https://phabricator.wikimedia.org/P17905 and previous config saved to /var/cache/conftool/dbconfig/20211130-160748-jynus.json
16:06 bblack: lvs2007 - repooling into service
16:01 bblack: lvs2007 - depooling for network maint - do not push LVS config changes please!
15:41 jbond@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts puppetboard2001.codfw.wmnet
15:41 jbond@cumin1001: START - Cookbook sre.hosts.decommission for hosts puppetboard2001.codfw.wmnet
15:38 jbond@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts puppetboard2001.codfw.wmnet
15:37 jbond@cumin1001: START - Cookbook sre.hosts.decommission for hosts puppetboard2001.codfw.wmnet
15:32 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:29 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:23 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:22 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:16 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:15 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:12 jforrester@deploy1002: Synchronized multiversion/MWMultiVersion.php: Add wikifunctions hard-coded value to setSiteInfoForWiki for Beta Cluster T284162 (duration: 00m 56s)
15:09 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:08 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
13:45 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade.
13:25 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade.
13:11 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17904 and previous config saved to /var/cache/conftool/dbconfig/20211130-131124-marostegui.json
13:05 topranks: Running homer against CR routers to adjust loopback4 filter enabling local NTP queries for status. T296623
12:56 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17903 and previous config saved to /var/cache/conftool/dbconfig/20211130-125620-marostegui.json
12:41 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17902 and previous config saved to /var/cache/conftool/dbconfig/20211130-124115-marostegui.json
12:26 marostegui@cumin1001: dbctl commit (dc=all): 'After maintenance db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17901 and previous config saved to /var/cache/conftool/dbconfig/20211130-122610-marostegui.json
12:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17900 and previous config saved to /var/cache/conftool/dbconfig/20211130-122555-marostegui.json
12:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2114.codfw.wmnet with reason: Maintenance T277354
12:25 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db2114.codfw.wmnet with reason: Maintenance T277354
12:09 jbond@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts puppetboard1001.eqiad.wmnet
12:02 jbond@cumin1001: START - Cookbook sre.hosts.decommission for hosts puppetboard1001.eqiad.wmnet
17:29 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:28 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:25 lucaswerkmeister-wmde@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' .
17:25 lucaswerkmeister-wmde@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' .
17:22 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:22 jforrester@deploy1002: Synchronized wmf-config/ProductionServices.php: Initial Beta Cluster deployment of Wikifunctions: II - Services for T289315 (duration: 00m 55s)
17:21 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:18 jforrester@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Initial Beta Cluster deployment of Wikifunctions: I - IS for T289315 (duration: 00m 55s)
17:00 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
16:59 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
16:56 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: bad34ed: Make foundationwiki a standard CentralAuth wiki (T205347) (duration: 00m 56s)
16:53 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:51 James_F: Running mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=enwiki en wikimedia wikifunctionswiki wikifunctions.beta.wmflabs.org in Beta Cluster for T284162
15:51 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2069.codfw.wmnet with OS buster
15:48 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:47 papaul: power down logstash2028 for IDRAC reset
15:47 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:15 moritzm: gnt-cluster renew-crypto --new-cluster-certificate for codfw Ganeti cluster T296622
14:40 mbsantos@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
14:38 mbsantos@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
14:37 mbsantos@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
08:00 elukey@deploy1002: Started deploy [ores/deploy@69ed061]: Upgrade of mwparserfromhell - T296563
07:31 elukey@deploy1002: Finished deploy [ores/deploy@69ed061]: Canary upgrade of mwparserfromhell - T296563 - (second attempt, no git update submodules the first time) (duration: 00m 04s)
07:31 elukey@deploy1002: Started deploy [ores/deploy@69ed061]: Canary upgrade of mwparserfromhell - T296563 - (second attempt, no git update submodules the first time)
06:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2014.codfw.wmnet with OS bullseye
05:39 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host pc2014.codfw.wmnet with OS bullseye
11:25 vgutierrez: restarting HAProxy on O:cache::(text|upload)_haproxy - T290005
10:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repool after fixing users T296274', diff saved to https://phabricator.wikimedia.org/P17880 and previous config saved to /var/cache/conftool/dbconfig/20211126-102340-ladsgroup.json
10:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1111 (T296274)', diff saved to https://phabricator.wikimedia.org/P17879 and previous config saved to /var/cache/conftool/dbconfig/20211126-101714-ladsgroup.json
10:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1111.eqiad.wmnet with reason: Maintenance T296274
10:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1111.eqiad.wmnet with reason: Maintenance T296274
10:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repool after fixing users T296274', diff saved to https://phabricator.wikimedia.org/P17878 and previous config saved to /var/cache/conftool/dbconfig/20211126-101423-ladsgroup.json
10:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1177 (T296274)', diff saved to https://phabricator.wikimedia.org/P17877 and previous config saved to /var/cache/conftool/dbconfig/20211126-100547-ladsgroup.json
10:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1177.eqiad.wmnet with reason: Maintenance T296274
10:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1177.eqiad.wmnet with reason: Maintenance T296274
10:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance T296143
10:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance T296143
08:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17876 and previous config saved to /var/cache/conftool/dbconfig/20211126-082834-ladsgroup.json
08:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17875 and previous config saved to /var/cache/conftool/dbconfig/20211126-081329-ladsgroup.json
07:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17874 and previous config saved to /var/cache/conftool/dbconfig/20211126-075824-ladsgroup.json
07:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17873 and previous config saved to /var/cache/conftool/dbconfig/20211126-074320-ladsgroup.json
06:28 Amir1: killing extensions/MachineVision/maintenance/fetchSuggestions.php in mwmaint
06:19 Amir1: killing lingering process from mwmaint to depooled db (db1160) that was depooled nine hours ago
2021-11-25
20:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17872 and previous config saved to /var/cache/conftool/dbconfig/20211125-204357-ladsgroup.json
20:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1160.eqiad.wmnet with reason: Maintenance T296143
20:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1160.eqiad.wmnet with reason: Maintenance T296143
19:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance T296143
19:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance T296143
19:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17871 and previous config saved to /var/cache/conftool/dbconfig/20211125-192850-ladsgroup.json
19:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17870 and previous config saved to /var/cache/conftool/dbconfig/20211125-191345-ladsgroup.json
18:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17869 and previous config saved to /var/cache/conftool/dbconfig/20211125-185841-ladsgroup.json
18:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17868 and previous config saved to /var/cache/conftool/dbconfig/20211125-184336-ladsgroup.json
17:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17867 and previous config saved to /var/cache/conftool/dbconfig/20211125-172714-ladsgroup.json
17:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1149.eqiad.wmnet with reason: Maintenance T296143
17:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1149.eqiad.wmnet with reason: Maintenance T296143
17:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1148 (T296143)', diff saved to https://phabricator.wikimedia.org/P17866 and previous config saved to /var/cache/conftool/dbconfig/20211125-172707-ladsgroup.json
12:11 jynus@cumin1001: dbctl commit (dc=all): 'Temp. depool db1163 fully', diff saved to https://phabricator.wikimedia.org/P17847 and previous config saved to /var/cache/conftool/dbconfig/20211125-121138-jynus.json
12:04 jynus@cumin1001: dbctl commit (dc=all): 'Reduce db1163 load even more', diff saved to https://phabricator.wikimedia.org/P17846 and previous config saved to /var/cache/conftool/dbconfig/20211125-120435-jynus.json
11:56 hnowlan@cumin1001: START - Cookbook sre.cassandra.roll-restart for nodes matching restbase202[1-3].codfw.wmnet: Restarting for certificate updates - hnowlan@cumin1001
11:56 jynus@cumin1001: dbctl commit (dc=all): 'Reduce db1163 load', diff saved to https://phabricator.wikimedia.org/P17845 and previous config saved to /var/cache/conftool/dbconfig/20211125-115602-jynus.json
11:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T296143)', diff saved to https://phabricator.wikimedia.org/P17844 and previous config saved to /var/cache/conftool/dbconfig/20211125-110443-ladsgroup.json
11:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1147.eqiad.wmnet with reason: Maintenance T296143
11:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1147.eqiad.wmnet with reason: Maintenance T296143
11:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17843 and previous config saved to /var/cache/conftool/dbconfig/20211125-110435-ladsgroup.json
10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17842 and previous config saved to /var/cache/conftool/dbconfig/20211125-104930-ladsgroup.json
10:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17841 and previous config saved to /var/cache/conftool/dbconfig/20211125-103425-ladsgroup.json
10:25 vgutierrez: rolling restart of varnish and HAProxy on cp2042.codfw.wmnet,cp1090.eqiad.wmnet,cp[5012].eqsin.wmnet,cp3065.esams.wmnet,cp[4026,4032].ulsfo.wmnet' to disable PROXY protocol - T290005
10:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17840 and previous config saved to /var/cache/conftool/dbconfig/20211125-101921-ladsgroup.json
07:17 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 32 hosts with reason: helm3 de-deploy T251305
07:17 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on 32 hosts with reason: helm3 de-deploy T251305
07:10 jelto: downtime PyBal backends health check on lvs1015 and lvs1016 for helm3 de-deploy T251305. I'm keeping an eye on icing and remove downtime as soon as I'm finished
07:09 jelto: start re-deploy procedure in eqiad Kubernetes T251305
06:31 marostegui: Restart tendril's DB
05:51 ryankemper: [WDQS Deploy] Deploy complete. Successful test query placed on query.wikidata.org, there's no relevant criticals in Icinga, and Grafana looks good
23:26 mutante: ganeti - bringing up new VM - sudo gnt-instance start gitlab-runner1001.eqiad.wmnet ; ran puppet on install1003; installing OS T295481
23:22 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2065.codfw.wmnet with OS buster
23:11 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2064.codfw.wmnet with OS buster
23:09 mutante: mwmaint1002 - sudo /usr/bin/find /var/lib/puppet/clientbucket/ -type f -size 1M -delete - to fix Icinga alert about large files in client bucket
23:08 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host gitlab-runner1001.eqiad.wmnet
23:03 mutante: wcqs1001 - sudo systemctl restart wcqs-blazegraph - after <+jinxer-wm> (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wcqs1001:9195 is burning free allocators
22:52 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm for new host gitlab-runner1001.eqiad.wmnet
22:50 mutante: Creating a new Ganeti VM and wondering which row to put it? [ganeti1009:~] $ for row in A B C D; do echo "row ${row}: $(sudo gnt-instance list -o name -F "pnode.group == 'row_${row}'" | wc -l) VMs"; done
22:43 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts gitlab-runner1001.wikimedia.org
22:41 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2064.codfw.wmnet with OS buster
22:39 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2063.codfw.wmnet with OS buster
22:38 mutante: running decom cookbook on gitlab-runner1001.wikimedia.org VM which was in state "ADMIN_down" and not used yet. to make room to recreate it as gitlab-runner1001.eqiad.wmnet T295481
22:36 dzahn@cumin1001: START - Cookbook sre.hosts.decommission for hosts gitlab-runner1001.wikimedia.org
22:08 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2063.codfw.wmnet with OS buster
22:03 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2062.codfw.wmnet with OS buster
21:40 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
21:37 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
21:35 legoktm@deploy1002: Synchronized wmf-config/: Improve docs on $wmgUseGlobalAbuseFilters and sort list of wikis (duration: 00m 57s)
21:33 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2062.codfw.wmnet with OS buster
21:21 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2061.codfw.wmnet with OS buster
21:00 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:58 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:54 legoktm@deploy1002: Synchronized wmf-config/: Update configuration related to disabling Score functionality (duration: 00m 57s)
20:51 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2061.codfw.wmnet with OS buster
19:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1144:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17834 and previous config saved to /var/cache/conftool/dbconfig/20211124-194857-ladsgroup.json
19:38 razzi: `sudo maintain-views --all-databases --replace-all` on clouddb1018 for T292594
19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1144:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17833 and previous config saved to /var/cache/conftool/dbconfig/20211124-193352-ladsgroup.json
19:19 razzi: run `maintain-views --all-databases --replace-all` on clouddb1013 for T292594
19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1144:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17832 and previous config saved to /var/cache/conftool/dbconfig/20211124-191847-ladsgroup.json
19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1144:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17831 and previous config saved to /var/cache/conftool/dbconfig/20211124-190343-ladsgroup.json
18:57 vgutierrez@cumin1001: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ncredir2002.codfw.wmnet
18:51 vgutierrez@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM ncredir2002.codfw.wmnet
18:48 vgutierrez@cumin1001: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ncredir2001.codfw.wmnet
18:43 vgutierrez@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM ncredir2001.codfw.wmnet
18:42 vgutierrez@cumin1001: END (FAIL) - Cookbook sre.ganeti.reboot-vm (exit_code=99) for VM ncredir2001.codfw.wmnet
18:42 vgutierrez@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM ncredir2001.codfw.wmnet
18:42 vgutierrez@cumin1001: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM acmechief-test2001.codfw.wmnet
18:36 vgutierrez@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM acmechief-test2001.codfw.wmnet
18:36 vgutierrez@cumin1001: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM acmechief2001.codfw.wmnet
18:30 vgutierrez@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM acmechief2001.codfw.wmnet
17:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17830 and previous config saved to /var/cache/conftool/dbconfig/20211124-174723-ladsgroup.json
17:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1144.eqiad.wmnet with reason: Maintenance T296143
17:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1144.eqiad.wmnet with reason: Maintenance T296143
17:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'After maintenance db1143 (T296143)', diff saved to https://phabricator.wikimedia.org/P17829 and previous config saved to /var/cache/conftool/dbconfig/20211124-174615-ladsgroup.json
09:22 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM failoid2002.codfw.wmnet
09:20 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM failoid2002.codfw.wmnet
09:19 elukey@cumin1001: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2001.codfw.wmnet
09:16 elukey@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2001.codfw.wmnet
09:16 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on zotero.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:16 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on zotero.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:16 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on wikifeeds.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on wikifeeds.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on termbox.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on termbox.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on tegola-vector-tiles.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on tegola-vector-tiles.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on similar-users.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on similar-users.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on shellbox-timeline.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on shellbox-timeline.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on shellbox-syntaxhighlight.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on shellbox-syntaxhighlight.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on shellbox-media.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on shellbox-media.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on shellbox-constraints.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on shellbox-constraints.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on shellbox.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on shellbox.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on sessionstore.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on sessionstore.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:15 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on recommendation-api.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on recommendation-api.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on push-notifications.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on push-notifications.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on proton.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on proton.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mobileapps.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on mobileapps.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mathoid.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on mathoid.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on linkrecommendation.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on linkrecommendation.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on eventstreams-internal.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on eventstreams-internal.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on eventstreams.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on eventstreams.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on eventgate-main.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on eventgate-main.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on eventgate-logging-external.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on eventgate-logging-external.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:14 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on eventgate-analytics-external.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on eventgate-analytics-external.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on eventgate-analytics.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on eventgate-analytics.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on echostore.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on echostore.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on cxserver.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on cxserver.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on citoid.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on citoid.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on blubberoid.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on blubberoid.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on apple-search.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on apple-search.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on api-gateway.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:13 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on api-gateway.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:11 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on apertium.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:11 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on apertium.svc.codfw.wmnet with reason: helm3 de-deploy T251305
09:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM deneb.codfw.wmnet
09:08 _joe_: switching search.wikimedia.org to be served by the apple-search servcie
09:04 jelto: start re-deploy procedure in codfw Kubernetes T251305
09:01 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM deneb.codfw.wmnet
08:59 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:56 _joe_: repooling cp2027
08:55 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:55 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'apple-search' for release 'main' .
08:51 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'apple-search' for release 'main' .
08:41 vgutierrez: depool cp2027
08:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1125.eqiad.wmnet with OS bullseye
07:40 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1125.eqiad.wmnet with OS bullseye
07:23 elukey: reboot kubernetes1018 (role::insetup) to verify negotiated speed of eth interface
07:12 elukey: drop /tmp/blockmgr-20fe4b2b-31fb-4a85-b5b1-bebe254120f8 and other blockmgr-* dirs on stat1006 to free space on the root partition
06:47 Amir1: running optimize table with replication on db1155:3314 (T296143)
06:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance (T296143)
06:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance (T296143)
06:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1121 (re)pooling @ 100%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17807 and previous config saved to /var/cache/conftool/dbconfig/20211124-063228-root.json
06:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1121 (re)pooling @ 75%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17806 and previous config saved to /var/cache/conftool/dbconfig/20211124-061725-root.json
17:47 sukhe@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM durum2002.codfw.wmnet
17:42 sukhe@cumin1001: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM durum2001.codfw.wmnet
17:35 ebernhardson: T295478 start snapshot of commonswiki_file from cirrus codfw -> swift eqiad
17:34 sukhe@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM durum2001.codfw.wmnet
17:33 sukhe@cumin1001: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM doh2002.wikimedia.org
17:31 cmjohnson1: upgrading msw's in row D eqiad T259758
17:28 sukhe@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM doh2002.wikimedia.org
17:26 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2012.codfw.wmnet with OS stretch
17:16 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on durum2002.codfw.wmnet with reason: apply new KVM machine settings
17:16 sukhe@cumin1001: START - Cookbook sre.hosts.downtime for 0:30:00 on durum2002.codfw.wmnet with reason: apply new KVM machine settings
17:16 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on durum2001.codfw.wmnet with reason: apply new KVM machine settings
17:16 sukhe@cumin1001: START - Cookbook sre.hosts.downtime for 0:30:00 on durum2001.codfw.wmnet with reason: apply new KVM machine settings
17:15 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on doh2002.wikimedia.org with reason: apply new KVM machine settings
17:15 sukhe@cumin1001: START - Cookbook sre.hosts.downtime for 0:30:00 on doh2002.wikimedia.org with reason: apply new KVM machine settings
17:15 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on doh2001.wikimedia.org with reason: apply new KVM machine settings
17:15 sukhe@cumin1001: START - Cookbook sre.hosts.downtime for 0:30:00 on doh2001.wikimedia.org with reason: apply new KVM machine settings
17:15 sukhe@cumin1001: END (ERROR) - Cookbook sre.ganeti.reboot-vm (exit_code=97) for VM doh2001.wikimedia.org
17:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM mwdebug2002.codfw.wmnet
17:14 sukhe@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM doh2001.wikimedia.org
17:14 sukhe@cumin1001: END (ERROR) - Cookbook sre.ganeti.reboot-vm (exit_code=97) for VM doh2001.wikimedia.org
17:11 sukhe@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM doh2001.wikimedia.org
17:10 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM mwdebug2002.codfw.wmnet
17:05 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM mwdebug2001.codfw.wmnet
17:02 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM mwdebug2001.codfw.wmnet
16:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM miscweb2002.codfw.wmnet
16:57 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM miscweb2002.codfw.wmnet
16:55 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM doc2001.codfw.wmnet
16:53 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM doc2001.codfw.wmnet
16:51 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host ms-fe2012.codfw.wmnet with OS stretch
16:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM pybal-test2001.codfw.wmnet
16:47 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM pybal-test2001.codfw.wmnet
16:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM pybal-test2003.codfw.wmnet
16:39 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM pybal-test2003.codfw.wmnet
16:28 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2011.codfw.wmnet with OS stretch
15:54 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host ms-fe2011.codfw.wmnet with OS stretch
15:46 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'apple-search' for release 'main' .
15:46 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2010.codfw.wmnet with OS stretch
15:41 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'apple-search' for release 'main' .
15:31 oblivian@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'apple-search' for release 'main' .
15:27 Emperor: rolling restart of thanos frontends T294380
15:01 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host ms-fe2010.codfw.wmnet with OS stretch
14:40 btullis@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) restart MirrorMaker for Kafka A:kafka-mirror-maker-test-eqiad cluster: Roll restart of jvm daemons.
14:30 btullis@cumin1001: START - Cookbook sre.kafka.roll-restart-mirror-maker restart MirrorMaker for Kafka A:kafka-mirror-maker-test-eqiad cluster: Roll restart of jvm daemons.
14:09 kharlan@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' .
14:09 kharlan@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'internal' .
14:03 kharlan@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'internal' .
14:03 kharlan@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' .
14:00 marostegui: Failover m5 from db1128 to db1132 - T288720
14:00 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host prometheus2006.codfw.wmnet with OS bullseye
13:50 godog: powercycle (again) ms-be2058
13:48 godog: add 80G to prometheus global in eqiad
13:31 filippo@cumin1001: START - Cookbook sre.hosts.reimage for host prometheus2006.codfw.wmnet with OS bullseye
13:29 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host prometheus2005.codfw.wmnet with OS bullseye
13:01 filippo@cumin1001: START - Cookbook sre.hosts.reimage for host prometheus2005.codfw.wmnet with OS bullseye
12:58 btullis@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching O:aqs: restarting to pick up new JRE - btullis@cumin1001
12:52 aborrero@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cloudbackup1002-dev.eqiad.wmnet
12:46 Lucas_WMDE: UTC morning backport+config window done
12:43 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:43 aborrero@cumin1001: START - Cookbook sre.ganeti.makevm for new host cloudbackup1002-dev.eqiad.wmnet
12:39 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
10:38 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
10:31 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
10:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance T296143
10:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance T296143
10:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1155.eqiad.wmnet with reason: Maintenance T296143
10:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1155.eqiad.wmnet with reason: Maintenance T296143
10:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on clouddb[1015,1019,1021].eqiad.wmnet with reason: Maintenance T296143
10:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on clouddb[1015,1019,1021].eqiad.wmnet with reason: Maintenance T296143
10:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T296143)', diff saved to https://phabricator.wikimedia.org/P17800 and previous config saved to /var/cache/conftool/dbconfig/20211123-102234-ladsgroup.json
10:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1121.eqiad.wmnet with reason: Maintenance T296143
10:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on db1121.eqiad.wmnet with reason: Maintenance T296143
09:52 kharlan@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' .
09:37 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1124.eqiad.wmnet with OS bullseye
09:27 Amir1: dropping useless GRANTs on s6 eqiad replicas without replication (T296274)
09:16 Amir1: dropping useless GRANTs on s6 eqiad master without replication (T296274)
09:09 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1124.eqiad.wmnet with OS bullseye
09:05 Amir1: fixing incorrect grants of wikiadmin on localhost in s6 master in codfw with replication
07:52 topranks: Adjusting BGP on cr1-eqiad and cr2-eqiad to keep MED unchanged in iBGP.
07:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1125.eqiad.wmnet with OS bullseye
06:41 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1125.eqiad.wmnet with OS bullseye
05:29 ryankemper: T295705 Downtimed `elastic2044` for one hour and doing a full reboot for good measure. Already ran the plugin upgrade: `DEBIAN_FRONTEND=noninteractive sudo apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install elasticsearch-oss wmf-elasticsearch-search-plugins`
05:26 ryankemper: T295705 Rolling restart of `codfw` complete. `elastic2044` was manually restarted earlier today so the cookbook didn't restart it (b/c we pass in a datetime cutoff threshold) so I'm manually upgrading and restarting that host
05:10 ryankemper@cumin1001: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) restart with plugin upgrade (2 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade + restart - ryankemper@cumin1001 - T295705
04:17 ryankemper: T295705 Properly disabled the sane-itizer; we don't want it running until after we (a) complete rolling restarts and (b) restore the missing `commonswikI_file` index (which is blocked on the restarts)
03:37 Amir1: rebuilding metadata of all djvu files outside of commons (T296001)
03:06 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation restart with plugin upgrade (2 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade + restart - ryankemper@cumin1001 - T295705
02:58 ryankemper: T295705 `elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPSConnectionPool(host='search.svc.codfw.wmnet', port=9243): Read timed out. (read timeout=60))` Probably transient failure; will wait 10 mins and try again
02:57 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart with plugin upgrade (2 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade + restart - ryankemper@cumin1001 - T295705
02:55 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation restart with plugin upgrade (2 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade + restart - ryankemper@cumin1001 - T295705
02:41 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
02:37 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
02:12 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
02:08 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
01:17 urbanecm: UTC late window done
01:17 urbanecm@deploy1002: Finished scap: 69aa4a7: 7c0e074: Revert "Create redirect Special Pages for delete and protect action" (T295611; T296203; 4/4) (duration: 25m 50s)
01:12 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
01:05 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
00:51 urbanecm@deploy1002: Started scap: 69aa4a7: 7c0e074: Revert "Create redirect Special Pages for delete and protect action" (T295611; T296203; 4/4)
00:50 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.9/autoload.php: 7c0e074: Revert "Create redirect Special Pages for delete and protect action" (T295611; T296203; 3/4) (duration: 00m 55s)
00:49 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.9/includes/specials/: 7c0e074: Revert "Create redirect Special Pages for delete and protect action" (T295611; T296203; 2/4) (duration: 00m 55s)
00:48 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.9/includes/specialpage/SpecialPageFactory.php: 7c0e074: Revert "Create redirect Special Pages for delete and protect action" (T295611; T296203; 1/4) (duration: 00m 56s)
23:48 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-fe2011.codfw.wmnet with OS stretch
23:22 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-fe2010.codfw.wmnet with OS stretch
23:20 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host ms-fe2011.codfw.wmnet with OS stretch
23:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2009.codfw.wmnet with OS stretch
22:53 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host ms-fe2010.codfw.wmnet with OS stretch
22:44 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host ms-fe2009.codfw.wmnet with OS stretch
22:20 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2028.codfw.wmnet with OS buster
21:51 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2028.codfw.wmnet with OS buster
21:42 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2027.codfw.wmnet with OS buster
21:07 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2027.codfw.wmnet with OS buster
20:43 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:39 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:28 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:24 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:23 legoktm@deploy1002: Synchronized wmf-config/PoolCounterSettings.php: Lower CirrusSearch maxqueues to be closer to number of workers (duration: 00m 56s)
20:01 ryankemper@cumin1001: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) restart with plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade + restart - ryankemper@cumin1001 - T295705
19:49 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation restart with plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade + restart - ryankemper@cumin1001 - T295705
19:49 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:46 urbanecm: Evening B&C window completed
19:45 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:44 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.9/extensions/ProofreadPage/: 10b8440: Use the WikiEditor ready hook instead of using() the lib (T296033) (duration: 00m 56s)
19:34 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:30 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:24 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: b6b05e3: kswiki: set wgTranslateNumerals to false (T296055) (duration: 00m 55s)
19:20 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:16 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 1c082be: Enable mapframe on the Indonesian Wikipedia (T295571) (duration: 00m 56s)
19:15 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:11 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:32 ebernhardson: restart both elasticsearch instances on elastic2044, reporting `connection refused` (after a brief period of `no route to host`) to masters even though the connection works outside elastic
16:51 jayme: fleet wide updated wmf-certificates to 0~20211122-1
16:50 vgutierrez: depol cp4032 to be reimaged as cache::text_haproxy - T290005
16:49 ryankemper: [Elastic] T295705 Downtimed relforge* for 2 hours in order to performing a manual rolling restart of the two hosts `relforge1003` and `relforge1004`
12:39 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:35 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:24 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:20 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:19 jforrester@deploy1002: Synchronized wmf-config/CommonSettings.php: ExtensionDistributor: 1.37.0 is out now, so there's no beta T289585 (duration: 01m 04s)
12:11 hashar@deploy1002: Synchronized php-1.38.0-wmf.9/skins/MinervaNeue: Fix banners to show CentralNotice - T296077 (duration: 01m 04s)
11:50 moritzm: installing qemu security updates on bullseye
11:46 oblivian@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:43 moritzm: installing krb5 security updates on stretch
11:34 hnowlan@cumin1001: START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Restarting to pick up Java security updates - hnowlan@cumin1001
11:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti-test2003.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
11:26 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2003.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
11:24 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
11:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet
11:21 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
11:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet
11:20 XioNoX: re-enable LibertyGlobal in esams
11:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1125.eqiad.wmnet with OS bullseye
11:12 XioNoX: Revert "prepend_as_out for esams/knams"
11:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti-test2003.codfw.wmnet with OS buster
10:54 elukey: apt-get purge up to linux-image-4.9.0-14-amd64 on sodium to free /boot space
10:49 elukey: `apt-get remove linux-image-4.9.0-5-amd64 linux-image-4.9.0-6-amd64` on sodium to free /boot
10:45 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1125.eqiad.wmnet with OS bullseye
10:41 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti-test2003.codfw.wmnet with OS buster
10:25 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1125.eqiad.wmnet with OS bullseye
10:16 jbond: restart snmp gracefully cr2-eqord
10:12 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1125.eqiad.wmnet with OS bullseye
09:45 oblivian@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'apple-search' for release 'main' .
09:40 oblivian@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'apple-search' for release 'main' .
09:38 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1125.eqiad.wmnet with OS bullseye
09:35 moritzm: installing Linux 4.9.272 updates on Stretch hosts
09:29 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
09:25 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
09:24 oblivian@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'apple-search' for release 'main' .
09:15 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 24b3a77: Growth: Disable filtering by unstarred mentees at arwiki, enwiki, fawiki (T293182) (duration: 01m 04s)
09:14 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1125.eqiad.wmnet with OS bullseye
09:12 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
09:08 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1125.eqiad.wmnet with OS bullseye
09:05 moritzm: installing 4.19.208-1 kernels on Stretch hosts with 4.19 kernels
09:05 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:49 moritzm: drain ganeti-test2003 for forthcoming reimage
08:44 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1125.eqiad.wmnet with OS bullseye
08:44 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/: 4418c43: ApiSetMentorStatus: Use READ_LATEST to request back timestamp (T295305) (duration: 01m 08s)
08:42 oblivian@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'apple-search' for release 'main' .
08:31 oblivian@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'apple-search' for release 'main' .
08:25 marostegui@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 100%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17793 and previous config saved to /var/cache/conftool/dbconfig/20211122-082525-root.json
08:15 oblivian@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'apple-search' for release 'main' .
08:10 marostegui@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 75%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17792 and previous config saved to /var/cache/conftool/dbconfig/20211122-081022-root.json
07:55 marostegui@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 50%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17791 and previous config saved to /var/cache/conftool/dbconfig/20211122-075518-root.json
07:40 marostegui@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 40%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17790 and previous config saved to /var/cache/conftool/dbconfig/20211122-074015-root.json
07:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance T296143
07:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance T296143
07:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2090.codfw.wmnet with reason: Maintenance T296143
07:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2090.codfw.wmnet with reason: Maintenance T296143
07:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2073.codfw.wmnet with reason: Maintenance T296143
07:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2073.codfw.wmnet with reason: Maintenance T296143
07:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance T296143
07:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance T296143
07:25 marostegui@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 25%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17789 and previous config saved to /var/cache/conftool/dbconfig/20211122-072511-root.json
07:17 Amir1: running optimize table on image table in commonswiki on codfw with replication enabled, it'll cause replication lag (T296143)
07:10 marostegui@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 20%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17788 and previous config saved to /var/cache/conftool/dbconfig/20211122-071006-root.json
06:55 marostegui@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 10%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17787 and previous config saved to /var/cache/conftool/dbconfig/20211122-065502-root.json
06:46 marostegui: Revoke dump grants for scholarships database T296166
06:39 marostegui@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 5%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17786 and previous config saved to /var/cache/conftool/dbconfig/20211122-063959-root.json
06:24 marostegui@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 1%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17785 and previous config saved to /var/cache/conftool/dbconfig/20211122-062455-root.json
03:30 Amir1: run optimize table on db2140 for image table (T296143)
2021-11-21
13:17 dcausse: restarting blazegraph on wdqs1007 (jvm stuck for 10h)
07:26 XioNoX: cr1-eqiad# deactivate protocols bgp group Confed_eqord
05:22 Amir1: running clean up of djvu files in all wikis (T275268)
05:13 Amir1: end of djvu metadata maint script run (T275268)
2021-11-20
01:02 mutante: lists1001 - restarted apache, icinga alerts for the web UI, but recovered
00:27 cdanis@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
17:21 andrew@deploy1002: Started deploy [horizon/deploy@ee83e27]: fixing sudo rule editing
17:19 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:10 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
16:54 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
16:50 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
16:42 thcipriani@deploy1002: rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.38.0-wmf.9 refs T293950T296098"
16:35 thcipriani: rolling back to group0 for T296098
16:20 hnowlan@cumin1001: START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Restarting to pick up Java security updates - hnowlan@cumin1001
15:31 akosiaris: roll restart wtp10* php7.2-fpm excluding wtp1025, wtp1041
15:29 akosiaris: depooling wtp1041, wtp1025 from traffic. The entire of the parsoid cluster is in a memory pressure situation, it looks like a rolling restart of php-fpm will alleviate the pressure and gives us some time to drill more on the problem before the pressure builds up again.
14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
14:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
14:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
14:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
14:30 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti-test2001.codfw.wmnet with OS buster
14:15 jayme: fleet wide updated wmf-certificates to 0~20211119-1
13:56 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti-test2001.codfw.wmnet with OS buster
13:23 moritzm: draining instances from ganeti-test2001 for reimage T284811
13:02 jgiannelos@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
12:10 jgiannelos@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
12:06 jgiannelos@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
11:54 hnowlan: roll-restarting cassandra on eqiad maps for java updates
11:36 jayme: imported wmf-certificates 0~20211119-1 to stretch-wikimedia,buster-wikimedia,bullseye-wikimedia
09:53 XioNoX: run `commit full` on asw-b-codfw - T295118
09:30 XioNoX: re-enable cr2-codfw<->asw-b7-codfw link after disabling inet6 on cr2-codfw:ae2 - T295118
09:06 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade.
08:46 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade.
08:31 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
08:30 ayounsi@cumin1001: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: update wmf-netbox - ayounsi@cumin1001
08:29 ayounsi@cumin1001: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: update wmf-netbox - ayounsi@cumin1001
08:27 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
23:28 dzahn@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'miscweb' for release 'main' .
23:27 dzahn@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'miscweb' for release 'main' .
22:52 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart with plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade + restart - ryankemper@cumin1001 - T295705
22:48 XioNoX: asw-b-codfw> request system power-off member 7
22:44 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation restart with plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade + restart - ryankemper@cumin1001 - T295705
22:28 mutante: icinga (alert1001) - manually fix IP of mw1488.mgmt (was 0.0.0.0 is: 10.65.1.26) in /etc/icinga/objects/puppet_hosts.cfg , running puppet
22:06 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts thumbor1003.eqiad.wmnet
21:53 legoktm@cumin1001: START - Cookbook sre.hosts.decommission for hosts thumbor1003.eqiad.wmnet
21:50 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts thumbor1004.eqiad.wmnet
21:36 legoktm@cumin1001: START - Cookbook sre.hosts.decommission for hosts thumbor1004.eqiad.wmnet
21:31 XioNoX: asw-b-codfw> request system power-off member 7
15:49 XioNoX: asw-b-codfw> request system power-off member 7 - T295118
15:39 XioNoX: lvs2007:~$ sudo service pybal stop - T295118
15:36 dzahn@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'miscweb' for release 'main' .
15:35 XioNoX: cr2-codfw# set interfaces et-1/0/3 disable - T295118
15:34 dzahn@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' .
15:33 btullis@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-jumbo-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade.
15:16 hnowlan: roll restarting cassandra on codfw maps for java updates
15:14 hnowlan@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:sessionstore: Restarting to pick up Java security updates - hnowlan@cumin1001
14:44 dzahn@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' .
14:38 hnowlan@cumin1001: START - Cookbook sre.cassandra.roll-restart for nodes matching A:sessionstore: Restarting to pick up Java security updates - hnowlan@cumin1001
14:37 hnowlan: roll-restarting sessionstore for java updates
14:19 moritzm: installing testvm2003
13:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2003.codfw.wmnet
13:34 moritzm: installing pam bugfix updates on bullseye hosts
13:26 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2003.codfw.wmnet
13:22 moritzm: failover ganeti master in test cluster to ganeti-test2002 T284811
12:23 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: cloudcephosd1016.wikimedia.org
12:23 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: cloudcephosd1016.wikimedia.org
12:21 btullis@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-jumbo-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade.
12:21 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
09:41 ema: cp4021: stop ats-be and clear its cache T288106
09:35 ema: cp4021: depool to enable single backend experiment T288106
09:32 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1090.eqiad.wmnet with OS buster
09:32 vgutierrez: pool cp1090 (upload) running HAProxy as TLS terminator - T290005
09:18 jayme: systemctl start prune-production-images.service on deneb - T287222
08:48 vgutierrez@cumin1001: START - Cookbook sre.hosts.reimage for host cp1090.eqiad.wmnet with OS buster
08:46 vgutierrez: depool cp1090 to be reimaged as cache::upload_haproxy - T290005
08:45 moritzm: installing mariadb-10.3 security updates on buster (as packaged in Debian, not the wmf-internal packages)
08:27 topranks: De-pool of Eqiad seems to be ok, transit/peering/transport links changed BW profile but nothing maxed, total LVS connections steady but have shifted to codfw. Proceeding to reconfigure iBGP policy on cr1-eqiad and cr2-eqiad maually.
08:01 topranks: Depooling eqiad in authdns to allow for reconfiguration of CR routers on site (T295672)
07:45 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
07:41 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
07:35 marostegui@cumin1001: dbctl commit (dc=all): 'db1112 (re)pooling @ 100%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17772 and previous config saved to /var/cache/conftool/dbconfig/20211118-073507-root.json
07:20 marostegui@cumin1001: dbctl commit (dc=all): 'db1112 (re)pooling @ 75%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17771 and previous config saved to /var/cache/conftool/dbconfig/20211118-072004-root.json
07:06 marostegui@cumin1001: dbctl commit (dc=all): 'Remove watchlist from s5 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17770 and previous config saved to /var/cache/conftool/dbconfig/20211118-070620-marostegui.json
07:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1156 (re)pooling @ 100%: After fixing GRANTs', diff saved to https://phabricator.wikimedia.org/P17769 and previous config saved to /var/cache/conftool/dbconfig/20211118-070559-root.json
07:05 marostegui@cumin1001: dbctl commit (dc=all): 'db1112 (re)pooling @ 50%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17768 and previous config saved to /var/cache/conftool/dbconfig/20211118-070500-root.json
06:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1156 (re)pooling @ 75%: After fixing GRANTs', diff saved to https://phabricator.wikimedia.org/P17767 and previous config saved to /var/cache/conftool/dbconfig/20211118-065055-root.json
06:49 marostegui@cumin1001: dbctl commit (dc=all): 'db1112 (re)pooling @ 40%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17766 and previous config saved to /var/cache/conftool/dbconfig/20211118-064957-root.json
06:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1156 (re)pooling @ 25%: After fixing GRANTs', diff saved to https://phabricator.wikimedia.org/P17765 and previous config saved to /var/cache/conftool/dbconfig/20211118-063552-root.json
06:34 marostegui@cumin1001: dbctl commit (dc=all): 'db1112 (re)pooling @ 25%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17764 and previous config saved to /var/cache/conftool/dbconfig/20211118-063453-root.json
06:31 Amir1: revoked all grants from wikiadmin and gave back an explicit list on db1102:3312 (T249683)
06:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1156 (re)pooling @ 10%: After fixing GRANTs', diff saved to https://phabricator.wikimedia.org/P17763 and previous config saved to /var/cache/conftool/dbconfig/20211118-062048-root.json
06:19 marostegui@cumin1001: dbctl commit (dc=all): 'db1112 (re)pooling @ 20%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17762 and previous config saved to /var/cache/conftool/dbconfig/20211118-061949-root.json
06:17 Amir1: revoked all grants from wikiadmin and gave back an explicit list on db1156 (T249683)
06:04 marostegui@cumin1001: dbctl commit (dc=all): 'db1112 (re)pooling @ 10%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17761 and previous config saved to /var/cache/conftool/dbconfig/20211118-060446-root.json
05:49 marostegui@cumin1001: dbctl commit (dc=all): 'db1112 (re)pooling @ 5%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17760 and previous config saved to /var/cache/conftool/dbconfig/20211118-054942-root.json
05:47 marostegui: Upgrade clouddb1014
05:34 marostegui@cumin1001: dbctl commit (dc=all): 'db1112 (re)pooling @ 1%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17759 and previous config saved to /var/cache/conftool/dbconfig/20211118-053438-root.json
05:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 due to network issues (T295952)', diff saved to https://phabricator.wikimedia.org/P17758 and previous config saved to /var/cache/conftool/dbconfig/20211118-050802-ladsgroup.json
04:23 dzahn@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' .
19:24 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:20 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:10 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
19:10 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 8e167a5: Disable local file upload on the Chinese Wikisource (T295265) (duration: 01m 05s)
19:09 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:05 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 7b3a1d9: Make reply tool available as opt-out on commonswiki (T295838) (duration: 01m 05s)
19:05 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:58 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2042.codfw.wmnet with OS buster
18:57 ejegg: updated fundraising CiviCRM from 9c5f0b69 -> 8054869b
18:56 vgutierrez: pool cp2042 (upload) running HAProxy as TLS terminator - T290005
18:06 vgutierrez@cumin1001: START - Cookbook sre.hosts.reimage for host cp2042.codfw.wmnet with OS buster
18:05 bblack@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:59 vgutierrez: depool cp2042 to be reimaged as an HAProxy cache upload node - T290005
17:41 btullis@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade.
17:25 cmooney@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host rpki2002.codfw.wmnet
17:11 XioNoX: repool Telia eqiad-codfw transport
17:10 cmooney@cumin2002: START - Cookbook sre.ganeti.makevm for new host rpki2002.codfw.wmnet
16:34 cmooney@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rpki2001.codfw.wmnet
16:32 mutante: LDAP - added jkieserman to wmf (T295693)
16:28 cmooney@cumin2002: START - Cookbook sre.hosts.decommission for hosts rpki2001.codfw.wmnet
16:28 XioNoX: drain Telia eqiad-codfw link
16:27 cmooney@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts rpki2001.codfw.wmnet
16:21 XioNoX: move cr1-codfw<->cr2-eqdfw link to BO cable
16:19 cmooney@cumin2002: START - Cookbook sre.hosts.decommission for hosts rpki2001.codfw.wmnet
16:06 XioNoX: move cr1-codfw:xe-5/3/0 to BO cable
16:04 XioNoX: re-enable Telia BGP on cr1-codfw
16:01 btullis@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade.
15:59 bblack: netbox: added ganeti01 and ganeti02 cluster definitions for drmrs
15:58 XioNoX: disable Telia BGP on cr1-codfw
15:55 XioNoX: move codfw-ulsfo link to break-out cable
15:46 mutante: restarting pybal on lvs1015
15:43 _joe_: restarting pybal on lvs2009
15:42 mutante: restarting pybal on lvs1016
15:39 _joe_: restarting pybal on lvs2010
15:35 XioNoX: drain ulsfo-codfw link
14:47 moritzm: installing perl bugfix updates from Bullseye point release
14:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on ganeti-test[2001-2003].codfw.wmnet with reason: Ganeti update tests
14:22 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on ganeti-test[2001-2003].codfw.wmnet with reason: Ganeti update tests
13:49 marostegui@cumin1001: dbctl commit (dc=all): 'Change weights on s5 special slaves in eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17755 and previous config saved to /var/cache/conftool/dbconfig/20211117-134942-marostegui.json
13:48 marostegui@cumin1001: dbctl commit (dc=all): 'Remove recentchanges from s5 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17754 and previous config saved to /var/cache/conftool/dbconfig/20211117-134835-marostegui.json
13:20 aborrero@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cloudbackup1001-dev.eqiad.wmnet
13:10 aborrero@cumin1001: START - Cookbook sre.ganeti.makevm for new host cloudbackup1001-dev.eqiad.wmnet
13:02 Lucas_WMDE: UTC morning backport+config window done
12:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet
12:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet
12:26 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:22 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:22 btullis@cumin1001: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto analytics cluster: Roll restart of all Presto's jvm daemons.
12:17 topranks: Re-pooling ulsfo after completing routing changes on cr3-ulsfo and cr4-ulsfo (T295672)
12:12 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:11 btullis@cumin1001: START - Cookbook sre.presto.roll-restart-workers for Presto analytics cluster: Roll restart of all Presto's jvm daemons.
12:11 moritzm: failover ganeti master in test cluster to ganeti-test2003
07:46 marostegui@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 20%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17747 and previous config saved to /var/cache/conftool/dbconfig/20211117-074606-root.json
07:44 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp6011.drmrs.wmnet with OS buster
07:34 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6009.drmrs.wmnet with OS buster
07:31 marostegui@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 10%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17746 and previous config saved to /var/cache/conftool/dbconfig/20211117-073102-root.json
07:31 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp6010.drmrs.wmnet with OS buster
07:29 elukey: `apt-get clean` on an-tool1005 to free space in the root partition
07:28 elukey: `sudo pkill -U jmixter` on stat100[5,8] to allow puppet to run and remove the offboarded user
07:22 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6008.drmrs.wmnet with OS buster
07:20 Amir1: start of clean up of autreview logs of ruwiki, deleting 3.5M rows (T285608)
07:15 marostegui@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 5%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17745 and previous config saved to /var/cache/conftool/dbconfig/20211117-071559-root.json
07:00 marostegui@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 1%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17744 and previous config saved to /var/cache/conftool/dbconfig/20211117-070055-root.json
06:58 marostegui: Upgrade db1180 to 10.4.22
06:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1180 for upgrade', diff saved to https://phabricator.wikimedia.org/P17743 and previous config saved to /var/cache/conftool/dbconfig/20211117-065740-marostegui.json
06:52 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp6009.drmrs.wmnet with OS buster
06:43 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp6008.drmrs.wmnet with OS buster
06:38 Amir1: start of deleting auto-review logs in arwiki (T285608) deleting 23M rows
06:33 marostegui: Upgrade clouddb1018
06:04 marostegui@cumin1001: dbctl commit (dc=all): 'Remove logpager from s5 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17742 and previous config saved to /var/cache/conftool/dbconfig/20211117-060426-marostegui.json
00:19 ryankemper: T276198 `ryankemper@cumin1001:~$ sudo cumin -b 3 '*elastic*' 'sudo run-puppet-agent --force'` Change looks good (no complaints from systemd), rolling out to rest of fleet / reenabling puppet
00:15 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor1006.eqiad.wmnet
00:06 legoktm@cumin1001: START - Cookbook sre.hosts.reboot-single for host thumbor1006.eqiad.wmnet
23:27 ryankemper: T276198 `ryankemper@elastic1049:~$ sudo run-puppet-agent --force`; `elasticsearch_6@production-search-eqiad.service ` didn't restart but it looks like there might be slightly wrong with the new `ExecPreStart` line => `Executable path is not absolute, ignoring: systemd-tmpfiles --create /usr/lib/tmpfiles.d/elasticsearch.conf`
14:44 cmooney@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host rpki2001.codfw.wmnet
14:31 cmooney@cumin2002: START - Cookbook sre.ganeti.makevm for new host rpki2001.codfw.wmnet
14:31 cmooney@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rpki2002.codfw.wmnet
14:24 jynus: re-adding backup user to db1108:analytics_meta T284150
14:22 cmooney@cumin2002: START - Cookbook sre.ganeti.makevm for new host rpki2002.codfw.wmnet
14:18 cmooney@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rpki2001.codfw.wmnet
14:09 cmooney@cumin2002: START - Cookbook sre.hosts.decommission for hosts rpki2001.codfw.wmnet
13:58 cmooney@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rpki2001.codfw.wmnet
13:51 cmooney@cumin2002: START - Cookbook sre.ganeti.makevm for new host rpki2001.codfw.wmnet
13:23 moritzm: installing debconf bugfix updates on buster
13:21 moritzm: prune unused packages from ping3001 T295767
13:18 moritzm: prune unused packages from ping1001/ping2001 T295767
13:05 moritzm: installing psmisc bugfix updates on buster hosts
13:04 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6007.drmrs.wmnet with OS buster
12:45 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:41 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:29 moritzm: installing Linux 4.19.208 updates on buster hosts (no reboots)
12:24 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp6007.drmrs.wmnet with OS buster
12:22 btullis@cumin1001: END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid public cluster: Roll restart of Druid jvm daemons.
12:13 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6006.drmrs.wmnet with OS buster
11:55 moritzm: failover ganeti master in test cluster to ganeti-test2002
11:34 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp6006.drmrs.wmnet with OS buster
11:31 btullis@cumin1001: START - Cookbook sre.druid.roll-restart-workers for Druid public cluster: Roll restart of Druid jvm daemons.
11:03 btullis@cumin1001: END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) restart masters for Hadoop analytics cluster: Restart of jvm daemons.
10:30 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6005.drmrs.wmnet with OS buster
10:21 ema: A:cp re-enable puppet after successful test on cp402[17] T293879
10:20 btullis@cumin1001: START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop analytics cluster: Restart of jvm daemons.
10:15 moritzm: installing testvm2001
10:06 arturo: updating deb packages on stretch-wikimedia/thirdparty/kubeadm-k8s-1-21 (T282942)
00:21 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
00:19 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 50d9f26: GrowthExperiments: Set up GEHomepageNewAccountVariantsByPlatform (T294737) (duration: 00m 56s)
00:11 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
00:07 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
2021-11-15
23:10 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor1005.eqiad.wmnet
22:59 legoktm@cumin1001: START - Cookbook sre.hosts.reboot-single for host thumbor1005.eqiad.wmnet
22:58 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on thumbor1005.eqiad.wmnet with reason: reboot after first puppet run
22:58 legoktm@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on thumbor1005.eqiad.wmnet with reason: reboot after first puppet run
21:46 bblack: dns6002 - reboot for another round of bios fixups
21:32 bblack: dns6001 - reboot for another round of bios fixups
21:21 legoktm: uploaded php7.4_7.4.25-1+wmf2+buster1_amd64.changes to apt.wm.o with patch for T293568
21:19 mutante: removing mediawiki font packages from remaining regular appservers globally (T294378)
20:49 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6001.drmrs.wmnet with OS buster
20:09 Amir1: revoked all grants from wikiadmin and gave back an explicit list on clouddb1013:3311 (T249683)
20:08 Amir1: revoked all grants from wikiadmin and gave back an explicit list on clouddb1021:3311 (T249683)
20:07 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp6001.drmrs.wmnet with OS buster
20:03 Amir1: revoked all grants from wikiadmin and gave back an explicit list on db1102:3312 (T249683)
19:57 mmandere@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp6001.drmrs.wmnet with OS buster
19:49 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:46 Amir1: revoked all grants from wikiadmins and gave back explicit list on db2101:3315 (T249683)
19:46 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:45 urbanecm: UTC evening B&C window done
19:37 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 898ebb1: Enable talk for mobile users on enwiki (T293946) (duration: 00m 57s)
19:35 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:31 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:31 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp6001.drmrs.wmnet with OS buster
19:30 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: cdac608: Change votewiki language back to English (T292685) (duration: 00m 56s)
19:06 mutante: removing font packages from MW API appservers T294378
18:58 mmandere@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp6001.drmrs.wmnet with OS buster
18:52 volans@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 40 days, 0:00:00 on ps1-d1-codfw with reason: Testing new PDU devices T265435
18:52 volans@cumin2002: START - Cookbook sre.hosts.downtime for 40 days, 0:00:00 on ps1-d1-codfw with reason: Testing new PDU devices T265435
18:41 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:37 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:32 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp6001.drmrs.wmnet with OS buster
18:31 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 1be2d39: Growth IP research survey: Fix platforms (T294568) (duration: 00m 55s)
18:27 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:23 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:21 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: d15948e: foundationwiki: Restrict editing in more namespaces (T294900) (duration: 00m 56s)
18:19 aokoth@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gitlab2001.wikimedia.org with reason: upgrade gitlab2001 to new version https://phabricator.wikimedia.org/T294580
18:19 aokoth@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on gitlab2001.wikimedia.org with reason: upgrade gitlab2001 to new version https://phabricator.wikimedia.org/T294580
18:08 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:07 arnoldokoth: upgrading gitlab version on gitlab2001 (T294580)
18:04 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
15:15 elukey: `reprepro --delete clearvanished` on apt1001 to clean-up thirdparty/amd-rocm38 (buster and stretch) - T295661
14:55 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
14:51 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
14:49 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 4f17e85: GrowthExperiments: Disable link recommendation frontend on dewiki (duration: 00m 56s)
14:45 btullis@cumin1001: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade.
14:21 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
14:17 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
13:36 jgiannelos@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
13:34 jgiannelos@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
13:25 topranks: Adding new policy-statement to CR routers via homer to set next-hop self on iBGP sessions (not yet configured for any peers).
12:46 btullis@cumin1001: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade.
12:45 btullis@cumin1001: END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid analytics cluster: Roll restart of Druid jvm daemons.
12:12 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:08 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
02:52 ryankemper: [Elastic] Downtimed relforge* for 2 hours in order to performing a rolling restart of the two hosts `relforge1003` and `relforge1004`
2021-11-12
21:00 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
08:48 marostegui@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 100%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17736 and previous config saved to /var/cache/conftool/dbconfig/20211112-084813-root.json
08:33 marostegui@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 75%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17735 and previous config saved to /var/cache/conftool/dbconfig/20211112-083310-root.json
08:27 moritzm: imported openjdk-8 8u312-b07-1~deb11u1 to component/jdk8 for bullseye-wikimedia (rebuild of latest Java 8 security release for Bullseye)
08:18 marostegui@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 50%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17734 and previous config saved to /var/cache/conftool/dbconfig/20211112-081806-root.json
08:03 marostegui@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 40%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17733 and previous config saved to /var/cache/conftool/dbconfig/20211112-080302-root.json
07:47 marostegui@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 25%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17732 and previous config saved to /var/cache/conftool/dbconfig/20211112-074759-root.json
07:32 marostegui@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 20%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17731 and previous config saved to /var/cache/conftool/dbconfig/20211112-073255-root.json
07:17 marostegui@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 10%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17730 and previous config saved to /var/cache/conftool/dbconfig/20211112-071752-root.json
07:02 marostegui@cumin1001: dbctl commit (dc=all): 'Add weight for db1104', diff saved to https://phabricator.wikimedia.org/P17729 and previous config saved to /var/cache/conftool/dbconfig/20211112-070236-marostegui.json
07:01 marostegui@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 5%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17728 and previous config saved to /var/cache/conftool/dbconfig/20211112-070141-root.json
00:19 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
00:15 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
14:52 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host db1139.eqiad.wmnet with OS buster
14:50 btullis@cumin1001: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. - btullis@cumin1001
14:46 moritzm: installing sqlalchemy security updates on stretch
14:42 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:38 btullis@cumin1001: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. - btullis@cumin1001
14:33 btullis@cumin1001: END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) restart masters for Hadoop test cluster: Restart of jvm daemons. - btullis@cumin1001
14:32 jynus@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1139.eqiad.wmnet with OS buster
12:50 lucaswerkmeister-wmde@deploy1002: Synchronized multiversion/buildConfigCache.php: Config: Don't need to keep all config in memory (resync, previous deploy for this file was missing `git rebase`) (duration: 00m 55s)
11:32 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol1004.wikimedia.org with reason: working on network tests
11:31 aborrero@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol1004.wikimedia.org with reason: working on network tests
11:28 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1001.eqiad.wmnet with OS buster
11:04 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host dbprov1001.eqiad.wmnet with OS buster
10:56 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2001.codfw.wmnet with OS buster
10:37 moritzm: updated routinator in thirdparty/routinator for bullseye-wikimedia to 0.10.12 T292503
10:24 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host dbprov2001.codfw.wmnet with OS buster
10:18 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3065.esams.wmnet with OS buster
10:15 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol1004.wikimedia.org with reason: working on network tests
10:15 aborrero@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol1004.wikimedia.org with reason: working on network tests
10:15 vgutierrez: pool cp3065 running haproxy - T290005
09:25 marostegui@cumin1001: dbctl commit (dc=all): 'Remove contributions from s5 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17725 and previous config saved to /var/cache/conftool/dbconfig/20211111-092528-marostegui.json
09:13 vgutierrez@cumin1001: START - Cookbook sre.hosts.reimage for host cp3065.esams.wmnet with OS buster
09:10 vgutierrez: depool cp3065 to be reimaged as cache::upload_haproxy - T290005
09:03 arturo: pull all packages for buster-wikimedia/thirdparty/kubeadm-k8s-1-21 (T282942)
06:56 elukey: `systemctl start prometheus-mysqld-exporter@analytics_meta` on db1108
06:37 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1104.eqiad.wmnet with OS buster
06:10 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1104.eqiad.wmnet with OS buster
06:06 marostegui: Stop replication on db1104 (old master) T294321
06:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1104 (old master) T294321', diff saved to https://phabricator.wikimedia.org/P17723 and previous config saved to /var/cache/conftool/dbconfig/20211111-060242-marostegui.json
06:01 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db1109 to s8 primary and set section read-write T294321', diff saved to https://phabricator.wikimedia.org/P17722 and previous config saved to /var/cache/conftool/dbconfig/20211111-060102-marostegui.json
06:00 marostegui@cumin1001: dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - T294321', diff saved to https://phabricator.wikimedia.org/P17721 and previous config saved to /var/cache/conftool/dbconfig/20211111-060031-marostegui.json
06:00 marostegui: Starting s8 eqiad failover from db1104 to db1109 - T294321
05:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 31 hosts with reason: Primary switchover s8 T294321
05:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 31 hosts with reason: Primary switchover s8 T294321
18:09 bblack: drmrs - rebooting a bunch of hosts to bios for further settings, please ignore any accidental alerts - they do *look* like they're alert-disabled)
18:08 vgutierrez: restart haproxy on cp4026 and cp5006 to enable hitless reloads - T290005
18:07 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:03 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:10 bblack@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dns6001.wikimedia.org with OS buster
16:49 bblack@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dns6002.wikimedia.org with OS buster
16:47 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
16:44 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
16:34 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
16:32 ebernhardson@deploy1002: Synchronized wmf-config/InitialiseSettings.php: T295480: Move all cirrussearch traffic to codfw (duration: 00m 55s)
16:30 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
16:28 elukey: move atskafka to the new CA bundle - T291905
16:26 elukey: move kafkatee instances (analytics-test,centralog) to the new CA bundle - T291905
16:14 bblack@cumin1001: START - Cookbook sre.hosts.reimage for host dns6002.wikimedia.org with OS buster
16:12 bblack@cumin1001: START - Cookbook sre.hosts.reimage for host dns6001.wikimedia.org with OS buster
15:52 ebernhardson@deploy1002: Synchronized wmf-config/InitialiseSettings.php: T295480: Move all cirrussearch traffic to codfw (duration: 00m 56s)
14:09 legoktm: restarted mailman3/mailman3-web to pick up new DNS for m5-master
14:08 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001
14:02 ayounsi@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:48 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001
12:18 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:07 urbanecm: wikiadmin@10.64.48.109(centralauth)> delete from globalnames where gn_name='DJemielniak (WMF)'; # to let OIT create that account globally, SULification of foundationwiki, T205347
12:07 urbanecm: wikiadmin@10.64.48.109(centralauth)> delete from localnames where ln_name='DJemielniak (WMF)' and ln_wiki='foundationwiki'; # to let OIT create that account globally, SULification of foundationwiki, T205347
12:07 urbanecm: wikiadmin@10.64.48.109(centralauth)> delete from localnames where ln_wiki='foundationwiki' and ln_name='AAnctil (WMF)'; # to let OIT create that account globally, SULification of foundationwiki, T205347
12:06 urbanecm: wikiadmin@10.64.48.109(centralauth)> select * from localnames where ln_name='AAnctil (WMF)'; # to let OIT create that account globally, SULification of foundationwiki, T205347
12:06 urbanecm: wikiadmin@10.64.48.109(centralauth)> delete from globalnames where gn_name='AAnctil (WMF)'; # to let OIT create that account globally, SULification of foundationwiki, T205347
09:38 marostegui: Upgrade db1124, db1125, db1133 and pc2014 to mariadb 10.4.22
09:22 volans@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS buster
08:43 volans@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS buster
08:39 volans@cumin1001: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti6004.drmrs.wmnet
08:22 volans@cumin1001: START - Cookbook sre.hosts.dhcp for host ganeti6004.drmrs.wmnet
06:41 marostegui@cumin1001: dbctl commit (dc=all): 'Set db1109 with weight 0 T294321', diff saved to https://phabricator.wikimedia.org/P17715 and previous config saved to /var/cache/conftool/dbconfig/20211110-064120-root.json
19:03 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:59 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:45 mbsantos@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
18:40 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:36 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:55 mutante: re-enabled puppet on mw* after deploying and testing gerrit:736595 on canary
17:37 mmandere@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns6001.wikimedia.org with OS buster
17:36 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:32 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:08 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host dns6001.wikimedia.org with OS buster
16:55 mmandere@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti6004.drmrs.wmnet with OS buster
16:50 mutante: snapshot* - disabling puppet - converting some crons
16:41 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS buster
16:38 mmandere@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti6004.drmrs.wmnet with OS buster
16:16 jgiannelos@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
16:12 jgiannelos@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
16:07 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS buster
16:07 jgiannelos@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
15:49 mmandere@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti6004.drmrs.wmnet with OS buster
15:08 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS buster
14:52 bblack: rebooting ganeti6003
14:21 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) restart MirrorMaker for Kafka A:kafka-mirror-maker-test-eqiad cluster: Roll restart of jvm daemons. - elukey@cumin1001
14:19 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS buster
14:11 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-mirror-maker restart MirrorMaker for Kafka A:kafka-mirror-maker-test-eqiad cluster: Roll restart of jvm daemons. - elukey@cumin1001
14:08 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001
13:51 vgutierrez: pool cp5006 (upload) running haproxy-tls - T290005
13:50 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5006.eqsin.wmnet with OS buster
13:47 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001
13:15 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS buster
13:09 mmandere@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti6003.drmrs.wmnet with OS buster
13:02 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS buster
12:24 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:22 Lucas_WMDE: UTC morning backport+config window done
12:21 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:18 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001
12:12 mmandere@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti6003.drmrs.wmnet with OS buster
12:11 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:07 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
11:57 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001
11:48 volans@cumin2002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: Release v0.2.9 - volans@cumin2002
11:48 vgutierrez@cumin1001: START - Cookbook sre.hosts.reimage for host cp5006.eqsin.wmnet with OS buster
11:47 volans@cumin2002: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: Release v0.2.9 - volans@cumin2002
11:45 vgutierrez: depool cp5006 to be reimaged as cache::upload_haproxy - T290005
20:02 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:58 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: e66bd53: Enable TheWikipediaLibrary on meta & testwiki (T288070) (duration: 00m 55s)
19:52 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:52 ottomata: an-coord1002: drop user 'admin'@'localhost'; start slave; to fix broken replication - T284150
19:49 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:48 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 1ca184b: Add a new "all assessments" option to MediaSearch assessments dropdown (T285349) (duration: 00m 55s)
19:46 sukhe: upload pdns-recursor 4.5.7-1wm1 to apt.wm.o (buster)
19:39 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.7/extensions/WikidataPageBanner/includes/WikidataPageBanner.php: 2c74457: WikidataPageBanner should disable table of contents using public functions (T295003) (duration: 00m 55s)
19:34 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:31 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.7/extensions/VisualEditor/modules/ve-mw/preinit/ve.init.mw.ArticleTargetSaver.js: 9d7cde4: ArticleTargetSaver: ve.init may be undefined (T294981) (duration: 00m 55s)
19:30 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:22 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: bf70a8b: Make reply tool available as opt-out on dewiki (T294591) (duration: 00m 56s)
19:20 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:17 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:19 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
12:02 marostegui@cumin1001: dbctl commit (dc=all): 'Adjust weights for s5 codfw replicas after removing special groups from them T263127', diff saved to https://phabricator.wikimedia.org/P17708 and previous config saved to /var/cache/conftool/dbconfig/20211108-120203-marostegui.json
11:59 marostegui@cumin1001: dbctl commit (dc=all): 'Remove contributions logpager recentchanges recentchangeslinked watchlist from s5 codfw T263127', diff saved to https://phabricator.wikimedia.org/P17707 and previous config saved to /var/cache/conftool/dbconfig/20211108-115945-marostegui.json
11:41 mmandere@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti6002.drmrs.wmnet with OS buster
11:32 vgutierrez@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4026.ulsfo.wmnet with OS buster
11:01 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS buster
10:53 hnowlan@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
10:53 hnowlan@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
10:49 hnowlan@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
10:49 hnowlan@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
10:49 vgutierrez@cumin1001: START - Cookbook sre.hosts.reimage for host cp4026.ulsfo.wmnet with OS buster
10:27 vgutierrez: depool cp4026 to be reimaged as a haproxy-tls test node - T290005
10:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet
18:20 legoktm: upgrading scap to 4.0.3 everywhere (T294966)
18:01 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS buster
17:22 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS buster
16:52 mmandere@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti6001.drmrs.wmnet with OS buster
16:30 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS buster
16:21 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001
16:01 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001
15:38 hnowlan@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
15:38 hnowlan@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
14:30 jayme: published docker-registry.discovery.wmnet/golang1.17:1.17-1
13:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet
13:28 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2001.codfw.wmnet
12:50 mmandere@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti6001.drmrs.wmnet with OS buster
12:22 moritzm: renamed Ganeti group of test cluster from "default" to "row_A" (following conventions in main DCs) T286206
12:10 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS buster
12:01 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2001.codfw.wmnet
11:40 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2001.codfw.wmnet
11:09 mmandere@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti6001.drmrs.wmnet with OS buster
10:29 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS buster
09:53 ema: cp[4033-4036]: upgrade varnish to 6.0.8-1wm2 T295120
09:43 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2002.codfw.wmnet
09:39 mmandere@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti6001.drmrs.wmnet with OS buster
09:27 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2002.codfw.wmnet
09:27 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2001.codfw.wmnet
23:11 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
23:07 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
21:26 bblack: cpNNNN: manual (cumin) removal of outdated digicert-2020 ocsp configuration and output files, to avoid icinga alerts and clean up
20:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026
20:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026
19:29 dduvall: 1.38.0-wmf.7 on all wikis. no new errors or increase in error rates (refs T293948)
19:25 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:21 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:16 dduvall@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.7 refs T293948
18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1153 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17703 and previous config saved to /var/cache/conftool/dbconfig/20211104-182655-root.json
18:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1153 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17701 and previous config saved to /var/cache/conftool/dbconfig/20211104-181151-root.json
18:11 legoktm: upgrading to scap 4.0.3 on canaries again (T294966)
18:11 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:08 legoktm: uploaded scap 4.0.3-2 to apt.wm.o for buster/stretch (T294966)
18:07 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026
17:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026
17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1153 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17700 and previous config saved to /var/cache/conftool/dbconfig/20211104-175606-ladsgroup.json
17:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1152 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17699 and previous config saved to /var/cache/conftool/dbconfig/20211104-175429-root.json
17:50 volans: restarted puppetdb.service on puppetdb2002
17:47 ryankemper: T288620 [Elastic] Rebooting `elastic1049.eqiad.wmnet` to uptake new gelf settings change
17:46 hnowlan: enabling puppet on C:cassandra after profile::java transition
17:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1152 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17698 and previous config saved to /var/cache/conftool/dbconfig/20211104-173926-root.json
17:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1152.eqiad.wmnet with reason: Maintenance T295026
17:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1152.eqiad.wmnet with reason: Maintenance T295026
17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1152 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17697 and previous config saved to /var/cache/conftool/dbconfig/20211104-172950-ladsgroup.json
17:29 ayounsi@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2143.codfw.wmnet with reason: Maintenance T295026
15:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2143.codfw.wmnet with reason: Maintenance T295026
15:30 XioNoX: drain codfw-ulsfo link
15:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2143 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17694 and previous config saved to /var/cache/conftool/dbconfig/20211104-152919-ladsgroup.json
15:26 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2003.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
15:26 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2003.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
15:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
15:05 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
15:04 jgiannelos@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
15:03 jgiannelos@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
14:50 XioNoX: disable cr1-codfw:et-0/0/0
14:49 hashar: Upgrading CI Jenkins
14:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
14:44 moritzm: imported jenkins 2.303.3 to thirdparty/ci for buster-wikimedia T294838
13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'db2144 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17691 and previous config saved to /var/cache/conftool/dbconfig/20211104-131916-root.json
13:17 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' .
13:16 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'mathoid' for release 'staging' .
13:15 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' .
13:14 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' .
13:14 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' .
13:12 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
13:11 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
13:10 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
13:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1124.eqiad.wmnet with reason: Testing with the test host
13:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db1124.eqiad.wmnet with reason: Testing with the test host
13:09 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' .
13:09 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
13:08 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'echostore' for release 'staging' .
13:06 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
13:05 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' .
13:04 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
13:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'db2144 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17690 and previous config saved to /var/cache/conftool/dbconfig/20211104-130412-root.json
13:03 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
13:03 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
13:02 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
13:01 jelto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'apertium' for release 'staging' .
12:44 Amir1: Upgrade db2144 (kernel and mariadb) T295026
12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2144 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17689 and previous config saved to /var/cache/conftool/dbconfig/20211104-122504-ladsgroup.json
12:09 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:02 marostegui@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17686 and previous config saved to /var/cache/conftool/dbconfig/20211104-090240-root.json
08:56 dcausse: restarting blazegraph on wdqs1012 (stuck for the past 6 hours)
08:47 marostegui@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17685 and previous config saved to /var/cache/conftool/dbconfig/20211104-084736-root.json
08:37 _joe_: ipvsadm -Dt 10.2.2.67:443 on lvs101{5,6}
08:32 marostegui@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17684 and previous config saved to /var/cache/conftool/dbconfig/20211104-083233-root.json
08:29 _joe_: restarting pybal on low-traffic nodes in eqiad and codfw
08:17 marostegui@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17683 and previous config saved to /var/cache/conftool/dbconfig/20211104-081729-root.json
08:17 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly pool db1163', diff saved to https://phabricator.wikimedia.org/P17682 and previous config saved to /var/cache/conftool/dbconfig/20211104-081726-marostegui.json
07:43 marostegui@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17681 and previous config saved to /var/cache/conftool/dbconfig/20211104-074346-root.json
05:54 marostegui@cumin1001: dbctl commit (dc=all): 'Increase weight for the old special replicas T263127', diff saved to https://phabricator.wikimedia.org/P17679 and previous config saved to /var/cache/conftool/dbconfig/20211104-055419-marostegui.json
19:19 dduvall: 1.38.0-wmf.7 now on group0. no new errors. leaving ~ 30 minutes before promoting group1 (T293948)
19:18 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:15 dduvall@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.7 refs T293948
19:15 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:10 tgr: UTC evening deploys done
19:05 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:01 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:59 razzi@cumin1001: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001
18:55 razzi@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001
18:51 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4035.ulsfo.wmnet with OS buster
18:51 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:48 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:40 legoktm: re-enabling puppet on lists1001
18:38 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
07:58 marostegui@cumin1001: dbctl commit (dc=all): 'Remove logpager replicas from s6 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17660 and previous config saved to /var/cache/conftool/dbconfig/20211103-075801-marostegui.json
07:58 elukey@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
07:57 elukey@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
07:50 marostegui: Drop oauth2_access_tokens oauth_accepted_consumer oauth_registered_consumer from foundationwiki T294595
06:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1163.eqiad.wmnet with OS buster
06:39 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
06:35 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
06:35 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 34888b0: Growth IP research survey: Fix coverage (T294568) (duration: 01m 04s)
06:13 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1163.eqiad.wmnet with OS buster
06:10 marostegui: Stop replication on db1163 T290865
06:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1163 until it's reimaged to buster T293964', diff saved to https://phabricator.wikimedia.org/P17659 and previous config saved to /var/cache/conftool/dbconfig/20211103-060644-root.json
06:02 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db1118 to s1 primary and set section read-write T293964', diff saved to https://phabricator.wikimedia.org/P17658 and previous config saved to /var/cache/conftool/dbconfig/20211103-060201-root.json
06:01 marostegui@cumin1001: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - T293964', diff saved to https://phabricator.wikimedia.org/P17657 and previous config saved to /var/cache/conftool/dbconfig/20211103-060114-root.json
06:00 marostegui: Starting s1 eqiad failover from db1163 to db1118 - T293964
05:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s1 T293964
05:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s1 T293964
01:53 milimetric@deploy1002: Started deploy [analytics/refinery@cf6095c]: Regular analytics weekly train [analytics/refinery@cf6095c]
2021-11-02
23:47 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
23:46 tgr: UTC late deploys done
23:45 tgr@deploy1002: Synchronized wmf-config: Config: Use page id for GrowthExperiments image recommendations, except for testwiki (736314736317 (T290949T292154) (duration: 01m 03s)
23:44 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
23:34 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:15 robh@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-db1001.eqiad.wmnet
18:14 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:11 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:01 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:59 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.6/extensions/DiscussionTools/modules/dt-ve/dt.ui.UsernameCompletionAction.js: 494af12: UsernameCompletion: Filter out users with indefinite sitewide blocks from API results (T294783) (duration: 00m 55s)
17:58 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:57 robh@cumin1001: START - Cookbook sre.hosts.decommission for hosts an-db1001.eqiad.wmnet
17:48 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:45 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:44 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: 339be07: foundationwiki: Set wgCentralAuthCookies to true (T205347) (duration: 00m 54s)
17:35 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
17:33 moritzm: installing opencv security updates
17:31 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
11:26 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudgw1002.eqiad.wmnet
11:06 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1003.eqiad.wmnet
11:00 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudnet1003.eqiad.wmnet
11:00 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1004.eqiad.wmnet
10:57 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1008.eqiad.wmnet
10:54 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudnet1004.eqiad.wmnet
10:53 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2002-dev.codfw.wmnet
10:48 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudgw2002-dev.codfw.wmnet
10:48 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2001-dev.codfw.wmnet
10:46 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1008.eqiad.wmnet
10:46 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1005.eqiad.wmnet
10:41 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudgw2001-dev.codfw.wmnet
10:40 aborrero@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudgw2001-dev.codfw.wmnet
10:40 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudgw2001-dev.codfw.wmnet
10:36 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1005.eqiad.wmnet
10:35 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
10:31 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
10:30 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: dbff998: dewiki: Set wgGEHomepageDefaultVariant to control (T294712) (duration: 00m 55s)
10:03 marostegui@cumin1001: dbctl commit (dc=all): 'Set db1118 with weight 0 T293964', diff saved to https://phabricator.wikimedia.org/P17652 and previous config saved to /var/cache/conftool/dbconfig/20211102-100348-root.json
09:46 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
09:42 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
09:40 legoktm: restarted apache2 on lists1001
09:39 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: b259434: QuickSurveys: Show Growth IP editors survey to 0.1% of users (T294568) (duration: 00m 57s)
09:03 marostegui@cumin1001: dbctl commit (dc=all): 'Remove recentchanges replicas from s6 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17651 and previous config saved to /var/cache/conftool/dbconfig/20211102-090306-marostegui.json
08:29 moritzm: installing sdl2 security updates
07:23 marostegui@cumin1001: dbctl commit (dc=all): 'Remove recentchangeslinked replicas from s6 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17650 and previous config saved to /var/cache/conftool/dbconfig/20211102-072320-marostegui.json
07:13 elukey: `apt-get purge dkms` (rc state) on stat100[5,8]
06:45 marostegui: Rename oauth2_access_tokens oauth_accepted_consumer oauth_registered_consumer tables on db1123 T294595
02:34 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
02:30 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
02:11 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
02:07 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
20:56 legoktm: upgrading PHP 7.2 on A:mw-canary servers
20:44 legoktm: upgrading PHP 7.2 on mwdebug* servers
20:34 mutante: mwmaint* - new timer/service mediawiki_job_growthexperiments-purgeExpiredMentorStatus created by puppet - T280307
20:33 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
20:32 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
20:30 legoktm@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
20:24 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .
20:22 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .
20:18 legoktm@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .
20:14 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .
20:12 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .
20:10 legoktm@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .
20:08 mutante: planet1002 - systemctl start update-en-planet after merging config change btw. legoktm: it should be included in a sec
19:35 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:31 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
19:29 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: cba805c: Prepare a QuickSurvey for Growth IP research (T294568) (duration: 00m 55s)
19:26 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .
19:23 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox' for release 'main' .
19:19 legoktm@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'shellbox' for release 'main' .
18:49 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' .
18:37 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' .
18:26 legoktm@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' .
18:25 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:19 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
18:09 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: fb433d6: Amend wordmark for the Meetei (Manipuri) Wikipedia (T294189; 2/2) (duration: 00m 55s)
