The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
2023-07-31
23:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T342617)', diff saved to https://phabricator.wikimedia.org/P49860 and previous config saved to /var/cache/conftool/dbconfig/20230731-235442-ladsgroup.json
23:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T342617)', diff saved to https://phabricator.wikimedia.org/P49859 and previous config saved to /var/cache/conftool/dbconfig/20230731-233039-ladsgroup.json
23:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance
23:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance
23:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T342617)', diff saved to https://phabricator.wikimedia.org/P49858 and previous config saved to /var/cache/conftool/dbconfig/20230731-233018-ladsgroup.json
23:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P49857 and previous config saved to /var/cache/conftool/dbconfig/20230731-231512-ladsgroup.json
23:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P49856 and previous config saved to /var/cache/conftool/dbconfig/20230731-230006-ladsgroup.json
22:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T342617)', diff saved to https://phabricator.wikimedia.org/P49855 and previous config saved to /var/cache/conftool/dbconfig/20230731-224500-ladsgroup.json
22:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1223 (T342617)', diff saved to https://phabricator.wikimedia.org/P49854 and previous config saved to /var/cache/conftool/dbconfig/20230731-223547-ladsgroup.json
22:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
22:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
22:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T342617)', diff saved to https://phabricator.wikimedia.org/P49853 and previous config saved to /var/cache/conftool/dbconfig/20230731-223526-ladsgroup.json
22:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P49852 and previous config saved to /var/cache/conftool/dbconfig/20230731-222020-ladsgroup.json
22:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P49851 and previous config saved to /var/cache/conftool/dbconfig/20230731-220514-ladsgroup.json
21:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T342617)', diff saved to https://phabricator.wikimedia.org/P49850 and previous config saved to /var/cache/conftool/dbconfig/20230731-215008-ladsgroup.json
21:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T342617)', diff saved to https://phabricator.wikimedia.org/P49849 and previous config saved to /var/cache/conftool/dbconfig/20230731-213017-ladsgroup.json
21:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
21:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
21:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance
21:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance
21:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T342617)', diff saved to https://phabricator.wikimedia.org/P49848 and previous config saved to /var/cache/conftool/dbconfig/20230731-212941-ladsgroup.json
21:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P49847 and previous config saved to /var/cache/conftool/dbconfig/20230731-211435-ladsgroup.json
20:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P49846 and previous config saved to /var/cache/conftool/dbconfig/20230731-205928-ladsgroup.json
20:45 volans@cumin1001: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
20:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T342617)', diff saved to https://phabricator.wikimedia.org/P49845 and previous config saved to /var/cache/conftool/dbconfig/20230731-204422-ladsgroup.json
20:37 volans@cumin1001: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
20:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1212 (T342617)', diff saved to https://phabricator.wikimedia.org/P49844 and previous config saved to /var/cache/conftool/dbconfig/20230731-203451-ladsgroup.json
20:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
20:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
20:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
20:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
20:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T342617)', diff saved to https://phabricator.wikimedia.org/P49843 and previous config saved to /var/cache/conftool/dbconfig/20230731-203413-ladsgroup.json
20:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P49842 and previous config saved to /var/cache/conftool/dbconfig/20230731-201907-ladsgroup.json
20:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P49841 and previous config saved to /var/cache/conftool/dbconfig/20230731-200401-ladsgroup.json
19:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T342617)', diff saved to https://phabricator.wikimedia.org/P49840 and previous config saved to /var/cache/conftool/dbconfig/20230731-194854-ladsgroup.json
19:03 xcollazo@deploy1002: Started deploy [airflow-dags/analytics@47f9458]: (no justification provided)
18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T342617)', diff saved to https://phabricator.wikimedia.org/P49839 and previous config saved to /var/cache/conftool/dbconfig/20230731-184200-ladsgroup.json
18:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
18:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
18:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T342617)', diff saved to https://phabricator.wikimedia.org/P49838 and previous config saved to /var/cache/conftool/dbconfig/20230731-184140-ladsgroup.json
18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P49837 and previous config saved to /var/cache/conftool/dbconfig/20230731-182633-ladsgroup.json
18:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T342617)', diff saved to https://phabricator.wikimedia.org/P49836 and previous config saved to /var/cache/conftool/dbconfig/20230731-182114-ladsgroup.json
18:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2149.codfw.wmnet with reason: Maintenance
18:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2149.codfw.wmnet with reason: Maintenance
18:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P49835 and previous config saved to /var/cache/conftool/dbconfig/20230731-181127-ladsgroup.json
17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T342617)', diff saved to https://phabricator.wikimedia.org/P49834 and previous config saved to /var/cache/conftool/dbconfig/20230731-175621-ladsgroup.json
17:04 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1108.eqiad.wmnet
17:04 btullis@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:04 btullis@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1108.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1001"
17:02 btullis@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1108.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1001"
17:00 btullis@cumin1001: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0)
17:00 btullis@cumin1001: Added views for new wiki: gpewiki T338678
16:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T342617)', diff saved to https://phabricator.wikimedia.org/P49833 and previous config saved to /var/cache/conftool/dbconfig/20230731-164759-ladsgroup.json
16:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance
16:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance
16:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T342617)', diff saved to https://phabricator.wikimedia.org/P49832 and previous config saved to /var/cache/conftool/dbconfig/20230731-164738-ladsgroup.json
16:42 btullis@cumin1001: START - Cookbook sre.hosts.decommission for hosts db1108.eqiad.wmnet
16:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P49831 and previous config saved to /var/cache/conftool/dbconfig/20230731-163232-ladsgroup.json
16:30 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-airflow1003.eqiad.wmnet
16:30 btullis@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:30 btullis@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-airflow1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1001"
16:28 btullis@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-airflow1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1001"
16:19 btullis@cumin1001: START - Cookbook sre.hosts.decommission for hosts an-airflow1003.eqiad.wmnet
16:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P49830 and previous config saved to /var/cache/conftool/dbconfig/20230731-161726-ladsgroup.json
16:07 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/datahub: sync on main
16:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance
16:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance
16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2127 (T342617)', diff saved to https://phabricator.wikimedia.org/P49829 and previous config saved to /var/cache/conftool/dbconfig/20230731-160500-ladsgroup.json
16:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T342617)', diff saved to https://phabricator.wikimedia.org/P49828 and previous config saved to /var/cache/conftool/dbconfig/20230731-160220-ladsgroup.json
15:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P49827 and previous config saved to /var/cache/conftool/dbconfig/20230731-154954-ladsgroup.json
15:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P49826 and previous config saved to /var/cache/conftool/dbconfig/20230731-153448-ladsgroup.json
15:20 volans: deploying python3-wmflib fleet wide
15:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2127 (T342617)', diff saved to https://phabricator.wikimedia.org/P49825 and previous config saved to /var/cache/conftool/dbconfig/20230731-151942-ladsgroup.json
14:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T342617)', diff saved to https://phabricator.wikimedia.org/P49824 and previous config saved to /var/cache/conftool/dbconfig/20230731-145252-ladsgroup.json
14:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
14:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
14:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T342617)', diff saved to https://phabricator.wikimedia.org/P49823 and previous config saved to /var/cache/conftool/dbconfig/20230731-145232-ladsgroup.json
14:47 sukhe: finished rolling out gdnsd 3.99.0~alpha2 upgrade
14:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P49821 and previous config saved to /var/cache/conftool/dbconfig/20230731-143725-ladsgroup.json
14:31 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/datahub: apply on main
14:26 volans: uploaded python3-wmflib_1.2.3 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia,bookworm-wikimedia
14:25 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P49820 and previous config saved to /var/cache/conftool/dbconfig/20230731-142220-ladsgroup.json
14:21 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
14:10 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T342617)', diff saved to https://phabricator.wikimedia.org/P49819 and previous config saved to /var/cache/conftool/dbconfig/20230731-140713-ladsgroup.json
14:05 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
14:03 jforrester@deploy1002: jforrester: Continuing with sync
14:03 jforrester@deploy1002: jforrester: Backport for Wikifunctions: Add WF as alias for NS_PROJECT (and WT for its talk) (T342964) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:57 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
13:52 sukhe: reprepro -C main include bullseye-wikimedia gdnsd_3.99.0~alpha2-1_amd64.changes
13:51 jforrester@deploy1002: jforrester: Continuing with sync
13:51 jforrester@deploy1002: jforrester: Backport for Wikifunctions: Disable the Collection extension for now, broken (T342931) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:37 marostegui@cumin1001: dbctl commit (dc=all): 'db2114 (re)pooling @ 100%: Repooling after migration', diff saved to https://phabricator.wikimedia.org/P49818 and previous config saved to /var/cache/conftool/dbconfig/20230731-133707-root.json
13:29 jforrester@deploy1002: jforrester and epicpupper: Continuing with sync
13:29 jforrester@deploy1002: jforrester and epicpupper: Backport for Remove F: namespace alias (T325910) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:24 moritzm: imported jenkins 2.401.3 to thirdparty/ci for bullseye-wikimedia T342572
13:22 marostegui@cumin1001: dbctl commit (dc=all): 'db2114 (re)pooling @ 75%: Repooling after migration', diff saved to https://phabricator.wikimedia.org/P49817 and previous config saved to /var/cache/conftool/dbconfig/20230731-132201-root.json
12:57 moritzm: installing 6.1.38 kernels on Bookworm hosts
12:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T342617)', diff saved to https://phabricator.wikimedia.org/P49815 and previous config saved to /var/cache/conftool/dbconfig/20230731-125513-ladsgroup.json
12:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
12:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
12:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1196.eqiad.wmnet with reason: Maint
12:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1196.eqiad.wmnet with reason: Maint
12:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depool db1196 T342284', diff saved to https://phabricator.wikimedia.org/P49814 and previous config saved to /var/cache/conftool/dbconfig/20230731-125252-ladsgroup.json
12:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2114 (re)pooling @ 25%: Repooling after migration', diff saved to https://phabricator.wikimedia.org/P49813 and previous config saved to /var/cache/conftool/dbconfig/20230731-125152-root.json
12:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2127 (T342617)', diff saved to https://phabricator.wikimedia.org/P49812 and previous config saved to /var/cache/conftool/dbconfig/20230731-124912-ladsgroup.json
12:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: Maintenance
12:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: Maintenance
12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T342617)', diff saved to https://phabricator.wikimedia.org/P49811 and previous config saved to /var/cache/conftool/dbconfig/20230731-124851-ladsgroup.json
12:36 marostegui@cumin1001: dbctl commit (dc=all): 'db2114 (re)pooling @ 10%: Repooling after migration', diff saved to https://phabricator.wikimedia.org/P49810 and previous config saved to /var/cache/conftool/dbconfig/20230731-123647-root.json
12:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P49809 and previous config saved to /var/cache/conftool/dbconfig/20230731-123345-ladsgroup.json
12:32 moritzm: installing xapian-core bugfix updates on Bullseye
12:23 moritzm: installing mariadb-10.5 updates from Bullseye 11.7 point release (libs/tools, unrelated to wmf-mariadb packages)
12:21 marostegui@cumin1001: dbctl commit (dc=all): 'db2114 (re)pooling @ 5%: Repooling after migration', diff saved to https://phabricator.wikimedia.org/P49808 and previous config saved to /var/cache/conftool/dbconfig/20230731-122142-root.json
12:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P49807 and previous config saved to /var/cache/conftool/dbconfig/20230731-121839-ladsgroup.json
12:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2114 (re)pooling @ 3%: Repooling after migration', diff saved to https://phabricator.wikimedia.org/P49806 and previous config saved to /var/cache/conftool/dbconfig/20230731-120638-root.json
12:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T342617)', diff saved to https://phabricator.wikimedia.org/P49805 and previous config saved to /var/cache/conftool/dbconfig/20230731-120332-ladsgroup.json
11:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2114 (re)pooling @ 1%: Repooling after migration', diff saved to https://phabricator.wikimedia.org/P49804 and previous config saved to /var/cache/conftool/dbconfig/20230731-115133-root.json
11:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2114 T334650', diff saved to https://phabricator.wikimedia.org/P49803 and previous config saved to /var/cache/conftool/dbconfig/20230731-114645-root.json
11:23 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
11:11 btullis@cumin1001: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0)
11:11 btullis@cumin1001: Added views for new wiki: wikifunctionswiki T289316
11:11 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
10:59 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
10:55 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
10:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
10:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
10:45 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
09:32 urbanecm: Unblock stuck global rename by running `extensions/CentralAuth/maintenance/fixStuckGlobalRename.php` (T343099)
09:29 ladsgroup@deploy1002: ladsgroup and isaranto: Backport for ores-extension: enable Lift Wing for most wikis (T342115) synced to the testservers mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
09:29 fabfur@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs1016.eqiad.wmnet with OS bookworm
09:29 fabfur@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1001"
09:21 ladsgroup@deploy1002: amire80 and ladsgroup: Continuing with sync
09:20 ladsgroup@deploy1002: amire80 and ladsgroup: Backport for Remove ak from wgImportSources (T333765) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
07:15 taavi@deploy1002: anzx and taavi: Continuing with sync
07:13 taavi@deploy1002: anzx and taavi: Backport for ruwikibooks: Set wgRestrictDisplayTitle to false (T342800) synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
10:29 elukey@deploy1002: helmfile [ml-serve-eqiad] 'sync' command on namespace 'ores-legacy' for release 'main' .
10:28 elukey@deploy1002: helmfile [ml-serve-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' .
10:25 elukey@deploy1002: helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' .
10:18 dcausse: T342924: created search indices for wikifunctions
10:00 aikochou@deploy1002: helmfile [ml-serve-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' .
09:57 aikochou@deploy1002: helmfile [ml-serve-eqiad] 'sync' command on namespace 'ores-legacy' for release 'main' .
09:07 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1108.eqiad.wmnet with reason: db1108 has been replaced with db1208 - leaving for a few days before decom
09:07 btullis@cumin1001: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1108.eqiad.wmnet with reason: db1108 has been replaced with db1208 - leaving for a few days before decom
00:11 tgr@deploy1002: tgr: Backport for help: Fix navigation in the help panel (T342927) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
21:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
21:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T342617)', diff saved to https://phabricator.wikimedia.org/P49790 and previous config saved to /var/cache/conftool/dbconfig/20230727-214302-ladsgroup.json
21:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P49789 and previous config saved to /var/cache/conftool/dbconfig/20230727-212756-ladsgroup.json
21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P49788 and previous config saved to /var/cache/conftool/dbconfig/20230727-211250-ladsgroup.json
20:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T342617)', diff saved to https://phabricator.wikimedia.org/P49787 and previous config saved to /var/cache/conftool/dbconfig/20230727-205744-ladsgroup.json
20:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1224 (T342617)', diff saved to https://phabricator.wikimedia.org/P49786 and previous config saved to /var/cache/conftool/dbconfig/20230727-203435-ladsgroup.json
20:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: Maintenance
20:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: Maintenance
20:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1213:3316 (T342617)', diff saved to https://phabricator.wikimedia.org/P49785 and previous config saved to /var/cache/conftool/dbconfig/20230727-203415-ladsgroup.json
20:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1213:3316', diff saved to https://phabricator.wikimedia.org/P49784 and previous config saved to /var/cache/conftool/dbconfig/20230727-201908-ladsgroup.json
20:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1213:3316', diff saved to https://phabricator.wikimedia.org/P49783 and previous config saved to /var/cache/conftool/dbconfig/20230727-200402-ladsgroup.json
19:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1213:3316 (T342617)', diff saved to https://phabricator.wikimedia.org/P49782 and previous config saved to /var/cache/conftool/dbconfig/20230727-194856-ladsgroup.json
19:35 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1014.eqiad.wmnet with OS bullseye
19:35 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
19:18 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage
19:15 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage
19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1213:3316 (T342617)', diff saved to https://phabricator.wikimedia.org/P49781 and previous config saved to /var/cache/conftool/dbconfig/20230727-190637-ladsgroup.json
19:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Maintenance
19:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Maintenance
19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T342617)', diff saved to https://phabricator.wikimedia.org/P49780 and previous config saved to /var/cache/conftool/dbconfig/20230727-190617-ladsgroup.json
18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P49779 and previous config saved to /var/cache/conftool/dbconfig/20230727-185110-ladsgroup.json
18:41 milimetric@deploy1002: Finished deploy [analytics/refinery@1af57de] (thin): Deploying to sync script updates and static files (duration: 00m 04s)
18:41 milimetric@deploy1002: Started deploy [analytics/refinery@1af57de] (thin): Deploying to sync script updates and static files
18:41 milimetric@deploy1002: Finished deploy [analytics/refinery@1af57de]: Deploying to sync script updates and static files (duration: 08m 25s)
18:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P49778 and previous config saved to /var/cache/conftool/dbconfig/20230727-183604-ladsgroup.json
18:33 milimetric@deploy1002: Started deploy [analytics/refinery@1af57de]: Deploying to sync script updates and static files
18:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T342617)', diff saved to https://phabricator.wikimedia.org/P49777 and previous config saved to /var/cache/conftool/dbconfig/20230727-182058-ladsgroup.json
18:13 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb1014.eqiad.wmnet with OS bullseye
17:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1201 (T342617)', diff saved to https://phabricator.wikimedia.org/P49776 and previous config saved to /var/cache/conftool/dbconfig/20230727-175659-ladsgroup.json
17:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance
17:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance
17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T342617)', diff saved to https://phabricator.wikimedia.org/P49775 and previous config saved to /var/cache/conftool/dbconfig/20230727-175638-ladsgroup.json
17:44 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage
17:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P49774 and previous config saved to /var/cache/conftool/dbconfig/20230727-174132-ladsgroup.json
17:41 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage
17:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P49773 and previous config saved to /var/cache/conftool/dbconfig/20230727-172626-ladsgroup.json
17:25 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb1013.eqiad.wmnet with OS bullseye
16:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T342617)', diff saved to https://phabricator.wikimedia.org/P49771 and previous config saved to /var/cache/conftool/dbconfig/20230727-164711-ladsgroup.json
16:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance
16:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance
16:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T342617)', diff saved to https://phabricator.wikimedia.org/P49770 and previous config saved to /var/cache/conftool/dbconfig/20230727-164650-ladsgroup.json
16:45 dancy@deploy1002: Installation of scap version "4.57.0" completed for 600 hosts
16:44 dancy@deploy1002: Installing scap version "4.57.0" for 600 hosts
16:33 ladsgroup@deploy1002: isaranto and ladsgroup: Backport for ores-extension: enable lw on itwiki and hewiki (T342115) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P49769 and previous config saved to /var/cache/conftool/dbconfig/20230727-163144-ladsgroup.json
16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P49768 and previous config saved to /var/cache/conftool/dbconfig/20230727-161638-ladsgroup.json
16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T342617)', diff saved to https://phabricator.wikimedia.org/P49766 and previous config saved to /var/cache/conftool/dbconfig/20230727-160132-ladsgroup.json
15:38 zabe@deploy1002: zabe and dreamyjazz: Backport for Revert "CheckUser event table migration: Write new on group0" (T342902) synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
15:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T342617)', diff saved to https://phabricator.wikimedia.org/P49764 and previous config saved to /var/cache/conftool/dbconfig/20230727-153649-ladsgroup.json
15:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance
15:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance
15:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1173 (T342617)', diff saved to https://phabricator.wikimedia.org/P49763 and previous config saved to /var/cache/conftool/dbconfig/20230727-153629-ladsgroup.json
15:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P49762 and previous config saved to /var/cache/conftool/dbconfig/20230727-152123-ladsgroup.json
15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P49761 and previous config saved to /var/cache/conftool/dbconfig/20230727-150616-ladsgroup.json
14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1173 (T342617)', diff saved to https://phabricator.wikimedia.org/P49759 and previous config saved to /var/cache/conftool/dbconfig/20230727-145110-ladsgroup.json
14:33 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
14:27 elukey@deploy1002: helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' .
14:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1173 (T342617)', diff saved to https://phabricator.wikimedia.org/P49758 and previous config saved to /var/cache/conftool/dbconfig/20230727-142721-ladsgroup.json
14:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance
14:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance
14:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T342617)', diff saved to https://phabricator.wikimedia.org/P49757 and previous config saved to /var/cache/conftool/dbconfig/20230727-142700-ladsgroup.json
14:26 cmooney@cumin1001: END (PASS) - Cookbook sre.network.provision (exit_code=0) for device ssw1-a1-codfw.mgmt.codfw.wmnet
14:25 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:25 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for ssw1-a1-codfw - cmooney@cumin1001"
14:24 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for ssw1-a1-codfw - cmooney@cumin1001"
14:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P49756 and previous config saved to /var/cache/conftool/dbconfig/20230727-141154-ladsgroup.json
13:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P49755 and previous config saved to /var/cache/conftool/dbconfig/20230727-135648-ladsgroup.json
13:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T342617)', diff saved to https://phabricator.wikimedia.org/P49754 and previous config saved to /var/cache/conftool/dbconfig/20230727-134141-ladsgroup.json
13:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T342617)', diff saved to https://phabricator.wikimedia.org/P49752 and previous config saved to /var/cache/conftool/dbconfig/20230727-131733-ladsgroup.json
13:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance
13:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance
13:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T342617)', diff saved to https://phabricator.wikimedia.org/P49751 and previous config saved to /var/cache/conftool/dbconfig/20230727-131712-ladsgroup.json
13:05 samtar@deploy1002: samtar and daniel: Backport for Re-enable PC writes for parsoid endpoints (T339867) synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P49750 and previous config saved to /var/cache/conftool/dbconfig/20230727-130206-ladsgroup.json
12:48 fabfur@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs6003.drmrs.wmnet
12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P49749 and previous config saved to /var/cache/conftool/dbconfig/20230727-124700-ladsgroup.json
12:45 fabfur@cumin1001: START - Cookbook sre.hosts.reboot-single for host lvs6003.drmrs.wmnet
12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T342617)', diff saved to https://phabricator.wikimedia.org/P49748 and previous config saved to /var/cache/conftool/dbconfig/20230727-123153-ladsgroup.json
12:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T342617)', diff saved to https://phabricator.wikimedia.org/P49747 and previous config saved to /var/cache/conftool/dbconfig/20230727-120710-ladsgroup.json
12:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
12:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
12:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance
12:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance
12:01 jforrester@deploy1002: jforrester: Backport for Wikifunctions: Also add square logo for Vector-2022 synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
11:52 ladsgroup@deploy1002: ladsgroup: Backport for rdbms: Avoid making wasteful memcached calls in CP (T314434) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
11:37 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-main-codfw cluster: Roll restart of jvm daemons.
11:24 ladsgroup@deploy1002: ladsgroup: Backport for CentralAuthUser: Don't load user information unless needed synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
08:34 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-main-eqiad cluster: Roll restart of jvm daemons.
08:16 jnuche@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.19 refs T340247
23:01 jforrester@deploy1002: Synchronized wmf-config/interwiki.php: Update interwiki cache now that wikifunctions is here (duration: 06m 52s)
21:53 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wcqs2001.codfw.wmnet
21:46 bking@cumin1001: START - Cookbook sre.hosts.reboot-single for host wcqs2001.codfw.wmnet
21:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T342617)', diff saved to https://phabricator.wikimedia.org/P49745 and previous config saved to /var/cache/conftool/dbconfig/20230726-212310-ladsgroup.json
21:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P49744 and previous config saved to /var/cache/conftool/dbconfig/20230726-210804-ladsgroup.json
21:04 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host rdb1013.eqiad.wmnet with OS bullseye
21:04 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb1013.eqiad.wmnet with OS bullseye
21:00 taavi: manually attach User:WikiLambda_system to SUL T342811
20:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P49743 and previous config saved to /var/cache/conftool/dbconfig/20230726-205257-ladsgroup.json
20:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T342617)', diff saved to https://phabricator.wikimedia.org/P49742 and previous config saved to /var/cache/conftool/dbconfig/20230726-203751-ladsgroup.json
20:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T342617)', diff saved to https://phabricator.wikimedia.org/P49741 and previous config saved to /var/cache/conftool/dbconfig/20230726-201554-ladsgroup.json
20:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance
20:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance
20:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T342617)', diff saved to https://phabricator.wikimedia.org/P49740 and previous config saved to /var/cache/conftool/dbconfig/20230726-201533-ladsgroup.json
20:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P49739 and previous config saved to /var/cache/conftool/dbconfig/20230726-200026-ladsgroup.json
19:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P49738 and previous config saved to /var/cache/conftool/dbconfig/20230726-194520-ladsgroup.json
19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T342617)', diff saved to https://phabricator.wikimedia.org/P49737 and previous config saved to /var/cache/conftool/dbconfig/20230726-193014-ladsgroup.json
18:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance
18:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T342617)', diff saved to https://phabricator.wikimedia.org/P49735 and previous config saved to /var/cache/conftool/dbconfig/20230726-184408-ladsgroup.json
18:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P49734 and previous config saved to /var/cache/conftool/dbconfig/20230726-182902-ladsgroup.json
18:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P49732 and previous config saved to /var/cache/conftool/dbconfig/20230726-181356-ladsgroup.json
17:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T342617)', diff saved to https://phabricator.wikimedia.org/P49731 and previous config saved to /var/cache/conftool/dbconfig/20230726-175850-ladsgroup.json
17:49 jforrester@deploy1002: Finished scap: Hopefully final update for wikifunctions.org initial config (duration: 07m 30s)
17:41 jforrester@deploy1002: Started scap: Hopefully final update for wikifunctions.org initial config
17:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T342617)', diff saved to https://phabricator.wikimedia.org/P49730 and previous config saved to /var/cache/conftool/dbconfig/20230726-171244-ladsgroup.json
17:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance
17:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance
17:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T342617)', diff saved to https://phabricator.wikimedia.org/P49729 and previous config saved to /var/cache/conftool/dbconfig/20230726-171223-ladsgroup.json
17:06 jforrester@deploy1002: jforrester: Backport for MWMultiVersion: Alert this code to wikifunctions.org existing (T275945) synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
16:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P49728 and previous config saved to /var/cache/conftool/dbconfig/20230726-165717-ladsgroup.json
16:43 fabfur@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1018.eqiad.wmnet
16:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P49727 and previous config saved to /var/cache/conftool/dbconfig/20230726-164211-ladsgroup.json
16:40 fabfur@cumin1001: START - Cookbook sre.hosts.reboot-single for host lvs1018.eqiad.wmnet
16:27 jforrester@deploy1002: Finished scap: Initial deploy of wikifunctionswiki in locked-down mode for T275945 (duration: 07m 49s)
16:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T342617)', diff saved to https://phabricator.wikimedia.org/P49726 and previous config saved to /var/cache/conftool/dbconfig/20230726-162705-ladsgroup.json
16:20 jforrester@deploy1002: Started scap: Initial deploy of wikifunctionswiki in locked-down mode for T275945
16:08 jforrester@deploy1002: jforrester: Backport for Add wikifunctions.org to prod wgLocalVirtualHosts (T275945) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
15:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T342617)', diff saved to https://phabricator.wikimedia.org/P49725 and previous config saved to /var/cache/conftool/dbconfig/20230726-154245-ladsgroup.json
15:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
15:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance
15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance
15:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T342617)', diff saved to https://phabricator.wikimedia.org/P49724 and previous config saved to /var/cache/conftool/dbconfig/20230726-154209-ladsgroup.json
15:39 jforrester@deploy1002: dcausse and jforrester: Backport for Load RescoreFunctions from the ExtensionRegistry (T342744) synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
15:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P49723 and previous config saved to /var/cache/conftool/dbconfig/20230726-152703-ladsgroup.json
15:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P49722 and previous config saved to /var/cache/conftool/dbconfig/20230726-151157-ladsgroup.json
15:10 fabfur@cumin1001: START - Cookbook sre.hosts.reboot-single for host lvs4009.ulsfo.wmnet
15:00 ladsgroup@deploy1002: isaranto and ladsgroup: Backport for ores-extension: enable lw on eswikiquotes and eswikibooks (T342115) synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
14:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T342617)', diff saved to https://phabricator.wikimedia.org/P49721 and previous config saved to /var/cache/conftool/dbconfig/20230726-145651-ladsgroup.json
14:33 hnowlan: enabling puppet on A:cp to deploy r/941440
14:32 jforrester@deploy1002: jforrester: Backport for Normalize the skin name when it comes from preferences or useskin (T342733) synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance
13:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance
13:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T342617)', diff saved to https://phabricator.wikimedia.org/P49719 and previous config saved to /var/cache/conftool/dbconfig/20230726-133104-ladsgroup.json
13:23 jforrester@deploy1002: jforrester and tsev: Backport for Add stream config for iOS schema (T341896) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:22 fab@deploy1002: Started deploy [airflow-dags/research@e7b9253]: (no justification provided)
13:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P49718 and previous config saved to /var/cache/conftool/dbconfig/20230726-131557-ladsgroup.json
13:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P49717 and previous config saved to /var/cache/conftool/dbconfig/20230726-130051-ladsgroup.json
12:52 jforrester@deploy1002: Synchronized php-1.41.0-wmf.19/extensions/WikiLambda/: Update WikiLambda wmf.19 branch to latest ahead of wikifunctions.org roll-out (duration: 07m 10s)
12:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T342617)', diff saved to https://phabricator.wikimedia.org/P49716 and previous config saved to /var/cache/conftool/dbconfig/20230726-124545-ladsgroup.json
12:41 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 13335
12:39 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 13335
12:30 jforrester@deploy1002: jforrester: Backport for ProductionServices: Define the wikifunctions orchestrator access point (T297314) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
11:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T342617)', diff saved to https://phabricator.wikimedia.org/P49714 and previous config saved to /var/cache/conftool/dbconfig/20230726-115528-ladsgroup.json
11:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2124.codfw.wmnet with reason: Maintenance
11:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2124.codfw.wmnet with reason: Maintenance
11:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T342617)', diff saved to https://phabricator.wikimedia.org/P49713 and previous config saved to /var/cache/conftool/dbconfig/20230726-115507-ladsgroup.json
11:40 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-logging-eqiad cluster: Roll restart of jvm daemons.
11:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P49712 and previous config saved to /var/cache/conftool/dbconfig/20230726-114001-ladsgroup.json
11:32 fabfur@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1016.eqiad.wmnet with OS bullseye
11:27 fabfur@cumin1001: START - Cookbook sre.hosts.reimage for host lvs1016.eqiad.wmnet with OS bullseye
11:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P49711 and previous config saved to /var/cache/conftool/dbconfig/20230726-112454-ladsgroup.json
11:19 jnuche@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.19 refs T340247
11:14 fabfur@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1016.eqiad.wmnet with OS bullseye
11:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T342617)', diff saved to https://phabricator.wikimedia.org/P49710 and previous config saved to /var/cache/conftool/dbconfig/20230726-110948-ladsgroup.json
11:05 eoghan@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts releases1002.eqiad.wmnet
11:05 eoghan@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:05 eoghan@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: releases1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eoghan@cumin1001"
11:03 eoghan@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: releases1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eoghan@cumin1001"
10:57 eoghan@cumin1001: START - Cookbook sre.hosts.decommission for hosts releases1002.eqiad.wmnet
10:56 eoghan@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts releases2002.codfw.wmnet
10:56 eoghan@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
10:56 eoghan@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: releases2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - eoghan@cumin1001"
10:56 fabfur@cumin1001: START - Cookbook sre.hosts.reimage for host lvs1016.eqiad.wmnet with OS bullseye
10:55 eoghan@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: releases2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - eoghan@cumin1001"
06:37 denisse@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts xhgui2001.codfw.wmnet,xhgui1001.eqiad.wmnet
06:37 denisse@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
06:37 denisse@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: xhgui2001.codfw.wmnet,xhgui1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - denisse@cumin1001"
06:34 marostegui: Stop mariadb on clouddb1021 T334651
06:33 denisse@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: xhgui2001.codfw.wmnet,xhgui1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - denisse@cumin1001"
20:23 taavi@deploy1002: taavi and bwang: Backport for Fix text showing on icon only buttons synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
16:41 fabfur@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs5005.eqsin.wmnet
16:40 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-main-eqiad cluster: Roll restart of jvm daemons.
16:37 fabfur@cumin1001: START - Cookbook sre.hosts.reboot-single for host lvs5005.eqsin.wmnet
16:30 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dns6001.wikimedia.org
16:26 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host dns6001.wikimedia.org
14:59 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
14:59 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
14:59 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
14:59 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
14:58 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-main-eqiad cluster: Roll restart of jvm daemons.
14:58 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
14:58 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
14:58 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-main-codfw cluster: Roll restart of jvm daemons.
14:58 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
14:00 sukhe: rolling out pdns-recursor update on A:dns-rec
13:42 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15 days, 0:00:00 on parse1002.eqiad.wmnet with reason: T339340 - hw troubleshooting
13:42 cgoubert@cumin1001: START - Cookbook sre.hosts.downtime for 15 days, 0:00:00 on parse1002.eqiad.wmnet with reason: T339340 - hw troubleshooting
13:41 urbanecm@deploy1002: urbanecm and dreamyjazz: Backport for Enable write new on testwiki for CheckUser event tables migration (T330158) synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:41 cgoubert@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "re-run to fix mw1486 - cgoubert@cumin1001"
13:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2114 (T342617)', diff saved to https://phabricator.wikimedia.org/P49704 and previous config saved to /var/cache/conftool/dbconfig/20230725-132121-ladsgroup.json
13:17 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-main-codfw cluster: Roll restart of jvm daemons.
13:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P49702 and previous config saved to /var/cache/conftool/dbconfig/20230725-130615-ladsgroup.json
12:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P49701 and previous config saved to /var/cache/conftool/dbconfig/20230725-125109-ladsgroup.json
12:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2114 (T342617)', diff saved to https://phabricator.wikimedia.org/P49700 and previous config saved to /var/cache/conftool/dbconfig/20230725-123602-ladsgroup.json
12:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2114 (T342617)', diff saved to https://phabricator.wikimedia.org/P49699 and previous config saved to /var/cache/conftool/dbconfig/20230725-120641-ladsgroup.json
12:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance
12:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance
11:32 akosiaris: T340087 wikidiff2 rollout done. 1 host is unreachable and will need to be reimaged or upgraded manually to pick this up, parse1002.eqiad.wmnet
11:25 akosiaris: T340087 keep a copy php-wikidiff2_1.13.0-1_amd64.deb in apt1001:/home/akosiaris/wd/ in case of emergency
11:24 akosiaris: T340087 starting wikidiff2 1.41.1 rollout to codfw
10:51 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 31 days, 0:00:00 on lvs[1013-1015].eqiad.wmnet with reason: test hosts
10:50 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime for 31 days, 0:00:00 on lvs[1013-1015].eqiad.wmnet with reason: test hosts
09:50 elukey: restart kafka on kafka-main1001 to pick up the new changes - T341558
09:47 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on kafka-main1001.eqiad.wmnet with reason: Apply a new setting to the Kafka broker
09:46 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 0:30:00 on kafka-main1001.eqiad.wmnet with reason: Apply a new setting to the Kafka broker
09:06 slyngs: Restart Tomcat / Apereo CAS on idp1002
09:01 jnuche@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.19 refs T340247
08:35 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on kafka-main1001.eqiad.wmnet with reason: Apply a new setting to the Kafka broker
08:35 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 0:30:00 on kafka-main1001.eqiad.wmnet with reason: Apply a new setting to the Kafka broker
08:03 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49696 and previous config saved to /var/cache/conftool/dbconfig/20230725-080326-root.json
08:03 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49695 and previous config saved to /var/cache/conftool/dbconfig/20230725-080315-root.json
07:57 jnuche@deploy1002: Started scap: testwikis wikis to 1.41.0-wmf.19 refs T340247
07:48 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49694 and previous config saved to /var/cache/conftool/dbconfig/20230725-074821-root.json
07:48 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49693 and previous config saved to /var/cache/conftool/dbconfig/20230725-074810-root.json
07:33 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49692 and previous config saved to /var/cache/conftool/dbconfig/20230725-073317-root.json
07:33 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49691 and previous config saved to /var/cache/conftool/dbconfig/20230725-073305-root.json
07:18 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49690 and previous config saved to /var/cache/conftool/dbconfig/20230725-071812-root.json
07:18 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49689 and previous config saved to /var/cache/conftool/dbconfig/20230725-071801-root.json
07:03 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49688 and previous config saved to /var/cache/conftool/dbconfig/20230725-070307-root.json
07:02 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49687 and previous config saved to /var/cache/conftool/dbconfig/20230725-070256-root.json
06:48 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49686 and previous config saved to /var/cache/conftool/dbconfig/20230725-064802-root.json
06:47 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49685 and previous config saved to /var/cache/conftool/dbconfig/20230725-064751-root.json
06:32 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49684 and previous config saved to /var/cache/conftool/dbconfig/20230725-063258-root.json
06:32 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49683 and previous config saved to /var/cache/conftool/dbconfig/20230725-063247-root.json
06:17 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49682 and previous config saved to /var/cache/conftool/dbconfig/20230725-061753-root.json
06:17 marostegui@cumin1001: dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49681 and previous config saved to /var/cache/conftool/dbconfig/20230725-061742-root.json
06:13 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1213 (s5, s6)', diff saved to https://phabricator.wikimedia.org/P49680 and previous config saved to /var/cache/conftool/dbconfig/20230725-061319-root.json
06:10 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wdqs[2004-2006].codfw.wmnet
06:10 ryankemper@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
06:10 ryankemper@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs[2004-2006].codfw.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin1001"
05:52 ryankemper@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs[2004-2006].codfw.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin1001"
14:11 samtar@deploy1002: samtar: Backport for Revert "Revert "Run a synthetic test for client side preferences"" (T336527 T339268) synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
14:00 jclark@cumin1001: START - Cookbook sre.hosts.reimage for host rdb1014.eqiad.wmnet with OS bullseye
13:56 samtar@deploy1002: samtar: Backport for Revert "Run a synthetic test for client side preferences" synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:56 marostegui@cumin1001: dbctl commit (dc=all): 'db2169:3316 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49676 and previous config saved to /var/cache/conftool/dbconfig/20230724-135604-root.json
13:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49675 and previous config saved to /var/cache/conftool/dbconfig/20230724-135557-root.json
13:47 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49674 and previous config saved to /var/cache/conftool/dbconfig/20230724-134721-root.json
13:44 jclark@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host rdb1014.eqiad.wmnet with OS bullseye
13:44 jclark@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host rdb1013.eqiad.wmnet with OS bullseye
13:41 marostegui@cumin1001: dbctl commit (dc=all): 'db2169:3316 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49673 and previous config saved to /var/cache/conftool/dbconfig/20230724-134059-root.json
13:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49672 and previous config saved to /var/cache/conftool/dbconfig/20230724-134052-root.json
13:38 taavi: run `taavi@mwmaint1002 ~ $ mwscript namespaceDupes.php mywiktionary --fix` after purging null editing page #131577 for T342516
13:34 samtar@deploy1002: samtar and mabualruz: Backport for Run a synthetic test for client side preferences (T336527 T339268) synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
10:51 eoghan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on releases2002.codfw.wmnet,releases1002.eqiad.wmnet with reason: Decommissioning prep
10:51 eoghan@cumin1001: START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on releases2002.codfw.wmnet,releases1002.eqiad.wmnet with reason: Decommissioning prep
18:00 dancy@deploy1002: daimona and dancy: Backport for Enable the CampaignEvents extension before loading CommonSettings-labs (T342452) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
09:06 marostegui@cumin1001: dbctl commit (dc=all): 'db1201 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49645 and previous config saved to /var/cache/conftool/dbconfig/20230721-090625-root.json
09:00 marostegui@cumin1001: dbctl commit (dc=all): 'db2171:3316 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49644 and previous config saved to /var/cache/conftool/dbconfig/20230721-090003-root.json
08:59 marostegui@cumin1001: dbctl commit (dc=all): 'db2171:3315 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49643 and previous config saved to /var/cache/conftool/dbconfig/20230721-085955-root.json
08:59 fabfur@cumin1001: START - Cookbook sre.hosts.reimage for host lvs1014.eqiad.wmnet with OS bookworm
08:56 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1108.eqiad.wmnet with reason: db1108 has been replaced with db1208 - leaving for a few days before decom
08:56 btullis@cumin1001: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1108.eqiad.wmnet with reason: db1108 has been replaced with db1208 - leaving for a few days before decom
07:44 marostegui@cumin1001: dbctl commit (dc=all): 'db2171:3316 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49629 and previous config saved to /var/cache/conftool/dbconfig/20230721-074440-root.json
07:44 marostegui@cumin1001: dbctl commit (dc=all): 'db2171:3315 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49628 and previous config saved to /var/cache/conftool/dbconfig/20230721-074431-root.json
07:35 marostegui@cumin1001: dbctl commit (dc=all): 'db1201 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49627 and previous config saved to /var/cache/conftool/dbconfig/20230721-073557-root.json
07:29 marostegui@cumin1001: dbctl commit (dc=all): 'db2171:3316 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49626 and previous config saved to /var/cache/conftool/dbconfig/20230721-072935-root.json
07:29 marostegui@cumin1001: dbctl commit (dc=all): 'db2171:3315 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49625 and previous config saved to /var/cache/conftool/dbconfig/20230721-072927-root.json
07:20 marostegui@cumin1001: dbctl commit (dc=all): 'db1201 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49624 and previous config saved to /var/cache/conftool/dbconfig/20230721-072052-root.json
07:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1201', diff saved to https://phabricator.wikimedia.org/P49623 and previous config saved to /var/cache/conftool/dbconfig/20230721-071623-root.json
07:14 marostegui@cumin1001: dbctl commit (dc=all): 'db2171:3316 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49622 and previous config saved to /var/cache/conftool/dbconfig/20230721-071430-root.json
07:14 marostegui@cumin1001: dbctl commit (dc=all): 'db2171:3315 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49621 and previous config saved to /var/cache/conftool/dbconfig/20230721-071422-root.json
07:12 marostegui: Upgrade dbstore1005 to mariadb 10.6 T334652
07:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2171 (s5 and s6)', diff saved to https://phabricator.wikimedia.org/P49620 and previous config saved to /var/cache/conftool/dbconfig/20230721-070110-root.json
06:40 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 3209
06:40 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 3209
06:40 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 398203
06:39 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 398203
06:38 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 139418
06:37 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 139418
06:36 ayounsi@cumin1001: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 139148
06:36 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 139148
04:00 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:01:00 on 10 hosts with reason: trying to remove downtime on these new hosts
04:00 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 0:01:00 on 10 hosts with reason: trying to remove downtime on these new hosts
13:57 bking@cumin1001: START - Cookbook sre.hosts.reimage for host flink-zk1003.eqiad.wmnet with OS bookworm
13:57 btullis@cumin1001: END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) restart masters for Hadoop test cluster: Restart of jvm daemons.
13:55 bking@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host flink-zk1003.eqiad.wmnet
13:55 bking@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
12:15 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for assigned switch gw ips. - cmooney@cumin1001"
12:13 zabe@deploy1002: zabe and dreamyjazz: Backport for SpecialUserRights: Check for username to be temporary (T340468 T342322) synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
10:53 klausman@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
10:40 klausman@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
10:33 klausman@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
09:50 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:50 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove reverse dns for IP allocated in error. - cmooney@cumin1001"
09:24 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove reverse dns for IP allocated in error. - cmooney@cumin1001"
09:17 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:17 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove reverse dns for IP allocated in error. - cmooney@cumin1001"
09:15 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove reverse dns for IP allocated in error. - cmooney@cumin1001"
07:29 ariel@deploy1002: oblivian and ariel: Backport for noc/db.php: use the new etcd fetch function (T341859) synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
07:17 ariel@deploy1002: oblivian and ariel: Backport for noc: add script to dump etcd db config (T341859) synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
07:04 ariel@deploy1002: ariel and soda: Backport for Enable EditInSequence in pawikisource synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
21:41 bking@cumin1001: START - Cookbook sre.hosts.reimage for host flink-zk1003.eqiad.wmnet with OS bookworm
21:38 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
21:37 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
21:37 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1003.eqiad.wmnet on all recursors
21:37 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1003.eqiad.wmnet on all recursors
21:37 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
21:37 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
21:36 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
21:20 bking@deploy1002: Started deploy [wdqs/wdqs@dff41b7]: 0.3.124
20:55 bking@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts flink-zk1003.eqiad.wmnet
20:55 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
20:55 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: flink-zk1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin1001"
20:54 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: flink-zk1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin1001"
20:12 samtar@deploy1002: samtar and hubaishan: Backport for Replace underscores with spaces in 4 Arabic sitenames (T337725) synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
19:45 sukhe@cumin2002: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum and A:durum
19:42 bking@cumin1001: START - Cookbook sre.hosts.reimage for host flink-zk1003.eqiad.wmnet with OS bookworm
19:41 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
19:40 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
19:40 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1003.eqiad.wmnet on all recursors
19:40 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1003.eqiad.wmnet on all recursors
19:40 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
19:40 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
19:39 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
19:37 bking@cumin1001: START - Cookbook sre.ganeti.makevm for new host flink-zk1003.eqiad.wmnet
19:36 bking@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts flink-zk1003.eqiad.wmnet
19:36 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
19:36 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: flink-zk1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin1001"
19:35 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: flink-zk1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin1001"
13:46 bking@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host flink-zk1003.eqiad.wmnet
13:46 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1003.eqiad.wmnet on all recursors
13:46 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1003.eqiad.wmnet on all recursors
13:46 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:46 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
13:45 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
13:42 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1003.eqiad.wmnet on all recursors
13:42 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1003.eqiad.wmnet on all recursors
13:42 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:42 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
13:42 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
13:04 lucaswerkmeister-wmde@deploy1002: ssastry and lucaswerkmeister-wmde: Backport for Fix incorrect use of UseLegacyMediaStyles (missing "wg" prefix) (T318433) synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
12:17 jbond@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) puppetboard.discovery.wmnet on all recursors
12:17 jbond@cumin1001: START - Cookbook sre.dns.wipe-cache puppetboard.discovery.wmnet on all recursors
12:17 jbond@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) puppetboard-next.discovery.wmnet on all recursors
12:17 jbond@cumin1001: START - Cookbook sre.dns.wipe-cache puppetboard-next.discovery.wmnet on all recursors
11:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy1016.eqiad.wmnet
11:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ladsgroup@cumin1001"
11:47 ladsgroup@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ladsgroup@cumin1001"
08:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49594 and previous config saved to /var/cache/conftool/dbconfig/20230719-083319-root.json
08:30 dcausse@deploy1002: dcausse: Backport for Use the LinksUpdate::isRecursive flag again to route cirrusSearchLinksUpdate synced to the testservers mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
08:04 dcausse@deploy1002: dcausse: Backport for Use the LinksUpdate::isRecursive flag again to route cirrusSearchLinksUpdate synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
08:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49590 and previous config saved to /var/cache/conftool/dbconfig/20230719-080309-root.json
07:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49586 and previous config saved to /var/cache/conftool/dbconfig/20230719-073300-root.json
07:26 marostegui@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49585 and previous config saved to /var/cache/conftool/dbconfig/20230719-072632-root.json
07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1180', diff saved to https://phabricator.wikimedia.org/P49584 and previous config saved to /var/cache/conftool/dbconfig/20230719-072207-root.json
07:20 dcausse@deploy1002: dcausse and abi: Backport for Add channel for TtmServerMessageUpdate of Translate extension synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
22:32 bking@cumin1001: START - Cookbook sre.ganeti.makevm for new host flink-zk1003.eqiad.wmnet
22:24 bking@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host flink-zk1003.eqiad.wmnet
22:24 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1003.eqiad.wmnet on all recursors
22:24 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1003.eqiad.wmnet on all recursors
22:24 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
22:24 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
22:23 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
22:18 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1003.eqiad.wmnet on all recursors
22:18 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1003.eqiad.wmnet on all recursors
22:18 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
22:18 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
22:16 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
21:15 urbanecm@deploy1002: jdlrobson and urbanecm: Backport for Don't log for documentElement (nodeType 9) (T340081) synced to the testservers mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
21:15 bking@cumin1001: START - Cookbook sre.hosts.reimage for host flink-zk1003.eqiad.wmnet with OS bookworm
21:15 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
21:14 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
21:14 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1003.eqiad.wmnet on all recursors
21:14 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1003.eqiad.wmnet on all recursors
21:14 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
21:14 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
21:13 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1003.eqiad.wmnet - bking@cumin1001"
21:10 bking@cumin1001: START - Cookbook sre.ganeti.makevm for new host flink-zk1003.eqiad.wmnet
21:03 bking@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host flink-zk1002.eqiad.wmnet
21:03 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1002.eqiad.wmnet on all recursors
21:03 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1002.eqiad.wmnet on all recursors
21:03 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
21:03 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM flink-zk1002.eqiad.wmnet - bking@cumin1001"
21:02 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM flink-zk1002.eqiad.wmnet - bking@cumin1001"
20:54 urbanecm@deploy1002: urbanecm and jdlrobson: Backport for Don't log for documentElement (nodeType 9) (T340081) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
20:43 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1002.eqiad.wmnet on all recursors
20:43 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1002.eqiad.wmnet on all recursors
20:43 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
20:43 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1002.eqiad.wmnet - bking@cumin1001"
20:41 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1002.eqiad.wmnet - bking@cumin1001"
20:07 urbanecm@deploy1002: jdlrobson and urbanecm: Backport for Deploy new logos (T341260 T341243 T341912) synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
17:25 bking@cumin1001: START - Cookbook sre.hosts.reimage for host flink-zk1002.eqiad.wmnet with OS bookworm
17:21 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM flink-zk1002.eqiad.wmnet - bking@cumin1001"
17:20 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM flink-zk1002.eqiad.wmnet - bking@cumin1001"
17:19 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1002.eqiad.wmnet on all recursors
17:19 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1002.eqiad.wmnet on all recursors
17:19 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:19 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1002.eqiad.wmnet - bking@cumin1001"
17:19 sukhe@cumin2002: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling reboot on A:wikidough-drmrs and A:wikidough
17:19 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1002.eqiad.wmnet - bking@cumin1001"
15:44 dancy@deploy1002: Started scap: testwikis wikis to 1.41.0-wmf.18 refs T340246
15:31 fabfur@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1016.eqiad.wmnet
15:31 fabfur@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:31 fabfur@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fabfur@cumin1001"
15:31 fabfur@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fabfur@cumin1001"
14:54 fabfur@cumin1001: START - Cookbook sre.hosts.decommission for hosts lvs1015.eqiad.wmnet
14:45 sukhe: dns2004 upgrade to pdns-rec 4.8.4: T341611
14:38 ayounsi@cumin1001: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.6.3 - ayounsi@cumin1001
14:37 fabfur@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1014.eqiad.wmnet
14:37 fabfur@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:37 fabfur@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fabfur@cumin1001"
14:36 fabfur@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fabfur@cumin1001"
14:36 ayounsi@cumin1001: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.6.3 - ayounsi@cumin1001
14:29 fabfur@cumin1001: START - Cookbook sre.hosts.decommission for hosts lvs1014.eqiad.wmnet
14:22 fabfur@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1013.eqiad.wmnet
14:22 fabfur@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:22 fabfur@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fabfur@cumin1001"
14:22 fabfur@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fabfur@cumin1001"
14:19 bking@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host flink-zk1001.eqiad.wmnet
14:19 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1001.eqiad.wmnet on all recursors
14:19 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1001.eqiad.wmnet on all recursors
14:19 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:19 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM flink-zk1001.eqiad.wmnet - bking@cumin1001"
14:18 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM flink-zk1001.eqiad.wmnet - bking@cumin1001"
14:16 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1001.eqiad.wmnet on all recursors
14:16 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1001.eqiad.wmnet on all recursors
14:16 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:16 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1001.eqiad.wmnet - bking@cumin1001"
14:10 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1001.eqiad.wmnet - bking@cumin1001"
14:05 XioNoX: asw2-esams# set interfaces xe-4/0/4 disable - T342121
14:04 fabfur@cumin1001: START - Cookbook sre.hosts.decommission for hosts lvs1013.eqiad.wmnet
13:58 jforrester@deploy1002: jforrester and daimona: Backport for private/readme.php: Add $wgCampaignEventsProgramsAndEventsDashboardAPISecret (T320260) synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:35 jforrester@deploy1002: daimona and jforrester: Backport for prod: Enable wgCampaignEventsProgramsAndEventsDashboardInstance (T320260) synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:20 stevemunene@deploy1002: Started deploy [airflow-dags/analytics_test@be05071]: (no justification provided)
13:20 jforrester@deploy1002: jforrester: Backport for Add wikifunctions.org to wgCentralNoticeContentSecurityPolicy (T275945) synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:06 jforrester@deploy1002: jforrester: Backport for Follow-up ca3aa70754: Drop 30x30px Notifications icons, unused for 7 years (T147219) synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:00 btullis@cumin1001: END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) restart masters for Hadoop test cluster: Restart of jvm daemons.
12:55 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host analytics1074.eqiad.wmnet with OS bullseye
12:42 btullis@cumin1001: START - Cookbook sre.hosts.dhcp for host analytics1073.eqiad.wmnet
12:40 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host analytics1073.eqiad.wmnet with OS bullseye
12:37 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
12:37 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
12:37 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
12:28 btullis@cumin1001: START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop test cluster: Restart of jvm daemons.
12:09 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
12:09 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
12:08 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
12:07 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1074.eqiad.wmnet with reason: host reimage
12:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy1015.eqiad.wmnet
12:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ladsgroup@cumin1001"
12:04 btullis@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1074.eqiad.wmnet with reason: host reimage
12:04 ladsgroup@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ladsgroup@cumin1001"
08:57 ladsgroup@deploy1002: isaranto and ladsgroup: Backport for ores: use envoy proxy for Lift Wing (T319170) synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
08:56 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2072.codfw.wmnet
08:56 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1072.eqiad.wmnet
06:36 cmooney@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cr3-knams,cr3-knams IPv6 with reason: Downtime cr3-knams ahead of remote hands moving router
06:36 cmooney@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cr3-knams,cr3-knams IPv6 with reason: Downtime cr3-knams ahead of remote hands moving router
21:15 bking@deploy1002: Started deploy [wdqs/wdqs@dff41b7]: 0.3.124
21:01 bking@cumin1001: START - Cookbook sre.hosts.reimage for host flink-zk1001.eqiad.wmnet with OS bookworm
21:00 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM flink-zk1001.eqiad.wmnet - bking@cumin1001"
21:00 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM flink-zk1001.eqiad.wmnet - bking@cumin1001"
20:59 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) flink-zk1001.eqiad.wmnet on all recursors
20:59 bking@cumin1001: START - Cookbook sre.dns.wipe-cache flink-zk1001.eqiad.wmnet on all recursors
20:59 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
20:59 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1001.eqiad.wmnet - bking@cumin1001"
20:51 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM flink-zk1001.eqiad.wmnet - bking@cumin1001"
20:43 bking@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
20:06 taavi@deploy1002: taavi and stang: Backport for bnwikiquote: Update wordmark (T341910) synced to the testservers mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
16:34 urbanecm@deploy1002: urbanecm: Backport for Fix UserDatabaseHelper::hasMainspaceEdits (T341994) synced to the testservers mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
13:33 btullis@cumin1001: START - Cookbook sre.hosts.reimage for host analytics1072.eqiad.wmnet with OS bullseye
13:32 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-be1059.eqiad.wmnet
13:28 taavi@deploy1002: taavi and urbanecm: Backport for NewImpact: fix undefined log function (T341865) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
12:54 elukey@deploy1002: helmfile [ml-serve-eqiad] 'sync' command on namespace 'ores-legacy' for release 'main' .
12:54 elukey@deploy1002: helmfile [ml-serve-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' .
12:53 elukey@deploy1002: helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' .
12:42 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2056.codfw.wmnet
12:01 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcontrol1005.wikimedia.org
12:01 aborrero@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:01 aborrero@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcontrol1005.wikimedia.org decommissioned, removing all IPs except the asset tag one - aborrero@cumin1001"
11:53 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1057.eqiad.wmnet
11:46 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-be1057.eqiad.wmnet
11:46 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2056.codfw.wmnet
11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2055.codfw.wmnet
11:45 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1056.eqiad.wmnet
11:39 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2055.codfw.wmnet
11:38 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-be1056.eqiad.wmnet
11:36 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1055.eqiad.wmnet
11:36 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2054.codfw.wmnet
11:35 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host analytics1070.eqiad.wmnet with OS bullseye
11:30 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-be1055.eqiad.wmnet
11:30 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2054.codfw.wmnet
11:29 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2053.codfw.wmnet
11:29 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1054.eqiad.wmnet
11:26 aborrero@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcontrol1005.wikimedia.org decommissioned, removing all IPs except the asset tag one - aborrero@cumin1001"
11:23 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2053.codfw.wmnet
15:33 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts sretest2002
15:33 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:33 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: sretest2002 decommissioned, removing all IPs except the asset tag one - pt1979@cumin2002"
15:32 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: sretest2002 decommissioned, removing all IPs except the asset tag one - pt1979@cumin2002"
07:47 ariel@deploy1002: ariel and superpes: Backport for [idwikiquote] Change the logo and add a wordmark (T341177) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
20:52 samtar@deploy1002: samtar and urbanecm: Backport for Fix mediawiki.special_diff_interactions configuration synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
20:24 samtar@deploy1002: jsn and samtar: Backport for log additional events on Special:Diff|MobileDiff (T326212) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
20:08 eevans@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase2013.codfw.wmnet: Applying JVM update - eevans@cumin1001
20:04 samtar@deploy1002: samtar: Backport for [ruwiki] Add permissions to 'editor' usergroup (T341707) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
16:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy1013.eqiad.wmnet
16:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ladsgroup@cumin1001"
16:38 ladsgroup@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ladsgroup@cumin1001"
15:11 btullis@cumin1001: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-druid-analytics cluster: Roll restart of jvm daemons.
15:08 bking@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
15:07 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host durum6001.drmrs.wmnet with OS bookworm
15:05 btullis@cumin1001: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-druid-analytics cluster: Roll restart of jvm daemons.
14:48 sukhe: upgrade dns2004 to gdnsd 3.99.0~alpha2
14:42 lucaswerkmeister-wmde@deploy1002: tgr and lucaswerkmeister-wmde: Backport for Temporarily allow OAuth on non-API entry points again (T341656) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
14:17 btullis@cumin1001: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-druid-public cluster: Roll restart of jvm daemons.
14:11 btullis@cumin1001: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-druid-public cluster: Roll restart of jvm daemons.
14:07 sukhe: dns4003: upgrade to pdns-rec 4.8.4: T341611
13:31 btullis@cumin1001: END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid analytics cluster: Roll restart of Druid jvm daemons.
13:31 lucaswerkmeister-wmde@deploy1002: daimona and lucaswerkmeister-wmde: Backport for Add new campaign_events.event_answers_status column (T341142) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
13:23 lucaswerkmeister-wmde@deploy1002: lucaswerkmeister-wmde: Backport for Html: Support more attr types in getTextInputAttributes() (T341566) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
13:22 bking@deploy1002: Started deploy [wdqs/wdqs@dff41b7]: 0.3.124
01:47 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6 days, 19:00:00 on wdqs[2013,2022].codfw.wmnet with reason: new host
01:46 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 6 days, 19:00:00 on wdqs[2013,2022].codfw.wmnet with reason: new host
2023-07-11
21:51 bking@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
20:26 urbanecm@deploy1002: jdlrobson and urbanecm: Backport for Logos: Fixes grantswiki and idwiktionary synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
15:54 lucaswerkmeister-wmde@deploy1002: jdlrobson and lucaswerkmeister-wmde: Backport for Add option for html label in Menu template (T340217) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
15:22 krinkle@deploy1002: phedenskog and krinkle: Backport for Remove oversampling for Navigation Timing extension. (T337858) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
14:49 btullis@cumin1001: END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid public cluster: Roll restart of Druid jvm daemons.
14:21 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/datahub: sync on main
14:19 btullis@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) restart MirrorMaker for Kafka A:kafka-mirror-maker-jumbo-eqiad cluster: Roll restart of jvm daemons.
14:17 moritzm: restarting apache on mw canaries
14:17 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/datahub: apply on main
14:15 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
14:12 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
14:02 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
14:00 btullis@cumin1001: START - Cookbook sre.kafka.roll-restart-mirror-maker restart MirrorMaker for Kafka A:kafka-mirror-maker-jumbo-eqiad cluster: Roll restart of jvm daemons.
13:59 moritzm: installing yajl security updates
13:59 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
13:57 btullis@cumin1001: START - Cookbook sre.druid.roll-restart-workers for Druid public cluster: Roll restart of Druid jvm daemons.
13:49 moritzm: rebalance ganeti group eqiad/d after reboots
09:43 hashar: Updating Zuul configuration which was stall to a version from March 29th after the switchover from contint2001 to contint2002 | T324659T341556
09:36 jbond: deploy gerrit:936273 enable submitting data to puppetdb7
09:30 jbond: disable puppet fleet wide to deploy 936273
09:08 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host kafkamon1003.eqiad.wmnet
08:40 volans: downtiming service 'Check no envoy runtime configuration is left persistent' on envoy hosts
08:39 jayme: disabled puppet on 'P{R:Package = envoyproxy}'
08:19 godog: upgrade prometheus to 2.24.1+ds-1+wmf2 on cloudmetrics*
08:03 hashar: Stopping Jenkins and Zuul for server switch over
08:01 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on contint2002.wikimedia.org with reason: Switch contint hosts for hardware replacement
08:01 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on contint2002.wikimedia.org with reason: Switch contint hosts for hardware replacement
08:01 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on contint2001.wikimedia.org with reason: Switch contint hosts for hardware replacement
08:01 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on contint2001.wikimedia.org with reason: Switch contint hosts for hardware replacement
07:55 kart_: Updated MinT to 2023-07-10-051738-production (T341335, T333969)
20:42 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
20:36 samtar@deploy1002: samtar: Backport for Revert "log additional events on Special:Diff|MobileDiff" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
20:03 samtar@deploy1002: samtar and jsn: Backport for log additional events on Special:Diff|MobileDiff (T326212) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
18:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.decommission for hosts dbproxy1012.eqiad.wmnet
18:50 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dns[1002-1003].wikimedia.org
18:50 sukhe@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
18:50 sukhe@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns[1002-1003].wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002"
18:49 ladsgroup@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts dbproxy1012.eqiad.wmnet
18:49 ladsgroup@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
18:49 sukhe@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns[1002-1003].wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002"
13:27 btullis@cumin1001: START - Cookbook sre.hosts.reimage for host karapace1002.eqiad.wmnet with OS bullseye
13:22 btullis@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM karapace1002.eqiad.wmnet - btullis@cumin1001"
13:21 btullis@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM karapace1002.eqiad.wmnet - btullis@cumin1001"
13:21 btullis@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) karapace1002.eqiad.wmnet on all recursors
13:21 btullis@cumin1001: START - Cookbook sre.dns.wipe-cache karapace1002.eqiad.wmnet on all recursors
13:21 btullis@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:21 btullis@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM karapace1002.eqiad.wmnet - btullis@cumin1001"
13:20 btullis@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM karapace1002.eqiad.wmnet - btullis@cumin1001"
13:20 ladsgroup@deploy1002: isaranto and ladsgroup: Backport for ores extension: deploy LiftWing usage on testwiki (T319170) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
13:16 btullis@cumin1001: START - Cookbook sre.ganeti.makevm for new host karapace1002.eqiad.wmnet
13:07 lucaswerkmeister-wmde@deploy1002: lucaswerkmeister-wmde and arlolra: Backport for Disable wgParserEnableLegacyMediaDOM on group2 wikis (T314318) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
09:52 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1033.eqiad.wmnet
09:50 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp2037.codfw.wmnet with reason: vgutierrez debugging
09:50 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cp2037.codfw.wmnet with reason: vgutierrez debugging
09:44 aborrero@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cloudlb1002.private.eqiad.wikimedia.cloud on all recursors
09:44 aborrero@cumin1001: START - Cookbook sre.dns.wipe-cache cloudlb1002.private.eqiad.wikimedia.cloud on all recursors
09:39 moritzm: rebalance ganeti group codfw/B after reboots
09:38 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudlb1001.eqiad.wmnet with reason: host reimage
08:04 moritzm: installing c-ares security updates on buster
08:04 hashar@deploy1002: hashar and mdsshakil: Backport for Deploy action blocks on bnwiki (T340904) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
07:54 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet
07:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet
07:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet
07:45 hashar@deploy1002: func and hashar: Backport for thwiki: Update logos from commons (T341407) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
14:51 apergos: swapped dumpsdata1003 in as the new nfs share for misc dumps; dumpsdata1002 is now a spare, to be decommissioned. 1003 is running bullseye.
04:04 apergos: rsync misc dumps output files from dumpsdata1002 to 1003, in ariel screen session on 1003, bwlimit to 1G
09:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1002.eqiad.wmnet
09:55 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetdb2003.codfw.wmnet
09:55 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow1002.eqiad.wmnet
09:52 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host debmonitor2003.codfw.wmnet
09:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2003.codfw.wmnet
09:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow2003.codfw.wmnet
09:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet
09:45 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hadoop.roll-restart-masters (exit_code=99) restart masters for Hadoop analytics cluster: Restart of jvm daemons.
09:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet
09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow6001.drmrs.wmnet
09:35 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet
09:34 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host lists1003.wikimedia.org
09:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow6001.drmrs.wmnet
09:29 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow6001.drmrs.wmnet
09:29 stevemunene@cumin1001: START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop analytics cluster: Restart of jvm daemons.
09:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow6001.drmrs.wmnet
09:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow3002.esams.wmnet
09:24 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host lists1003.wikimedia.org
09:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people1004.eqiad.wmnet
09:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host people1004.eqiad.wmnet
09:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow3002.esams.wmnet
09:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow5002.eqsin.wmnet
09:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people2003.codfw.wmnet
09:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host people2003.codfw.wmnet
09:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow5002.eqsin.wmnet
08:53 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
08:50 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
08:48 moritzm: installing bookworm kernel updates
08:47 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: xhgui2002.codfw.wmnet
08:47 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: xhgui2002.codfw.wmnet
08:46 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: xhgui1002.eqiad.wmnet
08:46 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: xhgui1002.eqiad.wmnet
08:05 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on kafka-test[1006-1010].eqiad.wmnet with reason: resetting cluster
08:05 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 0:30:00 on kafka-test[1006-1010].eqiad.wmnet with reason: resetting cluster
01:55 bking@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
23:14 mutante: mx1001 - rm /usr/local/bin/otrs_aliases ; rm /lib/systemd/system/generate_otrs_aliases.* after deploying gerrit:932316 which renamed script and timer without absenting them
23:08 mutante: mx2001 - rm /usr/local/bin/otrs_aliases ; rm /lib/systemd/system/generate_otrs_aliases.* after deploying gerrit:932316 which renamed script and timer without absenting them
21:12 thcipriani@deploy1002: Finished scap: Clean up font directory gerrit:723652 (duration: 06m 33s)
20:54 bking@deploy1002: Started deploy [wdqs/wdqs@dff41b7]: 0.3.124
20:53 thcipriani@deploy1002: stang and thcipriani: Backport for pawikibooks: Install Quiz extension (T340613) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
20:37 thcipriani@deploy1002: jdlrobson and thcipriani: Backport for Update more logos with available SVGs (T338162) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
17:04 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dns1001.wikimedia.org
17:04 sukhe@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:04 sukhe@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002"
17:02 sukhe@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002"
15:54 elukey: changeprop's kafka linger.ms set to 20s - T338357 (was 5ms, now changeprop waits a bit more to batch messages to send to kafka in one go)
14:51 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
14:49 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-worker1003.eqiad.wmnet with reason: host reimage
14:47 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
14:46 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts analytics1069.eqiad.wmnet
14:46 stevemunene@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:46 stevemunene@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1069.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
14:46 btullis@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-worker1003.eqiad.wmnet with reason: host reimage
14:45 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host dns1004.wikimedia.org with OS bullseye
14:45 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns1004.wikimedia.org with OS bullseye
14:42 stevemunene@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1069.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
14:31 stevemunene@cumin1001: START - Cookbook sre.hosts.decommission for hosts analytics1069.eqiad.wmnet
14:29 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts analytics1068.eqiad.wmnet
14:29 stevemunene@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:29 stevemunene@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1068.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
14:28 btullis@cumin1001: START - Cookbook sre.hosts.reimage for host an-test-worker1003.eqiad.wmnet with OS bullseye
14:27 stevemunene@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1068.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
14:27 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-test-worker1003.eqiad.wmnet with OS bullseye
14:20 stevemunene@cumin1001: START - Cookbook sre.hosts.decommission for hosts analytics1068.eqiad.wmnet
14:19 aborrero@cumin1001: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudlb1001
14:19 aborrero@cumin1001: START - Cookbook sre.network.configure-switch-interfaces for host cloudlb1001
14:18 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts analytics1067.eqiad.wmnet
14:18 stevemunene@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:18 stevemunene@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1067.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
14:14 stevemunene@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1067.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
14:14 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host dns1004.wikimedia.org with OS bullseye
14:13 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns1004.wikimedia.org with OS bullseye
14:13 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
13:54 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
13:51 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudlb1001.eqiad.wmnet
13:42 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
13:38 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host cloudlb1001.eqiad.wmnet
13:34 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
13:33 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zookeeper-test1002.eqiad.wmnet with OS bookworm
13:32 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host an-test-worker1003.eqiad.wmnet
13:30 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts analytics1066.eqiad.wmnet
13:30 stevemunene@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:30 stevemunene@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1066.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
13:29 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
13:29 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host an-test-worker1003.eqiad.wmnet
13:29 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host an-test-worker1003.eqiad.wmnet
13:24 stevemunene@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1066.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
13:17 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker1095.eqiad.wmnet with reason: Replacing RAID controller battery
13:17 btullis@cumin1001: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on an-worker1095.eqiad.wmnet with reason: Replacing RAID controller battery
13:14 stevemunene@cumin1001: START - Cookbook sre.hosts.decommission for hosts analytics1066.eqiad.wmnet
13:12 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts analytics1065.eqiad.wmnet
13:12 stevemunene@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:12 stevemunene@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1065.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
13:10 btullis@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons.
13:10 stevemunene@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1065.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
13:02 stevemunene@cumin1001: START - Cookbook sre.hosts.decommission for hosts analytics1065.eqiad.wmnet
13:00 aborrero@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudlb1001.eqiad.wmnet with OS bullseye
12:58 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zookeeper-test1002.eqiad.wmnet with reason: host reimage
12:58 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts analytics1064.eqiad.wmnet
12:58 stevemunene@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:58 stevemunene@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1064.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
12:56 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on zookeeper-test1002.eqiad.wmnet with reason: host reimage
12:43 stevemunene@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1064.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
12:42 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host zookeeper-test1002.eqiad.wmnet with OS bookworm
12:21 samtar@deploy1002: matmarex and samtar: Backport for Revert "Add tag when reference added to the page" (T341202) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
12:15 elukey@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zookeeper-test1002.eqiad.wmnet with OS bookworm
12:15 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host zookeeper-test1002.eqiad.wmnet with OS bookworm
11:30 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudswift1002.eqiad.wmnet
11:30 aborrero@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:30 aborrero@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudswift1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - aborrero@cumin1001"
11:29 aborrero@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudswift1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - aborrero@cumin1001"
11:27 lucaswerkmeister-wmde@deploy1002: migr and lucaswerkmeister-wmde: Backport for Beta-Wikidata: Always show mul on desktop Termbox (T339104) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
11:27 btullis@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons.
11:17 lucaswerkmeister-wmde@deploy1002: varnent and lucaswerkmeister-wmde: Backport for foundationwiki: Enable WikibaseClient (T321967) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
11:10 aborrero@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudswift1001.eqiad.wmnet
11:07 lucaswerkmeister-wmde@deploy1002: lucaswerkmeister-wmde: Backport for outreachwiki: Set wmgWikibaseSiteGroup synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
11:05 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts analytics1062.eqiad.wmnet
11:05 stevemunene@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:05 stevemunene@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1062.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
11:04 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
11:03 mvernon@cumin1001: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling reboot on A:swift-fe
10:54 stevemunene@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1062.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
10:53 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
10:51 taavi@deploy1002: taavi: Backport for extdist: REL1_40 is stable, REL1_38 is EOL synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
10:41 stevemunene@cumin1001: START - Cookbook sre.hosts.decommission for hosts analytics1062.eqiad.wmnet
10:10 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts analytics1061.eqiad.wmnet
10:10 stevemunene@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
10:10 stevemunene@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1061.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
10:08 stevemunene@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1061.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001"
08:45 fabfur: reenabled puppet on cp1075.eqiad.wmnet, cp2027.codfw.wmnet, cp3050.esams.wmnet
08:39 mvernon@cumin1001: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling reboot on A:thanos-fe
08:17 fabfur: disabling puppet temporary on cp1075.eqiad.wmnet, cp2027.codfw.wmnet, cp3050.esams.wmnet to apply 935760 (T340983)
08:03 jelto@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: GitLab minor version upgrade
07:31 kart_: Updated MinT to 2023-07-06-051402-production
07:29 mvernon@cumin1001: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling reboot on A:thanos-fe
07:04 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on 9 hosts with reason: Stopping puppet and hadoop-hdfs-datanode services then decommissioning the hosts
07:04 stevemunene@cumin1001: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on 9 hosts with reason: Stopping puppet and hadoop-hdfs-datanode services then decommissioning the hosts
06:54 jelto@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: GitLab minor version upgrade
22:52 bking@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
22:38 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host an-test-worker1003.eqiad.wmnet
22:36 denisse@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts xhgui2002
22:36 denisse@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
22:36 denisse@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: xhgui2002 decommissioned, removing all IPs except the asset tag one - denisse@cumin1001"
22:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host an-test-worker1003.eqiad.wmnet
22:35 denisse@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: xhgui2002 decommissioned, removing all IPs except the asset tag one - denisse@cumin1001"
22:28 denisse@cumin1001: START - Cookbook sre.hosts.decommission for hosts xhgui2002
22:28 denisse@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts xhgui1002
22:28 denisse@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
22:28 denisse@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: xhgui1002 decommissioned, removing all IPs except the asset tag one - denisse@cumin1001"
22:27 denisse@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: xhgui1002 decommissioned, removing all IPs except the asset tag one - denisse@cumin1001"
20:45 urbanecm@deploy1002: jdlrobson and urbanecm: Backport for Update various logos where SVGs are available (T338162) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
20:05 urbanecm@deploy1002: urbanecm and jdlrobson: Backport for Disable the Nearby feature on some sister projects (T341133) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
18:36 sukhe: re-enable puppet in A:dns-rec to finish merging CR 933497 and run-agent: T340479
18:25 denisse: disable puppet on webperf1003 to test PHP memory changes for XHGui
18:25 denisse: disable puppet on webperf1003
18:20 sukhe: disable puppet on A:dns-rec to merge CR 933497
17:40 jelto@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: GitLab minor version upgrade
16:33 jelto@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: GitLab minor version upgrade
16:25 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host an-test-worker1003.eqiad.wmnet
16:25 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host an-test-worker1003.eqiad.wmnet
16:06 jelto@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: GitLab minor version upgrade
15:55 fabfur: re-enabled puppet in all cp- hosts (done @2023-07-05 14:22:57 UTC)
14:48 btullis@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-jumbo-eqiad cluster: Roll restart of jvm daemons.
14:44 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2013.codfw.wmnet
14:44 vgutierrez@cumin1001: START - Cookbook sre.hosts.remove-downtime for lvs2013.codfw.wmnet
14:42 sukhe: re-enable puppet and start pybal on lvs2013
14:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1022.eqiad.wmnet
14:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1022.eqiad.wmnet
14:38 vgutierrez: pool cdn service in cp2027.codfw.wmnet,cp1075.eqiad.wmnet,cp3050.esams.wmnet
14:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1022.eqiad.wmnet
14:29 Lucas_WMDE: UTC afternoon backport+config window done
13:57 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1021.eqiad.wmnet
13:55 elukey: expand kafka topic partitions from 1 to 5 for {codfw,eqiad}.mediawiki.job.RecordLintJob and {eqiad,codfw}.mediawiki.job.refreshLinks on kafka-main eqiad/codfw - T338357
13:51 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs2013.codfw.wmnet with reason: mgmt interface issues
13:51 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on lvs2013.codfw.wmnet with reason: mgmt interface issues
13:45 lucaswerkmeister-wmde@deploy1002: lucaswerkmeister-wmde and anzx: Backport for Enable Extension:RelatedArticles for desktop on frwikinews (T341105) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
12:23 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow4002.ulsfo.wmnet
12:22 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1018.eqiad.wmnet
11:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts urldownloader1001.wikimedia.org
11:59 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:59 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: urldownloader1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
11:58 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: urldownloader1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
11:51 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts urldownloader1001.wikimedia.org
11:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts urldownloader1002.wikimedia.org
11:50 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:50 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: urldownloader1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
11:45 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: urldownloader1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
11:39 btullis@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-jumbo-eqiad cluster: Roll restart of jvm daemons.
11:37 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts urldownloader1002.wikimedia.org
11:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts urldownloader2002.wikimedia.org
11:36 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:36 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: urldownloader2002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
11:34 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: urldownloader2002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
11:32 btullis@cumin1001: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto analytics cluster: Roll restart of all Presto's jvm daemons.
11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1017.eqiad.wmnet
11:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1017.eqiad.wmnet
11:25 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts urldownloader2002.wikimedia.org
11:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts urldownloader2001.wikimedia.org
11:25 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:25 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: urldownloader2001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
11:21 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1017.eqiad.wmnet
11:09 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: urldownloader2001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
08:34 btullis@cumin1001: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
08:26 btullis@cumin1001: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1012.eqiad.wmnet
08:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1012.eqiad.wmnet
08:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1012.eqiad.wmnet
22:52 zabe@deploy1002: taavi and zabe: Backport for Remove migrateStewards.php reference synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
13:59 lucaswerkmeister-wmde@deploy1002: lucaswerkmeister-wmde and urbanecm: Backport for DeleteAction: Avoid displaying the form unconditionally (T341002) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
13:40 jayme@deploy1002: helmfile [eqiad] OK helmfile.d/services/zotero: apply
13:39 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
13:38 jayme@deploy1002: helmfile [eqiad] OK helmfile.d/services/wikifeeds: apply
13:37 jayme@deploy1002: helmfile [eqiad] OK helmfile.d/services/toolhub: apply
13:37 jayme@deploy1002: helmfile [eqiad] OK helmfile.d/services/termbox: apply
13:37 lucaswerkmeister-wmde@deploy1002: urbanecm and lucaswerkmeister-wmde: Backport for DeleteAction: Avoid displaying the form unconditionally (T341002) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
13:36 jayme@deploy1002: helmfile [eqiad] OK helmfile.d/services/tegola-vector-tiles: apply
13:36 jayme@deploy1002: helmfile [eqiad] OK helmfile.d/services/similar-users: apply
09:07 btullis@cumin1001: END (FAIL) - Cookbook sre.hadoop.roll-restart-workers (exit_code=99) restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade.
09:02 jnuche@deploy1002: Started scap: testwikis wikis to 1.41.0-wmf.16 refs T340244
08:56 btullis@cumin1001: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade.
20:18 jiji@cumin1001: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:eqiad and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad)
20:15 effie: restarting swift proxies
20:14 jiji@cumin1001: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:eqiad and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad)
13:22 volans@cumin1001: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
13:22 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1011.eqiad.wmnet
13:21 urbanecm: Run `wikiadmin2023@10.64.16.184(idwiki)> DELETE FROM `category` WHERE cat_title = ; ` (T336780)
13:20 urbanecm@deploy1002: func and urbanecm: Backport for SpecialLog: Fix issues related to IP users (T338042 T340929) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1010.eqiad.wmnet
13:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1010.eqiad.wmnet
13:04 urbanecm@deploy1002: jhsoby and urbanecm: Backport for Set wgCollectionDisableSidebarLink for nowiki (T340981) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
07:25 taavi@deploy1002: msz2001 and taavi: Backport for Update plwiki autopromote per consensus (T340397) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
07:13 taavi@deploy1002: soda and taavi: Backport for Enable edit-in-sequence in Italian Wikisource (T340847) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
