Server Admin Log/Archive 52

The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

2022-04-29

23:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T306560)', diff saved to https://phabricator.wikimedia.org/P27163 and previous config saved to /var/cache/conftool/dbconfig/20220429-231136-ladsgroup.json
22:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P27162 and previous config saved to /var/cache/conftool/dbconfig/20220429-225631-ladsgroup.json
22:49 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2003.codfw.wmnet with OS bullseye
22:48 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2004.codfw.wmnet with OS bullseye
22:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P27161 and previous config saved to /var/cache/conftool/dbconfig/20220429-224125-ladsgroup.json
22:37 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2003.codfw.wmnet with reason: host reimage
22:33 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2003.codfw.wmnet with reason: host reimage
22:28 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2003.codfw.wmnet with OS bullseye
22:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T306560)', diff saved to https://phabricator.wikimedia.org/P27160 and previous config saved to /var/cache/conftool/dbconfig/20220429-222620-ladsgroup.json
22:25 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2003.codfw.wmnet with OS bullseye
22:16 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2004.codfw.wmnet with OS bullseye
22:15 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2004.codfw.wmnet with OS bullseye
22:15 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2004.codfw.wmnet with OS bullseye
22:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T306560)', diff saved to https://phabricator.wikimedia.org/P27159 and previous config saved to /var/cache/conftool/dbconfig/20220429-221331-ladsgroup.json
22:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
22:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
22:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T306560)', diff saved to https://phabricator.wikimedia.org/P27158 and previous config saved to /var/cache/conftool/dbconfig/20220429-221323-ladsgroup.json
21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P27157 and previous config saved to /var/cache/conftool/dbconfig/20220429-215818-ladsgroup.json
21:54 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2003.codfw.wmnet with OS bullseye
21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P27156 and previous config saved to /var/cache/conftool/dbconfig/20220429-214313-ladsgroup.json
21:42 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2002.codfw.wmnet with OS bullseye
21:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
21:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
21:29 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2002.codfw.wmnet with reason: host reimage
21:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T306560)', diff saved to https://phabricator.wikimedia.org/P27155 and previous config saved to /var/cache/conftool/dbconfig/20220429-212808-ladsgroup.json
21:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
21:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
21:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
21:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
21:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T306560)', diff saved to https://phabricator.wikimedia.org/P27154 and previous config saved to /var/cache/conftool/dbconfig/20220429-212652-ladsgroup.json
21:26 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2002.codfw.wmnet with reason: host reimage
21:21 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2002.codfw.wmnet with OS bullseye
21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T306560)', diff saved to https://phabricator.wikimedia.org/P27153 and previous config saved to /var/cache/conftool/dbconfig/20220429-211609-ladsgroup.json
21:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
21:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T306560)', diff saved to https://phabricator.wikimedia.org/P27152 and previous config saved to /var/cache/conftool/dbconfig/20220429-211601-ladsgroup.json
21:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P27151 and previous config saved to /var/cache/conftool/dbconfig/20220429-211146-ladsgroup.json
21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P27150 and previous config saved to /var/cache/conftool/dbconfig/20220429-210055-ladsgroup.json
20:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P27149 and previous config saved to /var/cache/conftool/dbconfig/20220429-205641-ladsgroup.json
20:46 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2002.codfw.wmnet with OS bullseye
20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P27148 and previous config saved to /var/cache/conftool/dbconfig/20220429-204550-ladsgroup.json
20:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T306560)', diff saved to https://phabricator.wikimedia.org/P27147 and previous config saved to /var/cache/conftool/dbconfig/20220429-204136-ladsgroup.json
20:31 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2001.codfw.wmnet with OS bullseye
20:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T306560)', diff saved to https://phabricator.wikimedia.org/P27146 and previous config saved to /var/cache/conftool/dbconfig/20220429-203045-ladsgroup.json
20:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27145 and previous config saved to /var/cache/conftool/dbconfig/20220429-202824-ladsgroup.json
20:19 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2001.codfw.wmnet with reason: host reimage
20:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T306560)', diff saved to https://phabricator.wikimedia.org/P27144 and previous config saved to /var/cache/conftool/dbconfig/20220429-201753-ladsgroup.json
20:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
20:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
20:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T306560)', diff saved to https://phabricator.wikimedia.org/P27143 and previous config saved to /var/cache/conftool/dbconfig/20220429-201745-ladsgroup.json
20:16 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2001.codfw.wmnet with reason: host reimage
20:15 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2002.codfw.wmnet with OS bullseye
20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27142 and previous config saved to /var/cache/conftool/dbconfig/20220429-201319-ladsgroup.json
20:11 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2001.codfw.wmnet with OS bullseye
20:08 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2001.codfw.wmnet with OS bullseye
20:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P27141 and previous config saved to /var/cache/conftool/dbconfig/20220429-200240-ladsgroup.json
19:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27140 and previous config saved to /var/cache/conftool/dbconfig/20220429-195813-ladsgroup.json
19:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P27139 and previous config saved to /var/cache/conftool/dbconfig/20220429-194735-ladsgroup.json
19:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27138 and previous config saved to /var/cache/conftool/dbconfig/20220429-194308-ladsgroup.json
19:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T306560)', diff saved to https://phabricator.wikimedia.org/P27137 and previous config saved to /var/cache/conftool/dbconfig/20220429-194122-ladsgroup.json
19:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
19:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
19:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27136 and previous config saved to /var/cache/conftool/dbconfig/20220429-193649-ladsgroup.json
19:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
19:36 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2001.codfw.wmnet with OS bullseye
19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T306560)', diff saved to https://phabricator.wikimedia.org/P27135 and previous config saved to /var/cache/conftool/dbconfig/20220429-193624-ladsgroup.json
19:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27134 and previous config saved to /var/cache/conftool/dbconfig/20220429-193549-ladsgroup.json
19:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T306560)', diff saved to https://phabricator.wikimedia.org/P27133 and previous config saved to /var/cache/conftool/dbconfig/20220429-193230-ladsgroup.json
19:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P27132 and previous config saved to /var/cache/conftool/dbconfig/20220429-192119-ladsgroup.json
19:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27131 and previous config saved to /var/cache/conftool/dbconfig/20220429-192044-ladsgroup.json
19:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T306560)', diff saved to https://phabricator.wikimedia.org/P27130 and previous config saved to /var/cache/conftool/dbconfig/20220429-191932-ladsgroup.json
19:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
19:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
19:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
19:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P27129 and previous config saved to /var/cache/conftool/dbconfig/20220429-190614-ladsgroup.json
19:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27128 and previous config saved to /var/cache/conftool/dbconfig/20220429-190539-ladsgroup.json
18:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
18:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T306560)', diff saved to https://phabricator.wikimedia.org/P27127 and previous config saved to /var/cache/conftool/dbconfig/20220429-185705-ladsgroup.json
18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T306560)', diff saved to https://phabricator.wikimedia.org/P27126 and previous config saved to /var/cache/conftool/dbconfig/20220429-185109-ladsgroup.json
18:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27125 and previous config saved to /var/cache/conftool/dbconfig/20220429-185034-ladsgroup.json
18:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance
18:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance
18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298295)', diff saved to https://phabricator.wikimedia.org/P27124 and previous config saved to /var/cache/conftool/dbconfig/20220429-184506-ladsgroup.json
18:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27123 and previous config saved to /var/cache/conftool/dbconfig/20220429-184411-ladsgroup.json
18:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
18:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
18:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
18:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
18:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P27122 and previous config saved to /var/cache/conftool/dbconfig/20220429-184313-ladsgroup.json
18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P27121 and previous config saved to /var/cache/conftool/dbconfig/20220429-184200-ladsgroup.json
18:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance
18:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance
18:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance
18:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance
18:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1100.eqiad.wmnet with reason: Maintenance
18:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1100.eqiad.wmnet with reason: Maintenance
18:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P27120 and previous config saved to /var/cache/conftool/dbconfig/20220429-183001-ladsgroup.json
18:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P27119 and previous config saved to /var/cache/conftool/dbconfig/20220429-182807-ladsgroup.json
18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T306560)', diff saved to https://phabricator.wikimedia.org/P27118 and previous config saved to /var/cache/conftool/dbconfig/20220429-182714-ladsgroup.json
18:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
18:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
18:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
18:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T306560)', diff saved to https://phabricator.wikimedia.org/P27117 and previous config saved to /var/cache/conftool/dbconfig/20220429-182700-ladsgroup.json
18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P27116 and previous config saved to /var/cache/conftool/dbconfig/20220429-182653-ladsgroup.json
18:21 mvernon@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ms-be1040.eqiad.wmnet
18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P27115 and previous config saved to /var/cache/conftool/dbconfig/20220429-181456-ladsgroup.json
18:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P27114 and previous config saved to /var/cache/conftool/dbconfig/20220429-181302-ladsgroup.json
18:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P27113 and previous config saved to /var/cache/conftool/dbconfig/20220429-181153-ladsgroup.json
18:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T306560)', diff saved to https://phabricator.wikimedia.org/P27112 and previous config saved to /var/cache/conftool/dbconfig/20220429-181145-ladsgroup.json
18:11 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-be1040.eqiad.wmnet
17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298295)', diff saved to https://phabricator.wikimedia.org/P27111 and previous config saved to /var/cache/conftool/dbconfig/20220429-175951-ladsgroup.json
17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T306560)', diff saved to https://phabricator.wikimedia.org/P27110 and previous config saved to /var/cache/conftool/dbconfig/20220429-175903-ladsgroup.json
17:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
17:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
17:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T306560)', diff saved to https://phabricator.wikimedia.org/P27109 and previous config saved to /var/cache/conftool/dbconfig/20220429-175855-ladsgroup.json
17:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T298295)', diff saved to https://phabricator.wikimedia.org/P27108 and previous config saved to /var/cache/conftool/dbconfig/20220429-175841-ladsgroup.json
17:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
17:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
17:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298295)', diff saved to https://phabricator.wikimedia.org/P27107 and previous config saved to /var/cache/conftool/dbconfig/20220429-175833-ladsgroup.json
17:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P27106 and previous config saved to /var/cache/conftool/dbconfig/20220429-175757-ladsgroup.json
17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P27105 and previous config saved to /var/cache/conftool/dbconfig/20220429-175642-ladsgroup.json
17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P27104 and previous config saved to /var/cache/conftool/dbconfig/20220429-175122-ladsgroup.json
17:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
17:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27103 and previous config saved to /var/cache/conftool/dbconfig/20220429-175114-ladsgroup.json
17:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P27102 and previous config saved to /var/cache/conftool/dbconfig/20220429-174350-ladsgroup.json
17:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P27101 and previous config saved to /var/cache/conftool/dbconfig/20220429-174328-ladsgroup.json
17:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T306560)', diff saved to https://phabricator.wikimedia.org/P27100 and previous config saved to /var/cache/conftool/dbconfig/20220429-174136-ladsgroup.json
17:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T276292)', diff saved to https://phabricator.wikimedia.org/P27099 and previous config saved to /var/cache/conftool/dbconfig/20220429-174129-ladsgroup.json
17:36 Amir1: killed bnwiki's refresh links recommendation (T299021)
17:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27098 and previous config saved to /var/cache/conftool/dbconfig/20220429-173609-ladsgroup.json
17:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P27097 and previous config saved to /var/cache/conftool/dbconfig/20220429-172845-ladsgroup.json
17:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P27096 and previous config saved to /var/cache/conftool/dbconfig/20220429-172823-ladsgroup.json
17:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P27095 and previous config saved to /var/cache/conftool/dbconfig/20220429-172623-ladsgroup.json
17:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27094 and previous config saved to /var/cache/conftool/dbconfig/20220429-172104-ladsgroup.json
17:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T306560)', diff saved to https://phabricator.wikimedia.org/P27093 and previous config saved to /var/cache/conftool/dbconfig/20220429-171658-ladsgroup.json
17:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
17:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
17:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T306560)', diff saved to https://phabricator.wikimedia.org/P27092 and previous config saved to /var/cache/conftool/dbconfig/20220429-171650-ladsgroup.json
17:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T306560)', diff saved to https://phabricator.wikimedia.org/P27091 and previous config saved to /var/cache/conftool/dbconfig/20220429-171339-ladsgroup.json
17:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298295)', diff saved to https://phabricator.wikimedia.org/P27090 and previous config saved to /var/cache/conftool/dbconfig/20220429-171318-ladsgroup.json
17:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P27089 and previous config saved to /var/cache/conftool/dbconfig/20220429-171118-ladsgroup.json
17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27088 and previous config saved to /var/cache/conftool/dbconfig/20220429-170559-ladsgroup.json
17:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T306560)', diff saved to https://phabricator.wikimedia.org/P27087 and previous config saved to /var/cache/conftool/dbconfig/20220429-170205-ladsgroup.json
17:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
17:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
17:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
17:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
17:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
17:02 jynus@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup1002.eqiad.wmnet with OS buster
17:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
17:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
17:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
17:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P27086 and previous config saved to /var/cache/conftool/dbconfig/20220429-170145-ladsgroup.json
17:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
17:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
17:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
17:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
17:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
17:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
17:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298295)', diff saved to https://phabricator.wikimedia.org/P27085 and previous config saved to /var/cache/conftool/dbconfig/20220429-170125-ladsgroup.json
16:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27084 and previous config saved to /var/cache/conftool/dbconfig/20220429-165939-ladsgroup.json
16:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
16:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
16:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
16:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
16:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27083 and previous config saved to /var/cache/conftool/dbconfig/20220429-165839-ladsgroup.json
16:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T276292)', diff saved to https://phabricator.wikimedia.org/P27082 and previous config saved to /var/cache/conftool/dbconfig/20220429-165613-ladsgroup.json
16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T276292)', diff saved to https://phabricator.wikimedia.org/P27081 and previous config saved to /var/cache/conftool/dbconfig/20220429-165333-ladsgroup.json
16:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
16:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
16:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
16:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
16:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
16:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T306560)', diff saved to https://phabricator.wikimedia.org/P27080 and previous config saved to /var/cache/conftool/dbconfig/20220429-165035-ladsgroup.json
16:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P27079 and previous config saved to /var/cache/conftool/dbconfig/20220429-164640-ladsgroup.json
16:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P27078 and previous config saved to /var/cache/conftool/dbconfig/20220429-164620-ladsgroup.json
16:43 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab1004.wikimedia.org with OS bullseye
16:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27077 and previous config saved to /var/cache/conftool/dbconfig/20220429-164333-ladsgroup.json
16:41 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab1003.wikimedia.org with OS bullseye
16:37 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner1004.eqiad.wmnet with OS bullseye
16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P27076 and previous config saved to /var/cache/conftool/dbconfig/20220429-163530-ladsgroup.json
16:32 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab1004.wikimedia.org with reason: host reimage
16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T306560)', diff saved to https://phabricator.wikimedia.org/P27075 and previous config saved to /var/cache/conftool/dbconfig/20220429-163135-ladsgroup.json
16:31 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner1003.eqiad.wmnet with OS bullseye
16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P27074 and previous config saved to /var/cache/conftool/dbconfig/20220429-163115-ladsgroup.json
16:30 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host backup1002.eqiad.wmnet with OS buster
16:29 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2042.codfw.wmnet with OS bullseye
16:29 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab1003.wikimedia.org with reason: host reimage
16:29 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab1004.wikimedia.org with reason: host reimage
16:29 jynus@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup1002.eqiad.wmnet with OS bullseye
16:28 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner1002.eqiad.wmnet with OS bullseye
16:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27073 and previous config saved to /var/cache/conftool/dbconfig/20220429-162828-ladsgroup.json
16:26 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner1004.eqiad.wmnet with reason: host reimage
16:25 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab1003.wikimedia.org with reason: host reimage
16:23 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner1004.eqiad.wmnet with reason: host reimage
16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P27072 and previous config saved to /var/cache/conftool/dbconfig/20220429-162025-ladsgroup.json
16:19 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner1003.eqiad.wmnet with reason: host reimage
16:18 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab1004.wikimedia.org with OS bullseye
16:17 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner1002.eqiad.wmnet with reason: host reimage
16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298295)', diff saved to https://phabricator.wikimedia.org/P27071 and previous config saved to /var/cache/conftool/dbconfig/20220429-161610-ladsgroup.json
16:14 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab1003.wikimedia.org with OS bullseye
16:14 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner1003.eqiad.wmnet with reason: host reimage
16:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T298295)', diff saved to https://phabricator.wikimedia.org/P27070 and previous config saved to /var/cache/conftool/dbconfig/20220429-161400-ladsgroup.json
16:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
16:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
16:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298295)', diff saved to https://phabricator.wikimedia.org/P27069 and previous config saved to /var/cache/conftool/dbconfig/20220429-161352-ladsgroup.json
16:13 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner1002.eqiad.wmnet with reason: host reimage
16:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27068 and previous config saved to /var/cache/conftool/dbconfig/20220429-161323-ladsgroup.json
16:12 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab-runner1004.eqiad.wmnet with OS bullseye
16:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27067 and previous config saved to /var/cache/conftool/dbconfig/20220429-160702-ladsgroup.json
16:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
16:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
16:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
16:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
16:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27066 and previous config saved to /var/cache/conftool/dbconfig/20220429-160602-ladsgroup.json
16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T306560)', diff saved to https://phabricator.wikimedia.org/P27065 and previous config saved to /var/cache/conftool/dbconfig/20220429-160520-ladsgroup.json
16:04 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:03 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab-runner1003.eqiad.wmnet with OS bullseye
16:02 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab-runner1002.eqiad.wmnet with OS bullseye
15:59 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
15:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P27064 and previous config saved to /var/cache/conftool/dbconfig/20220429-155846-ladsgroup.json
15:56 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:53 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
15:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T306560)', diff saved to https://phabricator.wikimedia.org/P27063 and previous config saved to /var/cache/conftool/dbconfig/20220429-155142-ladsgroup.json
15:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
15:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
15:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T306560)', diff saved to https://phabricator.wikimedia.org/P27062 and previous config saved to /var/cache/conftool/dbconfig/20220429-155134-ladsgroup.json
15:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27061 and previous config saved to /var/cache/conftool/dbconfig/20220429-155057-ladsgroup.json
15:45 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P27060 and previous config saved to /var/cache/conftool/dbconfig/20220429-154341-ladsgroup.json
15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T306560)', diff saved to https://phabricator.wikimedia.org/P27059 and previous config saved to /var/cache/conftool/dbconfig/20220429-154253-ladsgroup.json
15:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
15:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T306560)', diff saved to https://phabricator.wikimedia.org/P27058 and previous config saved to /var/cache/conftool/dbconfig/20220429-154245-ladsgroup.json
15:41 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
15:39 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
15:37 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
15:36 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host backup1002.eqiad.wmnet with OS bullseye
15:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P27057 and previous config saved to /var/cache/conftool/dbconfig/20220429-153629-ladsgroup.json
15:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27056 and previous config saved to /var/cache/conftool/dbconfig/20220429-153551-ladsgroup.json
15:34 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:31 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
15:29 jynus: update NIC firmware for backup1002 T286722 T305446
15:28 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298295)', diff saved to https://phabricator.wikimedia.org/P27055 and previous config saved to /var/cache/conftool/dbconfig/20220429-152836-ladsgroup.json
15:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P27054 and previous config saved to /var/cache/conftool/dbconfig/20220429-152740-ladsgroup.json
15:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T298295)', diff saved to https://phabricator.wikimedia.org/P27053 and previous config saved to /var/cache/conftool/dbconfig/20220429-152628-ladsgroup.json
15:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
15:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
15:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298295)', diff saved to https://phabricator.wikimedia.org/P27052 and previous config saved to /var/cache/conftool/dbconfig/20220429-152620-ladsgroup.json
15:25 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2042.codfw.wmnet with reason: host reimage
15:25 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
15:21 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2042.codfw.wmnet with reason: host reimage
15:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P27051 and previous config saved to /var/cache/conftool/dbconfig/20220429-152124-ladsgroup.json
15:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27050 and previous config saved to /var/cache/conftool/dbconfig/20220429-152046-ladsgroup.json
15:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27049 and previous config saved to /var/cache/conftool/dbconfig/20220429-151424-ladsgroup.json
15:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
15:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
15:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
15:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
15:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27048 and previous config saved to /var/cache/conftool/dbconfig/20220429-151321-ladsgroup.json
15:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P27047 and previous config saved to /var/cache/conftool/dbconfig/20220429-151235-ladsgroup.json
15:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P27046 and previous config saved to /var/cache/conftool/dbconfig/20220429-151115-ladsgroup.json
15:07 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2042.codfw.wmnet with OS bullseye
15:07 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner1004.mgmt.eqiad.wmnet with reboot policy FORCED
15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T306560)', diff saved to https://phabricator.wikimedia.org/P27045 and previous config saved to /var/cache/conftool/dbconfig/20220429-150619-ladsgroup.json
15:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P27044 and previous config saved to /var/cache/conftool/dbconfig/20220429-150417-ladsgroup.json
14:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27043 and previous config saved to /var/cache/conftool/dbconfig/20220429-145816-ladsgroup.json
14:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T306560)', diff saved to https://phabricator.wikimedia.org/P27042 and previous config saved to /var/cache/conftool/dbconfig/20220429-145730-ladsgroup.json
14:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P27041 and previous config saved to /var/cache/conftool/dbconfig/20220429-145610-ladsgroup.json
14:52 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host gitlab-runner1004.mgmt.eqiad.wmnet with reboot policy FORCED
14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T306560)', diff saved to https://phabricator.wikimedia.org/P27040 and previous config saved to /var/cache/conftool/dbconfig/20220429-145148-ladsgroup.json
14:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
14:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T306560)', diff saved to https://phabricator.wikimedia.org/P27039 and previous config saved to /var/cache/conftool/dbconfig/20220429-144947-ladsgroup.json
14:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
14:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P27038 and previous config saved to /var/cache/conftool/dbconfig/20220429-144912-ladsgroup.json
14:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
14:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
14:43 jayme@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-wikikube-rw,name=eqiad
14:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27037 and previous config saved to /var/cache/conftool/dbconfig/20220429-144311-ladsgroup.json
14:43 jayme@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-wikikube-ro
14:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298295)', diff saved to https://phabricator.wikimedia.org/P27036 and previous config saved to /var/cache/conftool/dbconfig/20220429-144105-ladsgroup.json
14:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
14:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T306560)', diff saved to https://phabricator.wikimedia.org/P27035 and previous config saved to /var/cache/conftool/dbconfig/20220429-144028-ladsgroup.json
14:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T298295)', diff saved to https://phabricator.wikimedia.org/P27034 and previous config saved to /var/cache/conftool/dbconfig/20220429-143857-ladsgroup.json
14:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
14:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
14:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
14:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298295)', diff saved to https://phabricator.wikimedia.org/P27033 and previous config saved to /var/cache/conftool/dbconfig/20220429-143844-ladsgroup.json
14:35 jayme@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P27032 and previous config saved to /var/cache/conftool/dbconfig/20220429-143407-ladsgroup.json
14:32 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2041.codfw.wmnet with OS bullseye
14:31 jayme@cumin1001: START - Cookbook sre.dns.netbox
14:29 jayme@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-wikikube-ro
14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27031 and previous config saved to /var/cache/conftool/dbconfig/20220429-142806-ladsgroup.json
14:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P27030 and previous config saved to /var/cache/conftool/dbconfig/20220429-142523-ladsgroup.json
14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P27029 and previous config saved to /var/cache/conftool/dbconfig/20220429-142339-ladsgroup.json
14:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P27028 and previous config saved to /var/cache/conftool/dbconfig/20220429-142142-ladsgroup.json
14:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
14:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
14:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
14:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
14:19 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2041.codfw.wmnet with reason: host reimage
14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P27027 and previous config saved to /var/cache/conftool/dbconfig/20220429-141902-ladsgroup.json
14:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P27026 and previous config saved to /var/cache/conftool/dbconfig/20220429-141633-ladsgroup.json
14:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
14:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
14:16 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2041.codfw.wmnet with reason: host reimage
14:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P27025 and previous config saved to /var/cache/conftool/dbconfig/20220429-141017-ladsgroup.json
14:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P27024 and previous config saved to /var/cache/conftool/dbconfig/20220429-140834-ladsgroup.json
13:56 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2041.codfw.wmnet with OS bullseye
13:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T306560)', diff saved to https://phabricator.wikimedia.org/P27023 and previous config saved to /var/cache/conftool/dbconfig/20220429-135511-ladsgroup.json
13:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298295)', diff saved to https://phabricator.wikimedia.org/P27022 and previous config saved to /var/cache/conftool/dbconfig/20220429-135329-ladsgroup.json
13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298295)', diff saved to https://phabricator.wikimedia.org/P27021 and previous config saved to /var/cache/conftool/dbconfig/20220429-135121-ladsgroup.json
13:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
13:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298295)', diff saved to https://phabricator.wikimedia.org/P27020 and previous config saved to /var/cache/conftool/dbconfig/20220429-135111-ladsgroup.json
13:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti-test2003.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
13:42 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2003.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
13:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet
13:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P27019 and previous config saved to /var/cache/conftool/dbconfig/20220429-133606-ladsgroup.json
13:35 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet
13:32 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2040.codfw.wmnet
13:26 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-be2040.codfw.wmnet
13:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T306560)', diff saved to https://phabricator.wikimedia.org/P27018 and previous config saved to /var/cache/conftool/dbconfig/20220429-132619-ladsgroup.json
13:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
13:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
13:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T306560)', diff saved to https://phabricator.wikimedia.org/P27017 and previous config saved to /var/cache/conftool/dbconfig/20220429-132611-ladsgroup.json
13:25 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2040.codfw.wmnet
13:21 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-be2040.codfw.wmnet
13:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P27016 and previous config saved to /var/cache/conftool/dbconfig/20220429-132101-ladsgroup.json
13:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P27015 and previous config saved to /var/cache/conftool/dbconfig/20220429-131106-ladsgroup.json
13:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298295)', diff saved to https://phabricator.wikimedia.org/P27014 and previous config saved to /var/cache/conftool/dbconfig/20220429-130556-ladsgroup.json
12:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P27013 and previous config saved to /var/cache/conftool/dbconfig/20220429-125601-ladsgroup.json
12:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4004.ulsfo.wmnet
12:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T298295)', diff saved to https://phabricator.wikimedia.org/P27012 and previous config saved to /var/cache/conftool/dbconfig/20220429-125146-ladsgroup.json
12:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
12:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
12:49 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti4004.ulsfo.wmnet
12:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti-test2003.codfw.wmnet with OS bullseye
12:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T306560)', diff saved to https://phabricator.wikimedia.org/P27011 and previous config saved to /var/cache/conftool/dbconfig/20220429-124056-ladsgroup.json
12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase-dev2003.codfw.wmnet
12:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase-dev2003.codfw.wmnet
12:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti-test2003.codfw.wmnet with reason: host reimage
12:30 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet2005-dev.codfw.wmnet with OS bullseye
12:29 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti-test2003.codfw.wmnet with reason: host reimage
12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T306560)', diff saved to https://phabricator.wikimedia.org/P27010 and previous config saved to /var/cache/conftool/dbconfig/20220429-122805-ladsgroup.json
12:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
12:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
12:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase-dev2002.codfw.wmnet
12:23 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase-dev2002.codfw.wmnet
12:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase-dev2001.codfw.wmnet
12:14 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti-test2003.codfw.wmnet with OS bullseye
12:12 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet2005-dev.codfw.wmnet with reason: host reimage
12:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase-dev2001.codfw.wmnet
12:09 dcaro@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2005-dev.codfw.wmnet with reason: host reimage
12:04 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner1003.mgmt.eqiad.wmnet with reboot policy FORCED
12:04 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner1002.mgmt.eqiad.wmnet with reboot policy FORCED
12:04 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab1004.mgmt.eqiad.wmnet with reboot policy FORCED
11:57 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab1003.mgmt.eqiad.wmnet with reboot policy FORCED
11:51 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host gitlab-runner1003.mgmt.eqiad.wmnet with reboot policy FORCED
11:51 dcaro@cumin1001: START - Cookbook sre.hosts.reimage for host cloudnet2005-dev.codfw.wmnet with OS bullseye
11:50 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host gitlab-runner1002.mgmt.eqiad.wmnet with reboot policy FORCED
11:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host gitlab1004.mgmt.eqiad.wmnet with reboot policy FORCED
11:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host gitlab1003.mgmt.eqiad.wmnet with reboot policy FORCED
10:52 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
10:30 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1040.eqiad.wmnet with OS bullseye
10:11 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1040.eqiad.wmnet with reason: host reimage
10:08 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1040.eqiad.wmnet with reason: host reimage
10:06 kormat@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1164.eqiad.wmnet with OS bullseye
09:57 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be1040.eqiad.wmnet with OS bullseye
09:57 moritzm: drain ganeti-test2003 T306499
09:56 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet2006-dev.codfw.wmnet with OS bullseye
09:42 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db1164.eqiad.wmnet with OS bullseye
09:42 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet2006-dev.codfw.wmnet with reason: host reimage
09:39 dcaro@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2006-dev.codfw.wmnet with reason: host reimage
09:36 kormat@cumin1001: dbctl commit (dc=all): 'db1164 depooling: Rebooting for T303171', diff saved to https://phabricator.wikimedia.org/P27008 and previous config saved to /var/cache/conftool/dbconfig/20220429-093613-kormat.json
09:36 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1164.eqiad.wmnet with reason: Rebooting for T303171
09:36 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1164.eqiad.wmnet with reason: Rebooting for T303171
09:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti-test2002.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
09:33 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2002.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
09:27 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2040.codfw.wmnet with OS bullseye
09:25 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
09:20 dcaro@cumin1001: START - Cookbook sre.hosts.reimage for host cloudnet2006-dev.codfw.wmnet with OS bullseye
09:14 kormat@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 100%: Reboot T303171', diff saved to https://phabricator.wikimedia.org/P27007 and previous config saved to /var/cache/conftool/dbconfig/20220429-091401-kormat.json
09:02 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2040.codfw.wmnet with reason: host reimage
08:58 kormat@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 75%: Reboot T303171', diff saved to https://phabricator.wikimedia.org/P27006 and previous config saved to /var/cache/conftool/dbconfig/20220429-085858-kormat.json
08:58 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2040.codfw.wmnet with reason: host reimage
08:46 dcausse: restarting blazegraph on wdqs1006 (deadlocked for 18hours, T242453)
08:43 kormat@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 50%: Reboot T303171', diff saved to https://phabricator.wikimedia.org/P27005 and previous config saved to /var/cache/conftool/dbconfig/20220429-084354-kormat.json
08:33 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw1323.eqiad.wmnet
08:28 kormat@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 25%: Reboot T303171', diff saved to https://phabricator.wikimedia.org/P27004 and previous config saved to /var/cache/conftool/dbconfig/20220429-082850-kormat.json
08:27 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1163.eqiad.wmnet with OS bullseye
08:27 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2040.codfw.wmnet with OS bullseye
08:21 jelto: scap pull on mw1323
08:16 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1163.eqiad.wmnet with reason: host reimage
08:14 mvernon@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2040.codfw.wmnet with OS bullseye
08:13 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1163.eqiad.wmnet with reason: host reimage
08:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet
08:03 jelto@cumin1001: conftool action : set/pooled=no; selector: name=mw1323.eqiad.wmnet
08:02 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db1163.eqiad.wmnet with OS bullseye
08:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet
08:00 kormat@cumin1001: dbctl commit (dc=all): 'db1163 depooling: Rebooting for T303171', diff saved to https://phabricator.wikimedia.org/P27003 and previous config saved to /var/cache/conftool/dbconfig/20220429-080038-kormat.json
08:00 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1163.eqiad.wmnet with reason: Rebooting for T303171
08:00 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1163.eqiad.wmnet with reason: Rebooting for T303171
07:55 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2103.codfw.wmnet with OS bullseye
07:51 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2040.codfw.wmnet with OS bullseye
07:50 mvernon@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2040.codfw.wmnet with OS bullseye
07:43 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2103.codfw.wmnet with reason: host reimage
07:40 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2103.codfw.wmnet with reason: host reimage
07:27 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2103.codfw.wmnet with OS bullseye
07:26 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2040.codfw.wmnet with OS bullseye
07:09 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2103.codfw.wmnet with reason: Rebooting for T303171
07:09 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2103.codfw.wmnet with reason: Rebooting for T303171
07:08 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 14 hosts with reason: Reimaging db2103 T303171
07:07 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on 14 hosts with reason: Reimaging db2103 T303171
06:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1132 T301879', diff saved to https://phabricator.wikimedia.org/P27001 and previous config saved to /var/cache/conftool/dbconfig/20220429-063019-marostegui.json
03:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P27000 and previous config saved to /var/cache/conftool/dbconfig/20220429-035843-ladsgroup.json
03:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T306560)', diff saved to https://phabricator.wikimedia.org/P26999 and previous config saved to /var/cache/conftool/dbconfig/20220429-035415-ladsgroup.json
03:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26998 and previous config saved to /var/cache/conftool/dbconfig/20220429-034338-ladsgroup.json
03:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26997 and previous config saved to /var/cache/conftool/dbconfig/20220429-033910-ladsgroup.json
03:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26996 and previous config saved to /var/cache/conftool/dbconfig/20220429-032833-ladsgroup.json
03:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26995 and previous config saved to /var/cache/conftool/dbconfig/20220429-032405-ladsgroup.json
03:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P26994 and previous config saved to /var/cache/conftool/dbconfig/20220429-031328-ladsgroup.json
03:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T306560)', diff saved to https://phabricator.wikimedia.org/P26993 and previous config saved to /var/cache/conftool/dbconfig/20220429-030900-ladsgroup.json
03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T306560)', diff saved to https://phabricator.wikimedia.org/P26992 and previous config saved to /var/cache/conftool/dbconfig/20220429-030447-ladsgroup.json
03:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
03:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T306560)', diff saved to https://phabricator.wikimedia.org/P26991 and previous config saved to /var/cache/conftool/dbconfig/20220429-030439-ladsgroup.json
03:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P26990 and previous config saved to /var/cache/conftool/dbconfig/20220429-030303-ladsgroup.json
03:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
03:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
03:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P26989 and previous config saved to /var/cache/conftool/dbconfig/20220429-030250-ladsgroup.json
02:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26988 and previous config saved to /var/cache/conftool/dbconfig/20220429-024934-ladsgroup.json
02:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26987 and previous config saved to /var/cache/conftool/dbconfig/20220429-024745-ladsgroup.json
02:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26986 and previous config saved to /var/cache/conftool/dbconfig/20220429-023429-ladsgroup.json
02:32 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2012.mgmt.codfw.wmnet with reboot policy FORCED
02:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26985 and previous config saved to /var/cache/conftool/dbconfig/20220429-023240-ladsgroup.json
02:32 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2011.mgmt.codfw.wmnet with reboot policy FORCED
02:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T306560)', diff saved to https://phabricator.wikimedia.org/P26984 and previous config saved to /var/cache/conftool/dbconfig/20220429-021924-ladsgroup.json
02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P26983 and previous config saved to /var/cache/conftool/dbconfig/20220429-021735-ladsgroup.json
02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T306560)', diff saved to https://phabricator.wikimedia.org/P26982 and previous config saved to /var/cache/conftool/dbconfig/20220429-021710-ladsgroup.json
02:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
02:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
02:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
02:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
02:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T306560)', diff saved to https://phabricator.wikimedia.org/P26981 and previous config saved to /var/cache/conftool/dbconfig/20220429-021657-ladsgroup.json
02:09 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2012.mgmt.codfw.wmnet with reboot policy FORCED
02:08 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2011.mgmt.codfw.wmnet with reboot policy FORCED
02:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2010.mgmt.codfw.wmnet with reboot policy FORCED
02:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2009.mgmt.codfw.wmnet with reboot policy FORCED
02:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P26980 and previous config saved to /var/cache/conftool/dbconfig/20220429-020705-ladsgroup.json
02:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
02:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
02:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P26979 and previous config saved to /var/cache/conftool/dbconfig/20220429-020652-ladsgroup.json
02:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26978 and previous config saved to /var/cache/conftool/dbconfig/20220429-020151-ladsgroup.json
01:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26977 and previous config saved to /var/cache/conftool/dbconfig/20220429-015147-ladsgroup.json
01:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26976 and previous config saved to /var/cache/conftool/dbconfig/20220429-014646-ladsgroup.json
01:40 ejegg: updated fundraising IPN listener standalone (SmashPig) from a5d785fd to ffe5066d
01:37 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2010.mgmt.codfw.wmnet with reboot policy FORCED
01:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26975 and previous config saved to /var/cache/conftool/dbconfig/20220429-013642-ladsgroup.json
01:36 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2009.mgmt.codfw.wmnet with reboot policy FORCED
01:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2008.mgmt.codfw.wmnet with reboot policy FORCED
01:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T306560)', diff saved to https://phabricator.wikimedia.org/P26974 and previous config saved to /var/cache/conftool/dbconfig/20220429-013141-ladsgroup.json
01:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T306560)', diff saved to https://phabricator.wikimedia.org/P26973 and previous config saved to /var/cache/conftool/dbconfig/20220429-012827-ladsgroup.json
01:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
01:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
01:28 ejegg: updated Fundraising CiviCRM from a841cf55 to 852c1969
01:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
01:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
01:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
01:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
01:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
01:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
01:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
01:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
01:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
01:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
01:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26972 and previous config saved to /var/cache/conftool/dbconfig/20220429-012713-ladsgroup.json
01:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P26971 and previous config saved to /var/cache/conftool/dbconfig/20220429-012137-ladsgroup.json
01:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26970 and previous config saved to /var/cache/conftool/dbconfig/20220429-011207-ladsgroup.json
01:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P26969 and previous config saved to /var/cache/conftool/dbconfig/20220429-011046-ladsgroup.json
01:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
01:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
01:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P26968 and previous config saved to /var/cache/conftool/dbconfig/20220429-011033-ladsgroup.json
01:04 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2007.mgmt.codfw.wmnet with reboot policy FORCED
00:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26967 and previous config saved to /var/cache/conftool/dbconfig/20220429-005702-ladsgroup.json
00:56 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2008.mgmt.codfw.wmnet with reboot policy FORCED
00:56 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
00:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26966 and previous config saved to /var/cache/conftool/dbconfig/20220429-005528-ladsgroup.json
00:53 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
00:47 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2007.mgmt.codfw.wmnet with reboot policy FORCED
00:46 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
00:44 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
00:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26965 and previous config saved to /var/cache/conftool/dbconfig/20220429-004157-ladsgroup.json
00:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26964 and previous config saved to /var/cache/conftool/dbconfig/20220429-004023-ladsgroup.json
00:38 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
00:37 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
00:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P26963 and previous config saved to /var/cache/conftool/dbconfig/20220429-002518-ladsgroup.json
00:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26962 and previous config saved to /var/cache/conftool/dbconfig/20220429-001840-ladsgroup.json
00:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
00:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
00:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26961 and previous config saved to /var/cache/conftool/dbconfig/20220429-001832-ladsgroup.json
00:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P26960 and previous config saved to /var/cache/conftool/dbconfig/20220429-001333-ladsgroup.json
00:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
00:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
00:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P26959 and previous config saved to /var/cache/conftool/dbconfig/20220429-001320-ladsgroup.json
00:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26958 and previous config saved to /var/cache/conftool/dbconfig/20220429-000327-ladsgroup.json

2022-04-28

23:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26957 and previous config saved to /var/cache/conftool/dbconfig/20220428-235815-ladsgroup.json
23:49 dzahn@deploy1002: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
23:49 dzahn@deploy1002: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
23:48 dzahn@deploy1002: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
23:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26956 and previous config saved to /var/cache/conftool/dbconfig/20220428-234822-ladsgroup.json
23:47 dzahn@deploy1002: helmfile [codfw] START helmfile.d/services/push-notifications: apply
23:44 dzahn@deploy1002: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
23:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26955 and previous config saved to /var/cache/conftool/dbconfig/20220428-234310-ladsgroup.json
23:42 dzahn@deploy1002: helmfile [staging] START helmfile.d/services/push-notifications: apply
23:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26954 and previous config saved to /var/cache/conftool/dbconfig/20220428-233317-ladsgroup.json
23:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26953 and previous config saved to /var/cache/conftool/dbconfig/20220428-233103-ladsgroup.json
23:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
23:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
23:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26952 and previous config saved to /var/cache/conftool/dbconfig/20220428-233055-ladsgroup.json
23:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P26951 and previous config saved to /var/cache/conftool/dbconfig/20220428-232805-ladsgroup.json
23:18 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
23:18 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
23:17 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
23:17 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
23:17 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
23:17 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
23:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P26950 and previous config saved to /var/cache/conftool/dbconfig/20220428-231714-ladsgroup.json
23:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
23:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
23:17 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2003.mgmt.codfw.wmnet with reboot policy FORCED
23:17 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2004.mgmt.codfw.wmnet with reboot policy FORCED
23:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P26949 and previous config saved to /var/cache/conftool/dbconfig/20220428-231701-ladsgroup.json
23:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26948 and previous config saved to /var/cache/conftool/dbconfig/20220428-231550-ladsgroup.json
23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26947 and previous config saved to /var/cache/conftool/dbconfig/20220428-230156-ladsgroup.json
23:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26946 and previous config saved to /var/cache/conftool/dbconfig/20220428-230045-ladsgroup.json
22:54 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2004.mgmt.codfw.wmnet with reboot policy FORCED
22:54 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2003.mgmt.codfw.wmnet with reboot policy FORCED
22:48 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2002.mgmt.codfw.wmnet with reboot policy FORCED
22:48 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2001.mgmt.codfw.wmnet with reboot policy FORCED
22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26945 and previous config saved to /var/cache/conftool/dbconfig/20220428-224650-ladsgroup.json
22:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26944 and previous config saved to /var/cache/conftool/dbconfig/20220428-224540-ladsgroup.json
22:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26943 and previous config saved to /var/cache/conftool/dbconfig/20220428-224426-ladsgroup.json
22:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
22:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
22:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T306560)', diff saved to https://phabricator.wikimedia.org/P26942 and previous config saved to /var/cache/conftool/dbconfig/20220428-224417-ladsgroup.json
22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P26941 and previous config saved to /var/cache/conftool/dbconfig/20220428-223145-ladsgroup.json
22:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26940 and previous config saved to /var/cache/conftool/dbconfig/20220428-222912-ladsgroup.json
22:21 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2002.mgmt.codfw.wmnet with reboot policy FORCED
22:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P26939 and previous config saved to /var/cache/conftool/dbconfig/20220428-222035-ladsgroup.json
22:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
22:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
22:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P26938 and previous config saved to /var/cache/conftool/dbconfig/20220428-222022-ladsgroup.json
22:20 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2001.mgmt.codfw.wmnet with reboot policy FORCED
22:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26937 and previous config saved to /var/cache/conftool/dbconfig/20220428-221407-ladsgroup.json
22:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26936 and previous config saved to /var/cache/conftool/dbconfig/20220428-220517-ladsgroup.json
22:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T306560)', diff saved to https://phabricator.wikimedia.org/P26935 and previous config saved to /var/cache/conftool/dbconfig/20220428-220450-ladsgroup.json
21:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T306560)', diff saved to https://phabricator.wikimedia.org/P26934 and previous config saved to /var/cache/conftool/dbconfig/20220428-215902-ladsgroup.json
21:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 (T306560)', diff saved to https://phabricator.wikimedia.org/P26933 and previous config saved to /var/cache/conftool/dbconfig/20220428-215547-ladsgroup.json
21:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
21:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
21:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26932 and previous config saved to /var/cache/conftool/dbconfig/20220428-215012-ladsgroup.json
21:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26931 and previous config saved to /var/cache/conftool/dbconfig/20220428-214945-ladsgroup.json
21:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P26930 and previous config saved to /var/cache/conftool/dbconfig/20220428-213507-ladsgroup.json
21:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26929 and previous config saved to /var/cache/conftool/dbconfig/20220428-213440-ladsgroup.json
21:27 tgr: UTC late deploys done
21:26 tgr@deploy1002: Synchronized php-1.39.0-wmf.9/extensions/GrowthExperiments/includes/VariantHooks.php: Backport: Video landing page: Record campaign parameter for control users (T303785) (duration: 00m 54s)
21:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P26928 and previous config saved to /var/cache/conftool/dbconfig/20220428-212331-ladsgroup.json
21:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
21:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
21:20 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
21:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T306560)', diff saved to https://phabricator.wikimedia.org/P26927 and previous config saved to /var/cache/conftool/dbconfig/20220428-211934-ladsgroup.json
21:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T306560)', diff saved to https://phabricator.wikimedia.org/P26926 and previous config saved to /var/cache/conftool/dbconfig/20220428-211727-ladsgroup.json
21:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
21:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
21:15 pt1979@cumin2002: START - Cookbook sre.dns.netbox
21:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T306560)', diff saved to https://phabricator.wikimedia.org/P26925 and previous config saved to /var/cache/conftool/dbconfig/20220428-211507-ladsgroup.json
21:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:12 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:12 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:11 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance
21:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance
21:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1100.eqiad.wmnet with reason: Maintenance
21:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1100.eqiad.wmnet with reason: Maintenance
21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26924 and previous config saved to /var/cache/conftool/dbconfig/20220428-210002-ladsgroup.json
20:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26923 and previous config saved to /var/cache/conftool/dbconfig/20220428-204457-ladsgroup.json
20:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T306560)', diff saved to https://phabricator.wikimedia.org/P26922 and previous config saved to /var/cache/conftool/dbconfig/20220428-202952-ladsgroup.json
20:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:09 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
20:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:07 brennen: no trainee attendees for backport & config session; tgr self-serving some patches, calling end of training window
20:06 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
19:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298558)', diff saved to https://phabricator.wikimedia.org/P26921 and previous config saved to /var/cache/conftool/dbconfig/20220428-193109-ladsgroup.json
19:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T306560)', diff saved to https://phabricator.wikimedia.org/P26920 and previous config saved to /var/cache/conftool/dbconfig/20220428-192938-ladsgroup.json
19:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
19:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
19:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T306560)', diff saved to https://phabricator.wikimedia.org/P26919 and previous config saved to /var/cache/conftool/dbconfig/20220428-192930-ladsgroup.json
19:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26918 and previous config saved to /var/cache/conftool/dbconfig/20220428-191604-ladsgroup.json
19:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26917 and previous config saved to /var/cache/conftool/dbconfig/20220428-191425-ladsgroup.json
19:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26915 and previous config saved to /var/cache/conftool/dbconfig/20220428-190059-ladsgroup.json
18:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26914 and previous config saved to /var/cache/conftool/dbconfig/20220428-185920-ladsgroup.json
18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298558)', diff saved to https://phabricator.wikimedia.org/P26913 and previous config saved to /var/cache/conftool/dbconfig/20220428-184554-ladsgroup.json
18:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T306560)', diff saved to https://phabricator.wikimedia.org/P26912 and previous config saved to /var/cache/conftool/dbconfig/20220428-184415-ladsgroup.json
18:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298558)', diff saved to https://phabricator.wikimedia.org/P26911 and previous config saved to /var/cache/conftool/dbconfig/20220428-184338-ladsgroup.json
18:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
18:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
18:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298558)', diff saved to https://phabricator.wikimedia.org/P26910 and previous config saved to /var/cache/conftool/dbconfig/20220428-184330-ladsgroup.json
18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T306560)', diff saved to https://phabricator.wikimedia.org/P26909 and previous config saved to /var/cache/conftool/dbconfig/20220428-184207-ladsgroup.json
18:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
18:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T306560)', diff saved to https://phabricator.wikimedia.org/P26908 and previous config saved to /var/cache/conftool/dbconfig/20220428-184159-ladsgroup.json
18:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26907 and previous config saved to /var/cache/conftool/dbconfig/20220428-182825-ladsgroup.json
18:28 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1024.eqiad.wmnet with OS buster
18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26906 and previous config saved to /var/cache/conftool/dbconfig/20220428-182654-ladsgroup.json
18:25 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1023.eqiad.wmnet with OS buster
18:21 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1022.eqiad.wmnet with OS buster
18:19 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1021.eqiad.wmnet with OS buster
18:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:17 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1020.eqiad.wmnet with OS buster
18:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:16 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1024.eqiad.wmnet with reason: host reimage
18:13 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1023.eqiad.wmnet with reason: host reimage
18:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26905 and previous config saved to /var/cache/conftool/dbconfig/20220428-181320-ladsgroup.json
18:13 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1019.eqiad.wmnet with OS buster
18:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26904 and previous config saved to /var/cache/conftool/dbconfig/20220428-181149-ladsgroup.json
18:11 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1018.eqiad.wmnet with OS buster
18:10 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1024.eqiad.wmnet with reason: host reimage
18:10 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1023.eqiad.wmnet with reason: host reimage
18:08 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1017.eqiad.wmnet with OS buster
18:08 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1022.eqiad.wmnet with reason: host reimage
18:07 brennen@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.9 refs T305215
18:06 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab2002.wikimedia.org with OS bullseye
18:05 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1021.eqiad.wmnet with reason: host reimage
18:04 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab2003.wikimedia.org with OS bullseye
18:03 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1020.eqiad.wmnet with reason: host reimage
18:02 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1022.eqiad.wmnet with reason: host reimage
18:01 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1014.eqiad.wmnet with OS buster
18:01 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1011.eqiad.wmnet with OS buster
18:01 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1012.eqiad.wmnet with OS buster
18:01 brennen: train 1.39.0-wmf.9 (T305215): no current blockers, logs fairly clear, proceeding to all wikis as soon as i finish this burrito
18:00 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1019.eqiad.wmnet with reason: host reimage
18:00 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1021.eqiad.wmnet with reason: host reimage
17:59 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1024.eqiad.wmnet with OS buster
17:59 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1018.eqiad.wmnet with reason: host reimage
17:59 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1023.eqiad.wmnet with OS buster
17:59 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host parse1016.eqiad.wmnet with OS buster
17:58 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1015.eqiad.wmnet with OS buster
17:58 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1020.eqiad.wmnet with reason: host reimage
17:58 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1032.eqiad.wmnet with OS buster
17:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298558)', diff saved to https://phabricator.wikimedia.org/P26903 and previous config saved to /var/cache/conftool/dbconfig/20220428-175815-ladsgroup.json
17:57 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1017.eqiad.wmnet with reason: host reimage
17:57 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1018.eqiad.wmnet with reason: host reimage
17:56 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1019.eqiad.wmnet with reason: host reimage
17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T306560)', diff saved to https://phabricator.wikimedia.org/P26902 and previous config saved to /var/cache/conftool/dbconfig/20220428-175644-ladsgroup.json
17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T298558)', diff saved to https://phabricator.wikimedia.org/P26901 and previous config saved to /var/cache/conftool/dbconfig/20220428-175559-ladsgroup.json
17:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
17:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
17:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298558)', diff saved to https://phabricator.wikimedia.org/P26900 and previous config saved to /var/cache/conftool/dbconfig/20220428-175551-ladsgroup.json
17:55 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1013.eqiad.wmnet with OS buster
17:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T306560)', diff saved to https://phabricator.wikimedia.org/P26899 and previous config saved to /var/cache/conftool/dbconfig/20220428-175436-ladsgroup.json
17:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
17:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
17:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
17:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
17:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T306560)', diff saved to https://phabricator.wikimedia.org/P26898 and previous config saved to /var/cache/conftool/dbconfig/20220428-175423-ladsgroup.json
17:53 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab2002.wikimedia.org with reason: host reimage
17:53 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1017.eqiad.wmnet with reason: host reimage
17:51 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1022.eqiad.wmnet with OS buster
17:51 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab2003.wikimedia.org with reason: host reimage
17:51 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1031.eqiad.wmnet with OS buster
17:51 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1014.eqiad.wmnet with reason: host reimage
17:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1021.eqiad.wmnet with OS buster
17:48 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1030.eqiad.wmnet with OS buster
17:48 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1012.eqiad.wmnet with reason: host reimage
17:48 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1011.eqiad.wmnet with reason: host reimage
17:47 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1020.eqiad.wmnet with OS buster
17:47 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1010.eqiad.wmnet with OS buster
17:46 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab2003.wikimedia.org with reason: host reimage
17:46 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab2002.wikimedia.org with reason: host reimage
17:46 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1016.eqiad.wmnet with reason: host reimage
17:45 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1019.eqiad.wmnet with OS buster
17:45 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti1032.eqiad.wmnet with reason: host reimage
17:45 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1015.eqiad.wmnet with reason: host reimage
17:45 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1009.eqiad.wmnet with OS buster
17:45 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1018.eqiad.wmnet with OS buster
17:44 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1029.eqiad.wmnet with OS buster
17:43 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1013.eqiad.wmnet with reason: host reimage
17:42 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1017.eqiad.wmnet with OS buster
17:41 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1016.eqiad.wmnet with reason: host reimage
17:41 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1015.eqiad.wmnet with reason: host reimage
17:41 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1014.eqiad.wmnet with reason: host reimage
17:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26897 and previous config saved to /var/cache/conftool/dbconfig/20220428-174046-ladsgroup.json
17:40 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host parse1016.eqiad.wmnet with OS buster
17:40 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1016.eqiad.wmnet with OS buster
17:39 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti1031.eqiad.wmnet with reason: host reimage
17:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26896 and previous config saved to /var/cache/conftool/dbconfig/20220428-173918-ladsgroup.json
17:38 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1013.eqiad.wmnet with reason: host reimage
17:38 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1012.eqiad.wmnet with reason: host reimage
17:38 dancy: testing logging: gerrit:9999
17:38 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1011.eqiad.wmnet with reason: host reimage
17:37 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1030.eqiad.wmnet with reason: host reimage
17:36 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host parse1016.eqiad.wmnet with OS buster
17:36 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1016.eqiad.wmnet with OS buster
17:36 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host parse1016.eqiad.wmnet with OS buster
17:35 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1016.eqiad.wmnet with OS buster
17:35 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1032.eqiad.wmnet with reason: host reimage
17:35 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1010.eqiad.wmnet with reason: host reimage
17:32 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1029.eqiad.wmnet with reason: host reimage
17:31 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye
17:30 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1016.eqiad.wmnet with OS buster
17:30 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1007.eqiad.wmnet with OS buster
17:30 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1010.eqiad.wmnet with reason: host reimage
17:30 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host gitlab2002.wikimedia.org with OS bullseye
17:30 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1015.eqiad.wmnet with OS buster
17:30 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1014.eqiad.wmnet with OS buster
17:29 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1003.eqiad.wmnet with OS buster
17:29 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1008.eqiad.wmnet with OS buster
17:29 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1031.eqiad.wmnet with reason: host reimage
17:29 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1030.eqiad.wmnet with reason: host reimage
17:29 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1029.eqiad.wmnet with reason: host reimage
17:27 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1013.eqiad.wmnet with OS buster
17:27 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1009.eqiad.wmnet with reason: host reimage
17:27 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1012.eqiad.wmnet with OS buster
17:27 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1011.eqiad.wmnet with OS buster
17:26 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1005.eqiad.wmnet with OS buster
17:25 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1004.eqiad.wmnet with OS buster
17:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26895 and previous config saved to /var/cache/conftool/dbconfig/20220428-172540-ladsgroup.json
17:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26894 and previous config saved to /var/cache/conftool/dbconfig/20220428-172413-ladsgroup.json
17:24 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1009.eqiad.wmnet with reason: host reimage
17:23 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1006.eqiad.wmnet with OS buster
17:19 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1010.eqiad.wmnet with OS buster
17:19 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1007.eqiad.wmnet with reason: host reimage
17:19 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1002.eqiad.wmnet with OS buster
17:18 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1032.eqiad.wmnet with OS buster
17:17 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1031.eqiad.wmnet with OS buster
17:17 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1030.eqiad.wmnet with OS buster
17:17 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1029.eqiad.wmnet with OS buster
17:17 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1003.eqiad.wmnet with reason: host reimage
17:16 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab2003.wikimedia.org with OS bullseye
17:16 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1008.eqiad.wmnet with reason: host reimage
17:14 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1008.eqiad.wmnet with reason: host reimage
17:13 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1007.eqiad.wmnet with reason: host reimage
17:13 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1005.eqiad.wmnet with reason: host reimage
17:13 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1003.eqiad.wmnet with reason: host reimage
17:13 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1009.eqiad.wmnet with OS buster
17:12 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1001.eqiad.wmnet with OS buster
17:11 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1004.eqiad.wmnet with reason: host reimage
17:11 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1006.eqiad.wmnet with reason: host reimage
17:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298558)', diff saved to https://phabricator.wikimedia.org/P26893 and previous config saved to /var/cache/conftool/dbconfig/20220428-171035-ladsgroup.json
17:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T306560)', diff saved to https://phabricator.wikimedia.org/P26892 and previous config saved to /var/cache/conftool/dbconfig/20220428-170908-ladsgroup.json
17:08 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1002.eqiad.wmnet with reason: host reimage
17:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T298558)', diff saved to https://phabricator.wikimedia.org/P26891 and previous config saved to /var/cache/conftool/dbconfig/20220428-170820-ladsgroup.json
17:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
17:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
17:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
17:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
17:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
17:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
17:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
17:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
17:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
17:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
17:07 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1006.eqiad.wmnet with reason: host reimage
17:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T306560)', diff saved to https://phabricator.wikimedia.org/P26890 and previous config saved to /var/cache/conftool/dbconfig/20220428-170700-ladsgroup.json
17:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
17:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26889 and previous config saved to /var/cache/conftool/dbconfig/20220428-170652-ladsgroup.json
17:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
17:06 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1005.eqiad.wmnet with reason: host reimage
17:06 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1004.eqiad.wmnet with reason: host reimage
17:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
17:05 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
17:05 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1002.eqiad.wmnet with reason: host reimage
17:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
17:03 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1008.eqiad.wmnet with OS buster
17:03 dancy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
17:03 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1007.eqiad.wmnet with OS buster
17:02 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1003.eqiad.wmnet with OS buster
17:02 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye
17:01 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1001.eqiad.wmnet with reason: host reimage
16:57 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1001.eqiad.wmnet with reason: host reimage
16:57 dancy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
16:56 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1006.eqiad.wmnet with OS buster
16:55 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1005.eqiad.wmnet with OS buster
16:55 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1004.eqiad.wmnet with OS buster
16:54 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1002.eqiad.wmnet with OS buster
16:53 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner2004.codfw.wmnet with OS bullseye
16:53 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1032.mgmt.eqiad.wmnet with reboot policy FORCED
16:52 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1030.mgmt.eqiad.wmnet with reboot policy FORCED
16:52 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1029.mgmt.eqiad.wmnet with reboot policy FORCED
16:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26887 and previous config saved to /var/cache/conftool/dbconfig/20220428-165148-ladsgroup.json
16:46 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1001.eqiad.wmnet with OS buster
16:43 dancy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
16:42 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner2004.codfw.wmnet with reason: host reimage
16:41 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1031.mgmt.eqiad.wmnet with reboot policy FORCED
16:40 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1032.mgmt.eqiad.wmnet with reboot policy FORCED
16:39 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1031.mgmt.eqiad.wmnet with reboot policy FORCED
16:39 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner2004.codfw.wmnet with reason: host reimage
16:38 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1030.mgmt.eqiad.wmnet with reboot policy FORCED
16:38 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1029.mgmt.eqiad.wmnet with reboot policy FORCED
16:37 dancy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
16:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26886 and previous config saved to /var/cache/conftool/dbconfig/20220428-163643-ladsgroup.json
16:33 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner2003.codfw.wmnet with OS bullseye
16:31 dancy@deploy1002: Started scap: testing mediawiki container image build and deploy
16:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 (T298558)', diff saved to https://phabricator.wikimedia.org/P26885 and previous config saved to /var/cache/conftool/dbconfig/20220428-162159-ladsgroup.json
16:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26884 and previous config saved to /var/cache/conftool/dbconfig/20220428-162138-ladsgroup.json
16:21 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner2003.codfw.wmnet with reason: host reimage
16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 (T298558)', diff saved to https://phabricator.wikimedia.org/P26883 and previous config saved to /var/cache/conftool/dbconfig/20220428-162047-ladsgroup.json
16:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
16:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298558)', diff saved to https://phabricator.wikimedia.org/P26882 and previous config saved to /var/cache/conftool/dbconfig/20220428-162039-ladsgroup.json
16:20 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab-runner2004.codfw.wmnet with OS bullseye
16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26881 and previous config saved to /var/cache/conftool/dbconfig/20220428-161828-ladsgroup.json
16:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner2002.codfw.wmnet with OS bullseye
16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
16:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
16:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26880 and previous config saved to /var/cache/conftool/dbconfig/20220428-161748-ladsgroup.json
16:17 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner2003.codfw.wmnet with reason: host reimage
16:10 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1015.mgmt.eqiad.wmnet with reboot policy FORCED
16:07 dcaro@cumin1001: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host cloudbackup2002.codfw.wmnet
16:06 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner2002.codfw.wmnet with reason: host reimage
16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26879 and previous config saved to /var/cache/conftool/dbconfig/20220428-160533-ladsgroup.json
16:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26878 and previous config saved to /var/cache/conftool/dbconfig/20220428-160243-ladsgroup.json
16:02 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner2002.codfw.wmnet with reason: host reimage
16:01 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1015.mgmt.eqiad.wmnet with reboot policy FORCED
16:01 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1003.mgmt.eqiad.wmnet with reboot policy FORCED
15:59 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab-runner2003.codfw.wmnet with OS bullseye
15:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26877 and previous config saved to /var/cache/conftool/dbconfig/20220428-155028-ladsgroup.json
15:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26876 and previous config saved to /var/cache/conftool/dbconfig/20220428-154738-ladsgroup.json
15:45 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1003.mgmt.eqiad.wmnet with reboot policy FORCED
15:44 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab-runner2002.codfw.wmnet with OS bullseye
15:39 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2072.codfw.wmnet with OS bullseye
15:38 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2130.codfw.wmnet with OS bullseye
15:38 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:36 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2112.codfw.wmnet with OS bullseye
15:35 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2145.codfw.wmnet with OS bullseye
15:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298558)', diff saved to https://phabricator.wikimedia.org/P26875 and previous config saved to /var/cache/conftool/dbconfig/20220428-153523-ladsgroup.json
15:33 pt1979@cumin2002: START - Cookbook sre.dns.netbox
15:33 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2116.codfw.wmnet with OS bullseye
15:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T298558)', diff saved to https://phabricator.wikimedia.org/P26874 and previous config saved to /var/cache/conftool/dbconfig/20220428-153307-ladsgroup.json
15:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
15:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
15:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298558)', diff saved to https://phabricator.wikimedia.org/P26873 and previous config saved to /var/cache/conftool/dbconfig/20220428-153259-ladsgroup.json
15:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26872 and previous config saved to /var/cache/conftool/dbconfig/20220428-153233-ladsgroup.json
15:31 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2146.codfw.wmnet with OS bullseye
15:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26871 and previous config saved to /var/cache/conftool/dbconfig/20220428-152924-ladsgroup.json
15:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
15:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
15:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26870 and previous config saved to /var/cache/conftool/dbconfig/20220428-152916-ladsgroup.json
15:27 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2092.codfw.wmnet with OS bullseye
15:23 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2072.codfw.wmnet with reason: host reimage
15:20 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2130.codfw.wmnet with reason: host reimage
15:20 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup2002.codfw.wmnet
15:18 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2112.codfw.wmnet with reason: host reimage
15:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26869 and previous config saved to /var/cache/conftool/dbconfig/20220428-151754-ladsgroup.json
15:17 kormat@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db2145.codfw.wmnet with reason: host reimage
15:15 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2116.codfw.wmnet with reason: host reimage
15:14 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2130.codfw.wmnet with reason: host reimage
15:14 kormat@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db2146.codfw.wmnet with reason: host reimage
15:14 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2072.codfw.wmnet with reason: host reimage
15:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26868 and previous config saved to /var/cache/conftool/dbconfig/20220428-151411-ladsgroup.json
15:14 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2146.codfw.wmnet with reason: host reimage
15:13 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2145.codfw.wmnet with reason: host reimage
15:13 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2112.codfw.wmnet with reason: host reimage
15:11 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2092.codfw.wmnet with reason: host reimage
15:11 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2116.codfw.wmnet with reason: host reimage
15:08 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2092.codfw.wmnet with reason: host reimage
15:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26867 and previous config saved to /var/cache/conftool/dbconfig/20220428-150249-ladsgroup.json
15:00 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2072.codfw.wmnet with OS bullseye
15:00 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2146.codfw.wmnet with OS bullseye
14:59 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2145.codfw.wmnet with OS bullseye
14:59 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2130.codfw.wmnet with OS bullseye
14:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26866 and previous config saved to /var/cache/conftool/dbconfig/20220428-145906-ladsgroup.json
14:58 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2116.codfw.wmnet with OS bullseye
14:58 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:58 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2112.codfw.wmnet with OS bullseye
14:58 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2092.codfw.wmnet with OS bullseye
14:58 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:58 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:56 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host restbase2025.codfw.wmnet
14:55 ladsgroup@deploy1002: Synchronized php-1.39.0-wmf.9/includes/specials/SpecialExport.php: Backport: SpecialExport: Add page table once (T307037) (duration: 00m 51s)
14:53 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
14:51 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2071.codfw.wmnet with OS bullseye
14:49 mforns@deploy1002: Finished deploy [airflow-dags/analytics@8278877]: (no justification provided) (duration: 00m 07s)
14:49 mforns@deploy1002: Started deploy [airflow-dags/analytics@8278877]: (no justification provided)
14:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298558)', diff saved to https://phabricator.wikimedia.org/P26864 and previous config saved to /var/cache/conftool/dbconfig/20220428-144744-ladsgroup.json
14:46 moritzm: powercycling restbase2025
14:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298558)', diff saved to https://phabricator.wikimedia.org/P26863 and previous config saved to /var/cache/conftool/dbconfig/20220428-144528-ladsgroup.json
14:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
14:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
14:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
14:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
14:45 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti-test2002.codfw.wmnet with OS bullseye
14:44 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2072.codfw.wmnet with reason: Rebooting for T303171
14:44 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2072.codfw.wmnet with reason: Rebooting for T303171
14:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26862 and previous config saved to /var/cache/conftool/dbconfig/20220428-144401-ladsgroup.json
14:43 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2094.codfw.wmnet with reason: Reimaging db2072 T303171
14:43 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2094.codfw.wmnet with reason: Reimaging db2072 T303171
14:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26861 and previous config saved to /var/cache/conftool/dbconfig/20220428-144252-ladsgroup.json
14:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
14:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
14:42 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2146.codfw.wmnet with reason: Rebooting for T303171
14:42 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2146.codfw.wmnet with reason: Rebooting for T303171
14:41 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2145.codfw.wmnet with reason: Rebooting for T303171
14:41 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2145.codfw.wmnet with reason: Rebooting for T303171
14:40 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2116.codfw.wmnet with reason: Rebooting for T303171
14:40 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2116.codfw.wmnet with reason: Rebooting for T303171
14:40 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2130.codfw.wmnet with reason: Rebooting for T303171
14:40 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2130.codfw.wmnet with reason: Rebooting for T303171
14:40 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2112.codfw.wmnet with reason: Rebooting for T303171
14:40 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2112.codfw.wmnet with reason: Rebooting for T303171
14:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2025.codfw.wmnet
14:38 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2092.codfw.wmnet with reason: Rebooting for T303171
14:38 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2092.codfw.wmnet with reason: Rebooting for T303171
14:36 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2071.codfw.wmnet with reason: host reimage
14:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti-test2002.codfw.wmnet with reason: host reimage
14:32 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2071.codfw.wmnet with reason: host reimage
14:29 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti-test2002.codfw.wmnet with reason: host reimage
14:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2024.codfw.wmnet
14:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2024.codfw.wmnet
14:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:17 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
14:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
14:15 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti-test2002.codfw.wmnet with OS bullseye
14:14 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2071.codfw.wmnet with OS bullseye
14:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
14:09 kormat: reimaging s1 to bullseye on s2@codfw dbmaint (T303171)
14:08 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1013 as pc3 primary T307101 (duration: 00m 54s)
14:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2023.codfw.wmnet
14:07 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1004.eqiad.wmnet
14:06 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
14:06 Lucas_WMDE: UTC afternoon backport window done
14:06 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
14:05 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: itwiki: assign 'setmentor' to 'bot' usergroup (T307005) (duration: 00m 55s)
14:05 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
14:05 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
14:05 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
14:05 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
14:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:02 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2071.codfw.wmnet with reason: Rebooting for T303171
14:02 hnowlan@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
14:02 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2071.codfw.wmnet with reason: Rebooting for T303171
14:01 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/abusefilter.php: Config: zhwikiversity: Enable blocking feature of AbuseFilter (T307007) (duration: 00m 51s)
14:00 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:00 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:00 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup1004.eqiad.wmnet
13:59 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2023.codfw.wmnet
13:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:52 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Add kanoner.com to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T306795) (duration: 00m 50s)
13:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase[2023-2026].codfw.wmnet with reason: reboot
13:48 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase[2023-2026].codfw.wmnet with reason: reboot
13:45 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/logos.php: Config: zhwiki: Add comment to corresponding task of logo (T276694) (2/2, no-op) (duration: 00m 53s)
13:44 lucaswerkmeister-wmde@deploy1002: Synchronized logos/config.yaml: Config: zhwiki: Add comment to corresponding task of logo (T276694) (1/2, no-op) (duration: 00m 50s)
13:43 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on pc1013.eqiad.wmnet with reason: Rebooting for T303174
13:43 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on pc1013.eqiad.wmnet with reason: Rebooting for T303174
13:42 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2013.codfw.wmnet,pc[1013-1014].eqiad.wmnet with reason: Rebooting pc1013 T307101
13:42 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on pc2013.codfw.wmnet,pc[1013-1014].eqiad.wmnet with reason: Rebooting pc1013 T307101
13:42 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1014 as pc3 primary T307101 (duration: 00m 52s)
13:40 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2022.codfw.wmnet
13:37 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ printf 'https://en.wikipedia.org/static/images/project-logos/zhwiki-hans%s.png\n' '-1.5x' '-2x' | mwscript purgeList.php # T276694
13:36 lucaswerkmeister-wmde@deploy1002: Synchronized static/images/project-logos/zhwiki-hans-2x.png: Config: Revert Simplified Chinese logo of zhwiki (T276694) (3/3) (duration: 00m 56s)
13:35 lucaswerkmeister-wmde@deploy1002: Synchronized static/images/project-logos/zhwiki-hans-1.5x.png: Config: Revert Simplified Chinese logo of zhwiki (T276694) (2/3) (duration: 00m 49s)
13:33 lucaswerkmeister-wmde@deploy1002: Synchronized static/images/project-logos/zhwiki-hans.png: Config: Revert Simplified Chinese logo of zhwiki (T276694) (1/3) (duration: 01m 25s)
13:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2022.codfw.wmnet
13:31 moritzm: drain ganeti-test2002 T306499
13:29 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:22 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:22 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:15 elukey: upgrade scap to 4.7.1 to all nodes (except ores[12]* since they need to be upgraded to buster first)
13:15 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2021.codfw.wmnet
12:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2021.codfw.wmnet
12:54 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
12:54 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
12:53 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
12:53 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
12:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2020.codfw.wmnet
12:41 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2020.codfw.wmnet
12:39 moritzm: installing testvm2005 T306499
12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase[2020-2022].codfw.wmnet with reason: reboot
12:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase[2020-2022].codfw.wmnet with reason: reboot
12:32 krinkle@deploy1002: Synchronized w/static.php: I0bdf0b (duration: 01m 56s)
12:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2019.codfw.wmnet
12:28 jelto@cumin1001: conftool action : set/pooled=inactive; selector: name=mw1323.eqiad.wmnet
12:24 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2019.codfw.wmnet
12:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase2019.codfw.wmnet with reason: reboot
12:05 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase2019.codfw.wmnet with reason: reboot
12:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2018.codfw.wmnet
11:57 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2005.codfw.wmnet
11:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2018.codfw.wmnet
11:56 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
11:50 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw132[4-6].eqiad.wmnet
11:49 jelto: pool name=mw[1324-1326].eqiad.wmnet , manual puppet run and icinga green after reboot (cookbook failed because of mw1323 stuck in boot)
11:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:41 jmm@cumin2002: START - Cookbook sre.dns.netbox
11:41 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2005.codfw.wmnet
11:40 jelto@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1)
11:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2017.codfw.wmnet
11:35 jynus: applying NIC firmware update onto backup2002 T286722
11:33 moritzm: powercycling restbase2017
11:22 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2017.codfw.wmnet
11:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2016.codfw.wmnet
11:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2016.codfw.wmnet
11:01 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
10:56 moritzm: failover Ganeti master in ganeti-test to ganeti-test2001 (bullseye node) T306499
10:56 moritzm: failover Ganeti master in ganeti-test to ganeti-test1001 (bullseye node) T306499
10:55 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1012 as pc2 primary T306983 (duration: 00m 57s)
10:47 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
10:47 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on pc1012.eqiad.wmnet with reason: Rebooting for T303174
10:46 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on pc1012.eqiad.wmnet with reason: Rebooting for T303174
10:46 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
10:46 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
10:46 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1014 as pc2 primary T306983 (duration: 00m 52s)
10:45 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2012.codfw.wmnet,pc[1012,1014].eqiad.wmnet with reason: Rebooting pc1012 T306983
10:45 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on pc2012.codfw.wmnet,pc[1012,1014].eqiad.wmnet with reason: Rebooting pc1012 T306983
10:45 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
10:44 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
10:44 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
10:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase[2016-2018].codfw.wmnet with reason: reboot
10:43 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase[2016-2018].codfw.wmnet with reason: reboot
10:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2015.codfw.wmnet
10:28 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2015.codfw.wmnet
10:24 elukey@deploy1002: Finished deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided) (duration: 00m 18s)
10:24 elukey@deploy1002: Started deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided)
10:23 elukey: update scap to 4.7.1 on restbase1016 (canary) - T306998
10:20 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
10:15 elukey: update scap to 4.7.1 on A:mw-canary or A:parsoid-canary or A:mw-jobrunner-canary - T306998
10:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2014.codfw.wmnet
10:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2014.codfw.wmnet
09:55 moritzm: failover idp.wikimedia.org to idp1001 for CAS update
09:51 marostegui: Change pc2014 innodb_max_dirty_pages_pct from 90% to 75% T307082
09:41 marostegui: Change db1132 innodb_max_dirty_pages_pct from 90% to 75% T307082
09:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2013.codfw.wmnet
09:32 moritzm: uploadded ganeti 3.0.1-2+deb11u0 to apt.wikimedia.org (backport of Py2->Py3 regression) T306499
09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2013.codfw.wmnet
09:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase[2013-2015].codfw.wmnet with reason: reboot
09:23 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase[2013-2015].codfw.wmnet with reason: reboot
09:01 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host restbase2012.codfw.wmnet
08:58 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
08:58 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
08:58 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
08:57 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
08:56 jayme@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
08:55 jayme@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
08:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2012.codfw.wmnet
08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase-dev1006.eqiad.wmnet
08:31 elukey: upload scap 4.7.1 to {buster,stretch,bullseye}-wikimedia apt repos - T306998
08:28 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase-dev1006.eqiad.wmnet
08:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase-dev1005.eqiad.wmnet
08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase-dev1005.eqiad.wmnet
08:15 hashar: https://gerrit.wikimedia.org/ is now running 3.4.4 # T292759
08:12 hashar: Stopping Gerrit for version ugprade # T292759
08:10 hashar@deploy1002: Finished deploy [gerrit/gerrit@031f315]: Gerrit to 3.4.4 on gerrit1001 # T292759 (duration: 00m 09s)
08:10 hashar@deploy1002: Started deploy [gerrit/gerrit@031f315]: Gerrit to 3.4.4 on gerrit1001 # T292759
08:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase-dev1004.eqiad.wmnet
08:04 hashar@deploy1002: Finished deploy [gerrit/gerrit@031f315]: Gerrit to 3.4.4 on gerrit2001 # T292759 (duration: 00m 11s)
08:04 hashar@deploy1002: Started deploy [gerrit/gerrit@031f315]: Gerrit to 3.4.4 on gerrit2001 # T292759
08:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase-dev1004.eqiad.wmnet
07:56 apergos: UTC morning backport and config training window closed
07:34 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Increase db1132 weight T301879', diff saved to https://phabricator.wikimedia.org/P26860 and previous config saved to /var/cache/conftool/dbconfig/20220428-072200-marostegui.json
07:19 kartik@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Enable SectionTranslation in testwiki for Punjabi, Tsonga, Nepali, and Swahili (T304828) (duration: 00m 50s)
07:11 kartik@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Enable Section Translation for Basque Wikipedia (T304862) (duration: 00m 50s)
02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26859 and previous config saved to /var/cache/conftool/dbconfig/20220428-021733-ladsgroup.json
02:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26858 and previous config saved to /var/cache/conftool/dbconfig/20220428-020228-ladsgroup.json
01:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26857 and previous config saved to /var/cache/conftool/dbconfig/20220428-014723-ladsgroup.json
01:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26856 and previous config saved to /var/cache/conftool/dbconfig/20220428-013218-ladsgroup.json
01:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26855 and previous config saved to /var/cache/conftool/dbconfig/20220428-011007-ladsgroup.json
01:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
01:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
01:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26854 and previous config saved to /var/cache/conftool/dbconfig/20220428-010935-ladsgroup.json
00:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26853 and previous config saved to /var/cache/conftool/dbconfig/20220428-005430-ladsgroup.json
00:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26852 and previous config saved to /var/cache/conftool/dbconfig/20220428-003925-ladsgroup.json
00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26851 and previous config saved to /var/cache/conftool/dbconfig/20220428-002420-ladsgroup.json
00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T306560)', diff saved to https://phabricator.wikimedia.org/P26850 and previous config saved to /var/cache/conftool/dbconfig/20220428-002354-ladsgroup.json
00:15 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-web1001.eqiad.wmnet
00:10 razzi@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-web1001.eqiad.wmnet
00:09 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on an-web1001.eqiad.wmnet with reason: Restart for kernel upgrade
00:09 razzi@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on an-web1001.eqiad.wmnet with reason: Restart for kernel upgrade
00:09 razzi@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on an-web1001.eqiad.wmnet with reason: Restart for kernel upgrade
00:09 razzi@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on an-web1001.eqiad.wmnet with reason: Restart for kernel upgrade
00:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26849 and previous config saved to /var/cache/conftool/dbconfig/20220428-000849-ladsgroup.json
00:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26848 and previous config saved to /var/cache/conftool/dbconfig/20220428-000317-ladsgroup.json
00:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
00:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
00:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26847 and previous config saved to /var/cache/conftool/dbconfig/20220428-000244-ladsgroup.json

2022-04-27

23:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298563)', diff saved to https://phabricator.wikimedia.org/P26846 and previous config saved to /var/cache/conftool/dbconfig/20220427-235953-ladsgroup.json
23:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298558)', diff saved to https://phabricator.wikimedia.org/P26845 and previous config saved to /var/cache/conftool/dbconfig/20220427-235700-ladsgroup.json
23:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26844 and previous config saved to /var/cache/conftool/dbconfig/20220427-235344-ladsgroup.json
23:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26843 and previous config saved to /var/cache/conftool/dbconfig/20220427-234739-ladsgroup.json
23:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26842 and previous config saved to /var/cache/conftool/dbconfig/20220427-234448-ladsgroup.json
23:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26841 and previous config saved to /var/cache/conftool/dbconfig/20220427-234155-ladsgroup.json
23:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T306560)', diff saved to https://phabricator.wikimedia.org/P26840 and previous config saved to /var/cache/conftool/dbconfig/20220427-233839-ladsgroup.json
23:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T306560)', diff saved to https://phabricator.wikimedia.org/P26839 and previous config saved to /var/cache/conftool/dbconfig/20220427-233628-ladsgroup.json
23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance
23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance
23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
23:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
23:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
23:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
23:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P26838 and previous config saved to /var/cache/conftool/dbconfig/20220427-233541-ladsgroup.json
23:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26837 and previous config saved to /var/cache/conftool/dbconfig/20220427-233234-ladsgroup.json
23:31 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host gitlab-runner2002.codfw.wmnet with OS bullseye
23:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26836 and previous config saved to /var/cache/conftool/dbconfig/20220427-232942-ladsgroup.json
23:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26835 and previous config saved to /var/cache/conftool/dbconfig/20220427-232650-ladsgroup.json
23:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26834 and previous config saved to /var/cache/conftool/dbconfig/20220427-232036-ladsgroup.json
23:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26833 and previous config saved to /var/cache/conftool/dbconfig/20220427-231729-ladsgroup.json
23:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298563)', diff saved to https://phabricator.wikimedia.org/P26832 and previous config saved to /var/cache/conftool/dbconfig/20220427-231437-ladsgroup.json
23:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298558)', diff saved to https://phabricator.wikimedia.org/P26831 and previous config saved to /var/cache/conftool/dbconfig/20220427-231145-ladsgroup.json
23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T298558)', diff saved to https://phabricator.wikimedia.org/P26830 and previous config saved to /var/cache/conftool/dbconfig/20220427-231037-ladsgroup.json
23:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
23:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298558)', diff saved to https://phabricator.wikimedia.org/P26829 and previous config saved to /var/cache/conftool/dbconfig/20220427-231029-ladsgroup.json
23:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26828 and previous config saved to /var/cache/conftool/dbconfig/20220427-230531-ladsgroup.json
23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298563)', diff saved to https://phabricator.wikimedia.org/P26827 and previous config saved to /var/cache/conftool/dbconfig/20220427-230130-ladsgroup.json
23:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
23:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
23:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
23:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298563)', diff saved to https://phabricator.wikimedia.org/P26826 and previous config saved to /var/cache/conftool/dbconfig/20220427-230116-ladsgroup.json
22:59 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab-runner2002.codfw.wmnet with OS bullseye
22:58 pt1979@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host gitlab2002.wikimedia.org with OS bullseye
22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26825 and previous config saved to /var/cache/conftool/dbconfig/20220427-225524-ladsgroup.json
22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26824 and previous config saved to /var/cache/conftool/dbconfig/20220427-225517-ladsgroup.json
22:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
22:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
22:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26823 and previous config saved to /var/cache/conftool/dbconfig/20220427-225445-ladsgroup.json
22:53 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye
22:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P26822 and previous config saved to /var/cache/conftool/dbconfig/20220427-225026-ladsgroup.json
22:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P26821 and previous config saved to /var/cache/conftool/dbconfig/20220427-224711-ladsgroup.json
22:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
22:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
22:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
22:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T306560)', diff saved to https://phabricator.wikimedia.org/P26820 and previous config saved to /var/cache/conftool/dbconfig/20220427-224654-ladsgroup.json
22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26819 and previous config saved to /var/cache/conftool/dbconfig/20220427-224610-ladsgroup.json
22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26818 and previous config saved to /var/cache/conftool/dbconfig/20220427-224019-ladsgroup.json
22:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26817 and previous config saved to /var/cache/conftool/dbconfig/20220427-223940-ladsgroup.json
22:39 pt1979@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host gitlab2002.wikimedia.org with OS bullseye
22:34 mutante: kubernetes - Uprading release=namespaces/namspace-certificates which added developer-portal and image-suggestion namespaces - but only on staging-codfw - (T304891, T305155, T297140)
22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26816 and previous config saved to /var/cache/conftool/dbconfig/20220427-223149-ladsgroup.json
22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26815 and previous config saved to /var/cache/conftool/dbconfig/20220427-223105-ladsgroup.json
22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298558)', diff saved to https://phabricator.wikimedia.org/P26814 and previous config saved to /var/cache/conftool/dbconfig/20220427-222514-ladsgroup.json
22:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26813 and previous config saved to /var/cache/conftool/dbconfig/20220427-222435-ladsgroup.json
22:24 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye
22:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298558)', diff saved to https://phabricator.wikimedia.org/P26812 and previous config saved to /var/cache/conftool/dbconfig/20220427-222306-ladsgroup.json
22:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
22:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
22:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
22:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
22:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
22:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
22:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
22:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
22:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
22:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
22:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298558)', diff saved to https://phabricator.wikimedia.org/P26811 and previous config saved to /var/cache/conftool/dbconfig/20220427-222230-ladsgroup.json
22:19 dzahn@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
22:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner2004.mgmt.codfw.wmnet with reboot policy FORCED
22:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26810 and previous config saved to /var/cache/conftool/dbconfig/20220427-221644-ladsgroup.json
22:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298563)', diff saved to https://phabricator.wikimedia.org/P26809 and previous config saved to /var/cache/conftool/dbconfig/20220427-221600-ladsgroup.json
22:12 dzahn@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
22:09 mutante: running puppet on kubemasters - adding namespace to kubernetes for new service image-suggestion (T304891, T305155)
22:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26808 and previous config saved to /var/cache/conftool/dbconfig/20220427-220930-ladsgroup.json
22:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26807 and previous config saved to /var/cache/conftool/dbconfig/20220427-220725-ladsgroup.json
22:04 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
22:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T306560)', diff saved to https://phabricator.wikimedia.org/P26806 and previous config saved to /var/cache/conftool/dbconfig/20220427-220139-ladsgroup.json
21:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T298563)', diff saved to https://phabricator.wikimedia.org/P26805 and previous config saved to /var/cache/conftool/dbconfig/20220427-215914-ladsgroup.json
21:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1096.eqiad.wmnet with reason: Maintenance
21:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1096.eqiad.wmnet with reason: Maintenance
21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T306560)', diff saved to https://phabricator.wikimedia.org/P26804 and previous config saved to /var/cache/conftool/dbconfig/20220427-215828-ladsgroup.json
21:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
21:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T306560)', diff saved to https://phabricator.wikimedia.org/P26803 and previous config saved to /var/cache/conftool/dbconfig/20220427-215820-ladsgroup.json
21:56 pt1979@cumin2002: START - Cookbook sre.dns.netbox
21:54 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
21:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26802 and previous config saved to /var/cache/conftool/dbconfig/20220427-215220-ladsgroup.json
21:50 pt1979@cumin2002: START - Cookbook sre.dns.netbox
21:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26801 and previous config saved to /var/cache/conftool/dbconfig/20220427-214825-ladsgroup.json
21:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
21:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
21:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26800 and previous config saved to /var/cache/conftool/dbconfig/20220427-214752-ladsgroup.json
21:46 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=codfw,name=mw2412.codfw.wmnet
21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26799 and previous config saved to /var/cache/conftool/dbconfig/20220427-214315-ladsgroup.json
21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298558)', diff saved to https://phabricator.wikimedia.org/P26798 and previous config saved to /var/cache/conftool/dbconfig/20220427-213715-ladsgroup.json
21:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T298558)', diff saved to https://phabricator.wikimedia.org/P26797 and previous config saved to /var/cache/conftool/dbconfig/20220427-213507-ladsgroup.json
21:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
21:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
21:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298558)', diff saved to https://phabricator.wikimedia.org/P26796 and previous config saved to /var/cache/conftool/dbconfig/20220427-213458-ladsgroup.json
21:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26795 and previous config saved to /var/cache/conftool/dbconfig/20220427-213247-ladsgroup.json
21:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26794 and previous config saved to /var/cache/conftool/dbconfig/20220427-212810-ladsgroup.json
21:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26793 and previous config saved to /var/cache/conftool/dbconfig/20220427-211953-ladsgroup.json
21:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26792 and previous config saved to /var/cache/conftool/dbconfig/20220427-211742-ladsgroup.json
21:16 ebernhardson@deploy1002: Synchronized wmf-config/ProductionServices.php: Config: Revert: cirrus: Enable DeprecationLoggedHttps (T218994) (duration: 00m 58s)
21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 (T298563)', diff saved to https://phabricator.wikimedia.org/P26791 and previous config saved to /var/cache/conftool/dbconfig/20220427-211352-ladsgroup.json
21:13 ebernhardson@deploy1002: Synchronized wmf-config/ProductionServices.php: Config: cirrus: Enable DeprecationLoggedHttps (T218994) (duration: 00m 54s)
21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T306560)', diff saved to https://phabricator.wikimedia.org/P26790 and previous config saved to /var/cache/conftool/dbconfig/20220427-211305-ladsgroup.json
21:12 ebernhardson@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Forward CirrusSearchDeprecation logs to logstash (T218994) (duration: 00m 56s)
21:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T306560)', diff saved to https://phabricator.wikimedia.org/P26789 and previous config saved to /var/cache/conftool/dbconfig/20220427-211055-ladsgroup.json
21:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
21:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
21:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
21:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
21:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P26788 and previous config saved to /var/cache/conftool/dbconfig/20220427-211042-ladsgroup.json
21:09 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab-runner2004.mgmt.codfw.wmnet with reboot policy FORCED
21:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner2003.mgmt.codfw.wmnet with reboot policy FORCED
21:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26787 and previous config saved to /var/cache/conftool/dbconfig/20220427-210448-ladsgroup.json
21:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26786 and previous config saved to /var/cache/conftool/dbconfig/20220427-210237-ladsgroup.json
21:01 ebernhardson@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: cirrus: Turn on AB test of wbsearchentities profiles (T306644) (duration: 00m 53s)
21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 (T298563)', diff saved to https://phabricator.wikimedia.org/P26785 and previous config saved to /var/cache/conftool/dbconfig/20220427-210041-ladsgroup.json
21:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance
21:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance
20:59 dzahn@cumin2002: conftool action : set/pooled=yes; selector: dc=eqiad,name=ms-fe1009.eqiad.wmnet
20:56 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,name=ms-fe1009.eqiad.wmnet
20:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26784 and previous config saved to /var/cache/conftool/dbconfig/20220427-205537-ladsgroup.json
20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298558)', diff saved to https://phabricator.wikimedia.org/P26783 and previous config saved to /var/cache/conftool/dbconfig/20220427-204943-ladsgroup.json
20:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
20:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
20:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T298558)', diff saved to https://phabricator.wikimedia.org/P26782 and previous config saved to /var/cache/conftool/dbconfig/20220427-204736-ladsgroup.json
20:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
20:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
20:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298558)', diff saved to https://phabricator.wikimedia.org/P26781 and previous config saved to /var/cache/conftool/dbconfig/20220427-204728-ladsgroup.json
20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26780 and previous config saved to /var/cache/conftool/dbconfig/20220427-204032-ladsgroup.json
20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26779 and previous config saved to /var/cache/conftool/dbconfig/20220427-204031-ladsgroup.json
20:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
20:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26778 and previous config saved to /var/cache/conftool/dbconfig/20220427-203959-ladsgroup.json
20:39 mutante: alert1001 - systemctl start certspotter
20:36 mutante: ms-fe1009 - systemctl restart cron
20:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 8 hosts with reason: Maintenance
20:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 8 hosts with reason: Maintenance
20:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance
20:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance
20:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26777 and previous config saved to /var/cache/conftool/dbconfig/20220427-203223-ladsgroup.json
20:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P26776 and previous config saved to /var/cache/conftool/dbconfig/20220427-202527-ladsgroup.json
20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26775 and previous config saved to /var/cache/conftool/dbconfig/20220427-202454-ladsgroup.json
20:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P26774 and previous config saved to /var/cache/conftool/dbconfig/20220427-202314-ladsgroup.json
20:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
20:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
20:23 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab-runner2003.mgmt.codfw.wmnet with reboot policy FORCED
20:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T306560)', diff saved to https://phabricator.wikimedia.org/P26773 and previous config saved to /var/cache/conftool/dbconfig/20220427-202306-ladsgroup.json
20:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
20:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
20:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298563)', diff saved to https://phabricator.wikimedia.org/P26772 and previous config saved to /var/cache/conftool/dbconfig/20220427-202234-ladsgroup.json
20:21 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner2002.mgmt.codfw.wmnet with reboot policy FORCED
20:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26771 and previous config saved to /var/cache/conftool/dbconfig/20220427-201718-ladsgroup.json
20:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26770 and previous config saved to /var/cache/conftool/dbconfig/20220427-200949-ladsgroup.json
20:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26769 and previous config saved to /var/cache/conftool/dbconfig/20220427-200800-ladsgroup.json
20:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26768 and previous config saved to /var/cache/conftool/dbconfig/20220427-200729-ladsgroup.json
20:06 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Halt the DiscussionTools A/B test (T291873) (duration: 00m 51s)
20:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298558)', diff saved to https://phabricator.wikimedia.org/P26767 and previous config saved to /var/cache/conftool/dbconfig/20220427-200213-ladsgroup.json
20:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T298558)', diff saved to https://phabricator.wikimedia.org/P26766 and previous config saved to /var/cache/conftool/dbconfig/20220427-200006-ladsgroup.json
20:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
20:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
20:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
20:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
19:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298558)', diff saved to https://phabricator.wikimedia.org/P26765 and previous config saved to /var/cache/conftool/dbconfig/20220427-195953-ladsgroup.json
19:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26764 and previous config saved to /var/cache/conftool/dbconfig/20220427-195444-ladsgroup.json
19:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26763 and previous config saved to /var/cache/conftool/dbconfig/20220427-195255-ladsgroup.json
19:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26762 and previous config saved to /var/cache/conftool/dbconfig/20220427-195224-ladsgroup.json
19:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26761 and previous config saved to /var/cache/conftool/dbconfig/20220427-194448-ladsgroup.json
19:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T306560)', diff saved to https://phabricator.wikimedia.org/P26760 and previous config saved to /var/cache/conftool/dbconfig/20220427-193749-ladsgroup.json
19:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298563)', diff saved to https://phabricator.wikimedia.org/P26759 and previous config saved to /var/cache/conftool/dbconfig/20220427-193719-ladsgroup.json
19:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26758 and previous config saved to /var/cache/conftool/dbconfig/20220427-193233-ladsgroup.json
19:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
19:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
19:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26757 and previous config saved to /var/cache/conftool/dbconfig/20220427-193201-ladsgroup.json
19:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26756 and previous config saved to /var/cache/conftool/dbconfig/20220427-192943-ladsgroup.json
19:28 otto@deploy1002: Finished deploy [airflow-dags/analytics@6684963]: (no justification provided) (duration: 00m 09s)
19:28 otto@deploy1002: Started deploy [airflow-dags/analytics@6684963]: (no justification provided)
19:24 otto@deploy1002: Finished deploy [airflow-dags/analytics_test@6684963]: (no justification provided) (duration: 00m 11s)
19:24 otto@deploy1002: Started deploy [airflow-dags/analytics_test@6684963]: (no justification provided)
19:23 otto@deploy1002: Finished deploy [airflow-dags/analytics_test@6684963]: (no justification provided) (duration: 00m 03s)
19:23 otto@deploy1002: Started deploy [airflow-dags/analytics_test@6684963]: (no justification provided)
19:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T298563)', diff saved to https://phabricator.wikimedia.org/P26755 and previous config saved to /var/cache/conftool/dbconfig/20220427-192209-ladsgroup.json
19:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1110.eqiad.wmnet with reason: Maintenance
19:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1110.eqiad.wmnet with reason: Maintenance
19:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298563)', diff saved to https://phabricator.wikimedia.org/P26754 and previous config saved to /var/cache/conftool/dbconfig/20220427-192200-ladsgroup.json
19:21 otto@deploy1002: Finished deploy [airflow-dags/analytics_test@6684963]: (no justification provided) (duration: 00m 03s)
19:21 otto@deploy1002: Started deploy [airflow-dags/analytics_test@6684963]: (no justification provided)
19:18 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab-runner2002.mgmt.codfw.wmnet with reboot policy FORCED
19:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26753 and previous config saved to /var/cache/conftool/dbconfig/20220427-191656-ladsgroup.json
19:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298558)', diff saved to https://phabricator.wikimedia.org/P26752 and previous config saved to /var/cache/conftool/dbconfig/20220427-191438-ladsgroup.json
19:08 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab2003.mgmt.codfw.wmnet with reboot policy FORCED
19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26751 and previous config saved to /var/cache/conftool/dbconfig/20220427-190655-ladsgroup.json
19:02 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1004.eqiad.wmnet
19:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26750 and previous config saved to /var/cache/conftool/dbconfig/20220427-190151-ladsgroup.json
19:00 dancy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
19:00 dancy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26749 and previous config saved to /var/cache/conftool/dbconfig/20220427-185150-ladsgroup.json
18:51 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be1004.eqiad.wmnet
18:50 dancy@deploy1002: Started scap: testing mediawiki container image build and deploy
18:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26748 and previous config saved to /var/cache/conftool/dbconfig/20220427-184646-ladsgroup.json
18:46 otto@deploy1002: Finished deploy [airflow-dags/analytics_test@6684963]: (no justification provided) (duration: 00m 12s)
18:46 otto@deploy1002: Started deploy [airflow-dags/analytics_test@6684963]: (no justification provided)
18:45 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab2003.mgmt.codfw.wmnet with reboot policy FORCED
18:45 dancy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:45 dancy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:40 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:40 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:38 dancy@deploy1002: Started scap: (no justification provided)
18:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T306560)', diff saved to https://phabricator.wikimedia.org/P26746 and previous config saved to /var/cache/conftool/dbconfig/20220427-183735-ladsgroup.json
18:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
18:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
18:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P26745 and previous config saved to /var/cache/conftool/dbconfig/20220427-183727-ladsgroup.json
18:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298563)', diff saved to https://phabricator.wikimedia.org/P26744 and previous config saved to /var/cache/conftool/dbconfig/20220427-183645-ladsgroup.json
18:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab2002.mgmt.codfw.wmnet with reboot policy FORCED
18:26 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1003.eqiad.wmnet
18:25 brennen@deploy1002: Synchronized php: group1 wikis to 1.39.0-wmf.9 refs T305215 (duration: 00m 56s)
18:24 brennen@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.9 refs T305215
18:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T298563)', diff saved to https://phabricator.wikimedia.org/P26743 and previous config saved to /var/cache/conftool/dbconfig/20220427-182238-ladsgroup.json
18:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
18:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
18:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298563)', diff saved to https://phabricator.wikimedia.org/P26742 and previous config saved to /var/cache/conftool/dbconfig/20220427-182230-ladsgroup.json
18:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 (T298565)', diff saved to https://phabricator.wikimedia.org/P26741 and previous config saved to /var/cache/conftool/dbconfig/20220427-182226-ladsgroup.json
18:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26740 and previous config saved to /var/cache/conftool/dbconfig/20220427-182222-ladsgroup.json
18:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
18:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
18:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:20 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T298558)', diff saved to https://phabricator.wikimedia.org/P26739 and previous config saved to /var/cache/conftool/dbconfig/20220427-181423-ladsgroup.json
18:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
18:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298558)', diff saved to https://phabricator.wikimedia.org/P26738 and previous config saved to /var/cache/conftool/dbconfig/20220427-181415-ladsgroup.json
18:14 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be1003.eqiad.wmnet
18:12 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab2002.mgmt.codfw.wmnet with reboot policy FORCED
18:11 brennen@deploy1002: Synchronized php-1.39.0-wmf.9/skins/WikimediaApiPortal: Backport: Fix warnings relating to QuickTemplate (T306925) (duration: 02m 26s)
18:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26737 and previous config saved to /var/cache/conftool/dbconfig/20220427-180725-ladsgroup.json
18:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26736 and previous config saved to /var/cache/conftool/dbconfig/20220427-180717-ladsgroup.json
18:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:00 brennen: train 1.39.0-wmf.9 (T305215): no current blockers, proceeding to group1
17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26735 and previous config saved to /var/cache/conftool/dbconfig/20220427-175910-ladsgroup.json
17:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
17:54 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
17:54 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
17:54 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26734 and previous config saved to /var/cache/conftool/dbconfig/20220427-175220-ladsgroup.json
17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P26733 and previous config saved to /var/cache/conftool/dbconfig/20220427-175212-ladsgroup.json
17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P26732 and previous config saved to /var/cache/conftool/dbconfig/20220427-175100-ladsgroup.json
17:51 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
17:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
17:49 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
17:49 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
17:49 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
17:49 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
17:45 pt1979@cumin2002: START - Cookbook sre.dns.netbox
17:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26731 and previous config saved to /var/cache/conftool/dbconfig/20220427-174405-ladsgroup.json
17:41 ejegg: updated payments-wiki from 16c0c111 to 41e91033
17:39 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1002.eqiad.wmnet
17:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298563)', diff saved to https://phabricator.wikimedia.org/P26730 and previous config saved to /var/cache/conftool/dbconfig/20220427-173709-ladsgroup.json
17:36 dancy: dancy@deploy1002 Testing image build and deployment
17:35 dancy@deploy1002: Started scap: (no justification provided)
17:29 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be1002.eqiad.wmnet
17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298558)', diff saved to https://phabricator.wikimedia.org/P26729 and previous config saved to /var/cache/conftool/dbconfig/20220427-172900-ladsgroup.json
17:29 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: 88ac6b9: Enable Wikistories on enwiki beta (T303004; 2/2) (duration: 00m 50s)
17:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
17:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
17:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
17:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
17:28 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 88ac6b9: Enable Wikistories on enwiki beta (T303004; 1/2) (duration: 00m 51s)
17:27 urbanecm@deploy1002: Synchronized wmf-config/extension-list: 01dfaf0: Add Wikistories extension (T303004) (duration: 00m 49s)
17:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298558)', diff saved to https://phabricator.wikimedia.org/P26728 and previous config saved to /var/cache/conftool/dbconfig/20220427-172653-ladsgroup.json
17:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
17:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
17:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
17:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
17:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
17:23 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298563)', diff saved to https://phabricator.wikimedia.org/P26727 and previous config saved to /var/cache/conftool/dbconfig/20220427-172211-ladsgroup.json
17:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1113.eqiad.wmnet with reason: Maintenance
17:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1113.eqiad.wmnet with reason: Maintenance
17:10 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet[2002,2004]-dev.codfw.wmnet
17:01 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:58 andrew@cumin1001: START - Cookbook sre.dns.netbox
16:56 herron@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host thanos-be1001.eqiad.wmnet
16:53 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudnet[2002,2004]-dev.codfw.wmnet
16:52 andrew@cumin1001: END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) for hosts cloudnet[2002,2004]-dev.codfw.wmnet
16:52 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudnet[2002,2004]-dev.codfw.wmnet
16:46 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
16:43 dancy@deploy1002: Finished deploy [releng/phatality@d8e2adc]: (no justification provided) (duration: 00m 13s)
16:43 dancy@deploy1002: Started deploy [releng/phatality@d8e2adc]: (no justification provided)
16:41 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be1001.eqiad.wmnet
16:35 razzi@cumin1001: conftool action : set/pooled=inactive; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
16:34 razzi@cumin1001: conftool action : set/pooled=yes; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
16:28 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
16:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T306560)', diff saved to https://phabricator.wikimedia.org/P26725 and previous config saved to /var/cache/conftool/dbconfig/20220427-161459-ladsgroup.json
16:04 Amir1: foreachwikiindblist s6 mysql.php -- -e "desc querycache_info;" | grep -i qci_timestamp | grep -i varbinary | awk '{ print substr($1, 1, length($1)-1) }' | xargs -I {} sql {} --write -- -e 'ALTER TABLE /*_*/querycache_info CHANGE qci_timestamp qci_timestamp BINARY(14) DEFAULT '19700101000000' NOT NULL;' (T298559)
15:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26724 and previous config saved to /var/cache/conftool/dbconfig/20220427-155954-ladsgroup.json
15:58 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1003.eqiad.wmnet
15:56 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
15:56 jayme@deploy1002: helmfile [staging] START helmfile.d/services/miscweb: apply
15:52 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2004.codfw.wmnet
15:50 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup1003.eqiad.wmnet
15:50 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet
15:48 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup1002-dev.eqiad.wmnet
15:47 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1001-dev.eqiad.wmnet
15:45 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup1001-dev.eqiad.wmnet
15:45 nokafor@deploy1002: Finished deploy [airflow-dags/analytics@6684963]: (no justification provided) (duration: 00m 08s)
15:45 nokafor@deploy1002: Started deploy [airflow-dags/analytics@6684963]: (no justification provided)
15:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26723 and previous config saved to /var/cache/conftool/dbconfig/20220427-154449-ladsgroup.json
15:39 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be2004.codfw.wmnet
15:36 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
15:35 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
15:33 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
15:31 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
15:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T306560)', diff saved to https://phabricator.wikimedia.org/P26722 and previous config saved to /var/cache/conftool/dbconfig/20220427-152944-ladsgroup.json
15:02 moritzm: installing mariadb-10.5 updates (as packaged in Debian Bullseye, unrelated to wmf-mariadb packages)
14:57 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2003.codfw.wmnet
14:54 razzi@cumin1001: conftool action : set/pooled=inactive; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
14:54 razzi@cumin1001: conftool action : set/pooled=no; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
14:53 razzi@cumin1001: conftool action : set/pooled=yes; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
14:47 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be2003.codfw.wmnet
14:47 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:46 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:46 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
14:43 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Enable videojs in eswiki (T303785 T248418) (duration: 00m 51s)
14:42 moritzm: imported cas 6.4.6.3 to apt.wikimedia.org
14:27 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
14:23 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
14:22 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2002.codfw.wmnet
14:21 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T306560)', diff saved to https://phabricator.wikimedia.org/P26720 and previous config saved to /var/cache/conftool/dbconfig/20220427-141215-ladsgroup.json
14:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
14:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
14:09 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be2002.codfw.wmnet
14:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1100.eqiad.wmnet with reason: Maintenance
14:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1100.eqiad.wmnet with reason: Maintenance
14:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1173.eqiad.wmnet with reason: Maintenance
14:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1173.eqiad.wmnet with reason: Maintenance
14:07 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus6001.drmrs.wmnet
14:03 klausman@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
14:03 klausman@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
14:00 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host prometheus6001.drmrs.wmnet
13:59 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus4001.ulsfo.wmnet
13:54 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host prometheus4001.ulsfo.wmnet
13:53 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus5001.eqsin.wmnet
13:51 jayme@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:47 jayme@cumin1001: START - Cookbook sre.dns.netbox
13:46 moritzm: rebalance ganeti-test after adding new bullseye node T306499
13:46 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host prometheus5001.eqsin.wmnet
13:45 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus3001.esams.wmnet
13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T306560)', diff saved to https://phabricator.wikimedia.org/P26719 and previous config saved to /var/cache/conftool/dbconfig/20220427-134308-ladsgroup.json
13:40 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host prometheus3001.esams.wmnet
13:37 mvernon@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2040.codfw.wmnet with OS bullseye
13:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
13:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298556)', diff saved to https://phabricator.wikimedia.org/P26718 and previous config saved to /var/cache/conftool/dbconfig/20220427-133537-ladsgroup.json
13:35 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26717 and previous config saved to /var/cache/conftool/dbconfig/20220427-132802-ladsgroup.json
13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26716 and previous config saved to /var/cache/conftool/dbconfig/20220427-132032-ladsgroup.json
13:13 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2040.codfw.wmnet with OS bullseye
13:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26715 and previous config saved to /var/cache/conftool/dbconfig/20220427-131257-ladsgroup.json
13:11 mvernon@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2040.codfw.wmnet with OS bullseye
13:05 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
13:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26714 and previous config saved to /var/cache/conftool/dbconfig/20220427-130527-ladsgroup.json
12:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T306560)', diff saved to https://phabricator.wikimedia.org/P26712 and previous config saved to /var/cache/conftool/dbconfig/20220427-125752-ladsgroup.json
12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 (T306560)', diff saved to https://phabricator.wikimedia.org/P26710 and previous config saved to /var/cache/conftool/dbconfig/20220427-125427-ladsgroup.json
12:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
12:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
12:53 kormat: moved pc1014 to pc2 T303174
12:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298556)', diff saved to https://phabricator.wikimedia.org/P26709 and previous config saved to /var/cache/conftool/dbconfig/20220427-125022-ladsgroup.json
12:47 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2040.codfw.wmnet with OS bullseye
12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298556)', diff saved to https://phabricator.wikimedia.org/P26708 and previous config saved to /var/cache/conftool/dbconfig/20220427-124714-ladsgroup.json
12:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
12:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
12:42 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on eqiad
12:42 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2010.codfw.wmnet with OS bullseye
12:28 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2010.codfw.wmnet with reason: host reimage
12:25 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on drmrs
12:25 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2010.codfw.wmnet with reason: host reimage
12:15 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on esams
12:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
12:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
12:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
12:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
12:07 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe2010.codfw.wmnet with OS bullseye
12:07 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Set actor migration to READ NEW everywhere (T275246) (duration: 00m 53s)
11:58 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2011.codfw.wmnet with OS bullseye
11:44 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2011.codfw.wmnet with reason: host reimage
11:42 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
11:41 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2011.codfw.wmnet with reason: host reimage
11:24 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe2011.codfw.wmnet with OS bullseye
11:15 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2012.codfw.wmnet with OS bullseye
11:02 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2012.codfw.wmnet with reason: host reimage
10:58 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2012.codfw.wmnet with reason: host reimage
10:51 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on eqsin
10:40 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe2012.codfw.wmnet with OS bullseye
10:25 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on codfw
10:03 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1010.eqiad.wmnet with OS bullseye
09:50 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1010.eqiad.wmnet with reason: host reimage
09:47 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1010.eqiad.wmnet with reason: host reimage
09:44 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on ulsfo
09:33 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe1010.eqiad.wmnet with OS bullseye
09:30 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1446.eqiad.wmnet
09:30 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1445.eqiad.wmnet
09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1440.eqiad.wmnet
09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1439.eqiad.wmnet
09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1438.eqiad.wmnet
09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1437.eqiad.wmnet
09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1338.eqiad.wmnet
09:29 akosiaris: repool the hosts we split off on 2022-04-23 as dedicated videoscalers in the jobrunner cluster. The videoscaling load seems to be normal again.
09:22 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1011.eqiad.wmnet with OS bullseye
09:11 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1011.eqiad.wmnet with reason: host reimage
09:08 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1011.eqiad.wmnet with reason: host reimage
08:54 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe1011.eqiad.wmnet with OS bullseye
08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298554)', diff saved to https://phabricator.wikimedia.org/P26706 and previous config saved to /var/cache/conftool/dbconfig/20220427-084941-ladsgroup.json
08:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26705 and previous config saved to /var/cache/conftool/dbconfig/20220427-083436-ladsgroup.json
08:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26704 and previous config saved to /var/cache/conftool/dbconfig/20220427-081931-ladsgroup.json
08:17 marostegui@cumin1001: dbctl commit (dc=all): 'Increase db1132 weight T301879', diff saved to https://phabricator.wikimedia.org/P26703 and previous config saved to /var/cache/conftool/dbconfig/20220427-081727-marostegui.json
08:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298554)', diff saved to https://phabricator.wikimedia.org/P26702 and previous config saved to /var/cache/conftool/dbconfig/20220427-080425-ladsgroup.json
07:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti-test2001.codfw.wmnet with reason: bullseye update
07:49 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti-test2001.codfw.wmnet with reason: bullseye update
07:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298554)', diff saved to https://phabricator.wikimedia.org/P26701 and previous config saved to /var/cache/conftool/dbconfig/20220427-073114-ladsgroup.json
07:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
07:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
07:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298554)', diff saved to https://phabricator.wikimedia.org/P26700 and previous config saved to /var/cache/conftool/dbconfig/20220427-073106-ladsgroup.json
07:24 moritzm: installing libxml2 security updates
07:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26699 and previous config saved to /var/cache/conftool/dbconfig/20220427-071601-ladsgroup.json
07:11 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
07:11 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
07:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26698 and previous config saved to /var/cache/conftool/dbconfig/20220427-070056-ladsgroup.json
06:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298554)', diff saved to https://phabricator.wikimedia.org/P26697 and previous config saved to /var/cache/conftool/dbconfig/20220427-064551-ladsgroup.json
06:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T298554)', diff saved to https://phabricator.wikimedia.org/P26696 and previous config saved to /var/cache/conftool/dbconfig/20220427-061112-ladsgroup.json
06:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
06:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
06:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298554)', diff saved to https://phabricator.wikimedia.org/P26695 and previous config saved to /var/cache/conftool/dbconfig/20220427-061104-ladsgroup.json
05:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26694 and previous config saved to /var/cache/conftool/dbconfig/20220427-055559-ladsgroup.json
05:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26693 and previous config saved to /var/cache/conftool/dbconfig/20220427-054054-ladsgroup.json
05:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298556)', diff saved to https://phabricator.wikimedia.org/P26692 and previous config saved to /var/cache/conftool/dbconfig/20220427-052555-ladsgroup.json
05:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298554)', diff saved to https://phabricator.wikimedia.org/P26691 and previous config saved to /var/cache/conftool/dbconfig/20220427-052549-ladsgroup.json
05:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26690 and previous config saved to /var/cache/conftool/dbconfig/20220427-051050-ladsgroup.json
04:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26689 and previous config saved to /var/cache/conftool/dbconfig/20220427-045545-ladsgroup.json
04:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T298554)', diff saved to https://phabricator.wikimedia.org/P26688 and previous config saved to /var/cache/conftool/dbconfig/20220427-045153-ladsgroup.json
04:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
04:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
04:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298556)', diff saved to https://phabricator.wikimedia.org/P26687 and previous config saved to /var/cache/conftool/dbconfig/20220427-044040-ladsgroup.json
04:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T298556)', diff saved to https://phabricator.wikimedia.org/P26686 and previous config saved to /var/cache/conftool/dbconfig/20220427-043732-ladsgroup.json
04:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1168.eqiad.wmnet with reason: Maintenance
04:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1168.eqiad.wmnet with reason: Maintenance
04:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298556)', diff saved to https://phabricator.wikimedia.org/P26685 and previous config saved to /var/cache/conftool/dbconfig/20220427-043724-ladsgroup.json
04:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26684 and previous config saved to /var/cache/conftool/dbconfig/20220427-042219-ladsgroup.json
04:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
04:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
04:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26683 and previous config saved to /var/cache/conftool/dbconfig/20220427-040714-ladsgroup.json
04:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
04:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
03:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
03:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
03:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298556)', diff saved to https://phabricator.wikimedia.org/P26682 and previous config saved to /var/cache/conftool/dbconfig/20220427-035208-ladsgroup.json
03:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T298556)', diff saved to https://phabricator.wikimedia.org/P26681 and previous config saved to /var/cache/conftool/dbconfig/20220427-035001-ladsgroup.json
03:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
03:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
03:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1165.eqiad.wmnet with reason: Maintenance
03:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1165.eqiad.wmnet with reason: Maintenance
03:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298556)', diff saved to https://phabricator.wikimedia.org/P26680 and previous config saved to /var/cache/conftool/dbconfig/20220427-034948-ladsgroup.json
03:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26679 and previous config saved to /var/cache/conftool/dbconfig/20220427-033443-ladsgroup.json
03:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
03:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
03:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26678 and previous config saved to /var/cache/conftool/dbconfig/20220427-031938-ladsgroup.json
03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298556)', diff saved to https://phabricator.wikimedia.org/P26677 and previous config saved to /var/cache/conftool/dbconfig/20220427-030433-ladsgroup.json
03:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T298556)', diff saved to https://phabricator.wikimedia.org/P26676 and previous config saved to /var/cache/conftool/dbconfig/20220427-030225-ladsgroup.json
03:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1131.eqiad.wmnet with reason: Maintenance
03:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1131.eqiad.wmnet with reason: Maintenance
03:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298556)', diff saved to https://phabricator.wikimedia.org/P26675 and previous config saved to /var/cache/conftool/dbconfig/20220427-030217-ladsgroup.json
02:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26674 and previous config saved to /var/cache/conftool/dbconfig/20220427-024712-ladsgroup.json
02:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 (T298554)', diff saved to https://phabricator.wikimedia.org/P26673 and previous config saved to /var/cache/conftool/dbconfig/20220427-024409-ladsgroup.json
02:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26672 and previous config saved to /var/cache/conftool/dbconfig/20220427-023207-ladsgroup.json
02:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 (T298554)', diff saved to https://phabricator.wikimedia.org/P26671 and previous config saved to /var/cache/conftool/dbconfig/20220427-022053-ladsgroup.json
02:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
02:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
02:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298554)', diff saved to https://phabricator.wikimedia.org/P26670 and previous config saved to /var/cache/conftool/dbconfig/20220427-022045-ladsgroup.json
02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298556)', diff saved to https://phabricator.wikimedia.org/P26669 and previous config saved to /var/cache/conftool/dbconfig/20220427-021702-ladsgroup.json
02:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T298556)', diff saved to https://phabricator.wikimedia.org/P26668 and previous config saved to /var/cache/conftool/dbconfig/20220427-021450-ladsgroup.json
02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1096.eqiad.wmnet with reason: Maintenance
02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1096.eqiad.wmnet with reason: Maintenance
02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 8 hosts with reason: Maintenance
02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on 8 hosts with reason: Maintenance
02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2129.codfw.wmnet with reason: Maintenance
02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2129.codfw.wmnet with reason: Maintenance
02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1140.eqiad.wmnet with reason: Maintenance
02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1140.eqiad.wmnet with reason: Maintenance
02:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298556)', diff saved to https://phabricator.wikimedia.org/P26667 and previous config saved to /var/cache/conftool/dbconfig/20220427-021405-ladsgroup.json
02:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26666 and previous config saved to /var/cache/conftool/dbconfig/20220427-020540-ladsgroup.json
01:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26665 and previous config saved to /var/cache/conftool/dbconfig/20220427-015900-ladsgroup.json
01:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26664 and previous config saved to /var/cache/conftool/dbconfig/20220427-015035-ladsgroup.json
01:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26663 and previous config saved to /var/cache/conftool/dbconfig/20220427-014355-ladsgroup.json
01:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298554)', diff saved to https://phabricator.wikimedia.org/P26662 and previous config saved to /var/cache/conftool/dbconfig/20220427-013530-ladsgroup.json
01:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298556)', diff saved to https://phabricator.wikimedia.org/P26661 and previous config saved to /var/cache/conftool/dbconfig/20220427-012850-ladsgroup.json
01:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298556)', diff saved to https://phabricator.wikimedia.org/P26660 and previous config saved to /var/cache/conftool/dbconfig/20220427-012538-ladsgroup.json
01:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1098.eqiad.wmnet with reason: Maintenance
01:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1098.eqiad.wmnet with reason: Maintenance
01:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298556)', diff saved to https://phabricator.wikimedia.org/P26659 and previous config saved to /var/cache/conftool/dbconfig/20220427-012530-ladsgroup.json
01:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26658 and previous config saved to /var/cache/conftool/dbconfig/20220427-011025-ladsgroup.json
01:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T298554)', diff saved to https://phabricator.wikimedia.org/P26657 and previous config saved to /var/cache/conftool/dbconfig/20220427-010001-ladsgroup.json
01:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
01:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
00:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298554)', diff saved to https://phabricator.wikimedia.org/P26656 and previous config saved to /var/cache/conftool/dbconfig/20220427-005953-ladsgroup.json
00:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26655 and previous config saved to /var/cache/conftool/dbconfig/20220427-005520-ladsgroup.json
00:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26654 and previous config saved to /var/cache/conftool/dbconfig/20220427-004448-ladsgroup.json
00:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298556)', diff saved to https://phabricator.wikimedia.org/P26653 and previous config saved to /var/cache/conftool/dbconfig/20220427-004015-ladsgroup.json
00:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26652 and previous config saved to /var/cache/conftool/dbconfig/20220427-002943-ladsgroup.json
00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T306560)', diff saved to https://phabricator.wikimedia.org/P26651 and previous config saved to /var/cache/conftool/dbconfig/20220427-002432-ladsgroup.json
00:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298554)', diff saved to https://phabricator.wikimedia.org/P26650 and previous config saved to /var/cache/conftool/dbconfig/20220427-001438-ladsgroup.json
00:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P26649 and previous config saved to /var/cache/conftool/dbconfig/20220427-000927-ladsgroup.json

2022-04-26

23:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P26648 and previous config saved to /var/cache/conftool/dbconfig/20220426-235422-ladsgroup.json
23:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298554)', diff saved to https://phabricator.wikimedia.org/P26647 and previous config saved to /var/cache/conftool/dbconfig/20220426-234224-ladsgroup.json
23:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
23:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
23:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
23:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
23:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298556)', diff saved to https://phabricator.wikimedia.org/P26646 and previous config saved to /var/cache/conftool/dbconfig/20220426-234000-ladsgroup.json
23:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
23:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
23:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298556)', diff saved to https://phabricator.wikimedia.org/P26645 and previous config saved to /var/cache/conftool/dbconfig/20220426-233953-ladsgroup.json
23:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T306560)', diff saved to https://phabricator.wikimedia.org/P26644 and previous config saved to /var/cache/conftool/dbconfig/20220426-233917-ladsgroup.json
23:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 (T306560)', diff saved to https://phabricator.wikimedia.org/P26643 and previous config saved to /var/cache/conftool/dbconfig/20220426-233642-ladsgroup.json
23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 14 hosts with reason: Maintenance
23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 14 hosts with reason: Maintenance
23:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance
23:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance
23:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T306560)', diff saved to https://phabricator.wikimedia.org/P26642 and previous config saved to /var/cache/conftool/dbconfig/20220426-233545-ladsgroup.json
23:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26641 and previous config saved to /var/cache/conftool/dbconfig/20220426-232447-ladsgroup.json
23:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P26640 and previous config saved to /var/cache/conftool/dbconfig/20220426-232040-ladsgroup.json
23:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26639 and previous config saved to /var/cache/conftool/dbconfig/20220426-230942-ladsgroup.json
23:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P26638 and previous config saved to /var/cache/conftool/dbconfig/20220426-230535-ladsgroup.json
22:57 tgr@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/GrowthExperiments/extension.json: Backport: Enable SkinAddFooterLinks hook (duration: 00m 51s)
22:56 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
22:56 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
22:56 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
22:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
22:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298556)', diff saved to https://phabricator.wikimedia.org/P26637 and previous config saved to /var/cache/conftool/dbconfig/20220426-225437-ladsgroup.json
22:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T298556)', diff saved to https://phabricator.wikimedia.org/P26636 and previous config saved to /var/cache/conftool/dbconfig/20220426-225326-ladsgroup.json
22:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance
22:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance
22:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
22:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
22:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
22:51 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
22:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T306560)', diff saved to https://phabricator.wikimedia.org/P26635 and previous config saved to /var/cache/conftool/dbconfig/20220426-225030-ladsgroup.json
22:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1119 (T306560)', diff saved to https://phabricator.wikimedia.org/P26634 and previous config saved to /var/cache/conftool/dbconfig/20220426-224757-ladsgroup.json
22:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
22:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
22:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T306560)', diff saved to https://phabricator.wikimedia.org/P26633 and previous config saved to /var/cache/conftool/dbconfig/20220426-224749-ladsgroup.json
22:46 tgr@deploy1002: Finished scap: backport with i18n changes: gerrit:785944, gerrit:785941 (duration: 21m 40s)
22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P26632 and previous config saved to /var/cache/conftool/dbconfig/20220426-223244-ladsgroup.json
22:25 tgr@deploy1002: Started scap: backport with i18n changes: gerrit:785944, gerrit:785941
22:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P26631 and previous config saved to /var/cache/conftool/dbconfig/20220426-221739-ladsgroup.json
22:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T306560)', diff saved to https://phabricator.wikimedia.org/P26630 and previous config saved to /var/cache/conftool/dbconfig/20220426-220234-ladsgroup.json
22:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1135 (T306560)', diff saved to https://phabricator.wikimedia.org/P26629 and previous config saved to /var/cache/conftool/dbconfig/20220426-220001-ladsgroup.json
22:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
22:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
21:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T306560)', diff saved to https://phabricator.wikimedia.org/P26628 and previous config saved to /var/cache/conftool/dbconfig/20220426-215953-ladsgroup.json
21:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P26627 and previous config saved to /var/cache/conftool/dbconfig/20220426-214448-ladsgroup.json
21:38 aqu@deploy1002: Finished deploy [airflow-dags/analytics@e5fecc9]: Fix typo in mediarequest/hourly sensor [airflow-dags/analytics@e5fecc9] (duration: 00m 07s)
21:37 aqu@deploy1002: Started deploy [airflow-dags/analytics@e5fecc9]: Fix typo in mediarequest/hourly sensor [airflow-dags/analytics@e5fecc9]
21:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P26626 and previous config saved to /var/cache/conftool/dbconfig/20220426-212943-ladsgroup.json
21:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T306560)', diff saved to https://phabricator.wikimedia.org/P26625 and previous config saved to /var/cache/conftool/dbconfig/20220426-211437-ladsgroup.json
21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1134 (T306560)', diff saved to https://phabricator.wikimedia.org/P26624 and previous config saved to /var/cache/conftool/dbconfig/20220426-211204-ladsgroup.json
21:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
21:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
21:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T306560)', diff saved to https://phabricator.wikimedia.org/P26623 and previous config saved to /var/cache/conftool/dbconfig/20220426-211156-ladsgroup.json
21:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:05 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:05 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:03 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2418.codfw.wmnet
21:03 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2417.codfw.wmnet
21:02 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2416.codfw.wmnet
21:01 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: cab0062: fix wmgVectorMaxWidthOptionsNamespaces (T300182) (duration: 01m 00s)
21:00 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:00 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:00 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:00 mutante: mw2416, mw2417, mw2418 - scap pull
20:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P26621 and previous config saved to /var/cache/conftool/dbconfig/20220426-205651-ladsgroup.json
20:50 aqu@deploy1002: Finished deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87] (duration: 00m 07s)
20:50 aqu@deploy1002: Started deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87]
20:49 urbanecm@deploy1002: Synchronized wmf-config/SearchSettingsForWikidata.php: f76bc80: Correct wbsearchentities profiles (duration: 00m 57s)
20:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P26620 and previous config saved to /var/cache/conftool/dbconfig/20220426-204146-ladsgroup.json
20:40 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2415.codfw.wmnet
20:40 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:39 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2414.codfw.wmnet
20:39 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2413.codfw.wmnet
20:38 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2412.codfw.wmnet
20:37 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.9/skins/Vector/resources/skins.vector.styles/: 019a812: [ToC] Increase threshold for ToC collapsing to 1000px (T306904) (duration: 00m 50s)
20:36 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/skins/Vector/resources/skins.vector.styles/: 31ed884: [ToC] Increase threshold for ToC collapsing to 1000px (T306904) (duration: 00m 50s)
20:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:33 mutante: mw2412, mw2413, mw2414, mw2415 - scap pull, get into production the first time
20:29 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:29 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:29 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:29 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: fe0e119: Expand max-width to login, create account, disable on Wikidata (T300182, T306834; 2/2) (duration: 00m 54s)
20:28 aqu@deploy1002: Finished deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87] (duration: 00m 17s)
20:28 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: fe0e119: Expand max-width to login, create account, disable on Wikidata (T300182, T306834; 1/2) (duration: 00m 56s)
20:27 aqu@deploy1002: Started deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87]
20:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T306560)', diff saved to https://phabricator.wikimedia.org/P26619 and previous config saved to /var/cache/conftool/dbconfig/20220426-202641-ladsgroup.json
20:24 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: e3ce97b: Enable table of contents a/b test on euwiki and hewiki, enable reading depth (T306606) (duration: 00m 52s)
20:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:24 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:24 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:24 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1184 (T306560)', diff saved to https://phabricator.wikimedia.org/P26618 and previous config saved to /var/cache/conftool/dbconfig/20220426-202407-ladsgroup.json
20:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
20:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 (T306560)', diff saved to https://phabricator.wikimedia.org/P26617 and previous config saved to /var/cache/conftool/dbconfig/20220426-202359-ladsgroup.json
20:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298556)', diff saved to https://phabricator.wikimedia.org/P26616 and previous config saved to /var/cache/conftool/dbconfig/20220426-201610-ladsgroup.json
20:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:11 urbanecm@deploy1002: Synchronized wmf-config/: 9805e61: Add wbsearchentities profiles for testing (T306644) (duration: 00m 53s)
20:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P26615 and previous config saved to /var/cache/conftool/dbconfig/20220426-200854-ladsgroup.json
20:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:05 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 080b8fc: cirrus: Turn on retry_on_conflict quirk (duration: 00m 53s)
20:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26614 and previous config saved to /var/cache/conftool/dbconfig/20220426-200105-ladsgroup.json
19:54 dzahn@cumin2002: conftool action : set/pooled=yes; selector: dc=codfw,name=mw2419.codfw.wmnet
19:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P26612 and previous config saved to /var/cache/conftool/dbconfig/20220426-195349-ladsgroup.json
19:48 mutante: mw2419 - set weight to 25 in conftool, scap pull, first time in production, jobrunner/videoscaler T290192
19:46 dzahn@cumin2002: conftool action : set/weight=25; selector: dc=codfw,name=mw2419.codfw.wmnet
19:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26611 and previous config saved to /var/cache/conftool/dbconfig/20220426-194600-ladsgroup.json
19:45 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=codfw,name=mw2419.codfw.wmnet
19:42 aqu@deploy1002: Finished deploy [analytics/refinery@96a3934] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96a3934] (duration: 07m 19s)
19:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 (T306560)', diff saved to https://phabricator.wikimedia.org/P26610 and previous config saved to /var/cache/conftool/dbconfig/20220426-193844-ladsgroup.json
19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1164 (T306560)', diff saved to https://phabricator.wikimedia.org/P26609 and previous config saved to /var/cache/conftool/dbconfig/20220426-193610-ladsgroup.json
19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1164.eqiad.wmnet with reason: Maintenance
19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1164.eqiad.wmnet with reason: Maintenance
19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T306560)', diff saved to https://phabricator.wikimedia.org/P26608 and previous config saved to /var/cache/conftool/dbconfig/20220426-193602-ladsgroup.json
19:34 aqu@deploy1002: Started deploy [analytics/refinery@96a3934] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96a3934]
19:34 aqu@deploy1002: Finished deploy [analytics/refinery@96a3934] (thin): Regular analytics weekly train THIN [analytics/refinery@96a3934] (duration: 00m 07s)
19:34 aqu@deploy1002: Started deploy [analytics/refinery@96a3934] (thin): Regular analytics weekly train THIN [analytics/refinery@96a3934]
19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298556)', diff saved to https://phabricator.wikimedia.org/P26607 and previous config saved to /var/cache/conftool/dbconfig/20220426-193055-ladsgroup.json
19:30 aqu@deploy1002: Finished deploy [analytics/refinery@96a3934]: Regular analytics weekly train [analytics/refinery@96a3934] (duration: 24m 35s)
19:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298556)', diff saved to https://phabricator.wikimedia.org/P26606 and previous config saved to /var/cache/conftool/dbconfig/20220426-192841-ladsgroup.json
19:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
19:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
19:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1161.eqiad.wmnet with reason: Maintenance
19:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1161.eqiad.wmnet with reason: Maintenance
19:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298556)', diff saved to https://phabricator.wikimedia.org/P26605 and previous config saved to /var/cache/conftool/dbconfig/20220426-192828-ladsgroup.json
19:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P26604 and previous config saved to /var/cache/conftool/dbconfig/20220426-192057-ladsgroup.json
19:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26603 and previous config saved to /var/cache/conftool/dbconfig/20220426-191323-ladsgroup.json
19:06 aqu@deploy1002: Started deploy [analytics/refinery@96a3934]: Regular analytics weekly train [analytics/refinery@96a3934]
19:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P26602 and previous config saved to /var/cache/conftool/dbconfig/20220426-190552-ladsgroup.json
19:02 aqu: About to deploy analytics/refinery: Weekly deployment train + Artifacts to 0.1.27
18:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26601 and previous config saved to /var/cache/conftool/dbconfig/20220426-185818-ladsgroup.json
18:50 cmooney@cumin1001: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.4.1a - cmooney@cumin1001
18:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T306560)', diff saved to https://phabricator.wikimedia.org/P26599 and previous config saved to /var/cache/conftool/dbconfig/20220426-185047-ladsgroup.json
18:49 cmooney@cumin1001: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.4.1a - cmooney@cumin1001
18:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T306560)', diff saved to https://phabricator.wikimedia.org/P26598 and previous config saved to /var/cache/conftool/dbconfig/20220426-184815-ladsgroup.json
18:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
18:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
18:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1106.eqiad.wmnet with reason: Maintenance
18:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1106.eqiad.wmnet with reason: Maintenance
18:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance
18:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance
18:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 (T306560)', diff saved to https://phabricator.wikimedia.org/P26597 and previous config saved to /var/cache/conftool/dbconfig/20220426-184729-ladsgroup.json
18:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298556)', diff saved to https://phabricator.wikimedia.org/P26596 and previous config saved to /var/cache/conftool/dbconfig/20220426-184313-ladsgroup.json
18:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T298556)', diff saved to https://phabricator.wikimedia.org/P26595 and previous config saved to /var/cache/conftool/dbconfig/20220426-184058-ladsgroup.json
18:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1096.eqiad.wmnet with reason: Maintenance
18:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1096.eqiad.wmnet with reason: Maintenance
18:37 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:37 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:37 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P26594 and previous config saved to /var/cache/conftool/dbconfig/20220426-183224-ladsgroup.json
18:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P26593 and previous config saved to /var/cache/conftool/dbconfig/20220426-181719-ladsgroup.json
18:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:06 brennen@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.9 refs T305215
18:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 (T306560)', diff saved to https://phabricator.wikimedia.org/P26592 and previous config saved to /var/cache/conftool/dbconfig/20220426-180214-ladsgroup.json
17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1132 (T306560)', diff saved to https://phabricator.wikimedia.org/P26591 and previous config saved to /var/cache/conftool/dbconfig/20220426-175941-ladsgroup.json
17:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance
17:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance
17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T306560)', diff saved to https://phabricator.wikimedia.org/P26590 and previous config saved to /var/cache/conftool/dbconfig/20220426-175933-ladsgroup.json
17:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 (T298556)', diff saved to https://phabricator.wikimedia.org/P26589 and previous config saved to /var/cache/conftool/dbconfig/20220426-175536-ladsgroup.json
17:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 (T298556)', diff saved to https://phabricator.wikimedia.org/P26588 and previous config saved to /var/cache/conftool/dbconfig/20220426-175424-ladsgroup.json
17:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1130.eqiad.wmnet with reason: Maintenance
17:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1130.eqiad.wmnet with reason: Maintenance
17:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
17:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
17:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 8 hosts with reason: Maintenance
17:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on 8 hosts with reason: Maintenance
17:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2123.codfw.wmnet with reason: Maintenance
17:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2123.codfw.wmnet with reason: Maintenance
17:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1150.eqiad.wmnet with reason: Maintenance
17:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1150.eqiad.wmnet with reason: Maintenance
17:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298556)', diff saved to https://phabricator.wikimedia.org/P26587 and previous config saved to /var/cache/conftool/dbconfig/20220426-175322-ladsgroup.json
17:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P26586 and previous config saved to /var/cache/conftool/dbconfig/20220426-174428-ladsgroup.json
17:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26585 and previous config saved to /var/cache/conftool/dbconfig/20220426-173817-ladsgroup.json
17:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
17:36 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
17:36 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
17:36 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
17:30 brennen@deploy1002: Pruned MediaWiki: 1.39.0-wmf.7 (duration: 01m 29s)
17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P26584 and previous config saved to /var/cache/conftool/dbconfig/20220426-172923-ladsgroup.json
17:28 brennen@deploy1002: Finished scap: Re-running sync-world to see if timeouts recur for 32 hosts (T305215) (duration: 01m 43s)
17:26 brennen@deploy1002: Started scap: Re-running sync-world to see if timeouts recur for 32 hosts (T305215)
17:23 mutante: mw2309 - scap pull
17:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26583 and previous config saved to /var/cache/conftool/dbconfig/20220426-172312-ladsgroup.json
17:23 brennen@deploy1002: Finished scap: testwikis wikis to 1.39.0-wmf.9 refs T305215 (duration: 34m 37s)
17:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
17:21 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
17:21 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
17:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
17:15 mutante: wtp1046 - scap pull
17:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T306560)', diff saved to https://phabricator.wikimedia.org/P26582 and previous config saved to /var/cache/conftool/dbconfig/20220426-171418-ladsgroup.json
17:13 mutante: mw1362 - scap pull
17:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 (T306560)', diff saved to https://phabricator.wikimedia.org/P26580 and previous config saved to /var/cache/conftool/dbconfig/20220426-171144-ladsgroup.json
17:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
17:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
17:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
17:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
17:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
17:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
17:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T306560)', diff saved to https://phabricator.wikimedia.org/P26579 and previous config saved to /var/cache/conftool/dbconfig/20220426-171032-ladsgroup.json
17:09 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddb1021.eqiad.wmnet with OS bullseye
17:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298556)', diff saved to https://phabricator.wikimedia.org/P26578 and previous config saved to /var/cache/conftool/dbconfig/20220426-170807-ladsgroup.json
17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T298556)', diff saved to https://phabricator.wikimedia.org/P26577 and previous config saved to /var/cache/conftool/dbconfig/20220426-170553-ladsgroup.json
17:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1110.eqiad.wmnet with reason: Maintenance
17:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1110.eqiad.wmnet with reason: Maintenance
17:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298556)', diff saved to https://phabricator.wikimedia.org/P26576 and previous config saved to /var/cache/conftool/dbconfig/20220426-170545-ladsgroup.json
16:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P26575 and previous config saved to /var/cache/conftool/dbconfig/20220426-165526-ladsgroup.json
16:55 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1021.eqiad.wmnet with reason: host reimage
16:51 razzi@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on clouddb1021.eqiad.wmnet with reason: host reimage
16:50 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
16:50 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
16:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26574 and previous config saved to /var/cache/conftool/dbconfig/20220426-165040-ladsgroup.json
16:50 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
16:48 brennen@deploy1002: Started scap: testwikis wikis to 1.39.0-wmf.9 refs T305215
16:47 brennen: forgot SCAP=scap environment variable, re-running testwiki sync
16:46 brennen@deploy1002: stage-train aborted: (duration: 06m 04s)
16:46 brennen@deploy1002: deploy-promote aborted: (duration: 03m 22s)
16:44 brennen@deploy1002: Started scap: testwikis wikis to 1.39.0-wmf.9 refs T305215
16:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P26573 and previous config saved to /var/cache/conftool/dbconfig/20220426-164022-ladsgroup.json
16:36 klausman@cumin2002: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host ml-staging-ctrl2002.codfw.wmnet
16:35 razzi@cumin1001: START - Cookbook sre.hosts.reimage for host clouddb1021.eqiad.wmnet with OS bullseye
16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26572 and previous config saved to /var/cache/conftool/dbconfig/20220426-163535-ladsgroup.json
16:35 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
16:35 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
16:35 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
16:35 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
16:32 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Set actor migration to read new for medium wikis (T275246) (duration: 02m 01s)
16:30 klausman@cumin2002: START - Cookbook sre.hosts.reboot-single for host ml-staging-ctrl2002.codfw.wmnet
16:28 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1021.eqiad.wmnet with reason: Upgrade to bullseye
16:28 razzi@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb1021.eqiad.wmnet with reason: Upgrade to bullseye
16:27 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T306560)', diff saved to https://phabricator.wikimedia.org/P26571 and previous config saved to /var/cache/conftool/dbconfig/20220426-162517-ladsgroup.json
16:22 bd808: Toolhub upgrade to 18d94d and post-deploy data migrations complete
16:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1163 (T306560)', diff saved to https://phabricator.wikimedia.org/P26570 and previous config saved to /var/cache/conftool/dbconfig/20220426-162244-ladsgroup.json
16:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance
16:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance
16:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T306560)', diff saved to https://phabricator.wikimedia.org/P26569 and previous config saved to /var/cache/conftool/dbconfig/20220426-162236-ladsgroup.json
16:22 pt1979@cumin2002: START - Cookbook sre.dns.netbox
16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298556)', diff saved to https://phabricator.wikimedia.org/P26568 and previous config saved to /var/cache/conftool/dbconfig/20220426-162029-ladsgroup.json
16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T298556)', diff saved to https://phabricator.wikimedia.org/P26567 and previous config saved to /var/cache/conftool/dbconfig/20220426-161816-ladsgroup.json
16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1144.eqiad.wmnet with reason: Maintenance
16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1144.eqiad.wmnet with reason: Maintenance
16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298556)', diff saved to https://phabricator.wikimedia.org/P26566 and previous config saved to /var/cache/conftool/dbconfig/20220426-161808-ladsgroup.json
16:16 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:13 bd808@deploy1002: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
16:12 pt1979@cumin2002: START - Cookbook sre.dns.netbox
16:11 bd808@deploy1002: helmfile [eqiad] START helmfile.d/services/toolhub: apply
16:11 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:10 dancy@deploy1002: Finished deploy [restbase/deploy@0205f1d] (dev-cluster): testing (duration: 00m 17s)
16:09 dancy@deploy1002: Started deploy [restbase/deploy@0205f1d] (dev-cluster): testing
16:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P26564 and previous config saved to /var/cache/conftool/dbconfig/20220426-160731-ladsgroup.json
16:06 pt1979@cumin2002: START - Cookbook sre.dns.netbox
16:06 bd808@deploy1002: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
16:04 bd808@deploy1002: helmfile [codfw] START helmfile.d/services/toolhub: apply
16:03 bd808@deploy1002: helmfile [staging] DONE helmfile.d/services/toolhub: apply
16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26563 and previous config saved to /var/cache/conftool/dbconfig/20220426-160303-ladsgroup.json
16:01 bd808@deploy1002: helmfile [staging] START helmfile.d/services/toolhub: apply
16:00 dancy@deploy1002: Finished deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided) (duration: 02m 43s)
15:58 dancy@deploy1002: Started deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided)
15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P26562 and previous config saved to /var/cache/conftool/dbconfig/20220426-155226-ladsgroup.json
15:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26561 and previous config saved to /var/cache/conftool/dbconfig/20220426-154758-ladsgroup.json
15:42 cmooney@cumin1001: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.4.1 - cmooney@cumin1001
15:40 cmooney@cumin1001: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.4.1 - cmooney@cumin1001
15:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T306560)', diff saved to https://phabricator.wikimedia.org/P26560 and previous config saved to /var/cache/conftool/dbconfig/20220426-153720-ladsgroup.json
15:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1169 (T306560)', diff saved to https://phabricator.wikimedia.org/P26559 and previous config saved to /var/cache/conftool/dbconfig/20220426-153449-ladsgroup.json
15:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance
15:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance
15:34 klausman: Restarting pybal on lvs2009 to pick up change 786319 (ML staging k8s service setup)
15:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
15:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
15:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298556)', diff saved to https://phabricator.wikimedia.org/P26558 and previous config saved to /var/cache/conftool/dbconfig/20220426-153253-ladsgroup.json
15:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298556)', diff saved to https://phabricator.wikimedia.org/P26557 and previous config saved to /var/cache/conftool/dbconfig/20220426-153039-ladsgroup.json
15:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance
15:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance
15:28 klausman@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=ml-staging-ctrl2002.codfw.wmnet
15:27 klausman@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=ml-staging-ctrl2001.codfw.wmnet
15:24 klausman@puppetmaster1001: conftool action : set/pooled=yes,weight=10; selector: name=ml-staging-ctrl2002
15:24 klausman@puppetmaster1001: conftool action : set/pooled=yes,weight=10; selector: name=ml-staging-ctrl2001
15:14 klausman: Restarting pybal on lvs2010 to pick up change 786319 (ML staging k8s service setup)
15:12 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid1007.eqiad.wmnet
14:56 vgutierrez: upgrading trafficserver to 8.0.8-1wm6 on cp4032 - T304835
14:52 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host druid1007.eqiad.wmnet
14:49 vgutierrez: upgrading trafficserver to 8.0.8-1wm6 on cp4026 - T304835
14:44 vgutierrez: upload trafficserver 8.0.8-1wm6 to apt.wm.o (buster) - T304835
14:43 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:43 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:43 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:43 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
14:38 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:38 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:38 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
14:28 klausman@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:25 klausman@cumin1001: START - Cookbook sre.dns.netbox
14:24 urbanecm@deploy1002: Synchronized README: no op (duration: 02m 11s)
14:21 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/GrowthExperiments/: REVERT: Failed backports (duration: 01m 40s)
14:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
14:12 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
14:11 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
14:09 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
14:09 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
14:07 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
14:06 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
14:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
14:04 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
14:03 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
14:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
13:58 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:57 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:57 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:57 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:56 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host druid1006.eqiad.wmnet
13:56 tgr@deploy1002: scap failed: RuntimeError Scap failed!: 8/9 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING: canaries have not been rolled back. (duration: 02m 37s)
13:56 tgr@deploy1002: Scap failed!: 8/9 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING: canaries have not been rolled back.
13:53 tgr@deploy1002: Started scap: (no justification provided)
13:45 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host druid1006.eqiad.wmnet
13:37 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:37 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:37 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:36 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1011 as pc1 primary T306892 (duration: 01m 37s)
13:29 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
13:29 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
13:28 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
13:27 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:27 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:23 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on pc1011.eqiad.wmnet with reason: Rebooting for T303174
13:23 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on pc1011.eqiad.wmnet with reason: Rebooting for T303174
13:22 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Rebooting pc1011 T306892
13:21 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Rebooting pc1011 T306892
13:21 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1014 as pc1 primary T306892 (duration: 01m 07s)
13:18 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
13:18 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
13:16 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:14 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid1005.eqiad.wmnet
13:07 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host druid1005.eqiad.wmnet
12:55 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid1004.eqiad.wmnet
12:52 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26550 and previous config saved to /var/cache/conftool/dbconfig/20220426-125244-kormat.json
12:48 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host druid1004.eqiad.wmnet
12:46 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host archiva1002.wikimedia.org
12:44 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host archiva1002.wikimedia.org
12:37 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26547 and previous config saved to /var/cache/conftool/dbconfig/20220426-123740-kormat.json
12:24 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1015.eqiad.wmnet
12:22 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26546 and previous config saved to /var/cache/conftool/dbconfig/20220426-122235-kormat.json
12:14 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1015.eqiad.wmnet
12:07 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26545 and previous config saved to /var/cache/conftool/dbconfig/20220426-120731-kormat.json
12:07 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26544 and previous config saved to /var/cache/conftool/dbconfig/20220426-120727-kormat.json
12:03 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
12:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
12:00 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-backup1001.eqiad.wmnet with OS bullseye
11:52 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26543 and previous config saved to /var/cache/conftool/dbconfig/20220426-115223-kormat.json
11:49 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-backup1001.eqiad.wmnet with reason: host reimage
11:46 jynus@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-backup1001.eqiad.wmnet with reason: host reimage
11:42 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-backup2001.codfw.wmnet with OS bullseye
11:37 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26542 and previous config saved to /var/cache/conftool/dbconfig/20220426-113719-kormat.json
11:34 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host ms-backup1001.eqiad.wmnet with OS bullseye
11:30 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-backup2001.codfw.wmnet with reason: host reimage
11:27 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-backup2001.codfw.wmnet with reason: host reimage
11:23 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti-test2001.codfw.wmnet
11:22 topranks: Reconfigre routing policy lsw1-f1-eqiad, rename policies to use lower-case
11:22 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26541 and previous config saved to /var/cache/conftool/dbconfig/20220426-112215-kormat.json
11:17 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P26540 and previous config saved to /var/cache/conftool/dbconfig/20220426-111751-kormat.json
11:17 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P26539 and previous config saved to /var/cache/conftool/dbconfig/20220426-111741-kormat.json
11:17 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1170.eqiad.wmnet with reason: Rebooting for T303174
11:17 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1170.eqiad.wmnet with reason: Rebooting for T303174
11:16 topranks: Reconfigre routing policy lsw1-e1-eqiad, rename policies to use lower-case
11:13 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host ms-backup2001.codfw.wmnet with OS bullseye
11:11 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-backup1002.eqiad.wmnet with OS bullseye
11:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
11:09 topranks: Reconfigre routing policy lsw1-e2-eqiad, rename policies to use lower-case
11:09 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1014.eqiad.wmnet
11:08 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26538 and previous config saved to /var/cache/conftool/dbconfig/20220426-110819-kormat.json
11:05 topranks: Reconfigre routing policy lsw1-f2-eqiad, rename policies to use lower-case
11:01 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1014.eqiad.wmnet
11:00 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-backup1002.eqiad.wmnet with reason: host reimage
10:57 topranks: Reconfigre routing policy lsw1-e3-eqiad, rename policies to use lower-case
10:57 jynus@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-backup1002.eqiad.wmnet with reason: host reimage
10:53 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
10:53 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26537 and previous config saved to /var/cache/conftool/dbconfig/20220426-105315-kormat.json
10:45 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host ms-backup1002.eqiad.wmnet with OS bullseye
10:44 topranks: Reconfigre routing policy lsw1-f3-eqiad, rename policies to use lower-case
10:43 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-backup2002.codfw.wmnet with OS bullseye
10:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti-test2001.codfw.wmnet with OS bullseye
10:38 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26536 and previous config saved to /var/cache/conftool/dbconfig/20220426-103811-kormat.json
10:33 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
10:32 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-backup2002.codfw.wmnet with reason: host reimage
10:28 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-backup2002.codfw.wmnet with reason: host reimage
10:25 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1013.eqiad.wmnet
10:23 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26535 and previous config saved to /var/cache/conftool/dbconfig/20220426-102307-kormat.json
10:23 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26534 and previous config saved to /var/cache/conftool/dbconfig/20220426-102303-kormat.json
10:15 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1013.eqiad.wmnet
10:14 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host ms-backup2002.codfw.wmnet with OS bullseye
10:08 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26533 and previous config saved to /var/cache/conftool/dbconfig/20220426-100758-kormat.json
10:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti-test2001.codfw.wmnet with reason: host reimage
10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1122 into API', diff saved to https://phabricator.wikimedia.org/P26532 and previous config saved to /var/cache/conftool/dbconfig/20220426-100031-marostegui.json
10:00 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti-test2001.codfw.wmnet with reason: host reimage
09:59 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 100%: After reimage', diff saved to https://phabricator.wikimedia.org/P26531 and previous config saved to /var/cache/conftool/dbconfig/20220426-095957-root.json
09:56 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26530 and previous config saved to /var/cache/conftool/dbconfig/20220426-095627-kormat.json
09:52 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26528 and previous config saved to /var/cache/conftool/dbconfig/20220426-095254-kormat.json
09:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf1002.eqiad.wmnet
09:51 nokafor@deploy1002: Finished deploy [airflow-dags/analytics@9dbd5bc]: (no justification provided) (duration: 00m 07s)
09:51 nokafor@deploy1002: Started deploy [airflow-dags/analytics@9dbd5bc]: (no justification provided)
09:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf1002.eqiad.wmnet
09:47 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1012.eqiad.wmnet
09:45 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti-test2001.codfw.wmnet with OS bullseye
09:44 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 75%: After reimage', diff saved to https://phabricator.wikimedia.org/P26526 and previous config saved to /var/cache/conftool/dbconfig/20220426-094453-root.json
09:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf1001.eqiad.wmnet
09:41 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26525 and previous config saved to /var/cache/conftool/dbconfig/20220426-094123-kormat.json
09:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf1001.eqiad.wmnet
09:39 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
09:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
09:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
09:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
09:37 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26524 and previous config saved to /var/cache/conftool/dbconfig/20220426-093750-kormat.json
09:36 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1012.eqiad.wmnet
09:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf2002.codfw.wmnet
09:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
09:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
09:33 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
09:33 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
09:33 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P26523 and previous config saved to /var/cache/conftool/dbconfig/20220426-093314-kormat.json
09:33 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1146.eqiad.wmnet with reason: Rebooting for T303174
09:32 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1146.eqiad.wmnet with reason: Rebooting for T303174
09:32 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
09:32 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
09:31 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf2002.codfw.wmnet
09:30 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf2001.codfw.wmnet
09:29 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
09:29 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 50%: After reimage', diff saved to https://phabricator.wikimedia.org/P26522 and previous config saved to /var/cache/conftool/dbconfig/20220426-092949-root.json
09:29 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
09:27 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
09:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf2001.codfw.wmnet
09:26 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
09:26 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26521 and previous config saved to /var/cache/conftool/dbconfig/20220426-092619-kormat.json
09:25 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1115.eqiad.wmnet with reason: Rebooting for T303174
09:25 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1115.eqiad.wmnet with reason: Rebooting for T303174
09:25 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2093.codfw.wmnet,dborch1001.wikimedia.org with reason: Rebooting db1115 T303174
09:25 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db2093.codfw.wmnet,dborch1001.wikimedia.org with reason: Rebooting db1115 T303174
09:23 topranks: Reconfigure CR routers following bgp policy changes (no-op) CR785284
09:23 mvernon@cumin1001: conftool action : set/pooled=yes; selector: service=swift-fe,name=ms-fe1012.eqiad.wmnet
09:23 mvernon@cumin1001: conftool action : set/pooled=yes; selector: service=nginx,name=ms-fe1012.eqiad.wmnet
09:20 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe1012.eqiad.wmnet
09:14 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 25%: After reimage', diff saved to https://phabricator.wikimedia.org/P26520 and previous config saved to /var/cache/conftool/dbconfig/20220426-091445-root.json
09:14 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-fe1012.eqiad.wmnet
09:11 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26519 and previous config saved to /var/cache/conftool/dbconfig/20220426-091115-kormat.json
09:11 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26518 and previous config saved to /var/cache/conftool/dbconfig/20220426-091111-kormat.json
09:10 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P26517 and previous config saved to /var/cache/conftool/dbconfig/20220426-091015-kormat.json
09:10 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1146.eqiad.wmnet with reason: Rebooting for T303174
09:10 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1146.eqiad.wmnet with reason: Rebooting for T303174
08:59 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 10%: After reimage', diff saved to https://phabricator.wikimedia.org/P26516 and previous config saved to /var/cache/conftool/dbconfig/20220426-085941-root.json
08:56 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26515 and previous config saved to /var/cache/conftool/dbconfig/20220426-085607-kormat.json
08:47 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
08:44 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 1%: After reimage', diff saved to https://phabricator.wikimedia.org/P26514 and previous config saved to /var/cache/conftool/dbconfig/20220426-084437-root.json
08:43 jelto: pool name=mw229[7-9].codfw.wmnet, manual icinga recheck green after reboot
08:43 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw229[7-9].codfw.wmnet
08:41 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26513 and previous config saved to /var/cache/conftool/dbconfig/20220426-084103-kormat.json
08:34 moritzm: installing testvm2004 T306499
08:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1122.eqiad.wmnet with OS bullseye
08:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1002.eqiad.wmnet
08:31 jelto@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1)
08:26 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26512 and previous config saved to /var/cache/conftool/dbconfig/20220426-082559-kormat.json
08:25 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host deploy1002.eqiad.wmnet
08:22 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P26511 and previous config saved to /var/cache/conftool/dbconfig/20220426-082210-kormat.json
08:21 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P26510 and previous config saved to /var/cache/conftool/dbconfig/20220426-082155-kormat.json
08:21 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1113.eqiad.wmnet with reason: Rebooting for T303174
08:21 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1113.eqiad.wmnet with reason: Rebooting for T303174
08:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1122.eqiad.wmnet with reason: host reimage
08:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1122.eqiad.wmnet with reason: host reimage
08:08 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1122.eqiad.wmnet with OS bullseye
08:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2004.codfw.wmnet
08:03 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
07:56 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw2289.codfw.wmnet
07:56 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw2288.codfw.wmnet
07:56 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw2287.codfw.wmnet
07:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
07:48 jmm@cumin2002: START - Cookbook sre.dns.netbox
07:47 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2004.codfw.wmnet
07:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
07:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3005.wikimedia.org
07:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109 (T302185)', diff saved to https://phabricator.wikimedia.org/P26509 and previous config saved to /var/cache/conftool/dbconfig/20220426-073627-ladsgroup.json
07:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3005.wikimedia.org
07:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
07:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109', diff saved to https://phabricator.wikimedia.org/P26508 and previous config saved to /var/cache/conftool/dbconfig/20220426-072122-ladsgroup.json
07:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109', diff saved to https://phabricator.wikimedia.org/P26507 and previous config saved to /var/cache/conftool/dbconfig/20220426-070617-ladsgroup.json
07:01 marostegui: dbmaint s2@eqiad T298554
06:54 jayme@deploy1002: Finished deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided) (duration: 03m 05s)
06:51 jayme@deploy1002: Started deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided)
06:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109 (T302185)', diff saved to https://phabricator.wikimedia.org/P26506 and previous config saved to /var/cache/conftool/dbconfig/20220426-065112-ladsgroup.json
06:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1109.eqiad.wmnet with OS bullseye
06:45 jayme: imported scap 4.7.0 to stretch-/buster-/bullseye-wikimedia - T306827
06:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1109.eqiad.wmnet with reason: host reimage
06:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1109.eqiad.wmnet with reason: host reimage
06:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.reimage for host db1109.eqiad.wmnet with OS bullseye
06:16 marostegui: dbmaint s2@eqiad T300381
06:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1109 (T302185)', diff saved to https://phabricator.wikimedia.org/P26505 and previous config saved to /var/cache/conftool/dbconfig/20220426-061519-ladsgroup.json
06:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1109.eqiad.wmnet with reason: Maintenance
06:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1109.eqiad.wmnet with reason: Maintenance
06:14 marostegui: dbmaint s2@eqiad T298557
06:07 marostegui@cumin1001: dbctl commit (dc=all): 'Remove db1100, s5 master from API', diff saved to https://phabricator.wikimedia.org/P26504 and previous config saved to /var/cache/conftool/dbconfig/20220426-060734-marostegui.json
06:06 marostegui@cumin1001: dbctl commit (dc=all): 'db1162 is current s2 master, should not be in API T306417', diff saved to https://phabricator.wikimedia.org/P26503 and previous config saved to /var/cache/conftool/dbconfig/20220426-060602-marostegui.json
06:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1122 T306417', diff saved to https://phabricator.wikimedia.org/P26502 and previous config saved to /var/cache/conftool/dbconfig/20220426-060344-root.json
06:01 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db1162 to s2 primary and set section read-write T306417', diff saved to https://phabricator.wikimedia.org/P26501 and previous config saved to /var/cache/conftool/dbconfig/20220426-060058-marostegui.json
06:00 marostegui@cumin1001: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - T306417', diff saved to https://phabricator.wikimedia.org/P26500 and previous config saved to /var/cache/conftool/dbconfig/20220426-060033-marostegui.json
06:00 marostegui: Starting s2 eqiad failover from db1122 to db1162 - T306417
04:54 marostegui@cumin1001: dbctl commit (dc=all): 'Set db1162 with weight 0 T306417', diff saved to https://phabricator.wikimedia.org/P26498 and previous config saved to /var/cache/conftool/dbconfig/20220426-045406-root.json
04:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s2 T306417
04:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s2 T306417
04:38 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
04:38 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
04:38 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
04:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
02:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
02:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
02:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
02:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply

2022-04-25

23:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
23:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
23:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
23:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
23:01 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: ActorMigration: Read from rev_actor field in all of small wikis (T275246) (duration: 00m 57s)
22:59 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
22:59 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
22:59 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
22:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
22:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
22:54 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
22:54 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
22:54 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
22:49 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: TimedMediaHandler: Make videojs the only player on all group1 (T248418) (duration: 00m 54s)
22:04 dancy@deploy1002: Synchronized README: testing scap mods (duration: 00m 54s)
22:00 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudweb2001-dev.wikimedia.org
21:56 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
21:52 eileen: civicrm revision 7de7ddd4 -> a841cf55
21:49 andrew@cumin1001: START - Cookbook sre.dns.netbox
21:43 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudweb2001-dev.wikimedia.org
21:40 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephmon[2002-2003]-dev.codfw.wmnet
21:38 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
21:35 andrew@cumin1001: START - Cookbook sre.dns.netbox
21:26 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudcephmon[2002-2003]-dev.codfw.wmnet
21:25 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [Web scroll] Restore original sampling rate (T305442) (duration: 01m 01s)
21:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:23 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:59 dzahn@cumin2002: conftool action : set/pooled=inactive; selector: dc=codfw,name=mw2286.codfw.wmnet
20:58 mutante: rebooting mw2415
20:27 catrope@deploy1002: Synchronized wmf-config/wikitech.php: Config: labtestwiki: update labtest ldap server (T304881) (duration: 01m 39s)
20:23 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Enable TOC for all users opted into modern Vector outside of pilot wikis (T306608) (duration: 01m 40s)
20:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:17 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
19:57 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: fresh role user
19:57 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on 7 hosts with reason: fresh role user
19:53 mutante: rebooting mw2414 through mw2419
19:46 dancy@deploy1002: Finished scap: Config: Improve support for realms other than production and labs (duration: 12m 54s)
19:43 mutante: rebooting mw2412, mw2413
19:34 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mw2412.codfw.wmnet with reason: fresh role user
19:34 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 4:00:00 on mw2412.codfw.wmnet with reason: fresh role user
19:33 dancy@deploy1002: Started scap: Config: Improve support for realms other than production and labs
19:30 dancy@deploy1002: Started scap: Config: Improve support for realms other than production and labs
19:29 dancy@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: Improve support for realms other than production and labs (duration: 01m 43s)
19:27 dancy@deploy1002: Synchronized multiversion/MWConfigCacheGenerator.php: Config: Improve support for realms other than production and labs (duration: 01m 42s)
19:27 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
19:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
19:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
19:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
19:09 mutante: turning mw2412 through mw2419 into actual appservers - applying roles for the first time, will cause alerts probably
19:09 cwhite: install grafana-plugins 0.5 and restart grafana on grafana1002 T304583
18:47 cstone: payments-wiki revision changed from a3c69385 to 786dc94f
17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T306560)', diff saved to https://phabricator.wikimedia.org/P26497 and previous config saved to /var/cache/conftool/dbconfig/20220425-175957-ladsgroup.json
17:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26496 and previous config saved to /var/cache/conftool/dbconfig/20220425-174451-ladsgroup.json
17:39 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudservices[2002-2003]-dev.wikimedia.org
17:34 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:30 andrew@cumin1001: START - Cookbook sre.dns.netbox
17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26495 and previous config saved to /var/cache/conftool/dbconfig/20220425-172946-ladsgroup.json
17:24 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudservices[2002-2003]-dev.wikimedia.org
17:17 aokoth@cumin1001: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM otrs1001.eqiad.wmnet
17:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T306560)', diff saved to https://phabricator.wikimedia.org/P26494 and previous config saved to /var/cache/conftool/dbconfig/20220425-171441-ladsgroup.json
17:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T306560)', diff saved to https://phabricator.wikimedia.org/P26493 and previous config saved to /var/cache/conftool/dbconfig/20220425-171223-ladsgroup.json
17:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
17:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
17:04 aokoth@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM otrs1001.eqiad.wmnet
16:47 herron@cumin1001: END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka logging-eqiad cluster: Reboot kafka nodes
15:56 jbond@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet on all recursors
15:56 jbond@cumin1001: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet on all recursors
15:54 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1001.eqiad.wmnet
15:47 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki1001.eqiad.wmnet
15:46 jbond@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet on all recursors
15:46 jbond@cumin1001: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet on all recursors
15:41 jbond@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:37 herron@cumin1001: START - Cookbook sre.kafka.reboot-workers for Kafka logging-eqiad cluster: Reboot kafka nodes
15:36 jbond@cumin1001: START - Cookbook sre.dns.netbox
15:32 herron@cumin1001: END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka logging-codfw cluster: Reboot kafka nodes
15:25 jbond@cumin1001: START - Cookbook sre.dns.netbox
15:23 jbond@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:21 jbond@cumin1001: START - Cookbook sre.dns.netbox
15:20 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki-root1001.eqiad.wmnet
15:18 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki2001.codfw.wmnet
15:16 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki-root1001.eqiad.wmnet
15:15 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki2001.codfw.wmnet
15:15 jbond@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host pki2001.codfw.wmnet
15:15 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki2001.codfw.wmnet
15:15 jelto@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1)
15:13 jbond@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host pki2001.codfw.wmnet
15:12 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki2001.codfw.wmnet
15:10 jbond@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "test sync - jbond@cumin1001"
15:09 jbond@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "test sync - jbond@cumin1001"
14:52 krinkle@deploy1002: Synchronized wmf-config/InitialiseSettings.php: I22240af06d (duration: 01m 42s)
14:46 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:45 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:44 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:43 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
14:21 herron@cumin1001: START - Cookbook sre.kafka.reboot-workers for Kafka logging-codfw cluster: Reboot kafka nodes
14:13 jelto: mw2253: remove puppet lock of stuck puppet run due to reboot, run-puppet-agent
14:10 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1011.eqiad.wmnet
14:00 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1011.eqiad.wmnet
13:51 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1010.eqiad.wmnet
13:41 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1010.eqiad.wmnet
13:39 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1003.eqiad.wmnet
13:35 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-conf1003.eqiad.wmnet
13:31 jelto: maintenance (rolling reboot) on api_appserver in codfw (cookbook sre.hosts.reboot-cluster -D codfw -c api_appserver --percentage 5 --grace_sleep 60)
13:30 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
13:28 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1002.eqiad.wmnet
13:24 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-conf1002.eqiad.wmnet
13:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T306560)', diff saved to https://phabricator.wikimedia.org/P26492 and previous config saved to /var/cache/conftool/dbconfig/20220425-131411-ladsgroup.json
13:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:13 urbanecm: UTC afternoon B&C window done
13:12 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/CentralAuth/includes/User/GlobalUserSelectQueryBuilder.php: c4c4c32: GlobalUserSelectQueryBuilder: Do not fatal when no users are returned (T306535) (duration: 00m 54s)
13:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:04 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 0338c9b: GrowthExperiments: Do not use facebook in campaign pattern (T303785) (duration: 00m 51s)
13:02 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2001.wikimedia.org
12:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26491 and previous config saved to /var/cache/conftool/dbconfig/20220425-125906-ladsgroup.json
12:58 krinkle@deploy1002: Synchronized private/PrivateSettings.php: If4d7ea (duration: 00m 59s)
12:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2001.wikimedia.org
12:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26490 and previous config saved to /var/cache/conftool/dbconfig/20220425-124401-ladsgroup.json
12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T306560)', diff saved to https://phabricator.wikimedia.org/P26489 and previous config saved to /var/cache/conftool/dbconfig/20220425-122856-ladsgroup.json
12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T306560)', diff saved to https://phabricator.wikimedia.org/P26488 and previous config saved to /var/cache/conftool/dbconfig/20220425-122531-ladsgroup.json
12:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
12:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
12:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
12:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T306560)', diff saved to https://phabricator.wikimedia.org/P26487 and previous config saved to /var/cache/conftool/dbconfig/20220425-122518-ladsgroup.json
12:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26485 and previous config saved to /var/cache/conftool/dbconfig/20220425-121013-ladsgroup.json
12:02 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1001.eqiad.wmnet
11:58 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-conf1001.eqiad.wmnet
11:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26484 and previous config saved to /var/cache/conftool/dbconfig/20220425-115508-ladsgroup.json
11:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T306560)', diff saved to https://phabricator.wikimedia.org/P26483 and previous config saved to /var/cache/conftool/dbconfig/20220425-114003-ladsgroup.json
11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T306560)', diff saved to https://phabricator.wikimedia.org/P26482 and previous config saved to /var/cache/conftool/dbconfig/20220425-113138-ladsgroup.json
11:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
11:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T306560)', diff saved to https://phabricator.wikimedia.org/P26481 and previous config saved to /var/cache/conftool/dbconfig/20220425-113130-ladsgroup.json
11:29 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
11:24 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
11:20 moritzm: failover Ganeti master in codfw-test to ganeti-test2003 T306499
11:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26480 and previous config saved to /var/cache/conftool/dbconfig/20220425-111625-ladsgroup.json
11:13 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26479 and previous config saved to /var/cache/conftool/dbconfig/20220425-111315-kormat.json
11:11 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host an-master1001.eqiad.wmnet
11:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26478 and previous config saved to /var/cache/conftool/dbconfig/20220425-110119-ladsgroup.json
11:01 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-master1001.eqiad.wmnet
10:58 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26477 and previous config saved to /var/cache/conftool/dbconfig/20220425-105811-kormat.json
10:54 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2004.codfw.wmnet
10:54 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
10:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T306560)', diff saved to https://phabricator.wikimedia.org/P26476 and previous config saved to /var/cache/conftool/dbconfig/20220425-104614-ladsgroup.json
10:45 jmm@cumin2002: START - Cookbook sre.dns.netbox
10:45 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
10:43 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26475 and previous config saved to /var/cache/conftool/dbconfig/20220425-104307-kormat.json
10:38 jmm@cumin2002: START - Cookbook sre.dns.netbox
10:38 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2004.codfw.wmnet
10:35 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
10:31 jmm@cumin2002: START - Cookbook sre.dns.netbox
10:28 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26474 and previous config saved to /var/cache/conftool/dbconfig/20220425-102803-kormat.json
10:24 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for T303174
10:24 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for T303174
10:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flowspec1001.eqiad.wmnet
10:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host flowspec1001.eqiad.wmnet
10:14 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2004.codfw.wmnet
10:13 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
10:07 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2419.codfw.wmnet
10:05 jmm@cumin2002: START - Cookbook sre.dns.netbox
10:04 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
10:02 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2419.codfw.wmnet
10:02 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2418.codfw.wmnet
09:56 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2418.codfw.wmnet
09:56 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2417.codfw.wmnet
09:55 kormat@cumin1001: dbctl commit (dc=all): 'db1137 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P26472 and previous config saved to /var/cache/conftool/dbconfig/20220425-095543-kormat.json
09:55 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for T303174
09:55 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for T303174
09:51 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2417.codfw.wmnet
09:50 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2416.codfw.wmnet
09:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T306560)', diff saved to https://phabricator.wikimedia.org/P26471 and previous config saved to /var/cache/conftool/dbconfig/20220425-094600-ladsgroup.json
09:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
09:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
09:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T306560)', diff saved to https://phabricator.wikimedia.org/P26470 and previous config saved to /var/cache/conftool/dbconfig/20220425-094552-ladsgroup.json
09:43 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2416.codfw.wmnet
09:42 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2415.codfw.wmnet
09:41 jmm@cumin2002: START - Cookbook sre.dns.netbox
09:41 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2004.codfw.wmnet
09:37 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2415.codfw.wmnet
09:36 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2414.codfw.wmnet
09:31 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2414.codfw.wmnet
09:30 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2413.codfw.wmnet
09:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P26469 and previous config saved to /var/cache/conftool/dbconfig/20220425-093047-ladsgroup.json
09:28 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26468 and previous config saved to /var/cache/conftool/dbconfig/20220425-092807-kormat.json
09:25 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2413.codfw.wmnet
09:23 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2412.codfw.wmnet
09:16 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2412.codfw.wmnet
09:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P26467 and previous config saved to /var/cache/conftool/dbconfig/20220425-091542-ladsgroup.json
09:13 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26466 and previous config saved to /var/cache/conftool/dbconfig/20220425-091303-kormat.json
09:06 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1155.eqiad.wmnet with reason: Rebooting for T303174
09:06 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1155.eqiad.wmnet with reason: Rebooting for T303174
09:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T306560)', diff saved to https://phabricator.wikimedia.org/P26465 and previous config saved to /var/cache/conftool/dbconfig/20220425-090037-ladsgroup.json
08:59 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-master1002.eqiad.wmnet
08:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T306560)', diff saved to https://phabricator.wikimedia.org/P26464 and previous config saved to /var/cache/conftool/dbconfig/20220425-085822-ladsgroup.json
08:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
08:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
08:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1007.eqiad.wmnet
08:58 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26463 and previous config saved to /var/cache/conftool/dbconfig/20220425-085759-kormat.json
08:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
08:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
08:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
08:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
08:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
08:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
08:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
08:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
08:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26462 and previous config saved to /var/cache/conftool/dbconfig/20220425-085650-ladsgroup.json
08:55 vgutierrez: restart varnish and ats on cp2037 to clear daemon restart alerts
08:54 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-master1002.eqiad.wmnet
08:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host dumpsdata1007.eqiad.wmnet
08:48 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1154.eqiad.wmnet with reason: Rebooting for T303174
08:48 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1154.eqiad.wmnet with reason: Rebooting for T303174
08:48 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 9 hosts with reason: Rebooting db1154 T303174
08:48 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on 9 hosts with reason: Rebooting db1154 T303174
08:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T306560)', diff saved to https://phabricator.wikimedia.org/P26461 and previous config saved to /var/cache/conftool/dbconfig/20220425-084318-ladsgroup.json
08:42 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26460 and previous config saved to /var/cache/conftool/dbconfig/20220425-084256-kormat.json
08:42 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26459 and previous config saved to /var/cache/conftool/dbconfig/20220425-084251-kormat.json
08:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26458 and previous config saved to /var/cache/conftool/dbconfig/20220425-084145-ladsgroup.json
08:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26457 and previous config saved to /var/cache/conftool/dbconfig/20220425-082813-ladsgroup.json
08:27 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26456 and previous config saved to /var/cache/conftool/dbconfig/20220425-082747-kormat.json
08:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26455 and previous config saved to /var/cache/conftool/dbconfig/20220425-082640-ladsgroup.json
08:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26454 and previous config saved to /var/cache/conftool/dbconfig/20220425-081307-ladsgroup.json
08:12 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26453 and previous config saved to /var/cache/conftool/dbconfig/20220425-081244-kormat.json
08:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26452 and previous config saved to /var/cache/conftool/dbconfig/20220425-081135-ladsgroup.json
08:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26451 and previous config saved to /var/cache/conftool/dbconfig/20220425-080910-ladsgroup.json
08:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
08:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
08:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
08:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
08:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26450 and previous config saved to /var/cache/conftool/dbconfig/20220425-080838-ladsgroup.json
07:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host search-loader2001.codfw.wmnet
07:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T306560)', diff saved to https://phabricator.wikimedia.org/P26449 and previous config saved to /var/cache/conftool/dbconfig/20220425-075801-ladsgroup.json
07:57 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26448 and previous config saved to /var/cache/conftool/dbconfig/20220425-075740-kormat.json
07:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26447 and previous config saved to /var/cache/conftool/dbconfig/20220425-075333-ladsgroup.json
07:52 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host search-loader2001.codfw.wmnet
07:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host search-loader1001.eqiad.wmnet
07:51 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P26446 and previous config saved to /var/cache/conftool/dbconfig/20220425-075106-kormat.json
07:50 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P26445 and previous config saved to /var/cache/conftool/dbconfig/20220425-075045-kormat.json
07:50 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1144.eqiad.wmnet with reason: Rebooting for T303174
07:50 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1144.eqiad.wmnet with reason: Rebooting for T303174
07:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1001.wikimedia.org
07:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host search-loader1001.eqiad.wmnet
07:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T306560)', diff saved to https://phabricator.wikimedia.org/P26444 and previous config saved to /var/cache/conftool/dbconfig/20220425-074912-ladsgroup.json
07:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
07:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
07:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T306560)', diff saved to https://phabricator.wikimedia.org/P26443 and previous config saved to /var/cache/conftool/dbconfig/20220425-074904-ladsgroup.json
07:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt1001.wikimedia.org
07:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26442 and previous config saved to /var/cache/conftool/dbconfig/20220425-073828-ladsgroup.json
07:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26441 and previous config saved to /var/cache/conftool/dbconfig/20220425-073359-ladsgroup.json
07:31 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host seaborgium.wikimedia.org
07:26 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26440 and previous config saved to /var/cache/conftool/dbconfig/20220425-072323-ladsgroup.json
07:22 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host seaborgium.wikimedia.org
07:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26439 and previous config saved to /var/cache/conftool/dbconfig/20220425-072157-ladsgroup.json
07:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
07:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
07:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26438 and previous config saved to /var/cache/conftool/dbconfig/20220425-072149-ladsgroup.json
07:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:21 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:21 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster2005.codfw.wmnet
07:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26437 and previous config saved to /var/cache/conftool/dbconfig/20220425-071853-ladsgroup.json
07:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster2005.codfw.wmnet
07:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster2004.codfw.wmnet
07:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:11 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: ActorMigration: Start reading from rev_actor field in group0 (T275246) (duration: 00m 50s)
07:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:11 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:08 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster2004.codfw.wmnet
07:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26436 and previous config saved to /var/cache/conftool/dbconfig/20220425-070644-ladsgroup.json
07:06 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Add tothemoon.ser.asu.edu to the wgCopyUploadsDomains allowlist of commonswiki (T306671) (duration: 00m 52s)
07:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T306560)', diff saved to https://phabricator.wikimedia.org/P26435 and previous config saved to /var/cache/conftool/dbconfig/20220425-070348-ladsgroup.json
06:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1001.wikimedia.org
06:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T306560)', diff saved to https://phabricator.wikimedia.org/P26434 and previous config saved to /var/cache/conftool/dbconfig/20220425-065559-ladsgroup.json
06:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
06:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
06:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc1001.wikimedia.org
06:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26433 and previous config saved to /var/cache/conftool/dbconfig/20220425-065139-ladsgroup.json
06:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1132 into s1 T301879', diff saved to https://phabricator.wikimedia.org/P26432 and previous config saved to /var/cache/conftool/dbconfig/20220425-063823-marostegui.json
06:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26431 and previous config saved to /var/cache/conftool/dbconfig/20220425-063634-ladsgroup.json
06:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T306560)', diff saved to https://phabricator.wikimedia.org/P26430 and previous config saved to /var/cache/conftool/dbconfig/20220425-063409-ladsgroup.json
06:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
06:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
06:11 dcausse: depooling and restarting blazegraph on wdqs1007 (deadlocked for 4+days)
04:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26429 and previous config saved to /var/cache/conftool/dbconfig/20220425-045902-ladsgroup.json
04:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26428 and previous config saved to /var/cache/conftool/dbconfig/20220425-044357-ladsgroup.json
04:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26427 and previous config saved to /var/cache/conftool/dbconfig/20220425-042852-ladsgroup.json
04:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26426 and previous config saved to /var/cache/conftool/dbconfig/20220425-041347-ladsgroup.json
04:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26425 and previous config saved to /var/cache/conftool/dbconfig/20220425-040940-ladsgroup.json
04:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
04:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
04:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26424 and previous config saved to /var/cache/conftool/dbconfig/20220425-040926-ladsgroup.json
03:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26423 and previous config saved to /var/cache/conftool/dbconfig/20220425-035421-ladsgroup.json
03:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26422 and previous config saved to /var/cache/conftool/dbconfig/20220425-033916-ladsgroup.json
03:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26421 and previous config saved to /var/cache/conftool/dbconfig/20220425-032410-ladsgroup.json
03:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26420 and previous config saved to /var/cache/conftool/dbconfig/20220425-031959-ladsgroup.json
03:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
03:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
03:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26419 and previous config saved to /var/cache/conftool/dbconfig/20220425-031944-ladsgroup.json
03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26418 and previous config saved to /var/cache/conftool/dbconfig/20220425-030439-ladsgroup.json
02:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26417 and previous config saved to /var/cache/conftool/dbconfig/20220425-024934-ladsgroup.json
02:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26416 and previous config saved to /var/cache/conftool/dbconfig/20220425-023429-ladsgroup.json
02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26415 and previous config saved to /var/cache/conftool/dbconfig/20220425-023020-ladsgroup.json
02:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
02:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26414 and previous config saved to /var/cache/conftool/dbconfig/20220425-023012-ladsgroup.json
02:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26413 and previous config saved to /var/cache/conftool/dbconfig/20220425-021507-ladsgroup.json
02:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26412 and previous config saved to /var/cache/conftool/dbconfig/20220425-020002-ladsgroup.json
01:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26411 and previous config saved to /var/cache/conftool/dbconfig/20220425-014457-ladsgroup.json
01:40 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: update new codfw1dev host (duration: 00m 54s)
01:39 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: update new codfw1dev host
01:39 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: update new codfw1dev host
01:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26410 and previous config saved to /var/cache/conftool/dbconfig/20220425-010952-ladsgroup.json
01:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
01:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
01:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26409 and previous config saved to /var/cache/conftool/dbconfig/20220425-010938-ladsgroup.json
00:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26408 and previous config saved to /var/cache/conftool/dbconfig/20220425-005432-ladsgroup.json
00:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26407 and previous config saved to /var/cache/conftool/dbconfig/20220425-003927-ladsgroup.json
00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26406 and previous config saved to /var/cache/conftool/dbconfig/20220425-002422-ladsgroup.json
00:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26405 and previous config saved to /var/cache/conftool/dbconfig/20220425-001152-ladsgroup.json
00:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
00:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
00:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26404 and previous config saved to /var/cache/conftool/dbconfig/20220425-001144-ladsgroup.json

2022-04-24

23:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26403 and previous config saved to /var/cache/conftool/dbconfig/20220424-235639-ladsgroup.json
23:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26402 and previous config saved to /var/cache/conftool/dbconfig/20220424-234134-ladsgroup.json
23:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26401 and previous config saved to /var/cache/conftool/dbconfig/20220424-232629-ladsgroup.json
23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26400 and previous config saved to /var/cache/conftool/dbconfig/20220424-232219-ladsgroup.json
23:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
23:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26399 and previous config saved to /var/cache/conftool/dbconfig/20220424-232205-ladsgroup.json
23:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26398 and previous config saved to /var/cache/conftool/dbconfig/20220424-230700-ladsgroup.json
23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T306560)', diff saved to https://phabricator.wikimedia.org/P26397 and previous config saved to /var/cache/conftool/dbconfig/20220424-230136-ladsgroup.json
22:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26396 and previous config saved to /var/cache/conftool/dbconfig/20220424-225155-ladsgroup.json
22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26395 and previous config saved to /var/cache/conftool/dbconfig/20220424-224631-ladsgroup.json
22:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26394 and previous config saved to /var/cache/conftool/dbconfig/20220424-223650-ladsgroup.json
22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26393 and previous config saved to /var/cache/conftool/dbconfig/20220424-223240-ladsgroup.json
22:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
22:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26392 and previous config saved to /var/cache/conftool/dbconfig/20220424-223232-ladsgroup.json
22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26391 and previous config saved to /var/cache/conftool/dbconfig/20220424-223126-ladsgroup.json
22:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26390 and previous config saved to /var/cache/conftool/dbconfig/20220424-221727-ladsgroup.json
22:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T306560)', diff saved to https://phabricator.wikimedia.org/P26389 and previous config saved to /var/cache/conftool/dbconfig/20220424-221621-ladsgroup.json
22:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26388 and previous config saved to /var/cache/conftool/dbconfig/20220424-220222-ladsgroup.json
21:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26387 and previous config saved to /var/cache/conftool/dbconfig/20220424-214717-ladsgroup.json
21:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26386 and previous config saved to /var/cache/conftool/dbconfig/20220424-213440-ladsgroup.json
21:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
21:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
21:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26385 and previous config saved to /var/cache/conftool/dbconfig/20220424-213425-ladsgroup.json
21:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26384 and previous config saved to /var/cache/conftool/dbconfig/20220424-211920-ladsgroup.json
21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T306560)', diff saved to https://phabricator.wikimedia.org/P26383 and previous config saved to /var/cache/conftool/dbconfig/20220424-211607-ladsgroup.json
21:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
21:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T306560)', diff saved to https://phabricator.wikimedia.org/P26382 and previous config saved to /var/cache/conftool/dbconfig/20220424-211559-ladsgroup.json
21:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P26381 and previous config saved to /var/cache/conftool/dbconfig/20220424-210521-ladsgroup.json
21:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26380 and previous config saved to /var/cache/conftool/dbconfig/20220424-210415-ladsgroup.json
21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26379 and previous config saved to /var/cache/conftool/dbconfig/20220424-210052-ladsgroup.json
20:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26378 and previous config saved to /var/cache/conftool/dbconfig/20220424-205016-ladsgroup.json
20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26377 and previous config saved to /var/cache/conftool/dbconfig/20220424-204910-ladsgroup.json
20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26376 and previous config saved to /var/cache/conftool/dbconfig/20220424-204547-ladsgroup.json
20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26375 and previous config saved to /var/cache/conftool/dbconfig/20220424-203639-ladsgroup.json
20:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
20:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26374 and previous config saved to /var/cache/conftool/dbconfig/20220424-203630-ladsgroup.json
20:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26373 and previous config saved to /var/cache/conftool/dbconfig/20220424-203511-ladsgroup.json
20:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T306560)', diff saved to https://phabricator.wikimedia.org/P26372 and previous config saved to /var/cache/conftool/dbconfig/20220424-203042-ladsgroup.json
20:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26371 and previous config saved to /var/cache/conftool/dbconfig/20220424-202125-ladsgroup.json
20:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P26370 and previous config saved to /var/cache/conftool/dbconfig/20220424-202006-ladsgroup.json
20:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26369 and previous config saved to /var/cache/conftool/dbconfig/20220424-200620-ladsgroup.json
19:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26368 and previous config saved to /var/cache/conftool/dbconfig/20220424-195115-ladsgroup.json
19:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26367 and previous config saved to /var/cache/conftool/dbconfig/20220424-194705-ladsgroup.json
19:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
19:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
19:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26366 and previous config saved to /var/cache/conftool/dbconfig/20220424-194651-ladsgroup.json
19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P26365 and previous config saved to /var/cache/conftool/dbconfig/20220424-193635-ladsgroup.json
19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26364 and previous config saved to /var/cache/conftool/dbconfig/20220424-193611-ladsgroup.json
19:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26363 and previous config saved to /var/cache/conftool/dbconfig/20220424-193146-ladsgroup.json
19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T306560)', diff saved to https://phabricator.wikimedia.org/P26362 and previous config saved to /var/cache/conftool/dbconfig/20220424-193028-ladsgroup.json
19:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
19:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T306560)', diff saved to https://phabricator.wikimedia.org/P26361 and previous config saved to /var/cache/conftool/dbconfig/20220424-193020-ladsgroup.json
19:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26360 and previous config saved to /var/cache/conftool/dbconfig/20220424-192106-ladsgroup.json
19:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26359 and previous config saved to /var/cache/conftool/dbconfig/20220424-191641-ladsgroup.json
19:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26358 and previous config saved to /var/cache/conftool/dbconfig/20220424-191515-ladsgroup.json
19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26357 and previous config saved to /var/cache/conftool/dbconfig/20220424-190601-ladsgroup.json
19:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26356 and previous config saved to /var/cache/conftool/dbconfig/20220424-190135-ladsgroup.json
19:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26355 and previous config saved to /var/cache/conftool/dbconfig/20220424-190008-ladsgroup.json
18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P26354 and previous config saved to /var/cache/conftool/dbconfig/20220424-185724-ladsgroup.json
18:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
18:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26353 and previous config saved to /var/cache/conftool/dbconfig/20220424-185717-ladsgroup.json
18:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26352 and previous config saved to /var/cache/conftool/dbconfig/20220424-185056-ladsgroup.json
18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T306560)', diff saved to https://phabricator.wikimedia.org/P26351 and previous config saved to /var/cache/conftool/dbconfig/20220424-184503-ladsgroup.json
18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26350 and previous config saved to /var/cache/conftool/dbconfig/20220424-184212-ladsgroup.json
18:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T306560)', diff saved to https://phabricator.wikimedia.org/P26349 and previous config saved to /var/cache/conftool/dbconfig/20220424-183813-ladsgroup.json
18:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
18:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
18:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T306560)', diff saved to https://phabricator.wikimedia.org/P26348 and previous config saved to /var/cache/conftool/dbconfig/20220424-183805-ladsgroup.json
18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26347 and previous config saved to /var/cache/conftool/dbconfig/20220424-182707-ladsgroup.json
18:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26346 and previous config saved to /var/cache/conftool/dbconfig/20220424-182300-ladsgroup.json
18:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26345 and previous config saved to /var/cache/conftool/dbconfig/20220424-181201-ladsgroup.json
18:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26344 and previous config saved to /var/cache/conftool/dbconfig/20220424-180755-ladsgroup.json
18:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26343 and previous config saved to /var/cache/conftool/dbconfig/20220424-180555-ladsgroup.json
18:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
18:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
18:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26342 and previous config saved to /var/cache/conftool/dbconfig/20220424-180530-ladsgroup.json
18:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P26341 and previous config saved to /var/cache/conftool/dbconfig/20220424-180013-ladsgroup.json
18:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
18:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T306560)', diff saved to https://phabricator.wikimedia.org/P26340 and previous config saved to /var/cache/conftool/dbconfig/20220424-175250-ladsgroup.json
17:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26339 and previous config saved to /var/cache/conftool/dbconfig/20220424-175025-ladsgroup.json
17:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T306560)', diff saved to https://phabricator.wikimedia.org/P26338 and previous config saved to /var/cache/conftool/dbconfig/20220424-174553-ladsgroup.json
17:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
17:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
17:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
17:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
17:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
17:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
17:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T306560)', diff saved to https://phabricator.wikimedia.org/P26337 and previous config saved to /var/cache/conftool/dbconfig/20220424-173955-ladsgroup.json
17:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26336 and previous config saved to /var/cache/conftool/dbconfig/20220424-173520-ladsgroup.json
17:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P26335 and previous config saved to /var/cache/conftool/dbconfig/20220424-172450-ladsgroup.json
17:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26334 and previous config saved to /var/cache/conftool/dbconfig/20220424-172015-ladsgroup.json
17:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P26333 and previous config saved to /var/cache/conftool/dbconfig/20220424-170945-ladsgroup.json
16:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T306560)', diff saved to https://phabricator.wikimedia.org/P26332 and previous config saved to /var/cache/conftool/dbconfig/20220424-165439-ladsgroup.json
16:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T306560)', diff saved to https://phabricator.wikimedia.org/P26331 and previous config saved to /var/cache/conftool/dbconfig/20220424-164748-ladsgroup.json
16:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
16:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
16:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
16:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
16:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
16:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26330 and previous config saved to /var/cache/conftool/dbconfig/20220424-163151-ladsgroup.json
16:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
16:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
16:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
16:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
16:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
16:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
03:20 Amir1: optimizing flaggedtemplates on plwiki (s2) in db2088
01:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
01:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
01:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
01:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
01:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T306560)', diff saved to https://phabricator.wikimedia.org/P26329 and previous config saved to /var/cache/conftool/dbconfig/20220424-012830-ladsgroup.json
01:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P26328 and previous config saved to /var/cache/conftool/dbconfig/20220424-011325-ladsgroup.json
00:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P26327 and previous config saved to /var/cache/conftool/dbconfig/20220424-005820-ladsgroup.json
00:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T306560)', diff saved to https://phabricator.wikimedia.org/P26326 and previous config saved to /var/cache/conftool/dbconfig/20220424-004315-ladsgroup.json

2022-04-23

23:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T306560)', diff saved to https://phabricator.wikimedia.org/P26325 and previous config saved to /var/cache/conftool/dbconfig/20220423-232748-ladsgroup.json
23:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
23:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
23:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
23:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
23:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T306560)', diff saved to https://phabricator.wikimedia.org/P26324 and previous config saved to /var/cache/conftool/dbconfig/20220423-232735-ladsgroup.json
23:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P26323 and previous config saved to /var/cache/conftool/dbconfig/20220423-231230-ladsgroup.json
22:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P26322 and previous config saved to /var/cache/conftool/dbconfig/20220423-225725-ladsgroup.json
22:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T306560)', diff saved to https://phabricator.wikimedia.org/P26321 and previous config saved to /var/cache/conftool/dbconfig/20220423-224220-ladsgroup.json
21:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T306560)', diff saved to https://phabricator.wikimedia.org/P26320 and previous config saved to /var/cache/conftool/dbconfig/20220423-212332-ladsgroup.json
21:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
21:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
21:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T306560)', diff saved to https://phabricator.wikimedia.org/P26319 and previous config saved to /var/cache/conftool/dbconfig/20220423-212324-ladsgroup.json
21:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P26318 and previous config saved to /var/cache/conftool/dbconfig/20220423-210819-ladsgroup.json
20:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P26317 and previous config saved to /var/cache/conftool/dbconfig/20220423-205313-ladsgroup.json
20:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T306560)', diff saved to https://phabricator.wikimedia.org/P26316 and previous config saved to /var/cache/conftool/dbconfig/20220423-203808-ladsgroup.json
19:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T306560)', diff saved to https://phabricator.wikimedia.org/P26315 and previous config saved to /var/cache/conftool/dbconfig/20220423-191224-ladsgroup.json
19:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
19:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
19:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T306560)', diff saved to https://phabricator.wikimedia.org/P26314 and previous config saved to /var/cache/conftool/dbconfig/20220423-191216-ladsgroup.json
18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P26313 and previous config saved to /var/cache/conftool/dbconfig/20220423-185711-ladsgroup.json
18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P26312 and previous config saved to /var/cache/conftool/dbconfig/20220423-184206-ladsgroup.json
18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T306560)', diff saved to https://phabricator.wikimedia.org/P26311 and previous config saved to /var/cache/conftool/dbconfig/20220423-182701-ladsgroup.json
16:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T306560)', diff saved to https://phabricator.wikimedia.org/P26310 and previous config saved to /var/cache/conftool/dbconfig/20220423-165939-ladsgroup.json
16:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
16:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
16:25 akosiaris: increase mw1335 and mw1336 weights on the jobrunner cluster from 1 to 4 (they were at %25 CPU usage). That should direct more traffic to them and lighten the load on the rest.
16:24 akosiaris@cumin1001: conftool action : set/weight=4; selector: cluster=jobrunner,name=mw1336.eqiad.wmnet
16:24 akosiaris@cumin1001: conftool action : set/weight=4; selector: cluster=jobrunner,name=mw1335.eqiad.wmnet
16:22 akosiaris@cumin1001: conftool action : set/weight=8; selector: cluster=jobrunner,name=mw1335.eqiad.wmnet
16:22 akosiaris@cumin1001: conftool action : set/weight=8; selector: cluster=jobrunner,name=mw1336.eqiad.wmnet
16:18 akosiaris: depool the videoscalers from the jobrunner cluster. Effectively split the 2 clusters that way. This should isolate the rest of the jobs from the video transcoding jobs reducing the latency that they are experiencing
16:17 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1446.eqiad.wmnet
16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1445.eqiad.wmnet
16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1440.eqiad.wmnet
16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1439.eqiad.wmnet
16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1438.eqiad.wmnet
16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1437.eqiad.wmnet
16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1338.eqiad.wmnet
15:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
15:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
14:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
14:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
14:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T306560)', diff saved to https://phabricator.wikimedia.org/P26309 and previous config saved to /var/cache/conftool/dbconfig/20220423-142129-ladsgroup.json
14:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26308 and previous config saved to /var/cache/conftool/dbconfig/20220423-140624-ladsgroup.json
13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26307 and previous config saved to /var/cache/conftool/dbconfig/20220423-135119-ladsgroup.json
13:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T306560)', diff saved to https://phabricator.wikimedia.org/P26306 and previous config saved to /var/cache/conftool/dbconfig/20220423-133614-ladsgroup.json
12:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T306560)', diff saved to https://phabricator.wikimedia.org/P26305 and previous config saved to /var/cache/conftool/dbconfig/20220423-123558-ladsgroup.json
12:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
12:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
12:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T306560)', diff saved to https://phabricator.wikimedia.org/P26304 and previous config saved to /var/cache/conftool/dbconfig/20220423-123550-ladsgroup.json
12:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P26303 and previous config saved to /var/cache/conftool/dbconfig/20220423-122045-ladsgroup.json
12:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P26302 and previous config saved to /var/cache/conftool/dbconfig/20220423-120540-ladsgroup.json
11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T306560)', diff saved to https://phabricator.wikimedia.org/P26301 and previous config saved to /var/cache/conftool/dbconfig/20220423-115035-ladsgroup.json
11:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26300 and previous config saved to /var/cache/conftool/dbconfig/20220423-110511-ladsgroup.json
10:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26299 and previous config saved to /var/cache/conftool/dbconfig/20220423-105005-ladsgroup.json
10:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26298 and previous config saved to /var/cache/conftool/dbconfig/20220423-103500-ladsgroup.json
10:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T306560)', diff saved to https://phabricator.wikimedia.org/P26297 and previous config saved to /var/cache/conftool/dbconfig/20220423-103135-ladsgroup.json
10:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
10:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
10:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T306560)', diff saved to https://phabricator.wikimedia.org/P26296 and previous config saved to /var/cache/conftool/dbconfig/20220423-103127-ladsgroup.json
10:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26295 and previous config saved to /var/cache/conftool/dbconfig/20220423-101955-ladsgroup.json
10:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P26294 and previous config saved to /var/cache/conftool/dbconfig/20220423-101622-ladsgroup.json
10:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P26293 and previous config saved to /var/cache/conftool/dbconfig/20220423-100115-ladsgroup.json
09:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T306560)', diff saved to https://phabricator.wikimedia.org/P26292 and previous config saved to /var/cache/conftool/dbconfig/20220423-094610-ladsgroup.json
09:38 elukey: `apt-get clean` on an-airflow1001 to free some space
09:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26291 and previous config saved to /var/cache/conftool/dbconfig/20220423-093443-ladsgroup.json
09:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
09:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
09:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P26290 and previous config saved to /var/cache/conftool/dbconfig/20220423-093435-ladsgroup.json
09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26289 and previous config saved to /var/cache/conftool/dbconfig/20220423-091930-ladsgroup.json
09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26288 and previous config saved to /var/cache/conftool/dbconfig/20220423-090425-ladsgroup.json
08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P26287 and previous config saved to /var/cache/conftool/dbconfig/20220423-084920-ladsgroup.json
08:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P26286 and previous config saved to /var/cache/conftool/dbconfig/20220423-083545-ladsgroup.json
08:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
08:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
08:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
08:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
08:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T298565)', diff saved to https://phabricator.wikimedia.org/P26285 and previous config saved to /var/cache/conftool/dbconfig/20220423-083532-ladsgroup.json
08:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T306560)', diff saved to https://phabricator.wikimedia.org/P26284 and previous config saved to /var/cache/conftool/dbconfig/20220423-082735-ladsgroup.json
08:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
08:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
08:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T306560)', diff saved to https://phabricator.wikimedia.org/P26283 and previous config saved to /var/cache/conftool/dbconfig/20220423-082726-ladsgroup.json
08:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26282 and previous config saved to /var/cache/conftool/dbconfig/20220423-082027-ladsgroup.json
08:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26281 and previous config saved to /var/cache/conftool/dbconfig/20220423-081221-ladsgroup.json
08:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26280 and previous config saved to /var/cache/conftool/dbconfig/20220423-080522-ladsgroup.json
07:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26279 and previous config saved to /var/cache/conftool/dbconfig/20220423-075716-ladsgroup.json
07:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T298565)', diff saved to https://phabricator.wikimedia.org/P26278 and previous config saved to /var/cache/conftool/dbconfig/20220423-075017-ladsgroup.json
07:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T306560)', diff saved to https://phabricator.wikimedia.org/P26277 and previous config saved to /var/cache/conftool/dbconfig/20220423-074211-ladsgroup.json
07:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T298565)', diff saved to https://phabricator.wikimedia.org/P26276 and previous config saved to /var/cache/conftool/dbconfig/20220423-073656-ladsgroup.json
07:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
07:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
07:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P26275 and previous config saved to /var/cache/conftool/dbconfig/20220423-073648-ladsgroup.json
07:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26274 and previous config saved to /var/cache/conftool/dbconfig/20220423-072143-ladsgroup.json
07:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26273 and previous config saved to /var/cache/conftool/dbconfig/20220423-070638-ladsgroup.json
06:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P26272 and previous config saved to /var/cache/conftool/dbconfig/20220423-065133-ladsgroup.json
06:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T306560)', diff saved to https://phabricator.wikimedia.org/P26271 and previous config saved to /var/cache/conftool/dbconfig/20220423-062503-ladsgroup.json
06:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
06:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
06:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T306560)', diff saved to https://phabricator.wikimedia.org/P26270 and previous config saved to /var/cache/conftool/dbconfig/20220423-062455-ladsgroup.json
06:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P26269 and previous config saved to /var/cache/conftool/dbconfig/20220423-060950-ladsgroup.json
05:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P26268 and previous config saved to /var/cache/conftool/dbconfig/20220423-055445-ladsgroup.json
05:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P26267 and previous config saved to /var/cache/conftool/dbconfig/20220423-055118-ladsgroup.json
05:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
05:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
05:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T306560)', diff saved to https://phabricator.wikimedia.org/P26266 and previous config saved to /var/cache/conftool/dbconfig/20220423-053940-ladsgroup.json
05:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
05:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
05:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26265 and previous config saved to /var/cache/conftool/dbconfig/20220423-051219-ladsgroup.json
04:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26264 and previous config saved to /var/cache/conftool/dbconfig/20220423-045714-ladsgroup.json
04:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26263 and previous config saved to /var/cache/conftool/dbconfig/20220423-044209-ladsgroup.json
04:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26262 and previous config saved to /var/cache/conftool/dbconfig/20220423-042704-ladsgroup.json
04:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T306560)', diff saved to https://phabricator.wikimedia.org/P26261 and previous config saved to /var/cache/conftool/dbconfig/20220423-042001-ladsgroup.json
04:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
04:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
04:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T306560)', diff saved to https://phabricator.wikimedia.org/P26260 and previous config saved to /var/cache/conftool/dbconfig/20220423-041953-ladsgroup.json
04:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P26259 and previous config saved to /var/cache/conftool/dbconfig/20220423-040448-ladsgroup.json
03:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P26258 and previous config saved to /var/cache/conftool/dbconfig/20220423-034943-ladsgroup.json
03:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26257 and previous config saved to /var/cache/conftool/dbconfig/20220423-034558-ladsgroup.json
03:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
03:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
03:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26256 and previous config saved to /var/cache/conftool/dbconfig/20220423-034550-ladsgroup.json
03:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T306560)', diff saved to https://phabricator.wikimedia.org/P26255 and previous config saved to /var/cache/conftool/dbconfig/20220423-033438-ladsgroup.json
03:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26254 and previous config saved to /var/cache/conftool/dbconfig/20220423-033045-ladsgroup.json
03:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26253 and previous config saved to /var/cache/conftool/dbconfig/20220423-031540-ladsgroup.json
03:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26252 and previous config saved to /var/cache/conftool/dbconfig/20220423-030035-ladsgroup.json
02:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P26251 and previous config saved to /var/cache/conftool/dbconfig/20220423-021851-ladsgroup.json
02:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T306560)', diff saved to https://phabricator.wikimedia.org/P26250 and previous config saved to /var/cache/conftool/dbconfig/20220423-021826-ladsgroup.json
02:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
02:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
02:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26249 and previous config saved to /var/cache/conftool/dbconfig/20220423-021211-ladsgroup.json
02:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
02:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
02:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26248 and previous config saved to /var/cache/conftool/dbconfig/20220423-020346-ladsgroup.json
01:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26247 and previous config saved to /var/cache/conftool/dbconfig/20220423-014841-ladsgroup.json
01:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26246 and previous config saved to /var/cache/conftool/dbconfig/20220423-013450-ladsgroup.json
01:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
01:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
01:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P26245 and previous config saved to /var/cache/conftool/dbconfig/20220423-013336-ladsgroup.json
01:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26244 and previous config saved to /var/cache/conftool/dbconfig/20220423-011945-ladsgroup.json
01:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26243 and previous config saved to /var/cache/conftool/dbconfig/20220423-010440-ladsgroup.json
00:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
00:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
00:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance
00:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance
00:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
00:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
00:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26242 and previous config saved to /var/cache/conftool/dbconfig/20220423-005613-ladsgroup.json
00:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26241 and previous config saved to /var/cache/conftool/dbconfig/20220423-004935-ladsgroup.json
00:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26240 and previous config saved to /var/cache/conftool/dbconfig/20220423-004617-ladsgroup.json
00:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
00:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
00:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26239 and previous config saved to /var/cache/conftool/dbconfig/20220423-004108-ladsgroup.json
00:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26238 and previous config saved to /var/cache/conftool/dbconfig/20220423-002603-ladsgroup.json
00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P26237 and previous config saved to /var/cache/conftool/dbconfig/20220423-002352-ladsgroup.json
00:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
00:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P26236 and previous config saved to /var/cache/conftool/dbconfig/20220423-002344-ladsgroup.json
00:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26235 and previous config saved to /var/cache/conftool/dbconfig/20220423-001058-ladsgroup.json
00:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26234 and previous config saved to /var/cache/conftool/dbconfig/20220423-000839-ladsgroup.json

2022-04-22

23:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26233 and previous config saved to /var/cache/conftool/dbconfig/20220422-235334-ladsgroup.json
23:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P26232 and previous config saved to /var/cache/conftool/dbconfig/20220422-233829-ladsgroup.json
23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26231 and previous config saved to /var/cache/conftool/dbconfig/20220422-232210-ladsgroup.json
23:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
23:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
23:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26230 and previous config saved to /var/cache/conftool/dbconfig/20220422-232147-ladsgroup.json
23:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26229 and previous config saved to /var/cache/conftool/dbconfig/20220422-230642-ladsgroup.json
22:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P26228 and previous config saved to /var/cache/conftool/dbconfig/20220422-225735-ladsgroup.json
22:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
22:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
22:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26227 and previous config saved to /var/cache/conftool/dbconfig/20220422-225136-ladsgroup.json
22:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26226 and previous config saved to /var/cache/conftool/dbconfig/20220422-223631-ladsgroup.json
22:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
22:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
22:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P26225 and previous config saved to /var/cache/conftool/dbconfig/20220422-222203-ladsgroup.json
22:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26224 and previous config saved to /var/cache/conftool/dbconfig/20220422-220658-ladsgroup.json
21:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26223 and previous config saved to /var/cache/conftool/dbconfig/20220422-215153-ladsgroup.json
21:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P26222 and previous config saved to /var/cache/conftool/dbconfig/20220422-213648-ladsgroup.json
21:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26221 and previous config saved to /var/cache/conftool/dbconfig/20220422-213617-ladsgroup.json
21:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
21:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
21:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26220 and previous config saved to /var/cache/conftool/dbconfig/20220422-213609-ladsgroup.json
21:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26219 and previous config saved to /var/cache/conftool/dbconfig/20220422-212104-ladsgroup.json
21:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26218 and previous config saved to /var/cache/conftool/dbconfig/20220422-210559-ladsgroup.json
20:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T306560)', diff saved to https://phabricator.wikimedia.org/P26217 and previous config saved to /var/cache/conftool/dbconfig/20220422-205538-ladsgroup.json
20:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26216 and previous config saved to /var/cache/conftool/dbconfig/20220422-205053-ladsgroup.json
20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P26215 and previous config saved to /var/cache/conftool/dbconfig/20220422-204547-ladsgroup.json
20:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
20:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26214 and previous config saved to /var/cache/conftool/dbconfig/20220422-204033-ladsgroup.json
20:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26213 and previous config saved to /var/cache/conftool/dbconfig/20220422-202903-ladsgroup.json
20:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
20:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
20:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
20:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
20:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26212 and previous config saved to /var/cache/conftool/dbconfig/20220422-202528-ladsgroup.json
20:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T306560)', diff saved to https://phabricator.wikimedia.org/P26211 and previous config saved to /var/cache/conftool/dbconfig/20220422-201023-ladsgroup.json
20:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T306560)', diff saved to https://phabricator.wikimedia.org/P26210 and previous config saved to /var/cache/conftool/dbconfig/20220422-200605-ladsgroup.json
20:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
20:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
20:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
20:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
20:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
20:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
20:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
20:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
20:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
20:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
19:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance
19:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
19:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
19:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance
19:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
19:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
19:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 (T306560)', diff saved to https://phabricator.wikimedia.org/P26209 and previous config saved to /var/cache/conftool/dbconfig/20220422-191935-ladsgroup.json
19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 (T306560)', diff saved to https://phabricator.wikimedia.org/P26208 and previous config saved to /var/cache/conftool/dbconfig/20220422-191820-ladsgroup.json
19:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
19:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26207 and previous config saved to /var/cache/conftool/dbconfig/20220422-191812-ladsgroup.json
19:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
19:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26206 and previous config saved to /var/cache/conftool/dbconfig/20220422-190632-ladsgroup.json
19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26205 and previous config saved to /var/cache/conftool/dbconfig/20220422-190306-ladsgroup.json
18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26204 and previous config saved to /var/cache/conftool/dbconfig/20220422-185126-ladsgroup.json
18:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26203 and previous config saved to /var/cache/conftool/dbconfig/20220422-184801-ladsgroup.json
18:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26202 and previous config saved to /var/cache/conftool/dbconfig/20220422-183621-ladsgroup.json
18:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26201 and previous config saved to /var/cache/conftool/dbconfig/20220422-183256-ladsgroup.json
18:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26200 and previous config saved to /var/cache/conftool/dbconfig/20220422-182116-ladsgroup.json
17:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26199 and previous config saved to /var/cache/conftool/dbconfig/20220422-173242-ladsgroup.json
17:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
17:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
17:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26198 and previous config saved to /var/cache/conftool/dbconfig/20220422-173234-ladsgroup.json
17:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26197 and previous config saved to /var/cache/conftool/dbconfig/20220422-173031-ladsgroup.json
17:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
17:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
17:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26196 and previous config saved to /var/cache/conftool/dbconfig/20220422-173022-ladsgroup.json
17:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26195 and previous config saved to /var/cache/conftool/dbconfig/20220422-171727-ladsgroup.json
17:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26194 and previous config saved to /var/cache/conftool/dbconfig/20220422-171517-ladsgroup.json
17:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
17:05 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
17:05 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
17:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
17:04 krinkle@deploy1002: Synchronized static/: I5cf234 (duration: 00m 58s)
17:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26193 and previous config saved to /var/cache/conftool/dbconfig/20220422-170222-ladsgroup.json
17:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26192 and previous config saved to /var/cache/conftool/dbconfig/20220422-170012-ladsgroup.json
16:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26191 and previous config saved to /var/cache/conftool/dbconfig/20220422-164717-ladsgroup.json
16:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26190 and previous config saved to /var/cache/conftool/dbconfig/20220422-164507-ladsgroup.json
16:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26189 and previous config saved to /var/cache/conftool/dbconfig/20220422-164359-ladsgroup.json
16:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
16:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
16:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26188 and previous config saved to /var/cache/conftool/dbconfig/20220422-164350-ladsgroup.json
16:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26187 and previous config saved to /var/cache/conftool/dbconfig/20220422-162845-ladsgroup.json
16:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26186 and previous config saved to /var/cache/conftool/dbconfig/20220422-161340-ladsgroup.json
16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26185 and previous config saved to /var/cache/conftool/dbconfig/20220422-160342-ladsgroup.json
16:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
16:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
15:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26184 and previous config saved to /var/cache/conftool/dbconfig/20220422-155835-ladsgroup.json
15:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26183 and previous config saved to /var/cache/conftool/dbconfig/20220422-155617-ladsgroup.json
15:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
15:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
15:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T306560)', diff saved to https://phabricator.wikimedia.org/P26182 and previous config saved to /var/cache/conftool/dbconfig/20220422-155609-ladsgroup.json
15:42 Amir1: cleaning up all of old email tokens in s2
15:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26181 and previous config saved to /var/cache/conftool/dbconfig/20220422-154104-ladsgroup.json
15:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26180 and previous config saved to /var/cache/conftool/dbconfig/20220422-152559-ladsgroup.json
15:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
15:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
15:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P26179 and previous config saved to /var/cache/conftool/dbconfig/20220422-152401-ladsgroup.json
15:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T306560)', diff saved to https://phabricator.wikimedia.org/P26178 and previous config saved to /var/cache/conftool/dbconfig/20220422-151053-ladsgroup.json
15:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26177 and previous config saved to /var/cache/conftool/dbconfig/20220422-150856-ladsgroup.json
15:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T306560)', diff saved to https://phabricator.wikimedia.org/P26176 and previous config saved to /var/cache/conftool/dbconfig/20220422-150836-ladsgroup.json
15:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
15:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
15:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
15:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26175 and previous config saved to /var/cache/conftool/dbconfig/20220422-145351-ladsgroup.json
14:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
14:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P26174 and previous config saved to /var/cache/conftool/dbconfig/20220422-143846-ladsgroup.json
14:01 Amir1: removing all old user_email_token_expires rows in zhwiki
13:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P26173 and previous config saved to /var/cache/conftool/dbconfig/20220422-135334-ladsgroup.json
13:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
13:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
13:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T298565)', diff saved to https://phabricator.wikimedia.org/P26172 and previous config saved to /var/cache/conftool/dbconfig/20220422-135326-ladsgroup.json
13:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26171 and previous config saved to /var/cache/conftool/dbconfig/20220422-133820-ladsgroup.json
13:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26170 and previous config saved to /var/cache/conftool/dbconfig/20220422-132315-ladsgroup.json
13:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
13:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
13:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
13:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
13:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T298565)', diff saved to https://phabricator.wikimedia.org/P26169 and previous config saved to /var/cache/conftool/dbconfig/20220422-130810-ladsgroup.json
12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T298565)', diff saved to https://phabricator.wikimedia.org/P26168 and previous config saved to /var/cache/conftool/dbconfig/20220422-125447-ladsgroup.json
12:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
12:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P26167 and previous config saved to /var/cache/conftool/dbconfig/20220422-125439-ladsgroup.json
12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26166 and previous config saved to /var/cache/conftool/dbconfig/20220422-123934-ladsgroup.json
12:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26165 and previous config saved to /var/cache/conftool/dbconfig/20220422-122429-ladsgroup.json
12:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P26164 and previous config saved to /var/cache/conftool/dbconfig/20220422-120924-ladsgroup.json
11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P26163 and previous config saved to /var/cache/conftool/dbconfig/20220422-115626-ladsgroup.json
11:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
11:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
11:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
11:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
11:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26162 and previous config saved to /var/cache/conftool/dbconfig/20220422-115556-ladsgroup.json
11:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26161 and previous config saved to /var/cache/conftool/dbconfig/20220422-114051-ladsgroup.json
11:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26160 and previous config saved to /var/cache/conftool/dbconfig/20220422-112546-ladsgroup.json
11:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26159 and previous config saved to /var/cache/conftool/dbconfig/20220422-111041-ladsgroup.json
10:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
10:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
10:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
10:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
10:17 reedy@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/TimedMediaHandler/: T306697 (duration: 00m 50s)
10:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26158 and previous config saved to /var/cache/conftool/dbconfig/20220422-101026-ladsgroup.json
10:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
10:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
10:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P26157 and previous config saved to /var/cache/conftool/dbconfig/20220422-101018-ladsgroup.json
09:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26156 and previous config saved to /var/cache/conftool/dbconfig/20220422-095513-ladsgroup.json
09:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26155 and previous config saved to /var/cache/conftool/dbconfig/20220422-094008-ladsgroup.json
09:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P26154 and previous config saved to /var/cache/conftool/dbconfig/20220422-092503-ladsgroup.json
08:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P26153 and previous config saved to /var/cache/conftool/dbconfig/20220422-084431-ladsgroup.json
08:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
08:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
08:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
08:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
08:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T298565)', diff saved to https://phabricator.wikimedia.org/P26152 and previous config saved to /var/cache/conftool/dbconfig/20220422-084418-ladsgroup.json
08:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26151 and previous config saved to /var/cache/conftool/dbconfig/20220422-082913-ladsgroup.json
08:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26150 and previous config saved to /var/cache/conftool/dbconfig/20220422-081408-ladsgroup.json
07:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T298565)', diff saved to https://phabricator.wikimedia.org/P26149 and previous config saved to /var/cache/conftool/dbconfig/20220422-075903-ladsgroup.json
07:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T298565)', diff saved to https://phabricator.wikimedia.org/P26148 and previous config saved to /var/cache/conftool/dbconfig/20220422-074520-ladsgroup.json
07:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
07:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
07:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P26147 and previous config saved to /var/cache/conftool/dbconfig/20220422-074512-ladsgroup.json
07:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26146 and previous config saved to /var/cache/conftool/dbconfig/20220422-073007-ladsgroup.json
07:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26145 and previous config saved to /var/cache/conftool/dbconfig/20220422-071502-ladsgroup.json
06:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P26144 and previous config saved to /var/cache/conftool/dbconfig/20220422-065957-ladsgroup.json
06:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P26143 and previous config saved to /var/cache/conftool/dbconfig/20220422-065332-ladsgroup.json
06:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26142 and previous config saved to /var/cache/conftool/dbconfig/20220422-063827-ladsgroup.json
06:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26141 and previous config saved to /var/cache/conftool/dbconfig/20220422-062322-ladsgroup.json
06:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P26140 and previous config saved to /var/cache/conftool/dbconfig/20220422-061304-ladsgroup.json
06:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
06:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
06:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P26139 and previous config saved to /var/cache/conftool/dbconfig/20220422-060816-ladsgroup.json
05:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
05:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
05:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26138 and previous config saved to /var/cache/conftool/dbconfig/20220422-053246-ladsgroup.json
05:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26137 and previous config saved to /var/cache/conftool/dbconfig/20220422-051740-ladsgroup.json
05:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P26136 and previous config saved to /var/cache/conftool/dbconfig/20220422-050802-ladsgroup.json
05:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
05:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
05:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26135 and previous config saved to /var/cache/conftool/dbconfig/20220422-050235-ladsgroup.json
04:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26134 and previous config saved to /var/cache/conftool/dbconfig/20220422-044730-ladsgroup.json
04:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
04:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
04:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26133 and previous config saved to /var/cache/conftool/dbconfig/20220422-040325-ladsgroup.json
04:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
04:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
04:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26132 and previous config saved to /var/cache/conftool/dbconfig/20220422-040316-ladsgroup.json
03:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26131 and previous config saved to /var/cache/conftool/dbconfig/20220422-034811-ladsgroup.json
03:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
03:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
03:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26130 and previous config saved to /var/cache/conftool/dbconfig/20220422-033306-ladsgroup.json
03:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26129 and previous config saved to /var/cache/conftool/dbconfig/20220422-031801-ladsgroup.json
02:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
02:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
02:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
02:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
02:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P26127 and previous config saved to /var/cache/conftool/dbconfig/20220422-024512-ladsgroup.json
02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26126 and previous config saved to /var/cache/conftool/dbconfig/20220422-023007-ladsgroup.json
02:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P26125 and previous config saved to /var/cache/conftool/dbconfig/20220422-022544-ladsgroup.json
02:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
02:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
02:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26124 and previous config saved to /var/cache/conftool/dbconfig/20220422-021502-ladsgroup.json
01:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P26123 and previous config saved to /var/cache/conftool/dbconfig/20220422-015957-ladsgroup.json
01:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
01:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
01:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance
01:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance
01:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
01:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
01:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26122 and previous config saved to /var/cache/conftool/dbconfig/20220422-010645-ladsgroup.json
00:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P26121 and previous config saved to /var/cache/conftool/dbconfig/20220422-005942-ladsgroup.json
00:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
00:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
00:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P26120 and previous config saved to /var/cache/conftool/dbconfig/20220422-005934-ladsgroup.json
00:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26119 and previous config saved to /var/cache/conftool/dbconfig/20220422-005140-ladsgroup.json
00:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P26118 and previous config saved to /var/cache/conftool/dbconfig/20220422-004429-ladsgroup.json
00:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26117 and previous config saved to /var/cache/conftool/dbconfig/20220422-003634-ladsgroup.json
00:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P26116 and previous config saved to /var/cache/conftool/dbconfig/20220422-002924-ladsgroup.json
00:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26115 and previous config saved to /var/cache/conftool/dbconfig/20220422-002129-ladsgroup.json
00:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P26114 and previous config saved to /var/cache/conftool/dbconfig/20220422-001418-ladsgroup.json
00:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26113 and previous config saved to /var/cache/conftool/dbconfig/20220422-000732-ladsgroup.json
00:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
00:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
00:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26112 and previous config saved to /var/cache/conftool/dbconfig/20220422-000708-ladsgroup.json

2022-04-21

23:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P26111 and previous config saved to /var/cache/conftool/dbconfig/20220421-235814-ladsgroup.json
23:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
23:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
23:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26110 and previous config saved to /var/cache/conftool/dbconfig/20220421-235203-ladsgroup.json
23:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26109 and previous config saved to /var/cache/conftool/dbconfig/20220421-233658-ladsgroup.json
23:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26108 and previous config saved to /var/cache/conftool/dbconfig/20220421-233212-ladsgroup.json
23:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
23:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
23:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26107 and previous config saved to /var/cache/conftool/dbconfig/20220421-232153-ladsgroup.json
23:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
23:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
23:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
23:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
23:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T298565)', diff saved to https://phabricator.wikimedia.org/P26106 and previous config saved to /var/cache/conftool/dbconfig/20220421-231913-ladsgroup.json
23:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26105 and previous config saved to /var/cache/conftool/dbconfig/20220421-231707-ladsgroup.json
23:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
23:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P26104 and previous config saved to /var/cache/conftool/dbconfig/20220421-231049-ladsgroup.json
23:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P26103 and previous config saved to /var/cache/conftool/dbconfig/20220421-230408-ladsgroup.json
23:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26102 and previous config saved to /var/cache/conftool/dbconfig/20220421-230307-ladsgroup.json
23:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
23:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
23:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26101 and previous config saved to /var/cache/conftool/dbconfig/20220421-230243-ladsgroup.json
23:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26100 and previous config saved to /var/cache/conftool/dbconfig/20220421-230202-ladsgroup.json
22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26099 and previous config saved to /var/cache/conftool/dbconfig/20220421-225544-ladsgroup.json
22:52 mutante: gitlab - deleting runner 'ubuntu..something' that has been offline for 2 months, not sure who made it
22:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P26098 and previous config saved to /var/cache/conftool/dbconfig/20220421-224902-ladsgroup.json
22:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26097 and previous config saved to /var/cache/conftool/dbconfig/20220421-224738-ladsgroup.json
22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26096 and previous config saved to /var/cache/conftool/dbconfig/20220421-224657-ladsgroup.json
22:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26095 and previous config saved to /var/cache/conftool/dbconfig/20220421-224437-ladsgroup.json
22:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
22:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
22:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
22:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
22:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
22:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
22:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
22:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
22:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
22:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
22:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26094 and previous config saved to /var/cache/conftool/dbconfig/20220421-224322-ladsgroup.json
22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26093 and previous config saved to /var/cache/conftool/dbconfig/20220421-224039-ladsgroup.json
22:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T298565)', diff saved to https://phabricator.wikimedia.org/P26092 and previous config saved to /var/cache/conftool/dbconfig/20220421-223357-ladsgroup.json
22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26091 and previous config saved to /var/cache/conftool/dbconfig/20220421-223233-ladsgroup.json
22:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26090 and previous config saved to /var/cache/conftool/dbconfig/20220421-222817-ladsgroup.json
22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T298565)', diff saved to https://phabricator.wikimedia.org/P26089 and previous config saved to /var/cache/conftool/dbconfig/20220421-222550-ladsgroup.json
22:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
22:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P26088 and previous config saved to /var/cache/conftool/dbconfig/20220421-222542-ladsgroup.json
22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P26087 and previous config saved to /var/cache/conftool/dbconfig/20220421-222534-ladsgroup.json
22:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26086 and previous config saved to /var/cache/conftool/dbconfig/20220421-221728-ladsgroup.json
22:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26085 and previous config saved to /var/cache/conftool/dbconfig/20220421-221312-ladsgroup.json
22:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26084 and previous config saved to /var/cache/conftool/dbconfig/20220421-221037-ladsgroup.json
22:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P26083 and previous config saved to /var/cache/conftool/dbconfig/20220421-220552-ladsgroup.json
22:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
22:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
22:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
22:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
22:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P26082 and previous config saved to /var/cache/conftool/dbconfig/20220421-220539-ladsgroup.json
22:02 mutante: gitlab-runner2001 - systemctl start docker-resource-monitor ; systemctl start docker-gc
22:00 mutante: gitlab-runner2001 - installing apparmor ('apparmor' is the user utilities package and was NOT installed, libapparmor1 WAS installed), this caused bug https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1808456.html after upgrading gitlab-runner to bullseye because bullseye comes with libapparmor1 by default as opposed to before T297659
21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26081 and previous config saved to /var/cache/conftool/dbconfig/20220421-215807-ladsgroup.json
21:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26080 and previous config saved to /var/cache/conftool/dbconfig/20220421-215547-ladsgroup.json
21:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
21:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
21:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26079 and previous config saved to /var/cache/conftool/dbconfig/20220421-215540-ladsgroup.json
21:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26078 and previous config saved to /var/cache/conftool/dbconfig/20220421-215532-ladsgroup.json
21:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26077 and previous config saved to /var/cache/conftool/dbconfig/20220421-215034-ladsgroup.json
21:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26076 and previous config saved to /var/cache/conftool/dbconfig/20220421-214445-ladsgroup.json
21:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
21:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
21:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26075 and previous config saved to /var/cache/conftool/dbconfig/20220421-214422-ladsgroup.json
21:42 mutante: shutting down and reimaging gitlab-runner1001 T297659
21:40 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gitlab-runner1001.eqiad.wmnet with reason: reimage
21:40 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on gitlab-runner1001.eqiad.wmnet with reason: reimage
21:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26074 and previous config saved to /var/cache/conftool/dbconfig/20220421-214035-ladsgroup.json
21:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P26073 and previous config saved to /var/cache/conftool/dbconfig/20220421-214027-ladsgroup.json
21:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P26072 and previous config saved to /var/cache/conftool/dbconfig/20220421-213819-ladsgroup.json
21:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
21:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
21:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P26071 and previous config saved to /var/cache/conftool/dbconfig/20220421-213811-ladsgroup.json
21:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26070 and previous config saved to /var/cache/conftool/dbconfig/20220421-213529-ladsgroup.json
21:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26069 and previous config saved to /var/cache/conftool/dbconfig/20220421-212916-ladsgroup.json
21:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26068 and previous config saved to /var/cache/conftool/dbconfig/20220421-212523-ladsgroup.json
21:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26067 and previous config saved to /var/cache/conftool/dbconfig/20220421-212306-ladsgroup.json
21:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P26066 and previous config saved to /var/cache/conftool/dbconfig/20220421-212022-ladsgroup.json
21:19 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:19 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26065 and previous config saved to /var/cache/conftool/dbconfig/20220421-211411-ladsgroup.json
21:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26064 and previous config saved to /var/cache/conftool/dbconfig/20220421-211018-ladsgroup.json
21:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26063 and previous config saved to /var/cache/conftool/dbconfig/20220421-210801-ladsgroup.json
21:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T306560)', diff saved to https://phabricator.wikimedia.org/P26062 and previous config saved to /var/cache/conftool/dbconfig/20220421-210658-ladsgroup.json
21:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
21:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
21:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T306560)', diff saved to https://phabricator.wikimedia.org/P26061 and previous config saved to /var/cache/conftool/dbconfig/20220421-210650-ladsgroup.json
21:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P26060 and previous config saved to /var/cache/conftool/dbconfig/20220421-210414-ladsgroup.json
21:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
21:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
20:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26059 and previous config saved to /var/cache/conftool/dbconfig/20220421-205906-ladsgroup.json
20:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P26058 and previous config saved to /var/cache/conftool/dbconfig/20220421-205256-ladsgroup.json
20:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26057 and previous config saved to /var/cache/conftool/dbconfig/20220421-205145-ladsgroup.json
20:50 cdanis: re-enabled puppet and repooled cp2029
20:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P26056 and previous config saved to /var/cache/conftool/dbconfig/20220421-204709-ladsgroup.json
20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26055 and previous config saved to /var/cache/conftool/dbconfig/20220421-204532-ladsgroup.json
20:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
20:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26054 and previous config saved to /var/cache/conftool/dbconfig/20220421-204508-ladsgroup.json
20:45 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gitlab-runner2001.codfw.wmnet with reason: reimage
20:45 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on gitlab-runner2001.codfw.wmnet with reason: reimage
20:41 nokafor@deploy1002: Finished deploy [airflow-dags/analytics@bd28d80]: (no justification provided) (duration: 00m 07s)
20:41 nokafor@deploy1002: Started deploy [airflow-dags/analytics@bd28d80]: (no justification provided)
20:39 nokafor@deploy1002: Finished deploy [airflow-dags/analytics@bd28d80]: (no justification provided) (duration: 00m 27s)
20:39 nokafor@deploy1002: Started deploy [airflow-dags/analytics@bd28d80]: (no justification provided)
20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26053 and previous config saved to /var/cache/conftool/dbconfig/20220421-203640-ladsgroup.json
20:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26052 and previous config saved to /var/cache/conftool/dbconfig/20220421-203204-ladsgroup.json
20:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26051 and previous config saved to /var/cache/conftool/dbconfig/20220421-203003-ladsgroup.json
20:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P26050 and previous config saved to /var/cache/conftool/dbconfig/20220421-202826-ladsgroup.json
20:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
20:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
20:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T298565)', diff saved to https://phabricator.wikimedia.org/P26049 and previous config saved to /var/cache/conftool/dbconfig/20220421-202818-ladsgroup.json
20:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T306560)', diff saved to https://phabricator.wikimedia.org/P26048 and previous config saved to /var/cache/conftool/dbconfig/20220421-202135-ladsgroup.json
20:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T306560)', diff saved to https://phabricator.wikimedia.org/P26047 and previous config saved to /var/cache/conftool/dbconfig/20220421-201825-ladsgroup.json
20:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
20:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
20:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T306560)', diff saved to https://phabricator.wikimedia.org/P26046 and previous config saved to /var/cache/conftool/dbconfig/20220421-201817-ladsgroup.json
20:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26045 and previous config saved to /var/cache/conftool/dbconfig/20220421-201659-ladsgroup.json
20:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26044 and previous config saved to /var/cache/conftool/dbconfig/20220421-201455-ladsgroup.json
20:14 mutante: reimaging gitlab-runner2001.codfw.wmnet one more time to confirm things work from scratch now T297659
20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26043 and previous config saved to /var/cache/conftool/dbconfig/20220421-201313-ladsgroup.json
20:09 mutante: [ganeti2021:~] $ sudo gnt-instance shutdown gitlab-runner2001.codfw.wmnet
20:08 mutante: [puppetmaster1001:~] $ sudo puppet cert clean gitlab-runner2001.codfw.wmnet
20:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26042 and previous config saved to /var/cache/conftool/dbconfig/20220421-200312-ladsgroup.json
20:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P26041 and previous config saved to /var/cache/conftool/dbconfig/20220421-200154-ladsgroup.json
19:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26040 and previous config saved to /var/cache/conftool/dbconfig/20220421-195950-ladsgroup.json
19:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26039 and previous config saved to /var/cache/conftool/dbconfig/20220421-195808-ladsgroup.json
19:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26038 and previous config saved to /var/cache/conftool/dbconfig/20220421-194807-ladsgroup.json
19:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26037 and previous config saved to /var/cache/conftool/dbconfig/20220421-194419-ladsgroup.json
19:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
19:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
19:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T298565)', diff saved to https://phabricator.wikimedia.org/P26036 and previous config saved to /var/cache/conftool/dbconfig/20220421-194303-ladsgroup.json
19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T306560)', diff saved to https://phabricator.wikimedia.org/P26035 and previous config saved to /var/cache/conftool/dbconfig/20220421-193302-ladsgroup.json
19:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T306560)', diff saved to https://phabricator.wikimedia.org/P26034 and previous config saved to /var/cache/conftool/dbconfig/20220421-193052-ladsgroup.json
19:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
19:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
19:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
19:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T306560)', diff saved to https://phabricator.wikimedia.org/P26033 and previous config saved to /var/cache/conftool/dbconfig/20220421-193039-ladsgroup.json
19:23 cdanis: depooling & disabling puppet on cp2029 for some manual testing T303534
19:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P26032 and previous config saved to /var/cache/conftool/dbconfig/20220421-192330-ladsgroup.json
19:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
19:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
19:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P26031 and previous config saved to /var/cache/conftool/dbconfig/20220421-192322-ladsgroup.json
19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T298565)', diff saved to https://phabricator.wikimedia.org/P26030 and previous config saved to /var/cache/conftool/dbconfig/20220421-191847-ladsgroup.json
19:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
19:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
19:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
19:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
19:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26029 and previous config saved to /var/cache/conftool/dbconfig/20220421-191534-ladsgroup.json
19:08 ebernhardson: set index.unassigned.node_left.delayed_timeout to null for all indices on elasticsearch-eqiad-psi (:9200), reverting previous test of 10m back to defaults
19:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26028 and previous config saved to /var/cache/conftool/dbconfig/20220421-190817-ladsgroup.json
19:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26027 and previous config saved to /var/cache/conftool/dbconfig/20220421-190029-ladsgroup.json
18:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26026 and previous config saved to /var/cache/conftool/dbconfig/20220421-185312-ladsgroup.json
18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T306560)', diff saved to https://phabricator.wikimedia.org/P26025 and previous config saved to /var/cache/conftool/dbconfig/20220421-184523-ladsgroup.json
18:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P26024 and previous config saved to /var/cache/conftool/dbconfig/20220421-183807-ladsgroup.json
18:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P26023 and previous config saved to /var/cache/conftool/dbconfig/20220421-181614-ladsgroup.json
18:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
18:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
18:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
18:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
18:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P26022 and previous config saved to /var/cache/conftool/dbconfig/20220421-181601-ladsgroup.json
18:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:03 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.8 refs T305214
18:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26021 and previous config saved to /var/cache/conftool/dbconfig/20220421-180056-ladsgroup.json
17:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P26020 and previous config saved to /var/cache/conftool/dbconfig/20220421-175514-ladsgroup.json
17:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26019 and previous config saved to /var/cache/conftool/dbconfig/20220421-174551-ladsgroup.json
17:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T306560)', diff saved to https://phabricator.wikimedia.org/P26018 and previous config saved to /var/cache/conftool/dbconfig/20220421-174509-ladsgroup.json
17:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
17:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
17:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T306560)', diff saved to https://phabricator.wikimedia.org/P26017 and previous config saved to /var/cache/conftool/dbconfig/20220421-174501-ladsgroup.json
17:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26016 and previous config saved to /var/cache/conftool/dbconfig/20220421-174009-ladsgroup.json
17:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P26015 and previous config saved to /var/cache/conftool/dbconfig/20220421-173046-ladsgroup.json
17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26013 and previous config saved to /var/cache/conftool/dbconfig/20220421-172956-ladsgroup.json
17:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26012 and previous config saved to /var/cache/conftool/dbconfig/20220421-172504-ladsgroup.json
17:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26011 and previous config saved to /var/cache/conftool/dbconfig/20220421-171451-ladsgroup.json
17:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P26010 and previous config saved to /var/cache/conftool/dbconfig/20220421-170959-ladsgroup.json
17:05 kormat@cumin1001: dbctl commit (dc=all): 'db1120 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26009 and previous config saved to /var/cache/conftool/dbconfig/20220421-170551-kormat.json
16:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T306560)', diff saved to https://phabricator.wikimedia.org/P26008 and previous config saved to /var/cache/conftool/dbconfig/20220421-165946-ladsgroup.json
16:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T306560)', diff saved to https://phabricator.wikimedia.org/P26007 and previous config saved to /var/cache/conftool/dbconfig/20220421-165635-ladsgroup.json
16:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
16:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P26006 and previous config saved to /var/cache/conftool/dbconfig/20220421-165333-ladsgroup.json
16:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
16:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
16:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
16:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P26005 and previous config saved to /var/cache/conftool/dbconfig/20220421-165319-ladsgroup.json
16:50 kormat@cumin1001: dbctl commit (dc=all): 'db1120 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26004 and previous config saved to /var/cache/conftool/dbconfig/20220421-165047-kormat.json
16:45 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host parse1015.mgmt.eqiad.wmnet with reboot policy FORCED
16:43 XioNoX: replace mr1-eqiad - T294474
16:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26003 and previous config saved to /var/cache/conftool/dbconfig/20220421-163814-ladsgroup.json
16:35 kormat@cumin1001: dbctl commit (dc=all): 'db1120 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P26002 and previous config saved to /var/cache/conftool/dbconfig/20220421-163543-kormat.json
16:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P26001 and previous config saved to /var/cache/conftool/dbconfig/20220421-163031-ladsgroup.json
16:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
16:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
16:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26000 and previous config saved to /var/cache/conftool/dbconfig/20220421-162309-ladsgroup.json
16:20 kormat@cumin1001: dbctl commit (dc=all): 'db1120 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25999 and previous config saved to /var/cache/conftool/dbconfig/20220421-162039-kormat.json
16:17 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1120.eqiad.wmnet with reason: Rebooting for T303174
16:17 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1120.eqiad.wmnet with reason: Rebooting for T303174
16:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25998 and previous config saved to /var/cache/conftool/dbconfig/20220421-160804-ladsgroup.json
16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25997 and previous config saved to /var/cache/conftool/dbconfig/20220421-160133-ladsgroup.json
16:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
16:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25996 and previous config saved to /var/cache/conftool/dbconfig/20220421-160125-ladsgroup.json
15:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25995 and previous config saved to /var/cache/conftool/dbconfig/20220421-154620-ladsgroup.json
15:44 kormat@cumin1001: dbctl commit (dc=all): 'db1153 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25994 and previous config saved to /var/cache/conftool/dbconfig/20220421-154426-kormat.json
15:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
15:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25993 and previous config saved to /var/cache/conftool/dbconfig/20220421-154314-ladsgroup.json
15:42 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1146.eqiad.wmnet with OS buster
15:41 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1145.eqiad.wmnet with OS buster
15:41 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1144.eqiad.wmnet with OS buster
15:40 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply
15:39 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1143.eqiad.wmnet with OS buster
15:39 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply
15:38 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply
15:37 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply
15:36 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply
15:36 btullis@deploy1002: helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply
15:33 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply
15:33 btullis@deploy1002: helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply
15:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25992 and previous config saved to /var/cache/conftool/dbconfig/20220421-153115-ladsgroup.json
15:29 kormat@cumin1001: dbctl commit (dc=all): 'db1153 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25991 and previous config saved to /var/cache/conftool/dbconfig/20220421-152922-kormat.json
15:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25990 and previous config saved to /var/cache/conftool/dbconfig/20220421-152809-ladsgroup.json
15:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25989 and previous config saved to /var/cache/conftool/dbconfig/20220421-151610-ladsgroup.json
15:14 kormat@cumin1001: dbctl commit (dc=all): 'db1153 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25988 and previous config saved to /var/cache/conftool/dbconfig/20220421-151418-kormat.json
15:14 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1146.eqiad.wmnet with OS buster
15:13 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1145.eqiad.wmnet with OS buster
15:13 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1144.eqiad.wmnet with OS buster
15:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25987 and previous config saved to /var/cache/conftool/dbconfig/20220421-151303-ladsgroup.json
15:12 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1144.eqiad.wmnet with OS buster
15:12 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1145.eqiad.wmnet with OS buster
15:12 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1146.eqiad.wmnet with OS buster
15:12 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1146.eqiad.wmnet with OS buster
15:11 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1144.eqiad.wmnet with OS buster
15:11 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1143.eqiad.wmnet with OS buster
15:11 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1145.eqiad.wmnet with OS buster
15:10 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1143.eqiad.wmnet with OS buster
15:10 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1143.eqiad.wmnet with OS buster
15:10 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1143.eqiad.wmnet with OS buster
15:10 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1143.eqiad.wmnet with OS buster
15:10 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1143.eqiad.wmnet with OS buster
15:10 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1143.eqiad.wmnet with OS buster
15:10 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1143.eqiad.wmnet with OS buster
15:10 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1143.eqiad.wmnet with OS buster
15:10 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1143.eqiad.wmnet with OS buster
15:10 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1143.eqiad.wmnet with OS buster
15:10 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1143.eqiad.wmnet with OS buster
15:10 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1143.eqiad.wmnet with OS buster
15:09 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1143.eqiad.wmnet with OS buster
15:09 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-worker1143.eqiad.wmnet with OS buster
15:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25986 and previous config saved to /var/cache/conftool/dbconfig/20220421-150937-ladsgroup.json
15:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
15:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
15:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25985 and previous config saved to /var/cache/conftool/dbconfig/20220421-150929-ladsgroup.json
14:59 kormat@cumin1001: dbctl commit (dc=all): 'db1153 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25984 and previous config saved to /var/cache/conftool/dbconfig/20220421-145914-kormat.json
14:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25983 and previous config saved to /var/cache/conftool/dbconfig/20220421-145758-ladsgroup.json
14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25982 and previous config saved to /var/cache/conftool/dbconfig/20220421-145424-ladsgroup.json
14:53 kormat@cumin1001: dbctl commit (dc=all): 'db1153 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25981 and previous config saved to /var/cache/conftool/dbconfig/20220421-145303-kormat.json
14:53 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1153.eqiad.wmnet with reason: Rebooting for T303174
14:52 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1153.eqiad.wmnet with reason: Rebooting for T303174
14:52 kormat@cumin1001: dbctl commit (dc=all): 'db1152 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25980 and previous config saved to /var/cache/conftool/dbconfig/20220421-145231-kormat.json
14:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25979 and previous config saved to /var/cache/conftool/dbconfig/20220421-144145-ladsgroup.json
14:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
14:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
14:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25978 and previous config saved to /var/cache/conftool/dbconfig/20220421-144137-ladsgroup.json
14:40 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:40 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
14:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25977 and previous config saved to /var/cache/conftool/dbconfig/20220421-143918-ladsgroup.json
14:37 kormat@cumin1001: dbctl commit (dc=all): 'db1152 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25976 and previous config saved to /var/cache/conftool/dbconfig/20220421-143727-kormat.json
14:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor1006.eqiad.wmnet
14:37 ladsgroup@deploy1002: Synchronized wmf-config: Config: Re-enable article editing by anonymous users on fawiki (T292781) (duration: 00m 51s)
14:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host thumbor1006.eqiad.wmnet
14:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25975 and previous config saved to /var/cache/conftool/dbconfig/20220421-142631-ladsgroup.json
14:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor1005.eqiad.wmnet
14:25 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1117.eqiad.wmnet with reason: Rebooting for T303174
14:25 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1117.eqiad.wmnet with reason: Rebooting for T303174
14:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25974 and previous config saved to /var/cache/conftool/dbconfig/20220421-142413-ladsgroup.json
14:22 kormat@cumin1001: dbctl commit (dc=all): 'db1152 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25973 and previous config saved to /var/cache/conftool/dbconfig/20220421-142223-kormat.json
14:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25972 and previous config saved to /var/cache/conftool/dbconfig/20220421-141727-ladsgroup.json
14:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
14:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
14:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host thumbor1005.eqiad.wmnet
14:15 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor1002.eqiad.wmnet
14:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25971 and previous config saved to /var/cache/conftool/dbconfig/20220421-141126-ladsgroup.json
14:10 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1019.mgmt.eqiad.wmnet with reboot policy FORCED
14:10 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1020.mgmt.eqiad.wmnet with reboot policy FORCED
14:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
14:07 kormat@cumin1001: dbctl commit (dc=all): 'db1152 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25969 and previous config saved to /var/cache/conftool/dbconfig/20220421-140719-kormat.json
14:05 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host thumbor1002.eqiad.wmnet
14:03 kormat@cumin1001: dbctl commit (dc=all): 'db1152 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25968 and previous config saved to /var/cache/conftool/dbconfig/20220421-140309-kormat.json
14:03 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1152.eqiad.wmnet with reason: Rebooting for T303174
14:03 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1152.eqiad.wmnet with reason: Rebooting for T303174
14:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor1001.eqiad.wmnet
13:58 kormat@cumin1001: dbctl commit (dc=all): 'db1120 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25967 and previous config saved to /var/cache/conftool/dbconfig/20220421-135831-kormat.json
13:58 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1120.eqiad.wmnet with reason: Rebooting for T303174
13:58 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1120.eqiad.wmnet with reason: Rebooting for T303174
13:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25966 and previous config saved to /var/cache/conftool/dbconfig/20220421-135621-ladsgroup.json
13:55 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1020.mgmt.eqiad.wmnet with reboot policy FORCED
13:55 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1019.mgmt.eqiad.wmnet with reboot policy FORCED
13:54 moritzm: powercycling thumbor1001, stuck on reboot
13:45 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host thumbor1001.eqiad.wmnet
13:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25965 and previous config saved to /var/cache/conftool/dbconfig/20220421-133204-ladsgroup.json
13:31 taavi@deploy1002: Synchronized wmf-config/interwiki.php: Config: Update interwiki cache (duration: 00m 51s)
13:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25964 and previous config saved to /var/cache/conftool/dbconfig/20220421-132935-ladsgroup.json
13:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
13:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
13:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
13:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
13:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
13:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
13:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
13:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
13:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor2006.codfw.wmnet
13:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
13:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25963 and previous config saved to /var/cache/conftool/dbconfig/20220421-131902-ladsgroup.json
13:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25962 and previous config saved to /var/cache/conftool/dbconfig/20220421-131713-ladsgroup.json
13:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
13:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
13:09 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host thumbor2006.codfw.wmnet
13:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor2005.codfw.wmnet
13:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25961 and previous config saved to /var/cache/conftool/dbconfig/20220421-130357-ladsgroup.json
13:03 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 7d5114e: plwiki: Fix cascading protection configuration (T306300) (duration: 00m 55s)
13:02 vgutierrez: restart ats-be and varnish-fe on cp2036 to clear restarted service alerts
12:55 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host thumbor2005.codfw.wmnet
12:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor2004.codfw.wmnet
12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25960 and previous config saved to /var/cache/conftool/dbconfig/20220421-124852-ladsgroup.json
12:45 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host thumbor2004.codfw.wmnet
12:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor2003.codfw.wmnet
12:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host thumbor2003.codfw.wmnet
12:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25959 and previous config saved to /var/cache/conftool/dbconfig/20220421-123347-ladsgroup.json
12:30 moritzm: installing fribidi security updates
12:29 marostegui@cumin1001: dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 100%: After maintenance', diff saved to https://phabricator.wikimedia.org/P25958 and previous config saved to /var/cache/conftool/dbconfig/20220421-122859-root.json
12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25957 and previous config saved to /var/cache/conftool/dbconfig/20220421-122722-ladsgroup.json
12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
12:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25956 and previous config saved to /var/cache/conftool/dbconfig/20220421-122627-ladsgroup.json
12:25 moritzm: installing flac security updates
12:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
12:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
12:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
12:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
12:20 moritzm: installing openjpeg2 security updates
12:13 marostegui@cumin1001: dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 75%: After maintenance', diff saved to https://phabricator.wikimedia.org/P25955 and previous config saved to /var/cache/conftool/dbconfig/20220421-121355-root.json
12:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25954 and previous config saved to /var/cache/conftool/dbconfig/20220421-121122-ladsgroup.json
12:10 moritzm: installing subversion security updates
11:58 marostegui@cumin1001: dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 50%: After maintenance', diff saved to https://phabricator.wikimedia.org/P25953 and previous config saved to /var/cache/conftool/dbconfig/20220421-115851-root.json
11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25952 and previous config saved to /var/cache/conftool/dbconfig/20220421-115617-ladsgroup.json
11:43 marostegui@cumin1001: dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 25%: After maintenance', diff saved to https://phabricator.wikimedia.org/P25951 and previous config saved to /var/cache/conftool/dbconfig/20220421-114347-root.json
11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25950 and previous config saved to /var/cache/conftool/dbconfig/20220421-114112-ladsgroup.json
11:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
11:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
11:35 moritzm: installing zlib security updates on stretch (buster/bullseye already fixed)
11:34 kart_: Updated cxserver to 2022-04-21-081331-production (T287655, T304855, T304862, T304866, T305115)
11:30 kartik@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
11:29 kartik@deploy1002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
11:28 marostegui@cumin1001: dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 10%: After maintenance', diff saved to https://phabricator.wikimedia.org/P25949 and previous config saved to /var/cache/conftool/dbconfig/20220421-112843-root.json
11:28 kartik@deploy1002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
11:27 kartik@deploy1002: helmfile [codfw] START helmfile.d/services/cxserver: apply
11:26 marostegui@cumin1001: dbctl commit (dc=all): 'db1109 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P25948 and previous config saved to /var/cache/conftool/dbconfig/20220421-112648-root.json
11:23 kartik@deploy1002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
11:22 kartik@deploy1002: helmfile [staging] START helmfile.d/services/cxserver: apply
11:14 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2004.codfw.wmnet with OS bullseye
11:13 marostegui@cumin1001: dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 5%: After maintenance', diff saved to https://phabricator.wikimedia.org/P25947 and previous config saved to /var/cache/conftool/dbconfig/20220421-111340-root.json
11:13 marostegui: dbmaint s2@codfw T306604
11:11 marostegui@cumin1001: dbctl commit (dc=all): 'db1109 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P25946 and previous config saved to /var/cache/conftool/dbconfig/20220421-111144-root.json
11:05 jynus@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup1002.eqiad.wmnet with OS bullseye
10:58 marostegui@cumin1001: dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 1%: After maintenance', diff saved to https://phabricator.wikimedia.org/P25945 and previous config saved to /var/cache/conftool/dbconfig/20220421-105835-root.json
10:56 marostegui@cumin1001: dbctl commit (dc=all): 'db1109 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P25944 and previous config saved to /var/cache/conftool/dbconfig/20220421-105638-root.json
10:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
10:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
10:54 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2004.codfw.wmnet with reason: host reimage
10:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping3002.esams.wmnet
10:50 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on backup2004.codfw.wmnet with reason: host reimage
10:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ping3002.esams.wmnet
10:41 marostegui@cumin1001: dbctl commit (dc=all): 'db1109 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P25942 and previous config saved to /var/cache/conftool/dbconfig/20220421-104135-root.json
10:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25941 and previous config saved to /var/cache/conftool/dbconfig/20220421-104057-ladsgroup.json
10:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
10:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
10:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
10:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
10:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25940 and previous config saved to /var/cache/conftool/dbconfig/20220421-104044-ladsgroup.json
10:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25939 and previous config saved to /var/cache/conftool/dbconfig/20220421-103837-ladsgroup.json
10:32 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2004.codfw.wmnet with OS bullseye
10:30 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host backup1002.eqiad.wmnet with OS bullseye
10:26 marostegui@cumin1001: dbctl commit (dc=all): 'db1109 (re)pooling @ 10%: After schema change', diff saved to https://phabricator.wikimedia.org/P25938 and previous config saved to /var/cache/conftool/dbconfig/20220421-102631-root.json
10:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25937 and previous config saved to /var/cache/conftool/dbconfig/20220421-102539-ladsgroup.json
10:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25936 and previous config saved to /var/cache/conftool/dbconfig/20220421-102332-ladsgroup.json
10:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping2002.codfw.wmnet
10:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ping2002.codfw.wmnet
10:11 marostegui@cumin1001: dbctl commit (dc=all): 'db1109 (re)pooling @ 1%: After schema change', diff saved to https://phabricator.wikimedia.org/P25935 and previous config saved to /var/cache/conftool/dbconfig/20220421-101127-root.json
10:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25934 and previous config saved to /var/cache/conftool/dbconfig/20220421-101034-ladsgroup.json
10:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25933 and previous config saved to /var/cache/conftool/dbconfig/20220421-100827-ladsgroup.json
10:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance
10:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance
10:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
10:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
10:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
10:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
10:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P25932 and previous config saved to /var/cache/conftool/dbconfig/20220421-100359-ladsgroup.json
09:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25931 and previous config saved to /var/cache/conftool/dbconfig/20220421-095529-ladsgroup.json
09:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping1002.eqiad.wmnet
09:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25930 and previous config saved to /var/cache/conftool/dbconfig/20220421-095322-ladsgroup.json
09:52 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ping1002.eqiad.wmnet
09:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P25929 and previous config saved to /var/cache/conftool/dbconfig/20220421-094853-ladsgroup.json
09:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25928 and previous config saved to /var/cache/conftool/dbconfig/20220421-094807-ladsgroup.json
09:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
09:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
09:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
09:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
09:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
09:42 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2005.codfw.wmnet with OS bullseye
09:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
09:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
09:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
09:41 moritzm: upgrading the Ganeti test cluster to 3.0 T306499
09:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
09:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
09:35 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup1004.eqiad.wmnet with OS bullseye
09:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P25927 and previous config saved to /var/cache/conftool/dbconfig/20220421-093348-ladsgroup.json
09:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P25926 and previous config saved to /var/cache/conftool/dbconfig/20220421-091843-ladsgroup.json
09:12 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1004.eqiad.wmnet with reason: host reimage
09:10 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2005.codfw.wmnet with reason: host reimage
09:07 jynus@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on backup1004.eqiad.wmnet with reason: host reimage
09:06 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on backup2005.codfw.wmnet with reason: host reimage
08:55 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host backup1004.eqiad.wmnet with OS bullseye
08:53 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2005.codfw.wmnet with OS bullseye
08:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25925 and previous config saved to /var/cache/conftool/dbconfig/20220421-085307-ladsgroup.json
08:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
08:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
08:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P25924 and previous config saved to /var/cache/conftool/dbconfig/20220421-085259-ladsgroup.json
08:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25923 and previous config saved to /var/cache/conftool/dbconfig/20220421-085214-ladsgroup.json
08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25922 and previous config saved to /var/cache/conftool/dbconfig/20220421-084943-ladsgroup.json
08:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
08:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25921 and previous config saved to /var/cache/conftool/dbconfig/20220421-084935-ladsgroup.json
08:48 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup1005.eqiad.wmnet with OS bullseye
08:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25920 and previous config saved to /var/cache/conftool/dbconfig/20220421-083754-ladsgroup.json
08:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25919 and previous config saved to /var/cache/conftool/dbconfig/20220421-083430-ladsgroup.json
08:30 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
08:30 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
08:30 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
08:30 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
08:29 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1005.eqiad.wmnet with reason: host reimage
08:25 jynus@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on backup1005.eqiad.wmnet with reason: host reimage
08:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25918 and previous config saved to /var/cache/conftool/dbconfig/20220421-082249-ladsgroup.json
08:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25917 and previous config saved to /var/cache/conftool/dbconfig/20220421-081925-ladsgroup.json
08:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P25916 and previous config saved to /var/cache/conftool/dbconfig/20220421-081829-ladsgroup.json
08:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
08:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
08:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P25915 and previous config saved to /var/cache/conftool/dbconfig/20220421-081821-ladsgroup.json
08:11 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host backup1005.eqiad.wmnet with OS bullseye
08:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P25914 and previous config saved to /var/cache/conftool/dbconfig/20220421-080744-ladsgroup.json
08:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25913 and previous config saved to /var/cache/conftool/dbconfig/20220421-080420-ladsgroup.json
08:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P25912 and previous config saved to /var/cache/conftool/dbconfig/20220421-080316-ladsgroup.json
07:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25911 and previous config saved to /var/cache/conftool/dbconfig/20220421-075734-ladsgroup.json
07:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
07:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
07:53 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2006.codfw.wmnet with OS bullseye
07:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
07:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
07:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25910 and previous config saved to /var/cache/conftool/dbconfig/20220421-075300-ladsgroup.json
07:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P25909 and previous config saved to /var/cache/conftool/dbconfig/20220421-074811-ladsgroup.json
07:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25908 and previous config saved to /var/cache/conftool/dbconfig/20220421-073755-ladsgroup.json
07:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P25907 and previous config saved to /var/cache/conftool/dbconfig/20220421-073306-ladsgroup.json
07:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P25906 and previous config saved to /var/cache/conftool/dbconfig/20220421-073037-ladsgroup.json
07:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
07:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
07:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P25905 and previous config saved to /var/cache/conftool/dbconfig/20220421-073029-ladsgroup.json
07:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25904 and previous config saved to /var/cache/conftool/dbconfig/20220421-072249-ladsgroup.json
07:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P25903 and previous config saved to /var/cache/conftool/dbconfig/20220421-071524-ladsgroup.json
07:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25902 and previous config saved to /var/cache/conftool/dbconfig/20220421-070744-ladsgroup.json
07:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P25901 and previous config saved to /var/cache/conftool/dbconfig/20220421-070729-ladsgroup.json
07:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
07:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
07:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
07:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
07:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P25900 and previous config saved to /var/cache/conftool/dbconfig/20220421-070716-ladsgroup.json
07:06 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2006.codfw.wmnet with reason: host reimage
07:02 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on backup2006.codfw.wmnet with reason: host reimage
07:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25899 and previous config saved to /var/cache/conftool/dbconfig/20220421-070208-ladsgroup.json
07:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
07:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
07:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25898 and previous config saved to /var/cache/conftool/dbconfig/20220421-070113-ladsgroup.json
07:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P25897 and previous config saved to /var/cache/conftool/dbconfig/20220421-070019-ladsgroup.json
06:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P25896 and previous config saved to /var/cache/conftool/dbconfig/20220421-065211-ladsgroup.json
06:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25895 and previous config saved to /var/cache/conftool/dbconfig/20220421-064608-ladsgroup.json
06:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P25894 and previous config saved to /var/cache/conftool/dbconfig/20220421-064514-ladsgroup.json
06:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T306560)', diff saved to https://phabricator.wikimedia.org/P25893 and previous config saved to /var/cache/conftool/dbconfig/20220421-064245-ladsgroup.json
06:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
06:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
06:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
06:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
06:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T306560)', diff saved to https://phabricator.wikimedia.org/P25892 and previous config saved to /var/cache/conftool/dbconfig/20220421-064210-ladsgroup.json
06:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P25891 and previous config saved to /var/cache/conftool/dbconfig/20220421-063706-ladsgroup.json
06:34 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2006.codfw.wmnet with OS bullseye
06:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25890 and previous config saved to /var/cache/conftool/dbconfig/20220421-063103-ladsgroup.json
06:30 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup1006.eqiad.wmnet with OS bullseye
06:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P25889 and previous config saved to /var/cache/conftool/dbconfig/20220421-062705-ladsgroup.json
06:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P25888 and previous config saved to /var/cache/conftool/dbconfig/20220421-062201-ladsgroup.json
06:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25887 and previous config saved to /var/cache/conftool/dbconfig/20220421-061558-ladsgroup.json
06:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P25886 and previous config saved to /var/cache/conftool/dbconfig/20220421-061200-ladsgroup.json
06:11 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1006.eqiad.wmnet with reason: host reimage
06:08 jynus@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on backup1006.eqiad.wmnet with reason: host reimage
06:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1109 T303927', diff saved to https://phabricator.wikimedia.org/P25885 and previous config saved to /var/cache/conftool/dbconfig/20220421-060512-root.json
06:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Promote db1104 to s8 primary and set section read-write T303927', diff saved to https://phabricator.wikimedia.org/P25884 and previous config saved to /var/cache/conftool/dbconfig/20220421-060106-ladsgroup.json
06:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - T303927', diff saved to https://phabricator.wikimedia.org/P25883 and previous config saved to /var/cache/conftool/dbconfig/20220421-060023-ladsgroup.json
06:00 Amir1: Starting s8 eqiad failover from db1109 to db1104 - T303927
05:57 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host backup1006.eqiad.wmnet with OS bullseye
05:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T306560)', diff saved to https://phabricator.wikimedia.org/P25882 and previous config saved to /var/cache/conftool/dbconfig/20220421-055655-ladsgroup.json
05:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T306560)', diff saved to https://phabricator.wikimedia.org/P25881 and previous config saved to /var/cache/conftool/dbconfig/20220421-055441-ladsgroup.json
05:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
05:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
05:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T306560)', diff saved to https://phabricator.wikimedia.org/P25880 and previous config saved to /var/cache/conftool/dbconfig/20220421-055433-ladsgroup.json
05:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P25879 and previous config saved to /var/cache/conftool/dbconfig/20220421-053928-ladsgroup.json
05:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P25878 and previous config saved to /var/cache/conftool/dbconfig/20220421-052423-ladsgroup.json
05:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P25877 and previous config saved to /var/cache/conftool/dbconfig/20220421-052146-ladsgroup.json
05:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
05:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
05:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25876 and previous config saved to /var/cache/conftool/dbconfig/20220421-051543-ladsgroup.json
05:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
05:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
05:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
05:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
05:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25875 and previous config saved to /var/cache/conftool/dbconfig/20220421-051529-ladsgroup.json
05:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1132 T301879', diff saved to https://phabricator.wikimedia.org/P25874 and previous config saved to /var/cache/conftool/dbconfig/20220421-050931-marostegui.json
05:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T306560)', diff saved to https://phabricator.wikimedia.org/P25873 and previous config saved to /var/cache/conftool/dbconfig/20220421-050918-ladsgroup.json
05:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Set db1104 with weight 0 T303927', diff saved to https://phabricator.wikimedia.org/P25872 and previous config saved to /var/cache/conftool/dbconfig/20220421-050154-ladsgroup.json
05:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 31 hosts with reason: Primary switchover s8 T303927
05:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on 31 hosts with reason: Primary switchover s8 T303927
05:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25871 and previous config saved to /var/cache/conftool/dbconfig/20220421-050024-ladsgroup.json
04:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25870 and previous config saved to /var/cache/conftool/dbconfig/20220421-044519-ladsgroup.json
04:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
04:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
04:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25869 and previous config saved to /var/cache/conftool/dbconfig/20220421-043014-ladsgroup.json
04:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T306560)', diff saved to https://phabricator.wikimedia.org/P25868 and previous config saved to /var/cache/conftool/dbconfig/20220421-042545-ladsgroup.json
04:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
04:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
04:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T306560)', diff saved to https://phabricator.wikimedia.org/P25867 and previous config saved to /var/cache/conftool/dbconfig/20220421-042537-ladsgroup.json
04:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25866 and previous config saved to /var/cache/conftool/dbconfig/20220421-042142-ladsgroup.json
04:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
04:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
04:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
04:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
04:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25865 and previous config saved to /var/cache/conftool/dbconfig/20220421-041710-ladsgroup.json
04:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P25864 and previous config saved to /var/cache/conftool/dbconfig/20220421-041032-ladsgroup.json
04:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25863 and previous config saved to /var/cache/conftool/dbconfig/20220421-040204-ladsgroup.json
03:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P25862 and previous config saved to /var/cache/conftool/dbconfig/20220421-035526-ladsgroup.json
03:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25861 and previous config saved to /var/cache/conftool/dbconfig/20220421-034659-ladsgroup.json
03:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db[2074,2094,2109,2127,2149].codfw.wmnet with reason: Maintenance
03:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db[2074,2094,2109,2127,2149].codfw.wmnet with reason: Maintenance
03:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
03:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
03:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25860 and previous config saved to /var/cache/conftool/dbconfig/20220421-034404-ladsgroup.json
03:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T306560)', diff saved to https://phabricator.wikimedia.org/P25859 and previous config saved to /var/cache/conftool/dbconfig/20220421-034021-ladsgroup.json
03:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25858 and previous config saved to /var/cache/conftool/dbconfig/20220421-033154-ladsgroup.json
03:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T306560)', diff saved to https://phabricator.wikimedia.org/P25857 and previous config saved to /var/cache/conftool/dbconfig/20220421-032906-ladsgroup.json
03:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
03:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
03:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25856 and previous config saved to /var/cache/conftool/dbconfig/20220421-032859-ladsgroup.json
03:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
03:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
03:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
03:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
03:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P25855 and previous config saved to /var/cache/conftool/dbconfig/20220421-032753-ladsgroup.json
03:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T298563)', diff saved to https://phabricator.wikimedia.org/P25854 and previous config saved to /var/cache/conftool/dbconfig/20220421-032556-ladsgroup.json
03:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
03:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
03:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
03:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
03:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25853 and previous config saved to /var/cache/conftool/dbconfig/20220421-032503-ladsgroup.json
03:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
03:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
03:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25852 and previous config saved to /var/cache/conftool/dbconfig/20220421-031354-ladsgroup.json
02:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25851 and previous config saved to /var/cache/conftool/dbconfig/20220421-025849-ladsgroup.json
02:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25850 and previous config saved to /var/cache/conftool/dbconfig/20220421-023942-ladsgroup.json
02:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25849 and previous config saved to /var/cache/conftool/dbconfig/20220421-023710-ladsgroup.json
02:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
02:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
02:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
02:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
02:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
02:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
02:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
02:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
02:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
02:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
02:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25848 and previous config saved to /var/cache/conftool/dbconfig/20220421-022631-ladsgroup.json
02:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25847 and previous config saved to /var/cache/conftool/dbconfig/20220421-021126-ladsgroup.json
02:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25846 and previous config saved to /var/cache/conftool/dbconfig/20220421-020727-ladsgroup.json
02:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
02:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
01:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25845 and previous config saved to /var/cache/conftool/dbconfig/20220421-015621-ladsgroup.json
01:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25844 and previous config saved to /var/cache/conftool/dbconfig/20220421-014116-ladsgroup.json
01:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25843 and previous config saved to /var/cache/conftool/dbconfig/20220421-013456-ladsgroup.json
01:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
01:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
01:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25842 and previous config saved to /var/cache/conftool/dbconfig/20220421-013401-ladsgroup.json
01:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
01:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
01:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25841 and previous config saved to /var/cache/conftool/dbconfig/20220421-012235-ladsgroup.json
01:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25840 and previous config saved to /var/cache/conftool/dbconfig/20220421-011856-ladsgroup.json
01:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25839 and previous config saved to /var/cache/conftool/dbconfig/20220421-010730-ladsgroup.json
01:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25838 and previous config saved to /var/cache/conftool/dbconfig/20220421-010351-ladsgroup.json
00:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25837 and previous config saved to /var/cache/conftool/dbconfig/20220421-005225-ladsgroup.json
00:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25836 and previous config saved to /var/cache/conftool/dbconfig/20220421-004846-ladsgroup.json
00:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25835 and previous config saved to /var/cache/conftool/dbconfig/20220421-003720-ladsgroup.json
00:30 mutante: alert1001 - sudo systemctl start certspotter - another time, not on our end but should probably fail more gracefully
00:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25834 and previous config saved to /var/cache/conftool/dbconfig/20220421-002107-ladsgroup.json
00:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
00:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
00:09 mutante: alert1001 - sudo systemctl start certspotter (after an alert from Icinga itself that it failed. error was some temp error fetching data from comodo)

2022-04-20

23:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25833 and previous config saved to /var/cache/conftool/dbconfig/20220420-234831-ladsgroup.json
23:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
23:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
23:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
23:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
23:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25832 and previous config saved to /var/cache/conftool/dbconfig/20220420-234818-ladsgroup.json
23:36 mutante: kubernetes/puppetmaster: added deployment/user tokens for new service image-suggestion T304891
23:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
23:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
23:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25831 and previous config saved to /var/cache/conftool/dbconfig/20220420-233313-ladsgroup.json
23:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25830 and previous config saved to /var/cache/conftool/dbconfig/20220420-231808-ladsgroup.json
23:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25829 and previous config saved to /var/cache/conftool/dbconfig/20220420-231645-ladsgroup.json
23:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25828 and previous config saved to /var/cache/conftool/dbconfig/20220420-230303-ladsgroup.json
23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25827 and previous config saved to /var/cache/conftool/dbconfig/20220420-230140-ladsgroup.json
22:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25826 and previous config saved to /var/cache/conftool/dbconfig/20220420-225643-ladsgroup.json
22:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
22:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
22:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
22:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
22:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
22:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
22:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
22:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25825 and previous config saved to /var/cache/conftool/dbconfig/20220420-224634-ladsgroup.json
22:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
22:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25824 and previous config saved to /var/cache/conftool/dbconfig/20220420-223129-ladsgroup.json
22:14 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS buster
22:13 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2006-dev.codfw.wmnet with OS buster
22:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25823 and previous config saved to /var/cache/conftool/dbconfig/20220420-220048-ladsgroup.json
21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25822 and previous config saved to /var/cache/conftool/dbconfig/20220420-215818-ladsgroup.json
21:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
21:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25821 and previous config saved to /var/cache/conftool/dbconfig/20220420-215810-ladsgroup.json
21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25820 and previous config saved to /var/cache/conftool/dbconfig/20220420-214305-ladsgroup.json
21:38 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:38 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:38 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:33 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:33 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:33 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:32 jhuneidi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Revert "Revert "Create 'uploader' group for viwiki"" (duration: 00m 53s)
21:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25819 and previous config saved to /var/cache/conftool/dbconfig/20220420-213115-ladsgroup.json
21:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
21:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
21:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25818 and previous config saved to /var/cache/conftool/dbconfig/20220420-212800-ladsgroup.json
21:27 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:27 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25817 and previous config saved to /var/cache/conftool/dbconfig/20220420-211255-ladsgroup.json
21:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:07 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
21:05 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2006-dev.codfw.wmnet with reason: host reimage
21:02 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:01 andrew@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
21:01 andrew@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2006-dev.codfw.wmnet with reason: host reimage
20:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
20:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
20:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25816 and previous config saved to /var/cache/conftool/dbconfig/20220420-205732-ladsgroup.json
20:46 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudcephmon2006-dev.codfw.wmnet with OS buster
20:46 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS buster
20:46 jhuneidi@deploy1002: Synchronized static/images/project-logos/: Config: Revert "fawiki: Change wordmark & tagline (new Vector) and logo (legacy Vector)" (duration: 00m 51s)
20:44 jhuneidi@deploy1002: Synchronized static/images/mobile/copyright/: Config: Revert "fawiki: Change logo for 900K milestone" (duration: 00m 49s)
20:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25815 and previous config saved to /var/cache/conftool/dbconfig/20220420-204227-ladsgroup.json
20:40 jhuneidi@deploy1002: Synchronized wmf-config/logos.php: Config: Revert "fawiki: Change wordmark & tagline (new Vector) and logo (legacy Vector)" (duration: 00m 50s)
20:38 jhuneidi@deploy1002: Synchronized logos/config.yaml: Config: Revert "fawiki: Change wordmark & tagline (new Vector) and logo (legacy Vector)" (duration: 00m 51s)
20:37 jhuneidi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Revert "fawiki: Change logo for 900K milestone" Revert "fawiki: Change wordmark & tagline (new Vector) and logo (legacy Vector)" (duration: 00m 57s)
20:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:36 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:36 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:36 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:36 mutante: gitlab-runner2001 - mkdir /home/gitlab-runner (was: PANIC: mkdir /home/gitlab-runner: permission denied and other issues, trying if it's just the missing directory or more) T297659
20:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25814 and previous config saved to /var/cache/conftool/dbconfig/20220420-202722-ladsgroup.json
20:26 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25813 and previous config saved to /var/cache/conftool/dbconfig/20220420-201240-ladsgroup.json
20:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
20:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
20:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25812 and previous config saved to /var/cache/conftool/dbconfig/20220420-201232-ladsgroup.json
20:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25811 and previous config saved to /var/cache/conftool/dbconfig/20220420-201217-ladsgroup.json
20:10 gmodena@deploy1002: Finished deploy [airflow-dags/research@b029f10]: (no justification provided) (duration: 00m 06s)
20:10 gmodena@deploy1002: Started deploy [airflow-dags/research@b029f10]: (no justification provided)
19:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25810 and previous config saved to /var/cache/conftool/dbconfig/20220420-195727-ladsgroup.json
19:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25809 and previous config saved to /var/cache/conftool/dbconfig/20220420-195606-ladsgroup.json
19:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
19:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
19:50 gmodena@deploy1002: Finished deploy [airflow-dags/research@b029f10]: (no justification provided) (duration: 01m 12s)
19:48 gmodena@deploy1002: Started deploy [airflow-dags/research@b029f10]: (no justification provided)
19:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25808 and previous config saved to /var/cache/conftool/dbconfig/20220420-194222-ladsgroup.json
19:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25807 and previous config saved to /var/cache/conftool/dbconfig/20220420-193859-ladsgroup.json
19:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25806 and previous config saved to /var/cache/conftool/dbconfig/20220420-192717-ladsgroup.json
19:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25805 and previous config saved to /var/cache/conftool/dbconfig/20220420-192354-ladsgroup.json
19:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25804 and previous config saved to /var/cache/conftool/dbconfig/20220420-192029-ladsgroup.json
19:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
19:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
19:19 mutante: puppetmaster - cleaning cert for gitlab-runner2001, signing new request
19:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25803 and previous config saved to /var/cache/conftool/dbconfig/20220420-191934-ladsgroup.json
19:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25802 and previous config saved to /var/cache/conftool/dbconfig/20220420-190846-ladsgroup.json
19:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25801 and previous config saved to /var/cache/conftool/dbconfig/20220420-190429-ladsgroup.json
18:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25800 and previous config saved to /var/cache/conftool/dbconfig/20220420-185341-ladsgroup.json
18:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25799 and previous config saved to /var/cache/conftool/dbconfig/20220420-184925-ladsgroup.json
18:39 mutante: reimaging gitlab-runner2021.codfw.wmnet
18:36 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gitlab-runner2001.codfw.wmnet with reason: reimage
18:36 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on gitlab-runner2001.codfw.wmnet with reason: reimage
18:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25798 and previous config saved to /var/cache/conftool/dbconfig/20220420-183419-ladsgroup.json
18:17 kormat@cumin1001: dbctl commit (dc=all): 'es1025 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25797 and previous config saved to /var/cache/conftool/dbconfig/20220420-181720-kormat.json
18:15 kormat@cumin1001: dbctl commit (dc=all): 'es1028 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25796 and previous config saved to /var/cache/conftool/dbconfig/20220420-181515-kormat.json
18:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:05 jhuneidi@deploy1002: Synchronized php: group1 wikis to 1.39.0-wmf.8 refs T305214 (duration: 00m 51s)
18:04 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.8 refs T305214
18:02 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1024.mgmt.eqiad.wmnet with reboot policy FORCED
18:02 kormat@cumin1001: dbctl commit (dc=all): 'es1025 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25795 and previous config saved to /var/cache/conftool/dbconfig/20220420-180215-kormat.json
18:00 kormat@cumin1001: dbctl commit (dc=all): 'es1028 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25794 and previous config saved to /var/cache/conftool/dbconfig/20220420-180012-kormat.json
17:53 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1023.mgmt.eqiad.wmnet with reboot policy FORCED
17:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25793 and previous config saved to /var/cache/conftool/dbconfig/20220420-175327-ladsgroup.json
17:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
17:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
17:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25792 and previous config saved to /var/cache/conftool/dbconfig/20220420-175319-ladsgroup.json
17:50 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host parse1019.mgmt.eqiad.wmnet with reboot policy FORCED
17:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1024.mgmt.eqiad.wmnet with reboot policy FORCED
17:47 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1018.mgmt.eqiad.wmnet with reboot policy FORCED
17:47 kormat@cumin1001: dbctl commit (dc=all): 'es1025 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25791 and previous config saved to /var/cache/conftool/dbconfig/20220420-174711-kormat.json
17:46 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1017.mgmt.eqiad.wmnet with reboot policy FORCED
17:46 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1014.mgmt.eqiad.wmnet with reboot policy FORCED
17:46 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1021.mgmt.eqiad.wmnet with reboot policy FORCED
17:46 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1019.mgmt.eqiad.wmnet with reboot policy FORCED
17:46 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1013.mgmt.eqiad.wmnet with reboot policy FORCED
17:46 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1022.mgmt.eqiad.wmnet with reboot policy FORCED
17:46 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1016.mgmt.eqiad.wmnet with reboot policy FORCED
17:45 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1015.mgmt.eqiad.wmnet with reboot policy FORCED
17:45 kormat@cumin1001: dbctl commit (dc=all): 'es1028 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25790 and previous config saved to /var/cache/conftool/dbconfig/20220420-174508-kormat.json
17:40 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1023.mgmt.eqiad.wmnet with reboot policy FORCED
17:40 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1012.mgmt.eqiad.wmnet with reboot policy FORCED
17:39 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host parse1020.mgmt.eqiad.wmnet with reboot policy FORCED
17:39 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host parse1015.mgmt.eqiad.wmnet with reboot policy FORCED
17:39 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host parse1019.mgmt.eqiad.wmnet with reboot policy FORCED
17:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25789 and previous config saved to /var/cache/conftool/dbconfig/20220420-173814-ladsgroup.json
17:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25788 and previous config saved to /var/cache/conftool/dbconfig/20220420-173405-ladsgroup.json
17:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
17:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
17:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
17:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
17:34 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1019.mgmt.eqiad.wmnet with reboot policy FORCED
17:34 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1018.mgmt.eqiad.wmnet with reboot policy FORCED
17:34 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1016.mgmt.eqiad.wmnet with reboot policy FORCED
17:34 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1021.mgmt.eqiad.wmnet with reboot policy FORCED
17:34 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1022.mgmt.eqiad.wmnet with reboot policy FORCED
17:34 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1020.mgmt.eqiad.wmnet with reboot policy FORCED
17:33 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1014.mgmt.eqiad.wmnet with reboot policy FORCED
17:33 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1015.mgmt.eqiad.wmnet with reboot policy FORCED
17:33 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1017.mgmt.eqiad.wmnet with reboot policy FORCED
17:33 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1013.mgmt.eqiad.wmnet with reboot policy FORCED
17:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25787 and previous config saved to /var/cache/conftool/dbconfig/20220420-173304-ladsgroup.json
17:32 kormat@cumin1001: dbctl commit (dc=all): 'es1025 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25786 and previous config saved to /var/cache/conftool/dbconfig/20220420-173207-kormat.json
17:31 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host parse1003.mgmt.eqiad.wmnet with reboot policy FORCED
17:31 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gitlab-runner2001.codfw.wmnet with reason: reimage
17:31 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on gitlab-runner2001.codfw.wmnet with reason: reimage
17:31 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1007.mgmt.eqiad.wmnet with reboot policy FORCED
17:31 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1004.mgmt.eqiad.wmnet with reboot policy FORCED
17:31 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1010.mgmt.eqiad.wmnet with reboot policy FORCED
17:31 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1006.mgmt.eqiad.wmnet with reboot policy FORCED
17:31 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1002.mgmt.eqiad.wmnet with reboot policy FORCED
17:30 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1009.mgmt.eqiad.wmnet with reboot policy FORCED
17:30 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1011.mgmt.eqiad.wmnet with reboot policy FORCED
17:30 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1005.mgmt.eqiad.wmnet with reboot policy FORCED
17:30 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1008.mgmt.eqiad.wmnet with reboot policy FORCED
17:30 kormat@cumin1001: dbctl commit (dc=all): 'es1028 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25785 and previous config saved to /var/cache/conftool/dbconfig/20220420-173004-kormat.json
17:27 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1012.mgmt.eqiad.wmnet with reboot policy FORCED
17:26 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
17:26 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
17:26 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1028.eqiad.wmnet with reason: Rebooting for T303174
17:26 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1028.eqiad.wmnet with reason: Rebooting for T303174
17:26 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1001.mgmt.eqiad.wmnet with reboot policy FORCED
17:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25784 and previous config saved to /var/cache/conftool/dbconfig/20220420-172309-ladsgroup.json
17:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25783 and previous config saved to /var/cache/conftool/dbconfig/20220420-171759-ladsgroup.json
17:16 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1007.mgmt.eqiad.wmnet with reboot policy FORCED
17:16 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1008.mgmt.eqiad.wmnet with reboot policy FORCED
17:16 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1011.mgmt.eqiad.wmnet with reboot policy FORCED
17:16 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1010.mgmt.eqiad.wmnet with reboot policy FORCED
17:16 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1009.mgmt.eqiad.wmnet with reboot policy FORCED
17:16 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1006.mgmt.eqiad.wmnet with reboot policy FORCED
17:16 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1002.mgmt.eqiad.wmnet with reboot policy FORCED
17:16 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1004.mgmt.eqiad.wmnet with reboot policy FORCED
17:16 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1003.mgmt.eqiad.wmnet with reboot policy FORCED
17:16 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1005.mgmt.eqiad.wmnet with reboot policy FORCED
17:12 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1001.mgmt.eqiad.wmnet with reboot policy FORCED
17:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25782 and previous config saved to /var/cache/conftool/dbconfig/20220420-170804-ladsgroup.json
17:04 kormat@cumin1001: dbctl commit (dc=all): 'db1158 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25781 and previous config saved to /var/cache/conftool/dbconfig/20220420-170426-kormat.json
17:03 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/datahub: sync on main
17:03 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/datahub: apply on main
17:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25780 and previous config saved to /var/cache/conftool/dbconfig/20220420-170254-ladsgroup.json
17:02 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
17:02 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
17:02 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
17:02 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
17:01 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
17:01 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
16:59 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
16:59 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
16:59 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
16:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
16:51 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
16:51 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
16:51 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1028.eqiad.wmnet with reason: Rebooting for T303174
16:50 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1028.eqiad.wmnet with reason: Rebooting for T303174
16:50 jynus@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host backup1007.eqiad.wmnet with OS bullseye
16:49 kormat@cumin1001: dbctl commit (dc=all): 'db1158 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25779 and previous config saved to /var/cache/conftool/dbconfig/20220420-164922-kormat.json
16:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25778 and previous config saved to /var/cache/conftool/dbconfig/20220420-164749-ladsgroup.json
16:43 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-worker1143.mgmt.eqiad.wmnet with reboot policy FORCED
16:43 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-worker1144.mgmt.eqiad.wmnet with reboot policy FORCED
16:42 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host an-worker1144.mgmt.eqiad.wmnet with reboot policy FORCED
16:42 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host an-worker1143.mgmt.eqiad.wmnet with reboot policy FORCED
16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25777 and previous config saved to /var/cache/conftool/dbconfig/20220420-163537-ladsgroup.json
16:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
16:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
16:34 kormat@cumin1001: dbctl commit (dc=all): 'db1158 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25776 and previous config saved to /var/cache/conftool/dbconfig/20220420-163418-kormat.json
16:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
16:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
16:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
16:33 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
16:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
16:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
16:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
16:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
16:19 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
16:19 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
16:19 kormat@cumin1001: dbctl commit (dc=all): 'db1158 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25775 and previous config saved to /var/cache/conftool/dbconfig/20220420-161914-kormat.json
16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25774 and previous config saved to /var/cache/conftool/dbconfig/20220420-161828-ladsgroup.json
16:15 kormat@cumin1001: dbctl commit (dc=all): 'es1028 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25773 and previous config saved to /var/cache/conftool/dbconfig/20220420-161511-kormat.json
16:15 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1028.eqiad.wmnet with reason: Rebooting for T303174
16:15 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1028.eqiad.wmnet with reason: Rebooting for T303174
16:14 kormat@cumin1001: dbctl commit (dc=all): 'Change es3 'master' to es1031 T303174', diff saved to https://phabricator.wikimedia.org/P25772 and previous config saved to /var/cache/conftool/dbconfig/20220420-161453-kormat.json
16:13 kormat@cumin1001: dbctl commit (dc=all): 'db1158 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25771 and previous config saved to /var/cache/conftool/dbconfig/20220420-161353-kormat.json
16:13 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1158.eqiad.wmnet with reason: Rebooting for T303174
16:13 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1158.eqiad.wmnet with reason: Rebooting for T303174
16:13 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Rebooting db1158 T303174
16:13 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Rebooting db1158 T303174
16:13 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1007.eqiad.wmnet with reason: host reimage
16:12 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
16:12 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
16:11 kormat@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25770 and previous config saved to /var/cache/conftool/dbconfig/20220420-161123-kormat.json
16:09 jynus@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on backup1007.eqiad.wmnet with reason: host reimage
16:09 kormat@cumin1001: dbctl commit (dc=all): 'es1033 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25769 and previous config saved to /var/cache/conftool/dbconfig/20220420-160926-kormat.json
16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25768 and previous config saved to /var/cache/conftool/dbconfig/20220420-160322-ladsgroup.json
15:57 hnowlan@deploy1002: Finished deploy [restbase/deploy@0205f1d]: Bump mediawiki-title to 0.7.5 (duration: 15m 35s)
15:56 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host backup1007.eqiad.wmnet with OS bullseye
15:56 kormat@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25767 and previous config saved to /var/cache/conftool/dbconfig/20220420-155619-kormat.json
15:55 jynus@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup1007.eqiad.wmnet with OS bullseye
15:54 kormat@cumin1001: dbctl commit (dc=all): 'es1033 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25766 and previous config saved to /var/cache/conftool/dbconfig/20220420-155422-kormat.json
15:53 kormat@cumin1001: dbctl commit (dc=all): 'db1174 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25765 and previous config saved to /var/cache/conftool/dbconfig/20220420-155318-kormat.json
15:50 kormat@cumin1001: dbctl commit (dc=all): 'es1034 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25764 and previous config saved to /var/cache/conftool/dbconfig/20220420-155051-kormat.json
15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25763 and previous config saved to /var/cache/conftool/dbconfig/20220420-154817-ladsgroup.json
15:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25762 and previous config saved to /var/cache/conftool/dbconfig/20220420-154734-ladsgroup.json
15:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
15:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
15:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
15:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
15:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25761 and previous config saved to /var/cache/conftool/dbconfig/20220420-154635-ladsgroup.json
15:44 kormat@cumin1001: dbctl commit (dc=all): 'db1178 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25760 and previous config saved to /var/cache/conftool/dbconfig/20220420-154427-kormat.json
15:41 hnowlan@deploy1002: Started deploy [restbase/deploy@0205f1d]: Bump mediawiki-title to 0.7.5
15:41 kormat@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25759 and previous config saved to /var/cache/conftool/dbconfig/20220420-154115-kormat.json
15:39 kormat@cumin1001: dbctl commit (dc=all): 'es1033 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25758 and previous config saved to /var/cache/conftool/dbconfig/20220420-153918-kormat.json
15:38 kormat@cumin1001: dbctl commit (dc=all): 'db1174 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25757 and previous config saved to /var/cache/conftool/dbconfig/20220420-153814-kormat.json
15:35 kormat@cumin1001: dbctl commit (dc=all): 'es1034 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25756 and previous config saved to /var/cache/conftool/dbconfig/20220420-153547-kormat.json
15:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25755 and previous config saved to /var/cache/conftool/dbconfig/20220420-153312-ladsgroup.json
15:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25754 and previous config saved to /var/cache/conftool/dbconfig/20220420-153130-ladsgroup.json
15:29 kormat@cumin1001: dbctl commit (dc=all): 'db1178 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25753 and previous config saved to /var/cache/conftool/dbconfig/20220420-152923-kormat.json
15:26 kormat@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25752 and previous config saved to /var/cache/conftool/dbconfig/20220420-152611-kormat.json
15:24 kormat@cumin1001: dbctl commit (dc=all): 'es1033 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25751 and previous config saved to /var/cache/conftool/dbconfig/20220420-152414-kormat.json
15:23 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host backup1007.eqiad.wmnet with OS bullseye
15:23 kormat@cumin1001: dbctl commit (dc=all): 'db1174 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25750 and previous config saved to /var/cache/conftool/dbconfig/20220420-152310-kormat.json
15:20 kormat@cumin1001: dbctl commit (dc=all): 'es1025 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25749 and previous config saved to /var/cache/conftool/dbconfig/20220420-152044-kormat.json
15:20 kormat@cumin1001: dbctl commit (dc=all): 'es1034 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25748 and previous config saved to /var/cache/conftool/dbconfig/20220420-152043-kormat.json
15:20 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
15:20 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
15:20 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1022.eqiad.wmnet with reason: Rebooting for T303174
15:20 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1022.eqiad.wmnet with reason: Rebooting for T303174
15:20 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1033.eqiad.wmnet with reason: Rebooting for T303174
15:20 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1033.eqiad.wmnet with reason: Rebooting for T303174
15:18 moritzm: installing wireshark security updates
15:16 hnowlan@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad
15:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25747 and previous config saved to /var/cache/conftool/dbconfig/20220420-151625-ladsgroup.json
15:15 kormat@cumin1001: dbctl commit (dc=all): 'db1149 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25746 and previous config saved to /var/cache/conftool/dbconfig/20220420-151509-kormat.json
15:14 kormat@cumin1001: dbctl commit (dc=all): 'db1178 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25745 and previous config saved to /var/cache/conftool/dbconfig/20220420-151419-kormat.json
15:08 kormat@cumin1001: dbctl commit (dc=all): 'db1174 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25744 and previous config saved to /var/cache/conftool/dbconfig/20220420-150806-kormat.json
15:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host serpens.wikimedia.org
15:05 kormat@cumin1001: dbctl commit (dc=all): 'es1034 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25743 and previous config saved to /var/cache/conftool/dbconfig/20220420-150539-kormat.json
15:05 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
15:04 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
15:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host serpens.wikimedia.org
15:02 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1022.eqiad.wmnet with reason: Rebooting for T303174
15:02 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1022.eqiad.wmnet with reason: Rebooting for T303174
15:02 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1034.eqiad.wmnet with reason: Rebooting for T303174
15:02 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1034.eqiad.wmnet with reason: Rebooting for T303174
15:01 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1033.eqiad.wmnet with reason: Rebooting for T303174
15:01 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1033.eqiad.wmnet with reason: Rebooting for T303174
15:01 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1174.eqiad.wmnet with reason: Rebooting for T303174
15:01 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1174.eqiad.wmnet with reason: Rebooting for T303174
15:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25742 and previous config saved to /var/cache/conftool/dbconfig/20220420-150119-ladsgroup.json
15:00 taavi@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: updating wmf-puppet-dashboard for keystone authentication support T274666 (eqiad1) (duration: 05m 03s)
15:00 kormat@cumin1001: dbctl commit (dc=all): 'db1149 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25741 and previous config saved to /var/cache/conftool/dbconfig/20220420-150005-kormat.json
14:59 kormat@cumin1001: dbctl commit (dc=all): 'db1178 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25740 and previous config saved to /var/cache/conftool/dbconfig/20220420-145915-kormat.json
14:55 taavi@deploy1002: Started deploy [horizon/deploy@9d02cd6]: updating wmf-puppet-dashboard for keystone authentication support T274666 (eqiad1)
14:54 kormat@cumin1001: dbctl commit (dc=all): 'db1178 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25739 and previous config saved to /var/cache/conftool/dbconfig/20220420-145454-kormat.json
14:54 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1178.eqiad.wmnet with reason: Rebooting for T303174
14:54 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1178.eqiad.wmnet with reason: Rebooting for T303174
14:53 taavi@deploy1002: Finished deploy [horizon/deploy@9d02cd6] (dev): updating wmf-puppet-dashboard for keystone authentication support (codfw1dev) (duration: 02m 03s)
14:52 kormat@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25738 and previous config saved to /var/cache/conftool/dbconfig/20220420-145223-kormat.json
14:51 taavi@deploy1002: Started deploy [horizon/deploy@9d02cd6] (dev): updating wmf-puppet-dashboard for keystone authentication support (codfw1dev)
14:51 hnowlan@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad
14:50 kormat@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25737 and previous config saved to /var/cache/conftool/dbconfig/20220420-145057-kormat.json
14:47 kormat@cumin1001: dbctl commit (dc=all): 'es1025 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25735 and previous config saved to /var/cache/conftool/dbconfig/20220420-144730-kormat.json
14:47 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
14:47 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for T303174
14:46 kormat@cumin1001: dbctl commit (dc=all): 'es1022 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25734 and previous config saved to /var/cache/conftool/dbconfig/20220420-144615-kormat.json
14:46 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1022.eqiad.wmnet with reason: Rebooting for T303174
14:46 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1022.eqiad.wmnet with reason: Rebooting for T303174
14:46 kormat@cumin1001: dbctl commit (dc=all): 'es1034 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25733 and previous config saved to /var/cache/conftool/dbconfig/20220420-144557-kormat.json
14:46 taavi@deploy1002: Finished deploy [horizon/deploy@9d02cd6] (dev): updating wmf-puppet-dashboard for keystone authentication support (codwf1dev) (duration: 01m 59s)
14:45 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1034.eqiad.wmnet with reason: Rebooting for T303174
14:45 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1034.eqiad.wmnet with reason: Rebooting for T303174
14:45 kormat@cumin1001: dbctl commit (dc=all): 'es1033 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25732 and previous config saved to /var/cache/conftool/dbconfig/20220420-144511-kormat.json
14:45 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1033.eqiad.wmnet with reason: Rebooting for T303174
14:45 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1033.eqiad.wmnet with reason: Rebooting for T303174
14:45 kormat@cumin1001: dbctl commit (dc=all): 'db1149 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25731 and previous config saved to /var/cache/conftool/dbconfig/20220420-144501-kormat.json
14:44 kormat@cumin1001: dbctl commit (dc=all): 'db1174 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25730 and previous config saved to /var/cache/conftool/dbconfig/20220420-144443-kormat.json
14:44 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1174.eqiad.wmnet with reason: Rebooting for T303174
14:44 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1174.eqiad.wmnet with reason: Rebooting for T303174
14:43 taavi@deploy1002: Started deploy [horizon/deploy@9d02cd6] (dev): updating wmf-puppet-dashboard for keystone authentication support (codwf1dev)
14:43 kormat@cumin1001: dbctl commit (dc=all): 'es1024 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25729 and previous config saved to /var/cache/conftool/dbconfig/20220420-144352-kormat.json
14:42 kormat@cumin1001: dbctl commit (dc=all): 'es1021 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25728 and previous config saved to /var/cache/conftool/dbconfig/20220420-144252-kormat.json
14:42 kormat@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25727 and previous config saved to /var/cache/conftool/dbconfig/20220420-144200-kormat.json
14:42 kormat@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25726 and previous config saved to /var/cache/conftool/dbconfig/20220420-144159-kormat.json
14:41 kormat@cumin1001: dbctl commit (dc=all): 'db1127 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25725 and previous config saved to /var/cache/conftool/dbconfig/20220420-144134-kormat.json
14:38 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync
14:37 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: sync
14:37 kormat@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25724 and previous config saved to /var/cache/conftool/dbconfig/20220420-143719-kormat.json
14:35 kormat@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25723 and previous config saved to /var/cache/conftool/dbconfig/20220420-143554-kormat.json
14:33 taavi@deploy1002: Finished deploy [horizon/deploy@9d02cd6] (dev): updating wmf-puppet-dashboard for keystone authentication support (codwf1dev) (duration: 02m 08s)
14:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25722 and previous config saved to /var/cache/conftool/dbconfig/20220420-143258-ladsgroup.json
14:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
14:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
14:30 taavi@deploy1002: Started deploy [horizon/deploy@9d02cd6] (dev): updating wmf-puppet-dashboard for keystone authentication support (codwf1dev)
14:29 kormat@cumin1001: dbctl commit (dc=all): 'db1149 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25721 and previous config saved to /var/cache/conftool/dbconfig/20220420-142957-kormat.json
14:28 kormat@cumin1001: dbctl commit (dc=all): 'es1024 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25720 and previous config saved to /var/cache/conftool/dbconfig/20220420-142848-kormat.json
14:27 kormat@cumin1001: dbctl commit (dc=all): 'es1021 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25719 and previous config saved to /var/cache/conftool/dbconfig/20220420-142748-kormat.json
14:27 kormat@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25718 and previous config saved to /var/cache/conftool/dbconfig/20220420-142656-kormat.json
14:26 kormat@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25717 and previous config saved to /var/cache/conftool/dbconfig/20220420-142656-kormat.json
14:26 kormat@cumin1001: dbctl commit (dc=all): 'db1127 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25716 and previous config saved to /var/cache/conftool/dbconfig/20220420-142630-kormat.json
14:25 kormat@cumin1001: dbctl commit (dc=all): 'db1149 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25715 and previous config saved to /var/cache/conftool/dbconfig/20220420-142526-kormat.json
14:25 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1149.eqiad.wmnet with reason: Rebooting for T303174
14:25 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1149.eqiad.wmnet with reason: Rebooting for T303174
14:23 moritzm: installing webperf1004 T305460
14:23 kormat@cumin1001: dbctl commit (dc=all): 'db1148 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25714 and previous config saved to /var/cache/conftool/dbconfig/20220420-142310-kormat.json
14:22 kormat@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25713 and previous config saved to /var/cache/conftool/dbconfig/20220420-142215-kormat.json
14:20 kormat@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25712 and previous config saved to /var/cache/conftool/dbconfig/20220420-142050-kormat.json
14:13 kormat@cumin1001: dbctl commit (dc=all): 'es1024 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25711 and previous config saved to /var/cache/conftool/dbconfig/20220420-141345-kormat.json
14:12 kormat@cumin1001: dbctl commit (dc=all): 'es1021 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25710 and previous config saved to /var/cache/conftool/dbconfig/20220420-141244-kormat.json
14:12 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2090.codfw.wmnet with reason: Rebooting for T303174
14:12 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2090.codfw.wmnet with reason: Rebooting for T303174
14:11 kormat@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25709 and previous config saved to /var/cache/conftool/dbconfig/20220420-141152-kormat.json
14:11 kormat@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25708 and previous config saved to /var/cache/conftool/dbconfig/20220420-141152-kormat.json
14:11 kormat@cumin1001: dbctl commit (dc=all): 'db1127 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25707 and previous config saved to /var/cache/conftool/dbconfig/20220420-141127-kormat.json
14:08 kormat@cumin1001: dbctl commit (dc=all): 'db1148 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25706 and previous config saved to /var/cache/conftool/dbconfig/20220420-140806-kormat.json
14:07 kormat@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25705 and previous config saved to /var/cache/conftool/dbconfig/20220420-140711-kormat.json
14:05 kormat@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25704 and previous config saved to /var/cache/conftool/dbconfig/20220420-140546-kormat.json
14:01 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
14:01 kormat@cumin1001: dbctl commit (dc=all): 'db1180 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25703 and previous config saved to /var/cache/conftool/dbconfig/20220420-140123-kormat.json
14:01 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1180.eqiad.wmnet with reason: Rebooting for T303174
14:01 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1180.eqiad.wmnet with reason: Rebooting for T303174
14:01 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
14:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25702 and previous config saved to /var/cache/conftool/dbconfig/20220420-140105-ladsgroup.json
14:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
14:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
14:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
14:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
14:00 kormat@cumin1001: dbctl commit (dc=all): 'db1177 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25701 and previous config saved to /var/cache/conftool/dbconfig/20220420-140029-kormat.json
14:00 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1177.eqiad.wmnet with reason: Rebooting for T303174
14:00 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1177.eqiad.wmnet with reason: Rebooting for T303174
13:59 kormat@cumin1001: dbctl commit (dc=all): 'db1168 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25700 and previous config saved to /var/cache/conftool/dbconfig/20220420-135956-kormat.json
13:58 kormat@cumin1001: dbctl commit (dc=all): 'es1024 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25699 and previous config saved to /var/cache/conftool/dbconfig/20220420-135841-kormat.json
13:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T306269)', diff saved to https://phabricator.wikimedia.org/P25698 and previous config saved to /var/cache/conftool/dbconfig/20220420-135750-marostegui.json
13:57 kormat@cumin1001: dbctl commit (dc=all): 'es1021 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25697 and previous config saved to /var/cache/conftool/dbconfig/20220420-135740-kormat.json
13:56 kormat@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25696 and previous config saved to /var/cache/conftool/dbconfig/20220420-135648-kormat.json
13:56 kormat@cumin1001: dbctl commit (dc=all): 'db1127 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25695 and previous config saved to /var/cache/conftool/dbconfig/20220420-135623-kormat.json
13:54 kormat@cumin1001: dbctl commit (dc=all): 'es1032 (re)pooling @ 100%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25694 and previous config saved to /var/cache/conftool/dbconfig/20220420-135417-kormat.json
13:54 kormat@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25693 and previous config saved to /var/cache/conftool/dbconfig/20220420-135417-kormat.json
13:53 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1024.eqiad.wmnet with reason: Rebooting for T303174
13:53 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1024.eqiad.wmnet with reason: Rebooting for T303174
13:53 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
13:53 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
13:53 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1030.eqiad.wmnet with reason: Rebooting for T303174
13:53 kormat@cumin1001: dbctl commit (dc=all): 'db1148 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25692 and previous config saved to /var/cache/conftool/dbconfig/20220420-135302-kormat.json
13:53 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1030.eqiad.wmnet with reason: Rebooting for T303174
13:53 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T303174
13:53 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T303174
13:52 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1021.eqiad.wmnet with reason: Rebooting for T303174
13:52 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1021.eqiad.wmnet with reason: Rebooting for T303174
13:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:51 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:48 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: filebackend: Fix link to thumb url in testcommonswiki (T306139) (duration: 00m 53s)
13:44 kormat@cumin1001: dbctl commit (dc=all): 'db1168 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25691 and previous config saved to /var/cache/conftool/dbconfig/20220420-134452-kormat.json
13:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P25690 and previous config saved to /var/cache/conftool/dbconfig/20220420-134238-marostegui.json
13:39 kormat@cumin1001: dbctl commit (dc=all): 'es1032 (re)pooling @ 75%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25689 and previous config saved to /var/cache/conftool/dbconfig/20220420-133914-kormat.json
13:39 kormat@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25688 and previous config saved to /var/cache/conftool/dbconfig/20220420-133913-kormat.json
13:38 kormat@cumin1001: dbctl commit (dc=all): 'db1148 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25687 and previous config saved to /var/cache/conftool/dbconfig/20220420-133757-kormat.json
13:36 kormat@cumin1001: dbctl commit (dc=all): 'es1024 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25686 and previous config saved to /var/cache/conftool/dbconfig/20220420-133622-kormat.json
13:36 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1024.eqiad.wmnet with reason: Rebooting for T303174
13:36 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1024.eqiad.wmnet with reason: Rebooting for T303174
13:35 kormat@cumin1001: dbctl commit (dc=all): 'es1021 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25685 and previous config saved to /var/cache/conftool/dbconfig/20220420-133546-kormat.json
13:35 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1021.eqiad.wmnet with reason: Rebooting for T303174
13:35 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1021.eqiad.wmnet with reason: Rebooting for T303174
13:33 kormat@cumin1001: dbctl commit (dc=all): 'db1148 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25684 and previous config saved to /var/cache/conftool/dbconfig/20220420-133317-kormat.json
13:33 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1148.eqiad.wmnet with reason: Rebooting for T303174
13:33 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1148.eqiad.wmnet with reason: Rebooting for T303174
13:30 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1030.eqiad.wmnet with reason: Rebooting for T303174
13:30 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1030.eqiad.wmnet with reason: Rebooting for T303174
13:30 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T303174
13:30 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T303174
13:30 kormat@cumin1001: dbctl commit (dc=all): 'db1147 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25683 and previous config saved to /var/cache/conftool/dbconfig/20220420-133000-kormat.json
13:29 kormat@cumin1001: dbctl commit (dc=all): 'db1168 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25682 and previous config saved to /var/cache/conftool/dbconfig/20220420-132948-kormat.json
13:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P25681 and previous config saved to /var/cache/conftool/dbconfig/20220420-132733-marostegui.json
13:24 kormat@cumin1001: dbctl commit (dc=all): 'es1032 (re)pooling @ 50%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25680 and previous config saved to /var/cache/conftool/dbconfig/20220420-132410-kormat.json
13:24 kormat@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25679 and previous config saved to /var/cache/conftool/dbconfig/20220420-132409-kormat.json
13:23 kormat@cumin1001: dbctl commit (dc=all): 'es1031 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25678 and previous config saved to /var/cache/conftool/dbconfig/20220420-132325-kormat.json
13:23 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
13:23 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
13:14 kormat@cumin1001: dbctl commit (dc=all): 'db1147 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25677 and previous config saved to /var/cache/conftool/dbconfig/20220420-131456-kormat.json
13:14 kormat@cumin1001: dbctl commit (dc=all): 'db1168 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25676 and previous config saved to /var/cache/conftool/dbconfig/20220420-131444-kormat.json
13:14 vgutierrez: restarting pybal on lvs1017
13:12 kormat@cumin1001: dbctl commit (dc=all): 'es1030 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25675 and previous config saved to /var/cache/conftool/dbconfig/20220420-131251-kormat.json
13:12 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1030.eqiad.wmnet with reason: Rebooting for T303174
13:12 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1030.eqiad.wmnet with reason: Rebooting for T303174
13:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T306269)', diff saved to https://phabricator.wikimedia.org/P25674 and previous config saved to /var/cache/conftool/dbconfig/20220420-131228-marostegui.json
13:12 kormat@cumin1001: dbctl commit (dc=all): 'Change es2 'master' to es1026 T303174', diff saved to https://phabricator.wikimedia.org/P25673 and previous config saved to /var/cache/conftool/dbconfig/20220420-131222-kormat.json
13:11 vgutierrez: restarting pybal on lvs1018
13:10 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1168.eqiad.wmnet with reason: Rebooting for T303174
13:10 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1168.eqiad.wmnet with reason: Rebooting for T303174
13:10 elukey: restart etcdmirror on conf2005
13:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1135 (T306269)', diff saved to https://phabricator.wikimedia.org/P25672 and previous config saved to /var/cache/conftool/dbconfig/20220420-130914-marostegui.json
13:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
13:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
13:09 kormat@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25671 and previous config saved to /var/cache/conftool/dbconfig/20220420-130905-kormat.json
13:09 kormat@cumin1001: dbctl commit (dc=all): 'db1127 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25670 and previous config saved to /var/cache/conftool/dbconfig/20220420-130859-kormat.json
13:06 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T303174
13:06 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T303174
12:59 kormat@cumin1001: dbctl commit (dc=all): 'db1147 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25669 and previous config saved to /var/cache/conftool/dbconfig/20220420-125952-kormat.json
12:59 kormat@cumin1001: dbctl commit (dc=all): 'db1168 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25668 and previous config saved to /var/cache/conftool/dbconfig/20220420-125909-kormat.json
12:59 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1168.eqiad.wmnet with reason: Rebooting for T303174
12:59 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1168.eqiad.wmnet with reason: Rebooting for T303174
12:58 akosiaris: reboot conf2006, conf1006
12:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P25667 and previous config saved to /var/cache/conftool/dbconfig/20220420-125312-marostegui.json
12:49 kormat@cumin1001: dbctl commit (dc=all): 'es1032 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25666 and previous config saved to /var/cache/conftool/dbconfig/20220420-124926-kormat.json
12:49 kormat@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25665 and previous config saved to /var/cache/conftool/dbconfig/20220420-124920-kormat.json
12:45 kormat@cumin1001: dbctl commit (dc=all): 'es1032 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25664 and previous config saved to /var/cache/conftool/dbconfig/20220420-124537-kormat.json
12:45 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1032.eqiad.wmnet with reason: Rebooting for T303174
12:45 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1032.eqiad.wmnet with reason: Rebooting for T303174
12:45 kormat@cumin1001: dbctl commit (dc=all): 'db1172 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25663 and previous config saved to /var/cache/conftool/dbconfig/20220420-124502-kormat.json
12:44 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1172.eqiad.wmnet with reason: Rebooting for T303174
12:44 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1172.eqiad.wmnet with reason: Rebooting for T303174
12:44 kormat@cumin1001: dbctl commit (dc=all): 'db1147 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25662 and previous config saved to /var/cache/conftool/dbconfig/20220420-124448-kormat.json
12:40 moritzm: installing webperf1003 T305460
12:40 kormat@cumin1001: dbctl commit (dc=all): 'db1147 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25661 and previous config saved to /var/cache/conftool/dbconfig/20220420-124004-kormat.json
12:40 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1147.eqiad.wmnet with reason: Rebooting for T303174
12:39 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1147.eqiad.wmnet with reason: Rebooting for T303174
12:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P25660 and previous config saved to /var/cache/conftool/dbconfig/20220420-123807-marostegui.json
12:36 akosiaris: reboot conf2004, conf1004
12:33 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
12:33 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
12:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster1005.eqiad.wmnet
12:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T306269)', diff saved to https://phabricator.wikimedia.org/P25659 and previous config saved to /var/cache/conftool/dbconfig/20220420-122000-marostegui.json
12:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1134 (T306269)', diff saved to https://phabricator.wikimedia.org/P25658 and previous config saved to /var/cache/conftool/dbconfig/20220420-121745-marostegui.json
12:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
12:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
12:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T306269)', diff saved to https://phabricator.wikimedia.org/P25657 and previous config saved to /var/cache/conftool/dbconfig/20220420-121737-marostegui.json
12:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster1005.eqiad.wmnet
12:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe1002.eqiad.wmnet
12:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host moss-fe1002.eqiad.wmnet
12:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe1001.eqiad.wmnet
12:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P25656 and previous config saved to /var/cache/conftool/dbconfig/20220420-120232-marostegui.json
11:59 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host moss-fe1001.eqiad.wmnet
11:57 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe2002.codfw.wmnet
11:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host moss-fe2002.codfw.wmnet
11:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P25655 and previous config saved to /var/cache/conftool/dbconfig/20220420-114727-marostegui.json
11:43 kormat@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 100%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25654 and previous config saved to /var/cache/conftool/dbconfig/20220420-114326-kormat.json
11:42 kormat@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25653 and previous config saved to /var/cache/conftool/dbconfig/20220420-114159-kormat.json
11:35 kormat@cumin1001: dbctl commit (dc=all): 'es1032 (re)pooling @ 100%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25652 and previous config saved to /var/cache/conftool/dbconfig/20220420-113547-kormat.json
11:35 kormat@cumin1001: dbctl commit (dc=all): 'db1127 (re)pooling @ 100%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25651 and previous config saved to /var/cache/conftool/dbconfig/20220420-113503-kormat.json
11:34 kormat@cumin1001: dbctl commit (dc=all): 'db1165 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25650 and previous config saved to /var/cache/conftool/dbconfig/20220420-113432-kormat.json
11:34 hnowlan@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad
11:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe2001.codfw.wmnet
11:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T306269)', diff saved to https://phabricator.wikimedia.org/P25649 and previous config saved to /var/cache/conftool/dbconfig/20220420-113219-marostegui.json
11:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1119 (T306269)', diff saved to https://phabricator.wikimedia.org/P25648 and previous config saved to /var/cache/conftool/dbconfig/20220420-113000-marostegui.json
11:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
11:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
11:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T306269)', diff saved to https://phabricator.wikimedia.org/P25647 and previous config saved to /var/cache/conftool/dbconfig/20220420-112952-marostegui.json
11:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host moss-fe2001.codfw.wmnet
11:28 kormat@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 75%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25646 and previous config saved to /var/cache/conftool/dbconfig/20220420-112823-kormat.json
11:26 kormat@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25645 and previous config saved to /var/cache/conftool/dbconfig/20220420-112655-kormat.json
11:26 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 6 hosts with reason: postgres config change
11:26 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime for 0:15:00 on 6 hosts with reason: postgres config change
11:25 hnowlan@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad
11:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2002.codfw.wmnet
11:20 kormat@cumin1001: dbctl commit (dc=all): 'es1032 (re)pooling @ 75%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25644 and previous config saved to /var/cache/conftool/dbconfig/20220420-112043-kormat.json
11:20 kormat@cumin1001: dbctl commit (dc=all): 'db1127 (re)pooling @ 75%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25643 and previous config saved to /var/cache/conftool/dbconfig/20220420-111959-kormat.json
11:19 kormat@cumin1001: dbctl commit (dc=all): 'db1165 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25642 and previous config saved to /var/cache/conftool/dbconfig/20220420-111928-kormat.json
11:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host moss-be2002.codfw.wmnet
11:19 kormat@cumin1001: dbctl commit (dc=all): 'db1167 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25641 and previous config saved to /var/cache/conftool/dbconfig/20220420-111911-kormat.json
11:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2001.codfw.wmnet
11:16 kormat@cumin1001: dbctl commit (dc=all): 'db1156 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25640 and previous config saved to /var/cache/conftool/dbconfig/20220420-111626-kormat.json
11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P25639 and previous config saved to /var/cache/conftool/dbconfig/20220420-111447-marostegui.json
11:13 kormat@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 50%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25638 and previous config saved to /var/cache/conftool/dbconfig/20220420-111319-kormat.json
11:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host moss-be2001.codfw.wmnet
11:11 kormat@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25637 and previous config saved to /var/cache/conftool/dbconfig/20220420-111150-kormat.json
11:05 kormat@cumin1001: dbctl commit (dc=all): 'es1032 (re)pooling @ 50%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25636 and previous config saved to /var/cache/conftool/dbconfig/20220420-110539-kormat.json
11:04 kormat@cumin1001: dbctl commit (dc=all): 'db1127 (re)pooling @ 50%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25635 and previous config saved to /var/cache/conftool/dbconfig/20220420-110455-kormat.json
11:04 kormat@cumin1001: dbctl commit (dc=all): 'db1165 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25634 and previous config saved to /var/cache/conftool/dbconfig/20220420-110424-kormat.json
11:04 kormat@cumin1001: dbctl commit (dc=all): 'db1167 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25633 and previous config saved to /var/cache/conftool/dbconfig/20220420-110408-kormat.json
11:01 kormat@cumin1001: dbctl commit (dc=all): 'db1156 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25632 and previous config saved to /var/cache/conftool/dbconfig/20220420-110122-kormat.json
10:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P25631 and previous config saved to /var/cache/conftool/dbconfig/20220420-105942-marostegui.json
10:56 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
10:56 kormat@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25630 and previous config saved to /var/cache/conftool/dbconfig/20220420-105646-kormat.json
10:52 kormat@cumin1001: dbctl commit (dc=all): 'db1143 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25629 and previous config saved to /var/cache/conftool/dbconfig/20220420-105204-kormat.json
10:52 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1143.eqiad.wmnet with reason: Rebooting for T303174
10:51 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1143.eqiad.wmnet with reason: Rebooting for T303174
10:51 kormat@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 25%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25628 and previous config saved to /var/cache/conftool/dbconfig/20220420-105112-kormat.json
10:50 kormat@cumin1001: dbctl commit (dc=all): 'es1032 (re)pooling @ 25%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25627 and previous config saved to /var/cache/conftool/dbconfig/20220420-105035-kormat.json
10:49 kormat@cumin1001: dbctl commit (dc=all): 'db1127 (re)pooling @ 25%: repooling T303174', diff saved to https://phabricator.wikimedia.org/P25626 and previous config saved to /var/cache/conftool/dbconfig/20220420-104951-kormat.json
10:49 kormat@cumin1001: dbctl commit (dc=all): 'db1165 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25625 and previous config saved to /var/cache/conftool/dbconfig/20220420-104920-kormat.json
10:49 kormat@cumin1001: dbctl commit (dc=all): 'db1167 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25624 and previous config saved to /var/cache/conftool/dbconfig/20220420-104904-kormat.json
10:48 kormat@cumin1001: dbctl commit (dc=all): 'es1026 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25623 and previous config saved to /var/cache/conftool/dbconfig/20220420-104802-kormat.json
10:46 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
10:46 kormat@cumin1001: dbctl commit (dc=all): 'db1156 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25622 and previous config saved to /var/cache/conftool/dbconfig/20220420-104618-kormat.json
10:44 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T306269)', diff saved to https://phabricator.wikimedia.org/P25621 and previous config saved to /var/cache/conftool/dbconfig/20220420-104437-marostegui.json
10:43 kormat@cumin1001: dbctl commit (dc=all): 'db1165 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25620 and previous config saved to /var/cache/conftool/dbconfig/20220420-104310-kormat.json
10:43 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1165.eqiad.wmnet with reason: Rebooting for T303174
10:43 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1165.eqiad.wmnet with reason: Rebooting for T303174
10:42 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Rebooting db1165 T303174
10:42 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Rebooting db1165 T303174
10:42 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 (T306269)', diff saved to https://phabricator.wikimedia.org/P25619 and previous config saved to /var/cache/conftool/dbconfig/20220420-104214-marostegui.json
10:42 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
10:42 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
10:41 kormat@cumin1001: dbctl commit (dc=all): 'db1142 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25618 and previous config saved to /var/cache/conftool/dbconfig/20220420-104150-kormat.json
10:41 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
10:41 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
10:41 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1032.eqiad.wmnet with reason: Rebooting for T303174
10:41 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1032.eqiad.wmnet with reason: Rebooting for T303174
10:39 marostegui@cumin1001: dbctl commit (dc=all): 'db1184 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P25617 and previous config saved to /var/cache/conftool/dbconfig/20220420-103939-root.json
10:39 kormat@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25616 and previous config saved to /var/cache/conftool/dbconfig/20220420-103913-kormat.json
10:35 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
10:35 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
10:34 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
10:34 kormat@cumin1001: dbctl commit (dc=all): 'es1032 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25615 and previous config saved to /var/cache/conftool/dbconfig/20220420-103440-kormat.json
10:34 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1032.eqiad.wmnet with reason: Rebooting for T303174
10:34 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1032.eqiad.wmnet with reason: Rebooting for T303174
10:34 kormat@cumin1001: dbctl commit (dc=all): 'db1167 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25614 and previous config saved to /var/cache/conftool/dbconfig/20220420-103400-kormat.json
10:33 kormat@cumin1001: dbctl commit (dc=all): 'es1029 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25613 and previous config saved to /var/cache/conftool/dbconfig/20220420-103338-kormat.json
10:32 kormat@cumin1001: dbctl commit (dc=all): 'es1026 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25612 and previous config saved to /var/cache/conftool/dbconfig/20220420-103258-kormat.json
10:31 kormat@cumin1001: dbctl commit (dc=all): 'db1156 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25611 and previous config saved to /var/cache/conftool/dbconfig/20220420-103114-kormat.json
10:29 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
10:29 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
10:28 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1167.eqiad.wmnet with reason: Rebooting for T303174
10:28 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1167.eqiad.wmnet with reason: Rebooting for T303174
10:27 kormat@cumin1001: dbctl commit (dc=all): 'db1167 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25610 and previous config saved to /var/cache/conftool/dbconfig/20220420-102722-kormat.json
10:27 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1167.eqiad.wmnet with reason: Rebooting for T303174
10:27 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1167.eqiad.wmnet with reason: Rebooting for T303174
10:27 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Rebooting db1167 T303174
10:26 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Rebooting db1167 T303174
10:26 kormat@cumin1001: dbctl commit (dc=all): 'db1142 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25609 and previous config saved to /var/cache/conftool/dbconfig/20220420-102646-kormat.json
10:25 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1156.eqiad.wmnet with reason: Rebooting for T303174
10:25 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1156.eqiad.wmnet with reason: Rebooting for T303174
10:24 marostegui@cumin1001: dbctl commit (dc=all): 'db1184 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P25608 and previous config saved to /var/cache/conftool/dbconfig/20220420-102435-root.json
10:24 kormat@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25607 and previous config saved to /var/cache/conftool/dbconfig/20220420-102409-kormat.json
10:23 kormat@cumin1001: dbctl commit (dc=all): 'db1126 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25606 and previous config saved to /var/cache/conftool/dbconfig/20220420-102327-kormat.json
10:22 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
10:22 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
10:18 kormat@cumin1001: dbctl commit (dc=all): 'es1029 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25605 and previous config saved to /var/cache/conftool/dbconfig/20220420-101834-kormat.json
10:17 kormat@cumin1001: dbctl commit (dc=all): 'es1026 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25604 and previous config saved to /var/cache/conftool/dbconfig/20220420-101755-kormat.json
10:16 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host labweb1001.wikimedia.org
10:15 kormat@cumin1001: dbctl commit (dc=all): 'es1031 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25603 and previous config saved to /var/cache/conftool/dbconfig/20220420-101549-kormat.json
10:15 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
10:15 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for T303174
10:11 kormat@cumin1001: dbctl commit (dc=all): 'db1142 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25602 and previous config saved to /var/cache/conftool/dbconfig/20220420-101142-kormat.json
10:09 marostegui@cumin1001: dbctl commit (dc=all): 'db1184 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P25601 and previous config saved to /var/cache/conftool/dbconfig/20220420-100931-root.json
10:09 kormat@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25600 and previous config saved to /var/cache/conftool/dbconfig/20220420-100905-kormat.json
10:08 kormat@cumin1001: dbctl commit (dc=all): 'db1126 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25599 and previous config saved to /var/cache/conftool/dbconfig/20220420-100823-kormat.json
10:06 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
10:06 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
10:05 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
10:05 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
10:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm[2001-2003].codfw.wmnet with reason: reboot
10:05 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm[2001-2003].codfw.wmnet with reason: reboot
10:04 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T303174
10:04 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T303174
10:04 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1156.eqiad.wmnet with reason: Rebooting for T303174
10:04 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1156.eqiad.wmnet with reason: Rebooting for T303174
10:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host labweb1001.wikimedia.org
10:03 kormat@cumin1001: dbctl commit (dc=all): 'es1029 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25598 and previous config saved to /var/cache/conftool/dbconfig/20220420-100331-kormat.json
10:02 kormat@cumin1001: dbctl commit (dc=all): 'es1026 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25597 and previous config saved to /var/cache/conftool/dbconfig/20220420-100251-kormat.json
10:02 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host labweb1002.wikimedia.org
09:59 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1026.eqiad.wmnet with reason: Rebooting for T303174
09:59 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1026.eqiad.wmnet with reason: Rebooting for T303174
09:58 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T303174
09:58 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T303174
09:56 kormat@cumin1001: dbctl commit (dc=all): 'db1142 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25596 and previous config saved to /var/cache/conftool/dbconfig/20220420-095638-kormat.json
09:54 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1156.eqiad.wmnet with reason: Rebooting for T303174
09:54 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1156.eqiad.wmnet with reason: Rebooting for T303174
09:54 marostegui@cumin1001: dbctl commit (dc=all): 'db1184 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P25595 and previous config saved to /var/cache/conftool/dbconfig/20220420-095427-root.json
09:54 kormat@cumin1001: dbctl commit (dc=all): 'db1131 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25594 and previous config saved to /var/cache/conftool/dbconfig/20220420-095401-kormat.json
09:53 kormat@cumin1001: dbctl commit (dc=all): 'db1126 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25593 and previous config saved to /var/cache/conftool/dbconfig/20220420-095319-kormat.json
09:52 kormat@cumin1001: dbctl commit (dc=all): 'db1127 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25592 and previous config saved to /var/cache/conftool/dbconfig/20220420-095235-kormat.json
09:52 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T303174
09:52 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for T303174
09:52 kormat@cumin1001: dbctl commit (dc=all): 'db1142 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25591 and previous config saved to /var/cache/conftool/dbconfig/20220420-095209-kormat.json
09:52 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1142.eqiad.wmnet with reason: Rebooting for T303174
09:52 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1142.eqiad.wmnet with reason: Rebooting for T303174
09:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host labweb1002.wikimedia.org
09:50 kormat@cumin1001: dbctl commit (dc=all): 'db1131 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25590 and previous config saved to /var/cache/conftool/dbconfig/20220420-094958-kormat.json
09:49 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1131.eqiad.wmnet with reason: Rebooting for T303174
09:49 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1131.eqiad.wmnet with reason: Rebooting for T303174
09:48 kormat@cumin1001: dbctl commit (dc=all): 'db1156 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25589 and previous config saved to /var/cache/conftool/dbconfig/20220420-094857-kormat.json
09:48 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1156.eqiad.wmnet with reason: Rebooting for T303174
09:48 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1156.eqiad.wmnet with reason: Rebooting for T303174
09:48 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Rebooting db1156 T303174
09:48 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Rebooting db1156 T303174
09:48 kormat@cumin1001: dbctl commit (dc=all): 'es1029 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25588 and previous config saved to /var/cache/conftool/dbconfig/20220420-094827-kormat.json
09:45 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cloudweb2001-dev.wikimedia.org
09:44 kormat@cumin1001: dbctl commit (dc=all): 'es1029 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25587 and previous config saved to /var/cache/conftool/dbconfig/20220420-094435-kormat.json
09:44 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1029.eqiad.wmnet with reason: Rebooting for T303174
09:44 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1029.eqiad.wmnet with reason: Rebooting for T303174
09:43 kormat@cumin1001: dbctl commit (dc=all): 'Switch es1 'primary' T303174', diff saved to https://phabricator.wikimedia.org/P25586 and previous config saved to /var/cache/conftool/dbconfig/20220420-094354-kormat.json
09:38 kormat@cumin1001: dbctl commit (dc=all): 'db1126 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25585 and previous config saved to /var/cache/conftool/dbconfig/20220420-093815-kormat.json
09:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudweb2001-dev.wikimedia.org
09:27 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
09:26 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
09:23 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
09:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6001.wikimedia.org
09:19 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
09:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast6001.wikimedia.org
09:17 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
09:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster1004.eqiad.wmnet
09:16 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
09:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster1004.eqiad.wmnet
09:09 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host netbox-dev2001.wikimedia.org
09:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
09:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
09:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netbox-dev2001.wikimedia.org
09:00 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1029.eqiad.wmnet with reason: Rebooting for T303174
09:00 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1029.eqiad.wmnet with reason: Rebooting for T303174
09:00 kormat@cumin1001: dbctl commit (dc=all): 'db1126 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25583 and previous config saved to /var/cache/conftool/dbconfig/20220420-090010-kormat.json
09:00 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1126.eqiad.wmnet with reason: Rebooting for T303174
09:00 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1126.eqiad.wmnet with reason: Rebooting for T303174
08:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2001.codfw.wmnet
08:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb2001.codfw.wmnet
08:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25582 and previous config saved to /var/cache/conftool/dbconfig/20220420-085325-ladsgroup.json
08:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
08:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
08:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25581 and previous config saved to /var/cache/conftool/dbconfig/20220420-085231-ladsgroup.json
08:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox2001.wikimedia.org
08:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netbox2001.wikimedia.org
08:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1001.eqiad.wmnet
08:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T306269)', diff saved to https://phabricator.wikimedia.org/P25580 and previous config saved to /var/cache/conftool/dbconfig/20220420-084625-marostegui.json
08:43 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1184 (T306269)', diff saved to https://phabricator.wikimedia.org/P25579 and previous config saved to /var/cache/conftool/dbconfig/20220420-084312-marostegui.json
08:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
08:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
08:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb1001.eqiad.wmnet
08:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 (T306269)', diff saved to https://phabricator.wikimedia.org/P25578 and previous config saved to /var/cache/conftool/dbconfig/20220420-084303-marostegui.json
08:39 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host netbox1001.wikimedia.org
08:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25577 and previous config saved to /var/cache/conftool/dbconfig/20220420-083726-ladsgroup.json
08:31 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netbox1001.wikimedia.org
08:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P25576 and previous config saved to /var/cache/conftool/dbconfig/20220420-082758-marostegui.json
08:22 mmandere@cumin1001: END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for pybal-test2003.codfw.wmnet: Renew puppet certificate - mmandere@cumin1001
08:22 mmandere@cumin1001: START - Cookbook sre.puppet.renew-cert for pybal-test2003.codfw.wmnet: Renew puppet certificate - mmandere@cumin1001
08:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25575 and previous config saved to /var/cache/conftool/dbconfig/20220420-082221-ladsgroup.json
08:21 ariel@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1005.eqiad.wmnet
08:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
08:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
08:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25574 and previous config saved to /var/cache/conftool/dbconfig/20220420-082016-ladsgroup.json
08:15 ariel@cumin1001: START - Cookbook sre.hosts.reboot-single for host dumpsdata1005.eqiad.wmnet
08:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P25573 and previous config saved to /var/cache/conftool/dbconfig/20220420-081253-marostegui.json
08:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25572 and previous config saved to /var/cache/conftool/dbconfig/20220420-080716-ladsgroup.json
08:06 ariel@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1004.eqiad.wmnet
08:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25571 and previous config saved to /var/cache/conftool/dbconfig/20220420-080511-ladsgroup.json
08:01 mmandere: reimage pybal-test2003 as buster - T297187
08:01 ariel@cumin1001: START - Cookbook sre.hosts.reboot-single for host dumpsdata1004.eqiad.wmnet
07:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 (T306269)', diff saved to https://phabricator.wikimedia.org/P25570 and previous config saved to /var/cache/conftool/dbconfig/20220420-075747-marostegui.json
07:55 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1164 (T306269)', diff saved to https://phabricator.wikimedia.org/P25569 and previous config saved to /var/cache/conftool/dbconfig/20220420-075535-marostegui.json
07:55 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1164.eqiad.wmnet with reason: Maintenance
07:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1164.eqiad.wmnet with reason: Maintenance
07:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T306269)', diff saved to https://phabricator.wikimedia.org/P25568 and previous config saved to /var/cache/conftool/dbconfig/20220420-075527-marostegui.json
07:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25567 and previous config saved to /var/cache/conftool/dbconfig/20220420-075006-ladsgroup.json
07:49 dcausse: T305689: reset crosscluster settings of the elastic chi cluster in eqiad
07:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P25566 and previous config saved to /var/cache/conftool/dbconfig/20220420-074022-marostegui.json
07:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25565 and previous config saved to /var/cache/conftool/dbconfig/20220420-073501-ladsgroup.json
07:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P25564 and previous config saved to /var/cache/conftool/dbconfig/20220420-072516-marostegui.json
07:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25563 and previous config saved to /var/cache/conftool/dbconfig/20220420-071747-ladsgroup.json
07:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
07:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
07:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:10 kartik@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Enable SectionTranslation in Test WP for ckb, el, eu, and zh-yue (T304854 T304862 T304865 T304866) (duration: 01m 53s)
07:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T306269)', diff saved to https://phabricator.wikimedia.org/P25562 and previous config saved to /var/cache/conftool/dbconfig/20220420-071011-marostegui.json
07:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T306269)', diff saved to https://phabricator.wikimedia.org/P25561 and previous config saved to /var/cache/conftool/dbconfig/20220420-070906-marostegui.json
07:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
07:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
07:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1106.eqiad.wmnet with reason: Maintenance
07:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1106.eqiad.wmnet with reason: Maintenance
07:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance
07:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance
07:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance
07:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance
07:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T306269)', diff saved to https://phabricator.wikimedia.org/P25560 and previous config saved to /var/cache/conftool/dbconfig/20220420-070721-marostegui.json
07:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25559 and previous config saved to /var/cache/conftool/dbconfig/20220420-070702-ladsgroup.json
07:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
07:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
07:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
07:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
07:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25558 and previous config saved to /var/cache/conftool/dbconfig/20220420-070648-ladsgroup.json
07:05 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
07:02 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
07:00 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
06:59 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
06:57 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
06:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P25557 and previous config saved to /var/cache/conftool/dbconfig/20220420-065216-marostegui.json
06:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25556 and previous config saved to /var/cache/conftool/dbconfig/20220420-065143-ladsgroup.json
06:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P25555 and previous config saved to /var/cache/conftool/dbconfig/20220420-063711-marostegui.json
06:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25554 and previous config saved to /var/cache/conftool/dbconfig/20220420-063638-ladsgroup.json
06:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
06:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
06:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
06:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
06:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T306269)', diff saved to https://phabricator.wikimedia.org/P25553 and previous config saved to /var/cache/conftool/dbconfig/20220420-062206-marostegui.json
06:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25552 and previous config saved to /var/cache/conftool/dbconfig/20220420-062133-ladsgroup.json
06:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 (T306269)', diff saved to https://phabricator.wikimedia.org/P25551 and previous config saved to /var/cache/conftool/dbconfig/20220420-061848-marostegui.json
06:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
06:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
06:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
06:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
06:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
06:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
06:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T306269)', diff saved to https://phabricator.wikimedia.org/P25550 and previous config saved to /var/cache/conftool/dbconfig/20220420-061834-marostegui.json
06:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25549 and previous config saved to /var/cache/conftool/dbconfig/20220420-061433-ladsgroup.json
06:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
06:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
06:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25548 and previous config saved to /var/cache/conftool/dbconfig/20220420-061425-ladsgroup.json
06:07 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 100%: After reimage', diff saved to https://phabricator.wikimedia.org/P25547 and previous config saved to /var/cache/conftool/dbconfig/20220420-060732-root.json
06:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P25546 and previous config saved to /var/cache/conftool/dbconfig/20220420-060329-marostegui.json
05:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25545 and previous config saved to /var/cache/conftool/dbconfig/20220420-055920-ladsgroup.json
05:52 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 75%: After reimage', diff saved to https://phabricator.wikimedia.org/P25544 and previous config saved to /var/cache/conftool/dbconfig/20220420-055228-root.json
05:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P25543 and previous config saved to /var/cache/conftool/dbconfig/20220420-054824-marostegui.json
05:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25542 and previous config saved to /var/cache/conftool/dbconfig/20220420-054415-ladsgroup.json
05:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
05:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
05:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P25541 and previous config saved to /var/cache/conftool/dbconfig/20220420-053932-ladsgroup.json
05:39 ayounsi@cumin2002: END (PASS) - Cookbook sre.network.cf (exit_code=0)
05:39 ayounsi@cumin2002: START - Cookbook sre.network.cf
05:38 XioNoX: start CF in monitoring mode for drmrs
05:37 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 50%: After reimage', diff saved to https://phabricator.wikimedia.org/P25540 and previous config saved to /var/cache/conftool/dbconfig/20220420-053724-root.json
05:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T306269)', diff saved to https://phabricator.wikimedia.org/P25539 and previous config saved to /var/cache/conftool/dbconfig/20220420-053319-marostegui.json
05:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1163 (T306269)', diff saved to https://phabricator.wikimedia.org/P25538 and previous config saved to /var/cache/conftool/dbconfig/20220420-053006-marostegui.json
05:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance
05:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance
05:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T306269)', diff saved to https://phabricator.wikimedia.org/P25537 and previous config saved to /var/cache/conftool/dbconfig/20220420-052958-marostegui.json
05:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25536 and previous config saved to /var/cache/conftool/dbconfig/20220420-052910-ladsgroup.json
05:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P25535 and previous config saved to /var/cache/conftool/dbconfig/20220420-052427-ladsgroup.json
05:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25534 and previous config saved to /var/cache/conftool/dbconfig/20220420-052223-ladsgroup.json
05:22 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 25%: After reimage', diff saved to https://phabricator.wikimedia.org/P25533 and previous config saved to /var/cache/conftool/dbconfig/20220420-052220-root.json
05:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
05:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
05:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25532 and previous config saved to /var/cache/conftool/dbconfig/20220420-052215-ladsgroup.json
05:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P25531 and previous config saved to /var/cache/conftool/dbconfig/20220420-051453-marostegui.json
05:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P25530 and previous config saved to /var/cache/conftool/dbconfig/20220420-050921-ladsgroup.json
05:07 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 10%: After reimage', diff saved to https://phabricator.wikimedia.org/P25529 and previous config saved to /var/cache/conftool/dbconfig/20220420-050716-root.json
05:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25528 and previous config saved to /var/cache/conftool/dbconfig/20220420-050710-ladsgroup.json
04:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P25527 and previous config saved to /var/cache/conftool/dbconfig/20220420-045948-marostegui.json
04:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P25526 and previous config saved to /var/cache/conftool/dbconfig/20220420-045416-ladsgroup.json
04:52 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 5%: After reimage', diff saved to https://phabricator.wikimedia.org/P25525 and previous config saved to /var/cache/conftool/dbconfig/20220420-045212-root.json
04:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25524 and previous config saved to /var/cache/conftool/dbconfig/20220420-045205-ladsgroup.json
04:51 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1132 into s1 T301879', diff saved to https://phabricator.wikimedia.org/P25523 and previous config saved to /var/cache/conftool/dbconfig/20220420-045108-marostegui.json
04:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T306269)', diff saved to https://phabricator.wikimedia.org/P25522 and previous config saved to /var/cache/conftool/dbconfig/20220420-044443-marostegui.json
04:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1169 (T306269)', diff saved to https://phabricator.wikimedia.org/P25521 and previous config saved to /var/cache/conftool/dbconfig/20220420-044132-marostegui.json
04:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance
04:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance
04:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
04:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
04:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P25520 and previous config saved to /var/cache/conftool/dbconfig/20220420-043711-ladsgroup.json
04:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
04:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
04:37 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 1%: After reimage', diff saved to https://phabricator.wikimedia.org/P25519 and previous config saved to /var/cache/conftool/dbconfig/20220420-043702-root.json
04:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25518 and previous config saved to /var/cache/conftool/dbconfig/20220420-043700-ladsgroup.json
04:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25517 and previous config saved to /var/cache/conftool/dbconfig/20220420-043005-ladsgroup.json
04:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
04:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
04:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25516 and previous config saved to /var/cache/conftool/dbconfig/20220420-042152-ladsgroup.json
04:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25515 and previous config saved to /var/cache/conftool/dbconfig/20220420-040647-ladsgroup.json
03:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P25514 and previous config saved to /var/cache/conftool/dbconfig/20220420-035142-ladsgroup.json
03:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25513 and previous config saved to /var/cache/conftool/dbconfig/20220420-034443-ladsgroup.json
03:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25512 and previous config saved to /var/cache/conftool/dbconfig/20220420-034211-ladsgroup.json
03:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
03:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
03:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
03:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
03:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
03:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
03:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
03:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
03:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
03:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
03:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25511 and previous config saved to /var/cache/conftool/dbconfig/20220420-033126-ladsgroup.json
03:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P25510 and previous config saved to /var/cache/conftool/dbconfig/20220420-032157-ladsgroup.json
03:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
03:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
03:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
03:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
03:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25509 and previous config saved to /var/cache/conftool/dbconfig/20220420-031621-ladsgroup.json
03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25508 and previous config saved to /var/cache/conftool/dbconfig/20220420-030454-ladsgroup.json
03:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25507 and previous config saved to /var/cache/conftool/dbconfig/20220420-030116-ladsgroup.json
02:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25506 and previous config saved to /var/cache/conftool/dbconfig/20220420-024949-ladsgroup.json
02:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25505 and previous config saved to /var/cache/conftool/dbconfig/20220420-024611-ladsgroup.json
02:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25504 and previous config saved to /var/cache/conftool/dbconfig/20220420-023951-ladsgroup.json
02:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
02:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
02:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25503 and previous config saved to /var/cache/conftool/dbconfig/20220420-023857-ladsgroup.json
02:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25502 and previous config saved to /var/cache/conftool/dbconfig/20220420-023444-ladsgroup.json
02:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25501 and previous config saved to /var/cache/conftool/dbconfig/20220420-022352-ladsgroup.json
02:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25500 and previous config saved to /var/cache/conftool/dbconfig/20220420-021939-ladsgroup.json
02:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25499 and previous config saved to /var/cache/conftool/dbconfig/20220420-020846-ladsgroup.json
01:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25498 and previous config saved to /var/cache/conftool/dbconfig/20220420-015341-ladsgroup.json
01:31 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
01:28 pt1979@cumin2002: START - Cookbook sre.dns.netbox
01:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25497 and previous config saved to /var/cache/conftool/dbconfig/20220420-011925-ladsgroup.json
01:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
01:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
01:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P25496 and previous config saved to /var/cache/conftool/dbconfig/20220420-011917-ladsgroup.json
01:16 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudweb2002-dev.wikimedia.org with OS buster
01:05 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudweb2002-dev.wikimedia.org with reason: host reimage
01:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25495 and previous config saved to /var/cache/conftool/dbconfig/20220420-010412-ladsgroup.json
01:01 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudweb2002-dev.wikimedia.org with reason: host reimage
00:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25494 and previous config saved to /var/cache/conftool/dbconfig/20220420-005327-ladsgroup.json
00:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
00:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
00:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
00:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
00:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25493 and previous config saved to /var/cache/conftool/dbconfig/20220420-005314-ladsgroup.json
00:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25492 and previous config saved to /var/cache/conftool/dbconfig/20220420-004907-ladsgroup.json
00:46 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudweb2002-dev.wikimedia.org with OS buster
00:44 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudweb2002-dev.wikimedia.org with OS buster
00:44 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudweb2002-dev.wikimedia.org with OS buster
00:39 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudservices2005-dev.wikimedia.org with OS bullseye
00:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25491 and previous config saved to /var/cache/conftool/dbconfig/20220420-003809-ladsgroup.json
00:35 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudweb2002-dev.wikimedia.org with OS buster
00:35 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudweb2002-dev.wikimedia.org with OS buster
00:34 pt1979@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudweb2002-dev.wikimedia.org with OS bullseye
00:34 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudweb2002-dev.wikimedia.org with OS bullseye
00:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P25490 and previous config saved to /var/cache/conftool/dbconfig/20220420-003401-ladsgroup.json
00:28 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudservices2005-dev.wikimedia.org with reason: host reimage
00:25 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudservices2005-dev.wikimedia.org with reason: host reimage
00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25489 and previous config saved to /var/cache/conftool/dbconfig/20220420-002303-ladsgroup.json
00:10 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet2006-dev.codfw.wmnet with OS bullseye
00:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25488 and previous config saved to /var/cache/conftool/dbconfig/20220420-000758-ladsgroup.json
00:06 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudservices2005-dev.wikimedia.org with OS bullseye
00:04 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudservices2004-dev.wikimedia.org with OS bullseye
00:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25487 and previous config saved to /var/cache/conftool/dbconfig/20220420-000141-ladsgroup.json
00:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
00:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
00:00 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet2006-dev.codfw.wmnet with reason: host reimage

2022-04-19

23:56 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2006-dev.codfw.wmnet with reason: host reimage
23:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
23:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
23:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
23:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
23:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
23:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
23:54 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudservices2004-dev.wikimedia.org with reason: host reimage
23:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
23:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
23:49 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudservices2004-dev.wikimedia.org with reason: host reimage
23:34 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudnet2006-dev.codfw.wmnet with OS bullseye
23:34 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet2005-dev.codfw.wmnet with OS bullseye
23:30 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudservices2004-dev.wikimedia.org with OS bullseye
23:28 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2006-dev.codfw.wmnet with OS bullseye
23:24 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet2005-dev.codfw.wmnet with reason: host reimage
23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P25486 and previous config saved to /var/cache/conftool/dbconfig/20220419-232250-ladsgroup.json
23:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
23:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
23:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
23:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25485 and previous config saved to /var/cache/conftool/dbconfig/20220419-232237-ladsgroup.json
23:20 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2005-dev.codfw.wmnet with reason: host reimage
23:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2006-dev.codfw.wmnet with reason: host reimage
23:15 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2006-dev.codfw.wmnet with reason: host reimage
23:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25484 and previous config saved to /var/cache/conftool/dbconfig/20220419-230732-ladsgroup.json
23:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25483 and previous config saved to /var/cache/conftool/dbconfig/20220419-230459-ladsgroup.json
23:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25482 and previous config saved to /var/cache/conftool/dbconfig/20220419-230226-ladsgroup.json
23:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
23:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
23:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25481 and previous config saved to /var/cache/conftool/dbconfig/20220419-230218-ladsgroup.json
22:56 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudnet2005-dev.codfw.wmnet with OS bullseye
22:53 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcephmon2006-dev.codfw.wmnet with OS bullseye
22:53 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
22:53 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
22:53 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
22:53 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
22:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25480 and previous config saved to /var/cache/conftool/dbconfig/20220419-225227-ladsgroup.json
22:50 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
22:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25479 and previous config saved to /var/cache/conftool/dbconfig/20220419-224711-ladsgroup.json
22:42 bking@cumin1001: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_eqiad: Upgrading Elasticsearch to 6.8 in EQIAD - bking@cumin1001 - T301959
22:40 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
22:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25478 and previous config saved to /var/cache/conftool/dbconfig/20220419-223722-ladsgroup.json
22:36 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
22:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25477 and previous config saved to /var/cache/conftool/dbconfig/20220419-223356-ladsgroup.json
22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25476 and previous config saved to /var/cache/conftool/dbconfig/20220419-223206-ladsgroup.json
22:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
22:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
22:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
22:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
22:18 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.8 refs T305214
22:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25475 and previous config saved to /var/cache/conftool/dbconfig/20220419-221851-ladsgroup.json
22:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25474 and previous config saved to /var/cache/conftool/dbconfig/20220419-221701-ladsgroup.json
22:14 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
22:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25473 and previous config saved to /var/cache/conftool/dbconfig/20220419-221038-ladsgroup.json
22:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
22:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
22:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25472 and previous config saved to /var/cache/conftool/dbconfig/20220419-221030-ladsgroup.json
22:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25471 and previous config saved to /var/cache/conftool/dbconfig/20220419-220346-ladsgroup.json
21:58 ebernhardson: set indices.recovery.max_bytes_per_sec=240mb in elasticsearch-eqiad-psi
21:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25470 and previous config saved to /var/cache/conftool/dbconfig/20220419-215525-ladsgroup.json
21:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25469 and previous config saved to /var/cache/conftool/dbconfig/20220419-214841-ladsgroup.json
21:42 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:42 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:42 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:42 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:41 jhuneidi@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/LdapAuthentication/includes/LdapAuthenticationHooks.php: Backport: Hooks: return false rather than strings on failure (T305786) (duration: 01m 30s)
21:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25468 and previous config saved to /var/cache/conftool/dbconfig/20220419-214019-ladsgroup.json
21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25467 and previous config saved to /var/cache/conftool/dbconfig/20220419-213707-ladsgroup.json
21:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
21:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P25466 and previous config saved to /var/cache/conftool/dbconfig/20220419-213658-ladsgroup.json
21:25 ebernhardson: set index.unassigned.node_left.delayed_timeout to 10m for all indices in elasticsearch psi (:9200) cluster
21:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25465 and previous config saved to /var/cache/conftool/dbconfig/20220419-212514-ladsgroup.json
21:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P25464 and previous config saved to /var/cache/conftool/dbconfig/20220419-212153-ladsgroup.json
21:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25463 and previous config saved to /var/cache/conftool/dbconfig/20220419-211824-ladsgroup.json
21:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
21:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
21:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25462 and previous config saved to /var/cache/conftool/dbconfig/20220419-211817-ladsgroup.json
21:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P25460 and previous config saved to /var/cache/conftool/dbconfig/20220419-210648-ladsgroup.json
21:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25459 and previous config saved to /var/cache/conftool/dbconfig/20220419-210311-ladsgroup.json
20:52 urbanecm: UTC late B&C window done
20:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P25458 and previous config saved to /var/cache/conftool/dbconfig/20220419-205143-ladsgroup.json
20:49 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/GrowthExperiments/: e152df0: Revert "Skip welcome surveys for users in the no-homepage control group" (T305015) (duration: 00m 55s)
20:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25457 and previous config saved to /var/cache/conftool/dbconfig/20220419-204826-ladsgroup.json
20:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
20:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
20:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25456 and previous config saved to /var/cache/conftool/dbconfig/20220419-204818-ladsgroup.json
20:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25455 and previous config saved to /var/cache/conftool/dbconfig/20220419-204806-ladsgroup.json
20:46 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:46 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:46 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P25454 and previous config saved to /var/cache/conftool/dbconfig/20220419-203416-ladsgroup.json
20:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
20:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
20:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P25453 and previous config saved to /var/cache/conftool/dbconfig/20220419-203313-ladsgroup.json
20:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25452 and previous config saved to /var/cache/conftool/dbconfig/20220419-203301-ladsgroup.json
20:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:31 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:31 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:31 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:27 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/includes/page/UndeletePage.php: f1ebd29: DeletePage, UndeletePage: use plaintextParams when creating log message (T306431; 2/2) (duration: 00m 50s)
20:26 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/includes/page/DeletePage.php: f1ebd29: DeletePage, UndeletePage: use plaintextParams when creating log message (T306431; 1/2) (duration: 00m 50s)
20:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25451 and previous config saved to /var/cache/conftool/dbconfig/20220419-202618-ladsgroup.json
20:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
20:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
20:26 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:25 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25450 and previous config saved to /var/cache/conftool/dbconfig/20220419-202523-ladsgroup.json
20:24 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 0a87771: Add extendedconfirmed on elwiki (T306241) (duration: 00m 50s)
20:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P25449 and previous config saved to /var/cache/conftool/dbconfig/20220419-201808-ladsgroup.json
20:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:15 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:15 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:10 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: f55f817: Add video marketing campaign to $wgGECampaignPattern (T303785) (duration: 00m 54s)
20:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25448 and previous config saved to /var/cache/conftool/dbconfig/20220419-201018-ladsgroup.json
20:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25447 and previous config saved to /var/cache/conftool/dbconfig/20220419-200303-ladsgroup.json
19:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25446 and previous config saved to /var/cache/conftool/dbconfig/20220419-195513-ladsgroup.json
19:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25445 and previous config saved to /var/cache/conftool/dbconfig/20220419-195050-ladsgroup.json
19:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
19:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
19:50 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudweb2002-dev.mgmt.codfw.wmnet with reboot policy FORCED
19:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
19:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
19:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
19:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
19:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25444 and previous config saved to /var/cache/conftool/dbconfig/20220419-194008-ladsgroup.json
19:40 urbanecm: [urbanecm@mwmaint1002 ~]$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/T304461.php --delete # T304461
19:35 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=frwiki --delete # T304461
19:34 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=viwiki --delete # T304461
19:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
19:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25443 and previous config saved to /var/cache/conftool/dbconfig/20220419-193318-ladsgroup.json
19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25442 and previous config saved to /var/cache/conftool/dbconfig/20220419-193309-ladsgroup.json
19:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
19:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25441 and previous config saved to /var/cache/conftool/dbconfig/20220419-193301-ladsgroup.json
19:20 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudweb2002-dev.mgmt.codfw.wmnet with reboot policy FORCED
19:20 bking@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_eqiad: Upgrading Elasticsearch to 6.8 in EQIAD - bking@cumin1001 - T301959
19:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
19:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
19:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
19:20 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
19:19 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P25440 and previous config saved to /var/cache/conftool/dbconfig/20220419-191812-ladsgroup.json
19:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25439 and previous config saved to /var/cache/conftool/dbconfig/20220419-191756-ladsgroup.json
19:15 jhuneidi@deploy1002: Pruned MediaWiki: 1.39.0-wmf.6 (duration: 01m 31s)
19:14 pt1979@cumin2002: START - Cookbook sre.dns.netbox
19:10 jhuneidi@deploy1002: Finished scap: testwikis wikis to 1.39.0-wmf.8 refs T305214 (duration: 42m 16s)
19:09 bking@cumin1001: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_eqiad: Upgrading Elasticsearch to 6.8 in EQIAD - bking@cumin1001 - T301959
19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P25438 and previous config saved to /var/cache/conftool/dbconfig/20220419-190306-ladsgroup.json
19:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25437 and previous config saved to /var/cache/conftool/dbconfig/20220419-190250-ladsgroup.json
19:00 bking@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_eqiad: Upgrading Elasticsearch to 6.8 in EQIAD - bking@cumin1001 - T301959
18:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
18:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
18:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
18:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25436 and previous config saved to /var/cache/conftool/dbconfig/20220419-185602-ladsgroup.json
18:53 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudservices2005-dev.mgmt.codfw.wmnet with reboot policy FORCED
18:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25435 and previous config saved to /var/cache/conftool/dbconfig/20220419-184801-ladsgroup.json
18:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25434 and previous config saved to /var/cache/conftool/dbconfig/20220419-184745-ladsgroup.json
18:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25433 and previous config saved to /var/cache/conftool/dbconfig/20220419-184057-ladsgroup.json
18:39 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25432 and previous config saved to /var/cache/conftool/dbconfig/20220419-183544-ladsgroup.json
18:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
18:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
18:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P25431 and previous config saved to /var/cache/conftool/dbconfig/20220419-183536-ladsgroup.json
18:34 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudservices2005-dev.mgmt.codfw.wmnet with reboot policy FORCED
18:31 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudservices2004-dev.mgmt.codfw.wmnet with reboot policy FORCED
18:27 jhuneidi@deploy1002: Started scap: testwikis wikis to 1.39.0-wmf.8 refs T305214
18:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25430 and previous config saved to /var/cache/conftool/dbconfig/20220419-182552-ladsgroup.json
18:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P25429 and previous config saved to /var/cache/conftool/dbconfig/20220419-182031-ladsgroup.json
18:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25428 and previous config saved to /var/cache/conftool/dbconfig/20220419-181047-ladsgroup.json
18:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P25427 and previous config saved to /var/cache/conftool/dbconfig/20220419-180525-ladsgroup.json
18:05 brennen: train 1.38.0-wmf.9 (T305214): we're currently debugging some scap / train prep issues.
18:04 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudservices2004-dev.mgmt.codfw.wmnet with reboot policy FORCED
18:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:03 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudnet2006-dev.mgmt.codfw.wmnet with reboot policy FORCED
17:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25426 and previous config saved to /var/cache/conftool/dbconfig/20220419-175431-ladsgroup.json
17:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
17:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
17:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P25425 and previous config saved to /var/cache/conftool/dbconfig/20220419-175021-ladsgroup.json
17:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298565)', diff saved to https://phabricator.wikimedia.org/P25424 and previous config saved to /var/cache/conftool/dbconfig/20220419-174731-ladsgroup.json
17:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
17:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
17:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
17:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
17:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25423 and previous config saved to /var/cache/conftool/dbconfig/20220419-174717-ladsgroup.json
17:41 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:39 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
17:39 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudnet2006-dev.mgmt.codfw.wmnet with reboot policy FORCED
17:38 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
17:38 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
17:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P25422 and previous config saved to /var/cache/conftool/dbconfig/20220419-173836-ladsgroup.json
17:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
17:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
17:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25421 and previous config saved to /var/cache/conftool/dbconfig/20220419-173827-ladsgroup.json
17:38 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudnet2005-dev.mgmt.codfw.wmnet with reboot policy FORCED
17:38 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
17:38 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
17:38 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=arwiki --delete # T304461
17:37 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
17:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25420 and previous config saved to /var/cache/conftool/dbconfig/20220419-173706-ladsgroup.json
17:36 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=bnwiki --delete # T304461
17:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
17:33 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
17:33 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
17:33 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
17:33 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
17:33 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
17:32 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
17:32 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
17:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25419 and previous config saved to /var/cache/conftool/dbconfig/20220419-173212-ladsgroup.json
17:32 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
17:31 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
17:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P25418 and previous config saved to /var/cache/conftool/dbconfig/20220419-172321-ladsgroup.json
17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25417 and previous config saved to /var/cache/conftool/dbconfig/20220419-172200-ladsgroup.json
17:18 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
17:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25416 and previous config saved to /var/cache/conftool/dbconfig/20220419-171707-ladsgroup.json
17:14 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
17:14 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
17:11 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudnet2005-dev.mgmt.codfw.wmnet with reboot policy FORCED
17:11 cmooney@cumin1001: START - Cookbook sre.dns.netbox
17:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P25415 and previous config saved to /var/cache/conftool/dbconfig/20220419-170816-ladsgroup.json
17:07 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25414 and previous config saved to /var/cache/conftool/dbconfig/20220419-170655-ladsgroup.json
17:02 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephmon2006-dev.mgmt.codfw.wmnet with reboot policy FORCED
17:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25413 and previous config saved to /var/cache/conftool/dbconfig/20220419-170202-ladsgroup.json
16:56 kormat@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25412 and previous config saved to /var/cache/conftool/dbconfig/20220419-165641-kormat.json
16:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25411 and previous config saved to /var/cache/conftool/dbconfig/20220419-165511-ladsgroup.json
16:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
16:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
16:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25410 and previous config saved to /var/cache/conftool/dbconfig/20220419-165503-ladsgroup.json
16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25409 and previous config saved to /var/cache/conftool/dbconfig/20220419-165311-ladsgroup.json
16:53 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
16:53 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
16:53 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
16:53 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
16:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25408 and previous config saved to /var/cache/conftool/dbconfig/20220419-165150-ladsgroup.json
16:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25407 and previous config saved to /var/cache/conftool/dbconfig/20220419-164216-ladsgroup.json
16:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
16:42 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudcephmon2006-dev.mgmt.codfw.wmnet with reboot policy FORCED
16:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
16:41 kormat@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25406 and previous config saved to /var/cache/conftool/dbconfig/20220419-164137-kormat.json
16:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25405 and previous config saved to /var/cache/conftool/dbconfig/20220419-163958-ladsgroup.json
16:38 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
16:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25404 and previous config saved to /var/cache/conftool/dbconfig/20220419-163414-ladsgroup.json
16:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
16:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
16:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25403 and previous config saved to /var/cache/conftool/dbconfig/20220419-163406-ladsgroup.json
16:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
16:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
16:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25402 and previous config saved to /var/cache/conftool/dbconfig/20220419-163321-ladsgroup.json
16:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2012.codfw.wmnet
16:32 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
16:31 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
16:28 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
16:28 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2012.codfw.wmnet
16:28 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
16:27 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
16:26 kormat@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25401 and previous config saved to /var/cache/conftool/dbconfig/20220419-162633-kormat.json
16:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25400 and previous config saved to /var/cache/conftool/dbconfig/20220419-162453-ladsgroup.json
16:23 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=kowiki --delete # T304461
16:21 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=cswiki --delete # T304461
16:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25399 and previous config saved to /var/cache/conftool/dbconfig/20220419-161901-ladsgroup.json
16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P25398 and previous config saved to /var/cache/conftool/dbconfig/20220419-161816-ladsgroup.json
16:16 otto@deploy1002: Finished deploy [analytics/refinery@f136555] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f136555] (duration: 06m 49s)
16:15 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
16:14 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
16:13 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephmon2005-dev.mgmt.codfw.wmnet with reboot policy FORCED
16:11 kormat@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25397 and previous config saved to /var/cache/conftool/dbconfig/20220419-161129-kormat.json
16:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25396 and previous config saved to /var/cache/conftool/dbconfig/20220419-160948-ladsgroup.json
16:09 otto@deploy1002: Started deploy [analytics/refinery@f136555] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f136555]
16:09 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1019.eqiad.wmnet with OS bullseye
16:08 otto@deploy1002: Finished deploy [analytics/refinery@f136555] (thin): Regular analytics weekly train THIN [analytics/refinery@f136555] (duration: 00m 07s)
16:08 otto@deploy1002: Started deploy [analytics/refinery@f136555] (thin): Regular analytics weekly train THIN [analytics/refinery@f136555]
16:07 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
16:07 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1182.eqiad.wmnet with reason: Rebooting for T303174
16:07 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1182.eqiad.wmnet with reason: Rebooting for T303174
16:06 kormat@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25395 and previous config saved to /var/cache/conftool/dbconfig/20220419-160629-kormat.json
16:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298565)', diff saved to https://phabricator.wikimedia.org/P25394 and previous config saved to /var/cache/conftool/dbconfig/20220419-160409-ladsgroup.json
16:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
16:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25393 and previous config saved to /var/cache/conftool/dbconfig/20220419-160355-ladsgroup.json
16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P25392 and previous config saved to /var/cache/conftool/dbconfig/20220419-160311-ladsgroup.json
15:59 otto@deploy1002: Finished deploy [analytics/refinery@f136555]: weekly train (duration: 22m 21s)
15:57 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
15:57 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
15:57 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
15:55 kormat@cumin1001: dbctl commit (dc=all): 'db1114 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25391 and previous config saved to /var/cache/conftool/dbconfig/20220419-155531-kormat.json
15:54 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
15:54 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
15:54 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
15:54 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
15:51 kormat@cumin1001: dbctl commit (dc=all): 'es1026 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25390 and previous config saved to /var/cache/conftool/dbconfig/20220419-155146-kormat.json
15:51 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1026.eqiad.wmnet with reason: Rebooting for T303174
15:51 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1026.eqiad.wmnet with reason: Rebooting for T303174
15:51 kormat@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25389 and previous config saved to /var/cache/conftool/dbconfig/20220419-155125-kormat.json
15:51 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
15:51 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
15:50 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
15:50 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
15:50 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1019.eqiad.wmnet with reason: host reimage
15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25388 and previous config saved to /var/cache/conftool/dbconfig/20220419-154850-ladsgroup.json
15:48 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudcephmon2005-dev.mgmt.codfw.wmnet with reboot policy FORCED
15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25387 and previous config saved to /var/cache/conftool/dbconfig/20220419-154806-ladsgroup.json
15:47 andrew@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1019.eqiad.wmnet with reason: host reimage
15:46 damilare: payments-wiki revision changed from a9a1f2ee to a3c69385
15:45 damilare: localsettings revision changed from c8fee00c to e365fe0a
15:40 kormat@cumin1001: dbctl commit (dc=all): 'db1114 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25386 and previous config saved to /var/cache/conftool/dbconfig/20220419-154027-kormat.json
15:39 elukey: powercycle elastic1097 (still with role::insetup, but not reachable via ssh or mgmt console)
15:37 otto@deploy1002: Started deploy [analytics/refinery@f136555]: weekly train
15:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25385 and previous config saved to /var/cache/conftool/dbconfig/20220419-153707-ladsgroup.json
15:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
15:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
15:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P25384 and previous config saved to /var/cache/conftool/dbconfig/20220419-153659-ladsgroup.json
15:36 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2008.codfw.wmnet with OS bullseye
15:36 kormat@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25383 and previous config saved to /var/cache/conftool/dbconfig/20220419-153621-kormat.json
15:35 ariel@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host dumpsdata1003.eqiad.wmnet
15:35 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1019.eqiad.wmnet with OS bullseye
15:33 elukey: start rdb2008 from mgmt console (was powered down for relocation)
15:29 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host wdqs2011.codfw.wmnet
15:28 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:27 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2007.codfw.wmnet with OS bullseye
15:26 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
15:26 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
15:26 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
15:25 ariel@cumin1001: START - Cookbook sre.hosts.reboot-single for host dumpsdata1003.eqiad.wmnet
15:25 kormat@cumin1001: dbctl commit (dc=all): 'db1114 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25382 and previous config saved to /var/cache/conftool/dbconfig/20220419-152523-kormat.json
15:25 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
15:25 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
15:24 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2006.codfw.wmnet with OS bullseye
15:24 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
15:24 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2008.codfw.wmnet with reason: host reimage
15:24 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
15:24 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
15:24 pt1979@cumin2002: START - Cookbook sre.dns.netbox
15:24 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
15:24 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
15:23 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
15:23 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
15:22 ariel@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1001.eqiad.wmnet
15:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P25381 and previous config saved to /var/cache/conftool/dbconfig/20220419-152154-ladsgroup.json
15:21 kormat@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25380 and previous config saved to /var/cache/conftool/dbconfig/20220419-152117-kormat.json
15:19 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2008.codfw.wmnet with reason: host reimage
15:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25379 and previous config saved to /var/cache/conftool/dbconfig/20220419-151847-ladsgroup.json
15:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2011.codfw.wmnet
15:17 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host wdqs2010.codfw.wmnet
15:17 ariel@cumin1001: START - Cookbook sre.hosts.reboot-single for host dumpsdata1001.eqiad.wmnet
15:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 (T298565)', diff saved to https://phabricator.wikimedia.org/P25378 and previous config saved to /var/cache/conftool/dbconfig/20220419-151607-ladsgroup.json
15:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
15:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
15:15 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2007.codfw.wmnet with reason: host reimage
15:15 kormat@cumin1001: dbctl commit (dc=all): 'es1027 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25377 and previous config saved to /var/cache/conftool/dbconfig/20220419-151552-kormat.json
15:15 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1027.eqiad.wmnet with reason: Rebooting for T303174
15:15 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1027.eqiad.wmnet with reason: Rebooting for T303174
15:13 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2006.codfw.wmnet with reason: host reimage
15:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
15:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
15:10 kormat@cumin1001: dbctl commit (dc=all): 'db1114 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25376 and previous config saved to /var/cache/conftool/dbconfig/20220419-151019-kormat.json
15:10 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2005.codfw.wmnet with OS bullseye
15:10 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2007.codfw.wmnet with reason: host reimage
15:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
15:09 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2006.codfw.wmnet with reason: host reimage
15:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
15:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
15:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
15:09 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2010.codfw.wmnet
15:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2009.codfw.wmnet
15:07 kormat@cumin1001: dbctl commit (dc=all): 'db1182 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25375 and previous config saved to /var/cache/conftool/dbconfig/20220419-150717-kormat.json
15:07 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1182.eqiad.wmnet with reason: Rebooting for T303174
15:07 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1182.eqiad.wmnet with reason: Rebooting for T303174
15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P25374 and previous config saved to /var/cache/conftool/dbconfig/20220419-150649-ladsgroup.json
15:06 kormat@cumin1001: dbctl commit (dc=all): 'db1114 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25373 and previous config saved to /var/cache/conftool/dbconfig/20220419-150637-kormat.json
15:06 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1114.eqiad.wmnet with reason: Rebooting for T303174
15:06 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1114.eqiad.wmnet with reason: Rebooting for T303174
15:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
15:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
15:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25372 and previous config saved to /var/cache/conftool/dbconfig/20220419-150454-ladsgroup.json
15:03 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2008.codfw.wmnet with OS bullseye
15:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2009.codfw.wmnet
15:03 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2004.codfw.wmnet with OS bullseye
14:58 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2005.codfw.wmnet with reason: host reimage
14:56 kormat@cumin1001: dbctl commit (dc=all): 'db1111 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25371 and previous config saved to /var/cache/conftool/dbconfig/20220419-145658-kormat.json
14:54 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2005.codfw.wmnet with reason: host reimage
14:54 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2007.codfw.wmnet with OS bullseye
14:54 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2006.codfw.wmnet with OS bullseye
14:52 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2003.codfw.wmnet with OS bullseye
14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P25370 and previous config saved to /var/cache/conftool/dbconfig/20220419-145143-ladsgroup.json
14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25369 and previous config saved to /var/cache/conftool/dbconfig/20220419-144949-ladsgroup.json
14:49 kormat@cumin1001: dbctl commit (dc=all): 'db1129 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25368 and previous config saved to /var/cache/conftool/dbconfig/20220419-144941-kormat.json
14:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25367 and previous config saved to /var/cache/conftool/dbconfig/20220419-144836-ladsgroup.json
14:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
14:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
14:48 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2002.codfw.wmnet with OS bullseye
14:45 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2004.codfw.wmnet with reason: host reimage
14:42 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2004.codfw.wmnet with reason: host reimage
14:41 kormat@cumin1001: dbctl commit (dc=all): 'db1111 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25366 and previous config saved to /var/cache/conftool/dbconfig/20220419-144154-kormat.json
14:41 kormat@cumin1001: dbctl commit (dc=all): 'db1110 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25365 and previous config saved to /var/cache/conftool/dbconfig/20220419-144144-kormat.json
14:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P25364 and previous config saved to /var/cache/conftool/dbconfig/20220419-144105-ladsgroup.json
14:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
14:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P25363 and previous config saved to /var/cache/conftool/dbconfig/20220419-144057-ladsgroup.json
14:40 kormat@cumin1001: dbctl commit (dc=all): 'db1169 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25362 and previous config saved to /var/cache/conftool/dbconfig/20220419-144001-kormat.json
14:39 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2003.codfw.wmnet with reason: host reimage
14:39 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2005.codfw.wmnet with OS bullseye
14:38 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2001.codfw.wmnet with OS bullseye
14:36 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2002.codfw.wmnet with reason: host reimage
14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25361 and previous config saved to /var/cache/conftool/dbconfig/20220419-143444-ladsgroup.json
14:34 kormat@cumin1001: dbctl commit (dc=all): 'db1129 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25360 and previous config saved to /var/cache/conftool/dbconfig/20220419-143437-kormat.json
14:33 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2003.codfw.wmnet with reason: host reimage
14:33 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2002.codfw.wmnet with reason: host reimage
14:26 kormat@cumin1001: dbctl commit (dc=all): 'db1111 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25359 and previous config saved to /var/cache/conftool/dbconfig/20220419-142650-kormat.json
14:26 kormat@cumin1001: dbctl commit (dc=all): 'db1110 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25358 and previous config saved to /var/cache/conftool/dbconfig/20220419-142640-kormat.json
14:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P25357 and previous config saved to /var/cache/conftool/dbconfig/20220419-142552-ladsgroup.json
14:25 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2001.codfw.wmnet with reason: host reimage
14:25 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2004.codfw.wmnet with OS bullseye
14:24 kormat@cumin1001: dbctl commit (dc=all): 'db1169 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25356 and previous config saved to /var/cache/conftool/dbconfig/20220419-142457-kormat.json
14:22 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2001.codfw.wmnet with reason: host reimage
14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25355 and previous config saved to /var/cache/conftool/dbconfig/20220419-141937-ladsgroup.json
14:19 kormat@cumin1001: dbctl commit (dc=all): 'db1129 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25354 and previous config saved to /var/cache/conftool/dbconfig/20220419-141933-kormat.json
14:17 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2003.codfw.wmnet with OS bullseye
14:16 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2002.codfw.wmnet with OS bullseye
14:15 jynus: edited directly phab database to fix corrupt entry T305919
14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T298565)', diff saved to https://phabricator.wikimedia.org/P25353 and previous config saved to /var/cache/conftool/dbconfig/20220419-141303-ladsgroup.json
14:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
14:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
14:11 kormat@cumin1001: dbctl commit (dc=all): 'db1111 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25352 and previous config saved to /var/cache/conftool/dbconfig/20220419-141146-kormat.json
14:11 kormat@cumin1001: dbctl commit (dc=all): 'db1110 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25351 and previous config saved to /var/cache/conftool/dbconfig/20220419-141136-kormat.json
14:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P25350 and previous config saved to /var/cache/conftool/dbconfig/20220419-141047-ladsgroup.json
14:09 kormat@cumin1001: dbctl commit (dc=all): 'db1169 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25349 and previous config saved to /var/cache/conftool/dbconfig/20220419-140954-kormat.json
14:07 kormat@cumin1001: dbctl commit (dc=all): 'db1111 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25348 and previous config saved to /var/cache/conftool/dbconfig/20220419-140756-kormat.json
14:07 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1111.eqiad.wmnet with reason: Rebooting for T303174
14:07 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1111.eqiad.wmnet with reason: Rebooting for T303174
14:07 kormat@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 100%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25347 and previous config saved to /var/cache/conftool/dbconfig/20220419-140703-kormat.json
14:06 godog: start deleting tegola-cache/osm prefix from tegola-swift-container - T306424
14:05 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2001.codfw.wmnet with OS bullseye
14:04 kormat@cumin1001: dbctl commit (dc=all): 'db1129 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25346 and previous config saved to /var/cache/conftool/dbconfig/20220419-140430-kormat.json
14:01 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1129.eqiad.wmnet with reason: Rebooting for T303174
14:01 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1129.eqiad.wmnet with reason: Rebooting for T303174
13:56 kormat@cumin1001: dbctl commit (dc=all): 'db1110 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25345 and previous config saved to /var/cache/conftool/dbconfig/20220419-135632-kormat.json
13:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P25344 and previous config saved to /var/cache/conftool/dbconfig/20220419-135542-ladsgroup.json
13:55 hnowlan@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad
13:54 kormat@cumin1001: dbctl commit (dc=all): 'db1169 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25343 and previous config saved to /var/cache/conftool/dbconfig/20220419-135450-kormat.json
13:52 kormat@cumin1001: dbctl commit (dc=all): 'db1110 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25342 and previous config saved to /var/cache/conftool/dbconfig/20220419-135225-kormat.json
13:52 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1110.eqiad.wmnet with reason: Rebooting for T303174
13:52 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1110.eqiad.wmnet with reason: Rebooting for T303174
13:51 kormat@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 75%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25341 and previous config saved to /var/cache/conftool/dbconfig/20220419-135159-kormat.json
13:51 kormat@cumin1001: dbctl commit (dc=all): 'db1129 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25340 and previous config saved to /var/cache/conftool/dbconfig/20220419-135140-kormat.json
13:51 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1129.eqiad.wmnet with reason: Rebooting for T303174
13:51 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1129.eqiad.wmnet with reason: Rebooting for T303174
13:50 kormat@cumin1001: dbctl commit (dc=all): 'db1169 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25339 and previous config saved to /var/cache/conftool/dbconfig/20220419-135007-kormat.json
13:50 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1169.eqiad.wmnet with reason: Rebooting for T303174
13:50 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1169.eqiad.wmnet with reason: Rebooting for T303174
13:46 hnowlan@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad
13:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P25338 and previous config saved to /var/cache/conftool/dbconfig/20220419-134503-ladsgroup.json
13:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
13:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
13:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P25337 and previous config saved to /var/cache/conftool/dbconfig/20220419-134455-ladsgroup.json
13:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T306269)', diff saved to https://phabricator.wikimedia.org/P25336 and previous config saved to /var/cache/conftool/dbconfig/20220419-134139-marostegui.json
13:36 kormat@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 50%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25335 and previous config saved to /var/cache/conftool/dbconfig/20220419-133655-kormat.json
13:30 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2080.codfw.wmnet with reason: Rebooting for T303174
13:30 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2080.codfw.wmnet with reason: Rebooting for T303174
13:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P25334 and previous config saved to /var/cache/conftool/dbconfig/20220419-132949-ladsgroup.json
13:27 taavi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: mrwikisource: Add template editor and patroller user groups (T269067) (duration: 00m 50s)
13:27 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
13:26 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
13:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P25333 and previous config saved to /var/cache/conftool/dbconfig/20220419-132634-marostegui.json
13:26 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync
13:25 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: sync
13:25 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:25 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:25 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:25 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:21 kormat@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 25%: Reboot T303174', diff saved to https://phabricator.wikimedia.org/P25332 and previous config saved to /var/cache/conftool/dbconfig/20220419-132151-kormat.json
13:15 kormat@cumin1001: dbctl commit (dc=all): 'db1104 depooling: Rebooting for T303174', diff saved to https://phabricator.wikimedia.org/P25331 and previous config saved to /var/cache/conftool/dbconfig/20220419-131557-kormat.json
13:15 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1104.eqiad.wmnet with reason: Rebooting for T303174
13:15 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1104.eqiad.wmnet with reason: Rebooting for T303174
13:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P25330 and previous config saved to /var/cache/conftool/dbconfig/20220419-131444-ladsgroup.json
13:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P25329 and previous config saved to /var/cache/conftool/dbconfig/20220419-131128-marostegui.json
13:03 volans@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
13:03 volans@cumin1001: START - Cookbook sre.network.cf
12:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P25328 and previous config saved to /var/cache/conftool/dbconfig/20220419-125939-ladsgroup.json
12:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T306269)', diff saved to https://phabricator.wikimedia.org/P25327 and previous config saved to /var/cache/conftool/dbconfig/20220419-125623-marostegui.json
12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P25326 and previous config saved to /var/cache/conftool/dbconfig/20220419-124851-ladsgroup.json
12:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
12:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P25325 and previous config saved to /var/cache/conftool/dbconfig/20220419-124843-ladsgroup.json
12:47 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
12:46 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
12:46 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
12:46 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync
12:45 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: sync
12:41 jgiannelos@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
12:41 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
12:40 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
12:38 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
12:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P25324 and previous config saved to /var/cache/conftool/dbconfig/20220419-123337-ladsgroup.json
12:31 mmandere@cumin1001: END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for pybal-test2002.codfw.wmnet: Renew puppet certificate - mmandere@cumin1001
12:31 mmandere@cumin1001: START - Cookbook sre.puppet.renew-cert for pybal-test2002.codfw.wmnet: Renew puppet certificate - mmandere@cumin1001
12:23 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/datahub: sync on main
12:22 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/datahub: apply on main
12:21 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
12:20 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
12:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P25323 and previous config saved to /var/cache/conftool/dbconfig/20220419-121832-ladsgroup.json
12:16 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host db1136.eqiad.wmnet with OS bullseye
12:14 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
12:12 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
12:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P25322 and previous config saved to /var/cache/conftool/dbconfig/20220419-120327-ladsgroup.json
12:02 godog: create tegola-swift-fallback container in account tegola
12:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1136.eqiad.wmnet with reason: host reimage
11:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1136.eqiad.wmnet with reason: host reimage
11:56 hnowlan@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=codfw
11:56 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T306269)', diff saved to https://phabricator.wikimedia.org/P25321 and previous config saved to /var/cache/conftool/dbconfig/20220419-115609-marostegui.json
11:56 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
11:56 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
11:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T306269)', diff saved to https://phabricator.wikimedia.org/P25320 and previous config saved to /var/cache/conftool/dbconfig/20220419-115601-marostegui.json
11:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P25319 and previous config saved to /var/cache/conftool/dbconfig/20220419-115239-ladsgroup.json
11:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
11:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
11:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
11:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
11:47 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1136.eqiad.wmnet with OS bullseye
11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
11:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
11:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25318 and previous config saved to /var/cache/conftool/dbconfig/20220419-114311-ladsgroup.json
11:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25317 and previous config saved to /var/cache/conftool/dbconfig/20220419-114056-marostegui.json
11:32 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
11:30 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
11:28 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
11:28 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25316 and previous config saved to /var/cache/conftool/dbconfig/20220419-112806-ladsgroup.json
11:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25315 and previous config saved to /var/cache/conftool/dbconfig/20220419-112551-marostegui.json
11:25 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
11:25 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
11:24 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
11:23 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
11:21 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
11:21 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
11:18 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
11:18 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
11:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25314 and previous config saved to /var/cache/conftool/dbconfig/20220419-111301-ladsgroup.json
11:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T306269)', diff saved to https://phabricator.wikimedia.org/P25313 and previous config saved to /var/cache/conftool/dbconfig/20220419-111046-marostegui.json
11:10 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync
11:09 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: sync
11:08 hnowlan@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'similar-users' for release 'main' .
11:07 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
11:07 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
11:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T306269)', diff saved to https://phabricator.wikimedia.org/P25312 and previous config saved to /var/cache/conftool/dbconfig/20220419-110710-marostegui.json
11:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
11:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
11:05 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
11:05 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
11:05 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
11:05 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
11:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
11:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
11:04 moritzm: installing xz-utils/xzgrep security updates
11:04 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
11:03 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
11:02 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
11:02 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
11:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
11:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
11:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
11:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
10:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
10:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
10:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T306269)', diff saved to https://phabricator.wikimedia.org/P25311 and previous config saved to /var/cache/conftool/dbconfig/20220419-105948-marostegui.json
10:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25310 and previous config saved to /var/cache/conftool/dbconfig/20220419-105756-ladsgroup.json
10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P25309 and previous config saved to /var/cache/conftool/dbconfig/20220419-104443-marostegui.json
10:39 mmandere: reimage pybal-test2002 as buster - T297187
10:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P25308 and previous config saved to /var/cache/conftool/dbconfig/20220419-102938-marostegui.json
10:17 moritzm: installing gzip/zgrep security updates
10:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T306269)', diff saved to https://phabricator.wikimedia.org/P25306 and previous config saved to /var/cache/conftool/dbconfig/20220419-101433-marostegui.json
10:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T306269)', diff saved to https://phabricator.wikimedia.org/P25305 and previous config saved to /var/cache/conftool/dbconfig/20220419-101233-marostegui.json
10:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
10:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
10:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T306269)', diff saved to https://phabricator.wikimedia.org/P25304 and previous config saved to /var/cache/conftool/dbconfig/20220419-101225-marostegui.json
09:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25303 and previous config saved to /var/cache/conftool/dbconfig/20220419-095742-ladsgroup.json
09:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
09:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
09:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P25302 and previous config saved to /var/cache/conftool/dbconfig/20220419-095720-marostegui.json
09:50 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
09:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
09:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
09:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
09:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
09:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P25301 and previous config saved to /var/cache/conftool/dbconfig/20220419-094812-ladsgroup.json
09:43 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync
09:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P25300 and previous config saved to /var/cache/conftool/dbconfig/20220419-094215-marostegui.json
09:38 marostegui@cumin1001: dbctl commit (dc=all): 'db1162 (re)pooling @ 100%: After reboot', diff saved to https://phabricator.wikimedia.org/P25299 and previous config saved to /var/cache/conftool/dbconfig/20220419-093825-root.json
09:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P25298 and previous config saved to /var/cache/conftool/dbconfig/20220419-093307-ladsgroup.json
09:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T306269)', diff saved to https://phabricator.wikimedia.org/P25297 and previous config saved to /var/cache/conftool/dbconfig/20220419-092710-marostegui.json
09:23 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: sync
09:23 marostegui@cumin1001: dbctl commit (dc=all): 'db1162 (re)pooling @ 75%: After reboot', diff saved to https://phabricator.wikimedia.org/P25296 and previous config saved to /var/cache/conftool/dbconfig/20220419-092321-root.json
09:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T306269)', diff saved to https://phabricator.wikimedia.org/P25295 and previous config saved to /var/cache/conftool/dbconfig/20220419-092146-marostegui.json
09:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
09:21 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
09:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
09:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T306269)', diff saved to https://phabricator.wikimedia.org/P25294 and previous config saved to /var/cache/conftool/dbconfig/20220419-092138-marostegui.json
09:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P25293 and previous config saved to /var/cache/conftool/dbconfig/20220419-091802-ladsgroup.json
09:16 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
09:08 marostegui@cumin1001: dbctl commit (dc=all): 'db1162 (re)pooling @ 50%: After reboot', diff saved to https://phabricator.wikimedia.org/P25292 and previous config saved to /var/cache/conftool/dbconfig/20220419-090817-root.json
09:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P25291 and previous config saved to /var/cache/conftool/dbconfig/20220419-090633-marostegui.json
09:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic[1084-1088].eqiad.wmnet with reason: reboot
09:05 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic[1084-1088].eqiad.wmnet with reason: reboot
09:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P25290 and previous config saved to /var/cache/conftool/dbconfig/20220419-090256-ladsgroup.json
08:57 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic[2070-2072].codfw.wmnet with reason: reboot
08:57 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic[2070-2072].codfw.wmnet with reason: reboot
08:53 marostegui@cumin1001: dbctl commit (dc=all): 'db1162 (re)pooling @ 25%: After reboot', diff saved to https://phabricator.wikimedia.org/P25288 and previous config saved to /var/cache/conftool/dbconfig/20220419-085313-root.json
08:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P25287 and previous config saved to /var/cache/conftool/dbconfig/20220419-085148-ladsgroup.json
08:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
08:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
08:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
08:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
08:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P25286 and previous config saved to /var/cache/conftool/dbconfig/20220419-085135-ladsgroup.json
08:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P25285 and previous config saved to /var/cache/conftool/dbconfig/20220419-085128-marostegui.json
08:38 marostegui@cumin1001: dbctl commit (dc=all): 'db1162 (re)pooling @ 10%: After reboot', diff saved to https://phabricator.wikimedia.org/P25284 and previous config saved to /var/cache/conftool/dbconfig/20220419-083810-root.json
08:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P25283 and previous config saved to /var/cache/conftool/dbconfig/20220419-083630-ladsgroup.json
08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T306269)', diff saved to https://phabricator.wikimedia.org/P25282 and previous config saved to /var/cache/conftool/dbconfig/20220419-083623-marostegui.json
08:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T306269)', diff saved to https://phabricator.wikimedia.org/P25281 and previous config saved to /var/cache/conftool/dbconfig/20220419-083159-marostegui.json
08:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
08:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
08:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T306269)', diff saved to https://phabricator.wikimedia.org/P25280 and previous config saved to /var/cache/conftool/dbconfig/20220419-083151-marostegui.json
08:30 ayounsi@cumin2002: END (FAIL) - Cookbook sre.network.cf (exit_code=1)
08:29 ayounsi@cumin2002: START - Cookbook sre.network.cf
08:29 XioNoX: turn CF on for drmrs (test)
08:29 kormat: deploying monitoring change for db2093 T301315 https://gerrit.wikimedia.org/r/c/operations/puppet/+/775852
08:29 ayounsi@cumin2002: END (PASS) - Cookbook sre.network.cf (exit_code=0)
08:29 ayounsi@cumin2002: START - Cookbook sre.network.cf
08:23 marostegui@cumin1001: dbctl commit (dc=all): 'db1162 (re)pooling @ 5%: After reboot', diff saved to https://phabricator.wikimedia.org/P25279 and previous config saved to /var/cache/conftool/dbconfig/20220419-082306-root.json
08:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P25278 and previous config saved to /var/cache/conftool/dbconfig/20220419-082125-ladsgroup.json
08:20 elukey: systemctl restart kartotherian on maps1010
08:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P25277 and previous config saved to /var/cache/conftool/dbconfig/20220419-081646-marostegui.json
08:16 hashar: Restarting CI Jenkins on contint2001 for plugins updates
08:08 marostegui@cumin1001: dbctl commit (dc=all): 'db1162 (re)pooling @ 1%: After reboot', diff saved to https://phabricator.wikimedia.org/P25276 and previous config saved to /var/cache/conftool/dbconfig/20220419-080802-root.json
08:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P25275 and previous config saved to /var/cache/conftool/dbconfig/20220419-080620-ladsgroup.json
08:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P25273 and previous config saved to /var/cache/conftool/dbconfig/20220419-080141-marostegui.json
08:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1162', diff saved to https://phabricator.wikimedia.org/P25272 and previous config saved to /var/cache/conftool/dbconfig/20220419-080024-marostegui.json
07:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P25271 and previous config saved to /var/cache/conftool/dbconfig/20220419-075436-ladsgroup.json
07:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
07:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
07:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P25270 and previous config saved to /var/cache/conftool/dbconfig/20220419-075428-ladsgroup.json
07:53 elukey: restart tilerator on maps1010 (service down, following runbook)
07:52 elukey: restart tilerator on maps100[678] (service down, following runbook)
07:49 elukey: restart tilerator on maps1005 (service down, following runbook)
07:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 9 hosts with reason: reboot
07:49 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on 9 hosts with reason: reboot
07:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T306269)', diff saved to https://phabricator.wikimedia.org/P25269 and previous config saved to /var/cache/conftool/dbconfig/20220419-074636-marostegui.json
07:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T306269)', diff saved to https://phabricator.wikimedia.org/P25268 and previous config saved to /var/cache/conftool/dbconfig/20220419-074140-marostegui.json
07:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
07:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
07:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T306269)', diff saved to https://phabricator.wikimedia.org/P25267 and previous config saved to /var/cache/conftool/dbconfig/20220419-074132-marostegui.json
07:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P25266 and previous config saved to /var/cache/conftool/dbconfig/20220419-073923-ladsgroup.json
07:33 XioNoX: moving mr1-eqsin to new router
07:31 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging CGlenn out of all services on: 1229 hosts
07:31 jmm@cumin2002: START - Cookbook sre.idm.logout Logging CGlenn out of all services on: 1229 hosts
07:30 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging CGlenn out of all services on: 442 hosts
07:29 jmm@cumin2002: START - Cookbook sre.idm.logout Logging CGlenn out of all services on: 442 hosts
07:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P25265 and previous config saved to /var/cache/conftool/dbconfig/20220419-072627-marostegui.json
07:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P25264 and previous config saved to /var/cache/conftool/dbconfig/20220419-072418-ladsgroup.json
07:19 urbanecm: UTC morning B&C window done
07:19 marostegui: dbmaint s7@eqiad T301848
07:12 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.7/extensions/Translate/src/TranslatorInterface/Aid/TTMServerAid.php: 36c6682: TTMServerAid::getData: Do not swallow TranslationHelperException (T306233) (duration: 00m 51s)
07:11 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.7/extensions/Translate/ttmserver/ElasticSearchTTMServer.php: e966871: ElasticSearchTTMServer: tie break on wiki+localid (T305428, T306233) (duration: 00m 51s)
07:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P25263 and previous config saved to /var/cache/conftool/dbconfig/20220419-071122-marostegui.json
07:09 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Jason Linehan out of all services on: 1229 hosts
07:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P25262 and previous config saved to /var/cache/conftool/dbconfig/20220419-070913-ladsgroup.json
07:08 jmm@cumin2002: START - Cookbook sre.idm.logout Logging Jason Linehan out of all services on: 1229 hosts
07:08 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Jason Linehan out of all services on: 442 hosts
07:08 jmm@cumin2002: START - Cookbook sre.idm.logout Logging Jason Linehan out of all services on: 442 hosts
07:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
06:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
06:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
06:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P25261 and previous config saved to /var/cache/conftool/dbconfig/20220419-065833-ladsgroup.json
06:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
06:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
06:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P25260 and previous config saved to /var/cache/conftool/dbconfig/20220419-065825-ladsgroup.json
06:57 marostegui: dbmaint s7@eqiad T298554
06:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T306269)', diff saved to https://phabricator.wikimedia.org/P25259 and previous config saved to /var/cache/conftool/dbconfig/20220419-065617-marostegui.json
06:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T306269)', diff saved to https://phabricator.wikimedia.org/P25258 and previous config saved to /var/cache/conftool/dbconfig/20220419-065417-marostegui.json
06:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
06:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
06:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
06:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
06:51 marostegui: dbmaint s7@eqiad T305300
06:48 marostegui: dbmaint s7@eqiad T298563
06:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P25257 and previous config saved to /var/cache/conftool/dbconfig/20220419-064320-ladsgroup.json
06:41 XioNoX: eqiad: add missing Cloudflare route
06:37 XioNoX: drmrs: add tunnels to Cloudflare - T303152
06:35 ayounsi@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
06:30 ayounsi@cumin1001: START - Cookbook sre.dns.netbox
06:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P25256 and previous config saved to /var/cache/conftool/dbconfig/20220419-062815-ladsgroup.json
06:18 marostegui: dbmaint s7@eqiad T298557
06:13 marostegui: dbmaint s7@eqiad T300381
06:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P25255 and previous config saved to /var/cache/conftool/dbconfig/20220419-061310-ladsgroup.json
06:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
06:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
06:11 marostegui: dbmaint s7@eqiad T302658
06:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1136 T306001', diff saved to https://phabricator.wikimedia.org/P25254 and previous config saved to /var/cache/conftool/dbconfig/20220419-060559-marostegui.json
06:02 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db1181 to s7 primary and set section read-write T306001', diff saved to https://phabricator.wikimedia.org/P25253 and previous config saved to /var/cache/conftool/dbconfig/20220419-060226-marostegui.json
06:01 marostegui@cumin1001: dbctl commit (dc=all): 'Set s7 eqiad as read-only for maintenance - T306001', diff saved to https://phabricator.wikimedia.org/P25252 and previous config saved to /var/cache/conftool/dbconfig/20220419-060157-marostegui.json
06:01 marostegui: Starting s7 eqiad failover from db1136 to db1181 - T306001
06:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P25251 and previous config saved to /var/cache/conftool/dbconfig/20220419-060131-ladsgroup.json
06:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
06:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
06:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25250 and previous config saved to /var/cache/conftool/dbconfig/20220419-060123-ladsgroup.json
05:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P25249 and previous config saved to /var/cache/conftool/dbconfig/20220419-054618-ladsgroup.json
05:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P25248 and previous config saved to /var/cache/conftool/dbconfig/20220419-053113-ladsgroup.json
05:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25247 and previous config saved to /var/cache/conftool/dbconfig/20220419-051608-ladsgroup.json
05:09 marostegui: dbmaint s3@eqiad T306269
05:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25246 and previous config saved to /var/cache/conftool/dbconfig/20220419-050523-ladsgroup.json
05:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
05:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
05:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
05:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
04:58 marostegui@cumin1001: dbctl commit (dc=all): 'Set db1181 with weight 0 T306001', diff saved to https://phabricator.wikimedia.org/P25245 and previous config saved to /var/cache/conftool/dbconfig/20220419-045814-root.json
04:58 root@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 25 hosts with reason: Primary switchover s7 T306001
04:57 root@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on 25 hosts with reason: Primary switchover s7 T306001
04:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
04:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
04:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25244 and previous config saved to /var/cache/conftool/dbconfig/20220419-045635-ladsgroup.json
04:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P25243 and previous config saved to /var/cache/conftool/dbconfig/20220419-044130-ladsgroup.json
04:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P25242 and previous config saved to /var/cache/conftool/dbconfig/20220419-042625-ladsgroup.json
04:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
04:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
04:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25241 and previous config saved to /var/cache/conftool/dbconfig/20220419-041120-ladsgroup.json
04:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25240 and previous config saved to /var/cache/conftool/dbconfig/20220419-040024-ladsgroup.json
04:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
04:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
04:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P25239 and previous config saved to /var/cache/conftool/dbconfig/20220419-040017-ladsgroup.json
03:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P25238 and previous config saved to /var/cache/conftool/dbconfig/20220419-034512-ladsgroup.json
03:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P25237 and previous config saved to /var/cache/conftool/dbconfig/20220419-034204-ladsgroup.json
03:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P25236 and previous config saved to /var/cache/conftool/dbconfig/20220419-033006-ladsgroup.json
03:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P25235 and previous config saved to /var/cache/conftool/dbconfig/20220419-032659-ladsgroup.json
03:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
03:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
03:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
03:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
03:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P25234 and previous config saved to /var/cache/conftool/dbconfig/20220419-031501-ladsgroup.json
03:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P25233 and previous config saved to /var/cache/conftool/dbconfig/20220419-031154-ladsgroup.json
03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P25232 and previous config saved to /var/cache/conftool/dbconfig/20220419-030424-ladsgroup.json
03:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
03:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25231 and previous config saved to /var/cache/conftool/dbconfig/20220419-030416-ladsgroup.json
02:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P25230 and previous config saved to /var/cache/conftool/dbconfig/20220419-025649-ladsgroup.json
02:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P25229 and previous config saved to /var/cache/conftool/dbconfig/20220419-024911-ladsgroup.json
02:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P25228 and previous config saved to /var/cache/conftool/dbconfig/20220419-023406-ladsgroup.json
02:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
02:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
02:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
02:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
02:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25227 and previous config saved to /var/cache/conftool/dbconfig/20220419-021901-ladsgroup.json
02:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
02:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
02:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
02:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
02:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25226 and previous config saved to /var/cache/conftool/dbconfig/20220419-020703-ladsgroup.json
02:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
02:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
01:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
01:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
01:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P25225 and previous config saved to /var/cache/conftool/dbconfig/20220419-015635-ladsgroup.json
01:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
01:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
01:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P25224 and previous config saved to /var/cache/conftool/dbconfig/20220419-015627-ladsgroup.json
01:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
01:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
01:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25223 and previous config saved to /var/cache/conftool/dbconfig/20220419-014953-ladsgroup.json
01:47 mutante: [doc1001:~] $ sudo systemctl start rsync-doc-doc1002.eqiad.wmnet
01:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P25222 and previous config saved to /var/cache/conftool/dbconfig/20220419-014122-ladsgroup.json
01:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P25221 and previous config saved to /var/cache/conftool/dbconfig/20220419-013448-ladsgroup.json
01:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P25220 and previous config saved to /var/cache/conftool/dbconfig/20220419-012617-ladsgroup.json
01:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P25219 and previous config saved to /var/cache/conftool/dbconfig/20220419-011943-ladsgroup.json
01:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P25218 and previous config saved to /var/cache/conftool/dbconfig/20220419-011112-ladsgroup.json
01:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P25217 and previous config saved to /var/cache/conftool/dbconfig/20220419-010654-ladsgroup.json
01:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
01:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
01:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
01:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
01:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P25216 and previous config saved to /var/cache/conftool/dbconfig/20220419-010641-ladsgroup.json
01:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25215 and previous config saved to /var/cache/conftool/dbconfig/20220419-010438-ladsgroup.json
01:03 Amir1: turning off general logging in pc1012 (pc2) (T285993)
01:02 Amir1: turning on general logging in pc1012 (pc2) (T285993)
00:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25214 and previous config saved to /var/cache/conftool/dbconfig/20220419-005334-ladsgroup.json
00:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
00:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
00:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25213 and previous config saved to /var/cache/conftool/dbconfig/20220419-005320-ladsgroup.json
00:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P25212 and previous config saved to /var/cache/conftool/dbconfig/20220419-005136-ladsgroup.json
00:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P25211 and previous config saved to /var/cache/conftool/dbconfig/20220419-003815-ladsgroup.json
00:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P25210 and previous config saved to /var/cache/conftool/dbconfig/20220419-003631-ladsgroup.json
00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P25209 and previous config saved to /var/cache/conftool/dbconfig/20220419-002310-ladsgroup.json
00:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P25208 and previous config saved to /var/cache/conftool/dbconfig/20220419-002126-ladsgroup.json
00:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P25207 and previous config saved to /var/cache/conftool/dbconfig/20220419-001610-ladsgroup.json
00:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
00:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
00:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25206 and previous config saved to /var/cache/conftool/dbconfig/20220419-001602-ladsgroup.json
00:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25205 and previous config saved to /var/cache/conftool/dbconfig/20220419-000805-ladsgroup.json
00:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P25204 and previous config saved to /var/cache/conftool/dbconfig/20220419-000057-ladsgroup.json

2022-04-18

23:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25203 and previous config saved to /var/cache/conftool/dbconfig/20220418-235634-ladsgroup.json
23:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
23:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
23:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
23:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
23:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P25202 and previous config saved to /var/cache/conftool/dbconfig/20220418-234552-ladsgroup.json
23:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
23:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
23:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25201 and previous config saved to /var/cache/conftool/dbconfig/20220418-233848-ladsgroup.json
23:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25200 and previous config saved to /var/cache/conftool/dbconfig/20220418-233047-ladsgroup.json
23:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P25199 and previous config saved to /var/cache/conftool/dbconfig/20220418-232343-ladsgroup.json
23:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25198 and previous config saved to /var/cache/conftool/dbconfig/20220418-231750-ladsgroup.json
23:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
23:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
23:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25197 and previous config saved to /var/cache/conftool/dbconfig/20220418-231742-ladsgroup.json
23:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P25196 and previous config saved to /var/cache/conftool/dbconfig/20220418-230836-ladsgroup.json
23:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P25195 and previous config saved to /var/cache/conftool/dbconfig/20220418-230237-ladsgroup.json
22:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25194 and previous config saved to /var/cache/conftool/dbconfig/20220418-225331-ladsgroup.json
22:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P25193 and previous config saved to /var/cache/conftool/dbconfig/20220418-224732-ladsgroup.json
22:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25192 and previous config saved to /var/cache/conftool/dbconfig/20220418-224225-ladsgroup.json
22:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
22:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
22:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P25191 and previous config saved to /var/cache/conftool/dbconfig/20220418-224217-ladsgroup.json
22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25190 and previous config saved to /var/cache/conftool/dbconfig/20220418-223227-ladsgroup.json
22:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P25189 and previous config saved to /var/cache/conftool/dbconfig/20220418-222712-ladsgroup.json
22:23 mutante: contint1001 - re-enabling puppet that was disabled a week ago. to prevent more issues when it falls out of puppet DB, hopefully there wasn't a hard reason for this
22:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25188 and previous config saved to /var/cache/conftool/dbconfig/20220418-222022-ladsgroup.json
22:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
22:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
22:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P25187 and previous config saved to /var/cache/conftool/dbconfig/20220418-222014-ladsgroup.json
22:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P25186 and previous config saved to /var/cache/conftool/dbconfig/20220418-221206-ladsgroup.json
22:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P25185 and previous config saved to /var/cache/conftool/dbconfig/20220418-220509-ladsgroup.json
21:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P25184 and previous config saved to /var/cache/conftool/dbconfig/20220418-215701-ladsgroup.json
21:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P25183 and previous config saved to /var/cache/conftool/dbconfig/20220418-215004-ladsgroup.json
21:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P25182 and previous config saved to /var/cache/conftool/dbconfig/20220418-214610-ladsgroup.json
21:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
21:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
21:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25181 and previous config saved to /var/cache/conftool/dbconfig/20220418-214602-ladsgroup.json
21:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P25180 and previous config saved to /var/cache/conftool/dbconfig/20220418-213459-ladsgroup.json
21:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P25179 and previous config saved to /var/cache/conftool/dbconfig/20220418-213057-ladsgroup.json
21:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P25178 and previous config saved to /var/cache/conftool/dbconfig/20220418-213037-ladsgroup.json
21:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
21:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
21:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
21:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
21:16 mutante: mw2382 - iptables -Z INPUT 151 (zero'ing iptables rule for jobrunners, want to confirm for https://gerrit.wikimedia.org/r/c/operations/puppet/+//5/modules/profile/manifests/mediawiki/jobrunner.pp)
21:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P25177 and previous config saved to /var/cache/conftool/dbconfig/20220418-211552-ladsgroup.json
21:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
21:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
21:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
21:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
21:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
21:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
21:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25176 and previous config saved to /var/cache/conftool/dbconfig/20220418-210124-ladsgroup.json
21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25175 and previous config saved to /var/cache/conftool/dbconfig/20220418-210047-ladsgroup.json
20:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
20:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
20:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25174 and previous config saved to /var/cache/conftool/dbconfig/20220418-205021-ladsgroup.json
20:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25173 and previous config saved to /var/cache/conftool/dbconfig/20220418-204619-ladsgroup.json
20:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25172 and previous config saved to /var/cache/conftool/dbconfig/20220418-203755-ladsgroup.json
20:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
20:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
20:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25171 and previous config saved to /var/cache/conftool/dbconfig/20220418-203516-ladsgroup.json
20:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25170 and previous config saved to /var/cache/conftool/dbconfig/20220418-203114-ladsgroup.json
20:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
20:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
20:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25169 and previous config saved to /var/cache/conftool/dbconfig/20220418-202855-ladsgroup.json
20:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25168 and previous config saved to /var/cache/conftool/dbconfig/20220418-202011-ladsgroup.json
20:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25167 and previous config saved to /var/cache/conftool/dbconfig/20220418-201609-ladsgroup.json
20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25166 and previous config saved to /var/cache/conftool/dbconfig/20220418-201350-ladsgroup.json
20:10 urbanecm: UTC late backport window done
20:09 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 0efb2b2: Add WikiEditor Realtime Preview to BetaFeatures (T304596) (duration: 00m 51s)
20:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25165 and previous config saved to /var/cache/conftool/dbconfig/20220418-200506-ladsgroup.json
20:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25164 and previous config saved to /var/cache/conftool/dbconfig/20220418-200418-ladsgroup.json
20:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
20:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
20:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25163 and previous config saved to /var/cache/conftool/dbconfig/20220418-200404-ladsgroup.json
20:02 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddb1021.eqiad.wmnet with OS buster
19:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25162 and previous config saved to /var/cache/conftool/dbconfig/20220418-195845-ladsgroup.json
19:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25161 and previous config saved to /var/cache/conftool/dbconfig/20220418-194859-ladsgroup.json
19:46 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1021.eqiad.wmnet with reason: host reimage
19:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25160 and previous config saved to /var/cache/conftool/dbconfig/20220418-194340-ladsgroup.json
19:43 razzi@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on clouddb1021.eqiad.wmnet with reason: host reimage
19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25159 and previous config saved to /var/cache/conftool/dbconfig/20220418-193354-ladsgroup.json
19:32 razzi@cumin1001: START - Cookbook sre.hosts.reimage for host clouddb1021.eqiad.wmnet with OS buster
19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25158 and previous config saved to /var/cache/conftool/dbconfig/20220418-191849-ladsgroup.json
19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25157 and previous config saved to /var/cache/conftool/dbconfig/20220418-190640-ladsgroup.json
19:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
19:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P25156 and previous config saved to /var/cache/conftool/dbconfig/20220418-190632-ladsgroup.json
19:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25155 and previous config saved to /var/cache/conftool/dbconfig/20220418-190452-ladsgroup.json
19:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
19:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
19:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25154 and previous config saved to /var/cache/conftool/dbconfig/20220418-190444-ladsgroup.json
18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P25153 and previous config saved to /var/cache/conftool/dbconfig/20220418-185126-ladsgroup.json
18:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25152 and previous config saved to /var/cache/conftool/dbconfig/20220418-184939-ladsgroup.json
18:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25151 and previous config saved to /var/cache/conftool/dbconfig/20220418-184325-ladsgroup.json
18:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
18:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
18:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25150 and previous config saved to /var/cache/conftool/dbconfig/20220418-184317-ladsgroup.json
18:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P25149 and previous config saved to /var/cache/conftool/dbconfig/20220418-183621-ladsgroup.json
18:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25148 and previous config saved to /var/cache/conftool/dbconfig/20220418-183434-ladsgroup.json
18:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P25147 and previous config saved to /var/cache/conftool/dbconfig/20220418-182812-ladsgroup.json
18:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P25146 and previous config saved to /var/cache/conftool/dbconfig/20220418-182116-ladsgroup.json
18:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25145 and previous config saved to /var/cache/conftool/dbconfig/20220418-181929-ladsgroup.json
18:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P25144 and previous config saved to /var/cache/conftool/dbconfig/20220418-181307-ladsgroup.json
17:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25143 and previous config saved to /var/cache/conftool/dbconfig/20220418-175802-ladsgroup.json
17:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25142 and previous config saved to /var/cache/conftool/dbconfig/20220418-174704-ladsgroup.json
17:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
17:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
17:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P25141 and previous config saved to /var/cache/conftool/dbconfig/20220418-174656-ladsgroup.json
17:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P25140 and previous config saved to /var/cache/conftool/dbconfig/20220418-173151-ladsgroup.json
17:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P25139 and previous config saved to /var/cache/conftool/dbconfig/20220418-172101-ladsgroup.json
17:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
17:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
17:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25138 and previous config saved to /var/cache/conftool/dbconfig/20220418-171914-ladsgroup.json
17:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
17:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
17:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P25137 and previous config saved to /var/cache/conftool/dbconfig/20220418-171906-ladsgroup.json
17:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P25136 and previous config saved to /var/cache/conftool/dbconfig/20220418-171646-ladsgroup.json
17:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
17:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
17:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P25135 and previous config saved to /var/cache/conftool/dbconfig/20220418-170401-ladsgroup.json
17:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P25134 and previous config saved to /var/cache/conftool/dbconfig/20220418-170141-ladsgroup.json
17:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
17:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
17:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
17:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
16:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
16:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
16:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P25133 and previous config saved to /var/cache/conftool/dbconfig/20220418-165139-ladsgroup.json
16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P25132 and previous config saved to /var/cache/conftool/dbconfig/20220418-165053-ladsgroup.json
16:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P25131 and previous config saved to /var/cache/conftool/dbconfig/20220418-165044-ladsgroup.json
16:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P25130 and previous config saved to /var/cache/conftool/dbconfig/20220418-164856-ladsgroup.json
16:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P25129 and previous config saved to /var/cache/conftool/dbconfig/20220418-163634-ladsgroup.json
16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P25128 and previous config saved to /var/cache/conftool/dbconfig/20220418-163539-ladsgroup.json
16:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P25127 and previous config saved to /var/cache/conftool/dbconfig/20220418-163351-ladsgroup.json
16:26 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1020.eqiad.wmnet with OS bullseye
16:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P25126 and previous config saved to /var/cache/conftool/dbconfig/20220418-162129-ladsgroup.json
16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P25125 and previous config saved to /var/cache/conftool/dbconfig/20220418-162034-ladsgroup.json
16:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P25124 and previous config saved to /var/cache/conftool/dbconfig/20220418-161732-ladsgroup.json
16:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
16:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
16:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25123 and previous config saved to /var/cache/conftool/dbconfig/20220418-161724-ladsgroup.json
16:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P25122 and previous config saved to /var/cache/conftool/dbconfig/20220418-160624-ladsgroup.json
16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P25121 and previous config saved to /var/cache/conftool/dbconfig/20220418-160529-ladsgroup.json
16:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25120 and previous config saved to /var/cache/conftool/dbconfig/20220418-160219-ladsgroup.json
16:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P25119 and previous config saved to /var/cache/conftool/dbconfig/20220418-160203-ladsgroup.json
16:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
16:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25118 and previous config saved to /var/cache/conftool/dbconfig/20220418-160155-ladsgroup.json
15:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P25116 and previous config saved to /var/cache/conftool/dbconfig/20220418-155446-ladsgroup.json
15:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
15:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
15:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P25115 and previous config saved to /var/cache/conftool/dbconfig/20220418-155438-ladsgroup.json
15:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25114 and previous config saved to /var/cache/conftool/dbconfig/20220418-154714-ladsgroup.json
15:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P25113 and previous config saved to /var/cache/conftool/dbconfig/20220418-154650-ladsgroup.json
15:40 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1020.eqiad.wmnet with reason: host reimage
15:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P25112 and previous config saved to /var/cache/conftool/dbconfig/20220418-153933-ladsgroup.json
15:37 andrew@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1020.eqiad.wmnet with reason: host reimage
15:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25111 and previous config saved to /var/cache/conftool/dbconfig/20220418-153209-ladsgroup.json
15:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P25110 and previous config saved to /var/cache/conftool/dbconfig/20220418-153144-ladsgroup.json
15:25 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1020.eqiad.wmnet with OS bullseye
15:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P25109 and previous config saved to /var/cache/conftool/dbconfig/20220418-152428-ladsgroup.json
15:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25108 and previous config saved to /var/cache/conftool/dbconfig/20220418-151639-ladsgroup.json
15:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P25107 and previous config saved to /var/cache/conftool/dbconfig/20220418-150923-ladsgroup.json
14:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P25106 and previous config saved to /var/cache/conftool/dbconfig/20220418-145842-ladsgroup.json
14:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
14:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
14:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P25105 and previous config saved to /var/cache/conftool/dbconfig/20220418-145834-ladsgroup.json
14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25104 and previous config saved to /var/cache/conftool/dbconfig/20220418-145440-ladsgroup.json
14:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
14:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25103 and previous config saved to /var/cache/conftool/dbconfig/20220418-145432-ladsgroup.json
14:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P25102 and previous config saved to /var/cache/conftool/dbconfig/20220418-144329-ladsgroup.json
14:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P25101 and previous config saved to /var/cache/conftool/dbconfig/20220418-143927-ladsgroup.json
14:34 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
14:33 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
14:31 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
14:31 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P25100 and previous config saved to /var/cache/conftool/dbconfig/20220418-142824-ladsgroup.json
14:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P25099 and previous config saved to /var/cache/conftool/dbconfig/20220418-142752-ladsgroup.json
14:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
14:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
14:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P25098 and previous config saved to /var/cache/conftool/dbconfig/20220418-142744-ladsgroup.json
14:25 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:25 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:25 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:25 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
14:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P25097 and previous config saved to /var/cache/conftool/dbconfig/20220418-142421-ladsgroup.json
14:21 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: TimedMediaHandler: Make videojs the only player on Commons (T248418) (duration: 00m 50s)
14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P25096 and previous config saved to /var/cache/conftool/dbconfig/20220418-141319-ladsgroup.json
14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25095 and previous config saved to /var/cache/conftool/dbconfig/20220418-141239-ladsgroup.json
14:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25094 and previous config saved to /var/cache/conftool/dbconfig/20220418-140914-ladsgroup.json
13:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25093 and previous config saved to /var/cache/conftool/dbconfig/20220418-135812-ladsgroup.json
13:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
13:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
13:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P25092 and previous config saved to /var/cache/conftool/dbconfig/20220418-135804-ladsgroup.json
13:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25091 and previous config saved to /var/cache/conftool/dbconfig/20220418-135734-ladsgroup.json
13:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P25090 and previous config saved to /var/cache/conftool/dbconfig/20220418-135406-ladsgroup.json
13:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
13:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
13:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
13:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
13:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
13:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
13:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
13:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
13:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25089 and previous config saved to /var/cache/conftool/dbconfig/20220418-134444-ladsgroup.json
13:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P25088 and previous config saved to /var/cache/conftool/dbconfig/20220418-134259-ladsgroup.json
13:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P25087 and previous config saved to /var/cache/conftool/dbconfig/20220418-134229-ladsgroup.json
13:29 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:29 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:29 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P25086 and previous config saved to /var/cache/conftool/dbconfig/20220418-132939-ladsgroup.json
13:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P25085 and previous config saved to /var/cache/conftool/dbconfig/20220418-132754-ladsgroup.json
13:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:24 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:24 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:24 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P25084 and previous config saved to /var/cache/conftool/dbconfig/20220418-132407-ladsgroup.json
13:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
13:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
13:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
13:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
13:22 urbanecm@deploy1002: Synchronized logos/config.yaml: c927c3a: Wikispecies: update logo to prevent being obscured (T306037; 2/2) (duration: 00m 55s)
13:21 urbanecm@deploy1002: Synchronized static/images/project-logos/: c927c3a: Wikispecies: update logo to prevent being obscured (T306037; 1/2) (duration: 00m 51s)
13:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P25083 and previous config saved to /var/cache/conftool/dbconfig/20220418-131434-ladsgroup.json
13:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:14 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P25082 and previous config saved to /var/cache/conftool/dbconfig/20220418-131249-ladsgroup.json
13:09 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: c90079a: Increase autoconfirmed threshold to 10 edits on iswiki (T306305) (duration: 00m 53s)
13:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P25081 and previous config saved to /var/cache/conftool/dbconfig/20220418-130834-ladsgroup.json
13:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
13:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
13:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P25080 and previous config saved to /var/cache/conftool/dbconfig/20220418-130826-ladsgroup.json
12:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25079 and previous config saved to /var/cache/conftool/dbconfig/20220418-125929-ladsgroup.json
12:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P25078 and previous config saved to /var/cache/conftool/dbconfig/20220418-125321-ladsgroup.json
12:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P25077 and previous config saved to /var/cache/conftool/dbconfig/20220418-123816-ladsgroup.json
12:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P25076 and previous config saved to /var/cache/conftool/dbconfig/20220418-122309-ladsgroup.json
12:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P25075 and previous config saved to /var/cache/conftool/dbconfig/20220418-121856-ladsgroup.json
12:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
12:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
12:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
12:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
12:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25074 and previous config saved to /var/cache/conftool/dbconfig/20220418-121837-ladsgroup.json
12:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25073 and previous config saved to /var/cache/conftool/dbconfig/20220418-120332-ladsgroup.json
11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25072 and previous config saved to /var/cache/conftool/dbconfig/20220418-115914-ladsgroup.json
11:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
11:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
11:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
11:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
11:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
11:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
11:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P25071 and previous config saved to /var/cache/conftool/dbconfig/20220418-114947-ladsgroup.json
11:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25070 and previous config saved to /var/cache/conftool/dbconfig/20220418-114827-ladsgroup.json
11:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P25069 and previous config saved to /var/cache/conftool/dbconfig/20220418-113442-ladsgroup.json
11:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25068 and previous config saved to /var/cache/conftool/dbconfig/20220418-113322-ladsgroup.json
11:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P25067 and previous config saved to /var/cache/conftool/dbconfig/20220418-111937-ladsgroup.json
11:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P25066 and previous config saved to /var/cache/conftool/dbconfig/20220418-110432-ladsgroup.json
10:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P25065 and previous config saved to /var/cache/conftool/dbconfig/20220418-104323-ladsgroup.json
10:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
10:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
10:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
10:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
10:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P25064 and previous config saved to /var/cache/conftool/dbconfig/20220418-104311-ladsgroup.json
10:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25063 and previous config saved to /var/cache/conftool/dbconfig/20220418-103307-ladsgroup.json
10:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
10:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
10:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P25062 and previous config saved to /var/cache/conftool/dbconfig/20220418-103259-ladsgroup.json
10:30 marostegui: dbmaint s1@eqiad T297189
10:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P25061 and previous config saved to /var/cache/conftool/dbconfig/20220418-102806-ladsgroup.json
10:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P25060 and previous config saved to /var/cache/conftool/dbconfig/20220418-101754-ladsgroup.json
10:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P25059 and previous config saved to /var/cache/conftool/dbconfig/20220418-101301-ladsgroup.json
10:06 marostegui: dbmaint s3@eqiad T306270
10:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P25058 and previous config saved to /var/cache/conftool/dbconfig/20220418-100249-ladsgroup.json
09:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P25057 and previous config saved to /var/cache/conftool/dbconfig/20220418-095756-ladsgroup.json
09:51 marostegui: dbmaint s5@eqiad T306270
09:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P25056 and previous config saved to /var/cache/conftool/dbconfig/20220418-094743-ladsgroup.json
09:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P25055 and previous config saved to /var/cache/conftool/dbconfig/20220418-094722-ladsgroup.json
09:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
09:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
09:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P25054 and previous config saved to /var/cache/conftool/dbconfig/20220418-094714-ladsgroup.json
09:45 marostegui: dbmaint s4@eqiad T306270
09:44 marostegui: dbmaint s1@eqiad T306270
09:36 marostegui: dbmaint s2@eqiad T306270
09:34 marostegui: dbmaint s6@eqiad T306270
09:34 marostegui: dbmaint s7@eqiad T306270
09:34 marostegui: dbmaint s8@eqiad T306270
09:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P25053 and previous config saved to /var/cache/conftool/dbconfig/20220418-093209-ladsgroup.json
09:29 marostegui: dbmaint s5@eqiad T306269
09:25 marostegui: dbmaint s4@eqiad T306269
09:19 marostegui: dbmaint s2@eqiad T306269
09:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P25052 and previous config saved to /var/cache/conftool/dbconfig/20220418-091704-ladsgroup.json
09:14 marostegui: dbmaint s8@eqiad T306269
09:11 marostegui: dbmaint s7@eqiad T306269
09:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P25051 and previous config saved to /var/cache/conftool/dbconfig/20220418-090159-ladsgroup.json
08:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P25050 and previous config saved to /var/cache/conftool/dbconfig/20220418-085122-ladsgroup.json
08:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
08:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
08:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P25049 and previous config saved to /var/cache/conftool/dbconfig/20220418-085114-ladsgroup.json
08:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P25048 and previous config saved to /var/cache/conftool/dbconfig/20220418-084729-ladsgroup.json
08:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
08:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
08:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P25047 and previous config saved to /var/cache/conftool/dbconfig/20220418-084721-ladsgroup.json
08:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P25046 and previous config saved to /var/cache/conftool/dbconfig/20220418-083609-ladsgroup.json
08:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P25045 and previous config saved to /var/cache/conftool/dbconfig/20220418-083216-ladsgroup.json
08:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P25044 and previous config saved to /var/cache/conftool/dbconfig/20220418-082104-ladsgroup.json
08:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P25043 and previous config saved to /var/cache/conftool/dbconfig/20220418-081711-ladsgroup.json
08:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P25042 and previous config saved to /var/cache/conftool/dbconfig/20220418-080559-ladsgroup.json
08:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P25041 and previous config saved to /var/cache/conftool/dbconfig/20220418-080206-ladsgroup.json
07:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P25040 and previous config saved to /var/cache/conftool/dbconfig/20220418-075755-ladsgroup.json
07:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
07:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
07:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
07:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
07:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P25039 and previous config saved to /var/cache/conftool/dbconfig/20220418-075742-ladsgroup.json
07:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P25038 and previous config saved to /var/cache/conftool/dbconfig/20220418-075526-ladsgroup.json
07:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
07:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
07:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25037 and previous config saved to /var/cache/conftool/dbconfig/20220418-075518-ladsgroup.json
07:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P25036 and previous config saved to /var/cache/conftool/dbconfig/20220418-074237-ladsgroup.json
07:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P25035 and previous config saved to /var/cache/conftool/dbconfig/20220418-074013-ladsgroup.json
07:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P25034 and previous config saved to /var/cache/conftool/dbconfig/20220418-072732-ladsgroup.json
07:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P25033 and previous config saved to /var/cache/conftool/dbconfig/20220418-072508-ladsgroup.json
07:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P25032 and previous config saved to /var/cache/conftool/dbconfig/20220418-071227-ladsgroup.json
07:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25031 and previous config saved to /var/cache/conftool/dbconfig/20220418-071002-ladsgroup.json
07:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P25030 and previous config saved to /var/cache/conftool/dbconfig/20220418-070814-ladsgroup.json
07:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
07:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
07:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25029 and previous config saved to /var/cache/conftool/dbconfig/20220418-070806-ladsgroup.json
06:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P25028 and previous config saved to /var/cache/conftool/dbconfig/20220418-065921-ladsgroup.json
06:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
06:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
06:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25027 and previous config saved to /var/cache/conftool/dbconfig/20220418-065913-ladsgroup.json
06:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P25026 and previous config saved to /var/cache/conftool/dbconfig/20220418-065301-ladsgroup.json
06:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25025 and previous config saved to /var/cache/conftool/dbconfig/20220418-064408-ladsgroup.json
06:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P25024 and previous config saved to /var/cache/conftool/dbconfig/20220418-063756-ladsgroup.json
06:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25023 and previous config saved to /var/cache/conftool/dbconfig/20220418-062903-ladsgroup.json
06:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25022 and previous config saved to /var/cache/conftool/dbconfig/20220418-062251-ladsgroup.json
06:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25021 and previous config saved to /var/cache/conftool/dbconfig/20220418-061358-ladsgroup.json
06:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25020 and previous config saved to /var/cache/conftool/dbconfig/20220418-061204-ladsgroup.json
06:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
06:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
06:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25019 and previous config saved to /var/cache/conftool/dbconfig/20220418-061156-ladsgroup.json
06:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25018 and previous config saved to /var/cache/conftool/dbconfig/20220418-060216-ladsgroup.json
06:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
06:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
05:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P25017 and previous config saved to /var/cache/conftool/dbconfig/20220418-055651-ladsgroup.json
05:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
05:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
05:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25016 and previous config saved to /var/cache/conftool/dbconfig/20220418-055321-ladsgroup.json
05:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P25015 and previous config saved to /var/cache/conftool/dbconfig/20220418-054146-ladsgroup.json
05:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P25014 and previous config saved to /var/cache/conftool/dbconfig/20220418-053816-ladsgroup.json
05:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25013 and previous config saved to /var/cache/conftool/dbconfig/20220418-052641-ladsgroup.json
05:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P25012 and previous config saved to /var/cache/conftool/dbconfig/20220418-052311-ladsgroup.json
05:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25011 and previous config saved to /var/cache/conftool/dbconfig/20220418-051448-ladsgroup.json
05:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
05:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
05:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P25010 and previous config saved to /var/cache/conftool/dbconfig/20220418-051440-ladsgroup.json
05:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25009 and previous config saved to /var/cache/conftool/dbconfig/20220418-050806-ladsgroup.json
04:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P25008 and previous config saved to /var/cache/conftool/dbconfig/20220418-045935-ladsgroup.json
04:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25007 and previous config saved to /var/cache/conftool/dbconfig/20220418-044735-ladsgroup.json
04:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
04:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
04:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P25006 and previous config saved to /var/cache/conftool/dbconfig/20220418-044726-ladsgroup.json
04:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P25005 and previous config saved to /var/cache/conftool/dbconfig/20220418-044430-ladsgroup.json
04:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P25004 and previous config saved to /var/cache/conftool/dbconfig/20220418-043221-ladsgroup.json
04:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P25003 and previous config saved to /var/cache/conftool/dbconfig/20220418-042925-ladsgroup.json
04:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P25002 and previous config saved to /var/cache/conftool/dbconfig/20220418-042505-ladsgroup.json
04:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
04:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
04:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P25001 and previous config saved to /var/cache/conftool/dbconfig/20220418-041716-ladsgroup.json
04:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
04:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
04:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
04:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
04:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
04:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
04:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P25000 and previous config saved to /var/cache/conftool/dbconfig/20220418-040211-ladsgroup.json
03:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
03:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
03:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24999 and previous config saved to /var/cache/conftool/dbconfig/20220418-035551-ladsgroup.json
03:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24998 and previous config saved to /var/cache/conftool/dbconfig/20220418-035134-ladsgroup.json
03:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
03:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
03:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24997 and previous config saved to /var/cache/conftool/dbconfig/20220418-035126-ladsgroup.json
03:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24996 and previous config saved to /var/cache/conftool/dbconfig/20220418-034046-ladsgroup.json
03:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P24995 and previous config saved to /var/cache/conftool/dbconfig/20220418-033621-ladsgroup.json
03:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24994 and previous config saved to /var/cache/conftool/dbconfig/20220418-032541-ladsgroup.json
03:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P24993 and previous config saved to /var/cache/conftool/dbconfig/20220418-032116-ladsgroup.json
03:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24992 and previous config saved to /var/cache/conftool/dbconfig/20220418-031036-ladsgroup.json
03:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24991 and previous config saved to /var/cache/conftool/dbconfig/20220418-030610-ladsgroup.json
02:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24990 and previous config saved to /var/cache/conftool/dbconfig/20220418-025515-ladsgroup.json
02:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
02:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
02:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
02:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
02:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
02:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
02:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P24989 and previous config saved to /var/cache/conftool/dbconfig/20220418-023707-ladsgroup.json
02:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P24988 and previous config saved to /var/cache/conftool/dbconfig/20220418-022202-ladsgroup.json
02:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24987 and previous config saved to /var/cache/conftool/dbconfig/20220418-021021-ladsgroup.json
02:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
02:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
02:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24986 and previous config saved to /var/cache/conftool/dbconfig/20220418-021013-ladsgroup.json
02:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P24985 and previous config saved to /var/cache/conftool/dbconfig/20220418-020657-ladsgroup.json
01:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24984 and previous config saved to /var/cache/conftool/dbconfig/20220418-015508-ladsgroup.json
01:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P24983 and previous config saved to /var/cache/conftool/dbconfig/20220418-015152-ladsgroup.json
01:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24982 and previous config saved to /var/cache/conftool/dbconfig/20220418-014003-ladsgroup.json
01:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24981 and previous config saved to /var/cache/conftool/dbconfig/20220418-012458-ladsgroup.json
00:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P24980 and previous config saved to /var/cache/conftool/dbconfig/20220418-005138-ladsgroup.json
00:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
00:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
00:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
00:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
00:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
00:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
00:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24979 and previous config saved to /var/cache/conftool/dbconfig/20220418-003411-ladsgroup.json
00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24978 and previous config saved to /var/cache/conftool/dbconfig/20220418-002443-ladsgroup.json
00:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
00:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
00:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P24977 and previous config saved to /var/cache/conftool/dbconfig/20220418-001906-ladsgroup.json
00:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
00:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
00:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
00:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
00:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
00:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
00:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P24976 and previous config saved to /var/cache/conftool/dbconfig/20220418-000401-ladsgroup.json

2022-04-17

23:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
23:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
23:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24975 and previous config saved to /var/cache/conftool/dbconfig/20220417-235506-ladsgroup.json
23:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24974 and previous config saved to /var/cache/conftool/dbconfig/20220417-234856-ladsgroup.json
23:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24973 and previous config saved to /var/cache/conftool/dbconfig/20220417-234001-ladsgroup.json
23:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24972 and previous config saved to /var/cache/conftool/dbconfig/20220417-233747-ladsgroup.json
23:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
23:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
23:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24971 and previous config saved to /var/cache/conftool/dbconfig/20220417-233739-ladsgroup.json
23:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24970 and previous config saved to /var/cache/conftool/dbconfig/20220417-232456-ladsgroup.json
23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P24969 and previous config saved to /var/cache/conftool/dbconfig/20220417-232234-ladsgroup.json
23:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24968 and previous config saved to /var/cache/conftool/dbconfig/20220417-230951-ladsgroup.json
23:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P24967 and previous config saved to /var/cache/conftool/dbconfig/20220417-230729-ladsgroup.json
23:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24966 and previous config saved to /var/cache/conftool/dbconfig/20220417-230331-ladsgroup.json
23:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
23:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
23:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24965 and previous config saved to /var/cache/conftool/dbconfig/20220417-230323-ladsgroup.json
22:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24964 and previous config saved to /var/cache/conftool/dbconfig/20220417-225224-ladsgroup.json
22:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24963 and previous config saved to /var/cache/conftool/dbconfig/20220417-224818-ladsgroup.json
22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24962 and previous config saved to /var/cache/conftool/dbconfig/20220417-224045-ladsgroup.json
22:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
22:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24961 and previous config saved to /var/cache/conftool/dbconfig/20220417-224037-ladsgroup.json
22:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24960 and previous config saved to /var/cache/conftool/dbconfig/20220417-223313-ladsgroup.json
22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P24959 and previous config saved to /var/cache/conftool/dbconfig/20220417-222532-ladsgroup.json
22:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24958 and previous config saved to /var/cache/conftool/dbconfig/20220417-221808-ladsgroup.json
22:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P24957 and previous config saved to /var/cache/conftool/dbconfig/20220417-221026-ladsgroup.json
22:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24956 and previous config saved to /var/cache/conftool/dbconfig/20220417-220605-ladsgroup.json
22:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
22:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
22:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24955 and previous config saved to /var/cache/conftool/dbconfig/20220417-220557-ladsgroup.json
21:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24954 and previous config saved to /var/cache/conftool/dbconfig/20220417-215521-ladsgroup.json
21:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24953 and previous config saved to /var/cache/conftool/dbconfig/20220417-215052-ladsgroup.json
21:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24952 and previous config saved to /var/cache/conftool/dbconfig/20220417-214048-ladsgroup.json
21:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
21:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
21:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24951 and previous config saved to /var/cache/conftool/dbconfig/20220417-214040-ladsgroup.json
21:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24950 and previous config saved to /var/cache/conftool/dbconfig/20220417-213547-ladsgroup.json
21:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P24949 and previous config saved to /var/cache/conftool/dbconfig/20220417-212535-ladsgroup.json
21:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24948 and previous config saved to /var/cache/conftool/dbconfig/20220417-212042-ladsgroup.json
21:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P24947 and previous config saved to /var/cache/conftool/dbconfig/20220417-211029-ladsgroup.json
21:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24946 and previous config saved to /var/cache/conftool/dbconfig/20220417-210856-ladsgroup.json
21:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
21:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
21:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24945 and previous config saved to /var/cache/conftool/dbconfig/20220417-210848-ladsgroup.json
20:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24944 and previous config saved to /var/cache/conftool/dbconfig/20220417-205524-ladsgroup.json
20:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24943 and previous config saved to /var/cache/conftool/dbconfig/20220417-205343-ladsgroup.json
20:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24942 and previous config saved to /var/cache/conftool/dbconfig/20220417-204447-ladsgroup.json
20:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
20:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
20:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24941 and previous config saved to /var/cache/conftool/dbconfig/20220417-204439-ladsgroup.json
20:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24940 and previous config saved to /var/cache/conftool/dbconfig/20220417-203838-ladsgroup.json
20:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P24939 and previous config saved to /var/cache/conftool/dbconfig/20220417-202934-ladsgroup.json
20:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24938 and previous config saved to /var/cache/conftool/dbconfig/20220417-202333-ladsgroup.json
20:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24937 and previous config saved to /var/cache/conftool/dbconfig/20220417-201918-ladsgroup.json
20:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
20:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
20:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24936 and previous config saved to /var/cache/conftool/dbconfig/20220417-201910-ladsgroup.json
20:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P24935 and previous config saved to /var/cache/conftool/dbconfig/20220417-201429-ladsgroup.json
20:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24934 and previous config saved to /var/cache/conftool/dbconfig/20220417-200405-ladsgroup.json
19:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24933 and previous config saved to /var/cache/conftool/dbconfig/20220417-195924-ladsgroup.json
19:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24932 and previous config saved to /var/cache/conftool/dbconfig/20220417-194900-ladsgroup.json
19:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24931 and previous config saved to /var/cache/conftool/dbconfig/20220417-194829-ladsgroup.json
19:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
19:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
19:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24930 and previous config saved to /var/cache/conftool/dbconfig/20220417-194821-ladsgroup.json
19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24929 and previous config saved to /var/cache/conftool/dbconfig/20220417-193355-ladsgroup.json
19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P24928 and previous config saved to /var/cache/conftool/dbconfig/20220417-193316-ladsgroup.json
19:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24927 and previous config saved to /var/cache/conftool/dbconfig/20220417-192942-ladsgroup.json
19:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
19:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
19:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
19:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
19:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24926 and previous config saved to /var/cache/conftool/dbconfig/20220417-192923-ladsgroup.json
19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P24925 and previous config saved to /var/cache/conftool/dbconfig/20220417-191811-ladsgroup.json
19:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24924 and previous config saved to /var/cache/conftool/dbconfig/20220417-191418-ladsgroup.json
19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24923 and previous config saved to /var/cache/conftool/dbconfig/20220417-190306-ladsgroup.json
18:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24922 and previous config saved to /var/cache/conftool/dbconfig/20220417-185913-ladsgroup.json
18:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24921 and previous config saved to /var/cache/conftool/dbconfig/20220417-185216-ladsgroup.json
18:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
18:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
18:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24920 and previous config saved to /var/cache/conftool/dbconfig/20220417-185208-ladsgroup.json
18:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24919 and previous config saved to /var/cache/conftool/dbconfig/20220417-184408-ladsgroup.json
18:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P24918 and previous config saved to /var/cache/conftool/dbconfig/20220417-183703-ladsgroup.json
18:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P24917 and previous config saved to /var/cache/conftool/dbconfig/20220417-182158-ladsgroup.json
18:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24916 and previous config saved to /var/cache/conftool/dbconfig/20220417-180653-ladsgroup.json
17:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24915 and previous config saved to /var/cache/conftool/dbconfig/20220417-175515-ladsgroup.json
17:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
17:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
17:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24914 and previous config saved to /var/cache/conftool/dbconfig/20220417-175507-ladsgroup.json
17:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24913 and previous config saved to /var/cache/conftool/dbconfig/20220417-174353-ladsgroup.json
17:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
17:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
17:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24912 and previous config saved to /var/cache/conftool/dbconfig/20220417-174345-ladsgroup.json
17:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P24911 and previous config saved to /var/cache/conftool/dbconfig/20220417-174002-ladsgroup.json
17:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24910 and previous config saved to /var/cache/conftool/dbconfig/20220417-172840-ladsgroup.json
17:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P24909 and previous config saved to /var/cache/conftool/dbconfig/20220417-172457-ladsgroup.json
17:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24908 and previous config saved to /var/cache/conftool/dbconfig/20220417-171335-ladsgroup.json
17:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24907 and previous config saved to /var/cache/conftool/dbconfig/20220417-170952-ladsgroup.json
16:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24906 and previous config saved to /var/cache/conftool/dbconfig/20220417-165909-ladsgroup.json
16:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
16:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
16:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24905 and previous config saved to /var/cache/conftool/dbconfig/20220417-165901-ladsgroup.json
16:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24904 and previous config saved to /var/cache/conftool/dbconfig/20220417-165830-ladsgroup.json
16:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P24903 and previous config saved to /var/cache/conftool/dbconfig/20220417-164356-ladsgroup.json
16:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P24902 and previous config saved to /var/cache/conftool/dbconfig/20220417-162851-ladsgroup.json
16:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24901 and previous config saved to /var/cache/conftool/dbconfig/20220417-161346-ladsgroup.json
16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24900 and previous config saved to /var/cache/conftool/dbconfig/20220417-160146-ladsgroup.json
16:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
16:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
16:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
16:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
15:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24899 and previous config saved to /var/cache/conftool/dbconfig/20220417-155816-ladsgroup.json
15:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
15:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
15:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24898 and previous config saved to /var/cache/conftool/dbconfig/20220417-155808-ladsgroup.json
15:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
15:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
15:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
15:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
15:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
15:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P24897 and previous config saved to /var/cache/conftool/dbconfig/20220417-154356-ladsgroup.json
15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24896 and previous config saved to /var/cache/conftool/dbconfig/20220417-154303-ladsgroup.json
15:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P24895 and previous config saved to /var/cache/conftool/dbconfig/20220417-152851-ladsgroup.json
15:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24894 and previous config saved to /var/cache/conftool/dbconfig/20220417-152758-ladsgroup.json
15:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P24893 and previous config saved to /var/cache/conftool/dbconfig/20220417-152738-ladsgroup.json
15:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P24892 and previous config saved to /var/cache/conftool/dbconfig/20220417-151346-ladsgroup.json
15:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24891 and previous config saved to /var/cache/conftool/dbconfig/20220417-151253-ladsgroup.json
15:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P24890 and previous config saved to /var/cache/conftool/dbconfig/20220417-151233-ladsgroup.json
14:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P24889 and previous config saved to /var/cache/conftool/dbconfig/20220417-145841-ladsgroup.json
14:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24888 and previous config saved to /var/cache/conftool/dbconfig/20220417-145734-ladsgroup.json
14:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
14:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P24887 and previous config saved to /var/cache/conftool/dbconfig/20220417-145728-ladsgroup.json
14:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
14:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
14:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
14:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24886 and previous config saved to /var/cache/conftool/dbconfig/20220417-145722-ladsgroup.json
14:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P24885 and previous config saved to /var/cache/conftool/dbconfig/20220417-144223-ladsgroup.json
14:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24884 and previous config saved to /var/cache/conftool/dbconfig/20220417-144217-ladsgroup.json
14:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24883 and previous config saved to /var/cache/conftool/dbconfig/20220417-142712-ladsgroup.json
14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P24882 and previous config saved to /var/cache/conftool/dbconfig/20220417-142316-ladsgroup.json
14:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
14:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
14:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
14:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24881 and previous config saved to /var/cache/conftool/dbconfig/20220417-141206-ladsgroup.json
14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24880 and previous config saved to /var/cache/conftool/dbconfig/20220417-140754-ladsgroup.json
14:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
14:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24879 and previous config saved to /var/cache/conftool/dbconfig/20220417-140746-ladsgroup.json
13:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P24878 and previous config saved to /var/cache/conftool/dbconfig/20220417-135827-ladsgroup.json
13:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
13:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
13:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24877 and previous config saved to /var/cache/conftool/dbconfig/20220417-135241-ladsgroup.json
13:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
13:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
13:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
13:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
13:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
13:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
13:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24876 and previous config saved to /var/cache/conftool/dbconfig/20220417-133901-ladsgroup.json
13:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24875 and previous config saved to /var/cache/conftool/dbconfig/20220417-133736-ladsgroup.json
13:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P24874 and previous config saved to /var/cache/conftool/dbconfig/20220417-132356-ladsgroup.json
13:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24873 and previous config saved to /var/cache/conftool/dbconfig/20220417-132230-ladsgroup.json
13:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24872 and previous config saved to /var/cache/conftool/dbconfig/20220417-131143-ladsgroup.json
13:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
13:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
13:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24871 and previous config saved to /var/cache/conftool/dbconfig/20220417-131135-ladsgroup.json
13:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P24870 and previous config saved to /var/cache/conftool/dbconfig/20220417-130851-ladsgroup.json
12:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24869 and previous config saved to /var/cache/conftool/dbconfig/20220417-125630-ladsgroup.json
12:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24868 and previous config saved to /var/cache/conftool/dbconfig/20220417-125346-ladsgroup.json
12:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24867 and previous config saved to /var/cache/conftool/dbconfig/20220417-124125-ladsgroup.json
12:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24866 and previous config saved to /var/cache/conftool/dbconfig/20220417-124109-ladsgroup.json
12:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
12:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
12:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
12:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
12:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24865 and previous config saved to /var/cache/conftool/dbconfig/20220417-124056-ladsgroup.json
12:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24864 and previous config saved to /var/cache/conftool/dbconfig/20220417-122619-ladsgroup.json
12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P24863 and previous config saved to /var/cache/conftool/dbconfig/20220417-122551-ladsgroup.json
12:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24862 and previous config saved to /var/cache/conftool/dbconfig/20220417-121417-ladsgroup.json
12:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
12:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
12:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24861 and previous config saved to /var/cache/conftool/dbconfig/20220417-121409-ladsgroup.json
12:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P24860 and previous config saved to /var/cache/conftool/dbconfig/20220417-121046-ladsgroup.json
11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24859 and previous config saved to /var/cache/conftool/dbconfig/20220417-115904-ladsgroup.json
11:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24858 and previous config saved to /var/cache/conftool/dbconfig/20220417-115541-ladsgroup.json
11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24857 and previous config saved to /var/cache/conftool/dbconfig/20220417-114419-ladsgroup.json
11:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
11:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24856 and previous config saved to /var/cache/conftool/dbconfig/20220417-114411-ladsgroup.json
11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24855 and previous config saved to /var/cache/conftool/dbconfig/20220417-114359-ladsgroup.json
11:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P24854 and previous config saved to /var/cache/conftool/dbconfig/20220417-112905-ladsgroup.json
11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24853 and previous config saved to /var/cache/conftool/dbconfig/20220417-112854-ladsgroup.json
11:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24852 and previous config saved to /var/cache/conftool/dbconfig/20220417-112432-ladsgroup.json
11:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
11:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
11:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
11:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
11:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P24851 and previous config saved to /var/cache/conftool/dbconfig/20220417-111400-ladsgroup.json
11:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
11:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
11:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
11:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24850 and previous config saved to /var/cache/conftool/dbconfig/20220417-105855-ladsgroup.json
10:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
10:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24849 and previous config saved to /var/cache/conftool/dbconfig/20220417-105534-ladsgroup.json
10:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24848 and previous config saved to /var/cache/conftool/dbconfig/20220417-104727-ladsgroup.json
10:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
10:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
10:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24847 and previous config saved to /var/cache/conftool/dbconfig/20220417-104718-ladsgroup.json
10:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24846 and previous config saved to /var/cache/conftool/dbconfig/20220417-104029-ladsgroup.json
10:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P24845 and previous config saved to /var/cache/conftool/dbconfig/20220417-103213-ladsgroup.json
10:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24844 and previous config saved to /var/cache/conftool/dbconfig/20220417-102524-ladsgroup.json
10:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P24843 and previous config saved to /var/cache/conftool/dbconfig/20220417-101708-ladsgroup.json
10:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24842 and previous config saved to /var/cache/conftool/dbconfig/20220417-101019-ladsgroup.json
10:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24841 and previous config saved to /var/cache/conftool/dbconfig/20220417-100203-ladsgroup.json
09:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24840 and previous config saved to /var/cache/conftool/dbconfig/20220417-094937-ladsgroup.json
09:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
09:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
09:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24839 and previous config saved to /var/cache/conftool/dbconfig/20220417-094929-ladsgroup.json
09:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P24838 and previous config saved to /var/cache/conftool/dbconfig/20220417-093424-ladsgroup.json
09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P24837 and previous config saved to /var/cache/conftool/dbconfig/20220417-091919-ladsgroup.json
09:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24836 and previous config saved to /var/cache/conftool/dbconfig/20220417-091002-ladsgroup.json
09:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
09:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
09:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24835 and previous config saved to /var/cache/conftool/dbconfig/20220417-090954-ladsgroup.json
09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24834 and previous config saved to /var/cache/conftool/dbconfig/20220417-090414-ladsgroup.json
08:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24833 and previous config saved to /var/cache/conftool/dbconfig/20220417-085449-ladsgroup.json
08:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24832 and previous config saved to /var/cache/conftool/dbconfig/20220417-085239-ladsgroup.json
08:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
08:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
08:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24831 and previous config saved to /var/cache/conftool/dbconfig/20220417-085231-ladsgroup.json
08:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24830 and previous config saved to /var/cache/conftool/dbconfig/20220417-083944-ladsgroup.json
08:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P24829 and previous config saved to /var/cache/conftool/dbconfig/20220417-083725-ladsgroup.json
08:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24828 and previous config saved to /var/cache/conftool/dbconfig/20220417-082439-ladsgroup.json
08:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P24827 and previous config saved to /var/cache/conftool/dbconfig/20220417-082220-ladsgroup.json
08:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24826 and previous config saved to /var/cache/conftool/dbconfig/20220417-080715-ladsgroup.json
07:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24825 and previous config saved to /var/cache/conftool/dbconfig/20220417-075601-ladsgroup.json
07:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
07:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
07:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24824 and previous config saved to /var/cache/conftool/dbconfig/20220417-075553-ladsgroup.json
07:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P24823 and previous config saved to /var/cache/conftool/dbconfig/20220417-074048-ladsgroup.json
07:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P24822 and previous config saved to /var/cache/conftool/dbconfig/20220417-072543-ladsgroup.json
07:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24821 and previous config saved to /var/cache/conftool/dbconfig/20220417-072425-ladsgroup.json
07:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
07:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
07:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
07:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
07:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24820 and previous config saved to /var/cache/conftool/dbconfig/20220417-071038-ladsgroup.json
07:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
07:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
07:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
07:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
07:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24819 and previous config saved to /var/cache/conftool/dbconfig/20220417-070037-ladsgroup.json
07:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
07:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
07:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24818 and previous config saved to /var/cache/conftool/dbconfig/20220417-070029-ladsgroup.json
06:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
06:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
06:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24817 and previous config saved to /var/cache/conftool/dbconfig/20220417-065532-ladsgroup.json
06:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P24816 and previous config saved to /var/cache/conftool/dbconfig/20220417-064524-ladsgroup.json
06:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24815 and previous config saved to /var/cache/conftool/dbconfig/20220417-064027-ladsgroup.json
06:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P24814 and previous config saved to /var/cache/conftool/dbconfig/20220417-063019-ladsgroup.json
06:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24813 and previous config saved to /var/cache/conftool/dbconfig/20220417-062522-ladsgroup.json
06:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24812 and previous config saved to /var/cache/conftool/dbconfig/20220417-061514-ladsgroup.json
06:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24811 and previous config saved to /var/cache/conftool/dbconfig/20220417-061017-ladsgroup.json
06:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24810 and previous config saved to /var/cache/conftool/dbconfig/20220417-060600-ladsgroup.json
06:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
06:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
06:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24809 and previous config saved to /var/cache/conftool/dbconfig/20220417-060552-ladsgroup.json
06:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24808 and previous config saved to /var/cache/conftool/dbconfig/20220417-060354-ladsgroup.json
06:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
06:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
06:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24807 and previous config saved to /var/cache/conftool/dbconfig/20220417-060346-ladsgroup.json
05:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24806 and previous config saved to /var/cache/conftool/dbconfig/20220417-055047-ladsgroup.json
05:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P24805 and previous config saved to /var/cache/conftool/dbconfig/20220417-054841-ladsgroup.json
05:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24804 and previous config saved to /var/cache/conftool/dbconfig/20220417-053542-ladsgroup.json
05:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P24803 and previous config saved to /var/cache/conftool/dbconfig/20220417-053336-ladsgroup.json
05:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24802 and previous config saved to /var/cache/conftool/dbconfig/20220417-052037-ladsgroup.json
05:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24801 and previous config saved to /var/cache/conftool/dbconfig/20220417-051831-ladsgroup.json
05:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24800 and previous config saved to /var/cache/conftool/dbconfig/20220417-050652-ladsgroup.json
05:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
05:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
05:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24799 and previous config saved to /var/cache/conftool/dbconfig/20220417-050644-ladsgroup.json
05:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24798 and previous config saved to /var/cache/conftool/dbconfig/20220417-050553-ladsgroup.json
05:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
05:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
04:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
04:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
04:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24797 and previous config saved to /var/cache/conftool/dbconfig/20220417-045139-ladsgroup.json
04:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
04:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
04:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
04:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
04:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24796 and previous config saved to /var/cache/conftool/dbconfig/20220417-043634-ladsgroup.json
04:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
04:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
04:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24795 and previous config saved to /var/cache/conftool/dbconfig/20220417-042815-ladsgroup.json
04:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24794 and previous config saved to /var/cache/conftool/dbconfig/20220417-042129-ladsgroup.json
04:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P24793 and previous config saved to /var/cache/conftool/dbconfig/20220417-041310-ladsgroup.json
04:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24792 and previous config saved to /var/cache/conftool/dbconfig/20220417-040956-ladsgroup.json
04:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
04:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
04:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24791 and previous config saved to /var/cache/conftool/dbconfig/20220417-040948-ladsgroup.json
03:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P24790 and previous config saved to /var/cache/conftool/dbconfig/20220417-035805-ladsgroup.json
03:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24789 and previous config saved to /var/cache/conftool/dbconfig/20220417-035443-ladsgroup.json
03:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24788 and previous config saved to /var/cache/conftool/dbconfig/20220417-034300-ladsgroup.json
03:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24787 and previous config saved to /var/cache/conftool/dbconfig/20220417-033938-ladsgroup.json
03:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24786 and previous config saved to /var/cache/conftool/dbconfig/20220417-033104-ladsgroup.json
03:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
03:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
03:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24785 and previous config saved to /var/cache/conftool/dbconfig/20220417-033056-ladsgroup.json
03:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24784 and previous config saved to /var/cache/conftool/dbconfig/20220417-032433-ladsgroup.json
03:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P24783 and previous config saved to /var/cache/conftool/dbconfig/20220417-031551-ladsgroup.json
03:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24782 and previous config saved to /var/cache/conftool/dbconfig/20220417-031117-ladsgroup.json
03:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
03:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
03:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24781 and previous config saved to /var/cache/conftool/dbconfig/20220417-031109-ladsgroup.json
03:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P24780 and previous config saved to /var/cache/conftool/dbconfig/20220417-030045-ladsgroup.json
02:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24779 and previous config saved to /var/cache/conftool/dbconfig/20220417-025604-ladsgroup.json
02:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24778 and previous config saved to /var/cache/conftool/dbconfig/20220417-024540-ladsgroup.json
02:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24777 and previous config saved to /var/cache/conftool/dbconfig/20220417-024059-ladsgroup.json
02:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24776 and previous config saved to /var/cache/conftool/dbconfig/20220417-023403-ladsgroup.json
02:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
02:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
02:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24775 and previous config saved to /var/cache/conftool/dbconfig/20220417-023354-ladsgroup.json
02:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24774 and previous config saved to /var/cache/conftool/dbconfig/20220417-022554-ladsgroup.json
02:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24773 and previous config saved to /var/cache/conftool/dbconfig/20220417-022143-ladsgroup.json
02:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
02:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
02:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
02:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
02:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24772 and previous config saved to /var/cache/conftool/dbconfig/20220417-022124-ladsgroup.json
02:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P24771 and previous config saved to /var/cache/conftool/dbconfig/20220417-021849-ladsgroup.json
02:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24770 and previous config saved to /var/cache/conftool/dbconfig/20220417-020619-ladsgroup.json
02:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P24769 and previous config saved to /var/cache/conftool/dbconfig/20220417-020344-ladsgroup.json
01:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24768 and previous config saved to /var/cache/conftool/dbconfig/20220417-015114-ladsgroup.json
01:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24767 and previous config saved to /var/cache/conftool/dbconfig/20220417-014839-ladsgroup.json
01:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24766 and previous config saved to /var/cache/conftool/dbconfig/20220417-013713-ladsgroup.json
01:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
01:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
01:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24765 and previous config saved to /var/cache/conftool/dbconfig/20220417-013705-ladsgroup.json
01:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24764 and previous config saved to /var/cache/conftool/dbconfig/20220417-013609-ladsgroup.json
01:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P24763 and previous config saved to /var/cache/conftool/dbconfig/20220417-012200-ladsgroup.json
01:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P24762 and previous config saved to /var/cache/conftool/dbconfig/20220417-010655-ladsgroup.json
00:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24761 and previous config saved to /var/cache/conftool/dbconfig/20220417-005150-ladsgroup.json
00:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24760 and previous config saved to /var/cache/conftool/dbconfig/20220417-004013-ladsgroup.json
00:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
00:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
00:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24759 and previous config saved to /var/cache/conftool/dbconfig/20220417-004004-ladsgroup.json
00:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24758 and previous config saved to /var/cache/conftool/dbconfig/20220417-003554-ladsgroup.json
00:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
00:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
00:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24757 and previous config saved to /var/cache/conftool/dbconfig/20220417-003546-ladsgroup.json
00:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P24756 and previous config saved to /var/cache/conftool/dbconfig/20220417-002459-ladsgroup.json
00:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24755 and previous config saved to /var/cache/conftool/dbconfig/20220417-002041-ladsgroup.json
00:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P24754 and previous config saved to /var/cache/conftool/dbconfig/20220417-000954-ladsgroup.json
00:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24753 and previous config saved to /var/cache/conftool/dbconfig/20220417-000536-ladsgroup.json

2022-04-16

23:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24752 and previous config saved to /var/cache/conftool/dbconfig/20220416-235449-ladsgroup.json
23:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24751 and previous config saved to /var/cache/conftool/dbconfig/20220416-235031-ladsgroup.json
23:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P24750 and previous config saved to /var/cache/conftool/dbconfig/20220416-234956-ladsgroup.json
23:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24749 and previous config saved to /var/cache/conftool/dbconfig/20220416-234307-ladsgroup.json
23:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
23:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
23:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24748 and previous config saved to /var/cache/conftool/dbconfig/20220416-234259-ladsgroup.json
23:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P24747 and previous config saved to /var/cache/conftool/dbconfig/20220416-233451-ladsgroup.json
23:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P24746 and previous config saved to /var/cache/conftool/dbconfig/20220416-232754-ladsgroup.json
23:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P24745 and previous config saved to /var/cache/conftool/dbconfig/20220416-231946-ladsgroup.json
23:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P24744 and previous config saved to /var/cache/conftool/dbconfig/20220416-231249-ladsgroup.json
23:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P24743 and previous config saved to /var/cache/conftool/dbconfig/20220416-230441-ladsgroup.json
22:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24742 and previous config saved to /var/cache/conftool/dbconfig/20220416-225744-ladsgroup.json
22:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24741 and previous config saved to /var/cache/conftool/dbconfig/20220416-225017-ladsgroup.json
22:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
22:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
22:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24740 and previous config saved to /var/cache/conftool/dbconfig/20220416-225009-ladsgroup.json
22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T298565)', diff saved to https://phabricator.wikimedia.org/P24739 and previous config saved to /var/cache/conftool/dbconfig/20220416-224618-ladsgroup.json
22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24738 and previous config saved to /var/cache/conftool/dbconfig/20220416-224617-ladsgroup.json
22:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
22:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
22:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
22:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P24737 and previous config saved to /var/cache/conftool/dbconfig/20220416-224610-ladsgroup.json
22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24736 and previous config saved to /var/cache/conftool/dbconfig/20220416-224610-ladsgroup.json
22:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24735 and previous config saved to /var/cache/conftool/dbconfig/20220416-223504-ladsgroup.json
22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P24734 and previous config saved to /var/cache/conftool/dbconfig/20220416-223105-ladsgroup.json
22:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24733 and previous config saved to /var/cache/conftool/dbconfig/20220416-221958-ladsgroup.json
22:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P24732 and previous config saved to /var/cache/conftool/dbconfig/20220416-221601-ladsgroup.json
22:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P24731 and previous config saved to /var/cache/conftool/dbconfig/20220416-221600-ladsgroup.json
22:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24730 and previous config saved to /var/cache/conftool/dbconfig/20220416-220453-ladsgroup.json
22:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24729 and previous config saved to /var/cache/conftool/dbconfig/20220416-220055-ladsgroup.json
22:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24728 and previous config saved to /var/cache/conftool/dbconfig/20220416-220034-ladsgroup.json
22:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
22:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
22:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
22:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
22:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24727 and previous config saved to /var/cache/conftool/dbconfig/20220416-220021-ladsgroup.json
21:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24726 and previous config saved to /var/cache/conftool/dbconfig/20220416-214926-ladsgroup.json
21:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
21:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
21:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24725 and previous config saved to /var/cache/conftool/dbconfig/20220416-214918-ladsgroup.json
21:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24724 and previous config saved to /var/cache/conftool/dbconfig/20220416-214516-ladsgroup.json
21:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P24723 and previous config saved to /var/cache/conftool/dbconfig/20220416-214429-ladsgroup.json
21:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
21:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
21:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P24722 and previous config saved to /var/cache/conftool/dbconfig/20220416-214421-ladsgroup.json
21:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P24721 and previous config saved to /var/cache/conftool/dbconfig/20220416-213413-ladsgroup.json
21:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24720 and previous config saved to /var/cache/conftool/dbconfig/20220416-213011-ladsgroup.json
21:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P24719 and previous config saved to /var/cache/conftool/dbconfig/20220416-212916-ladsgroup.json
21:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P24718 and previous config saved to /var/cache/conftool/dbconfig/20220416-211908-ladsgroup.json
21:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24717 and previous config saved to /var/cache/conftool/dbconfig/20220416-211506-ladsgroup.json
21:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P24716 and previous config saved to /var/cache/conftool/dbconfig/20220416-211411-ladsgroup.json
21:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24715 and previous config saved to /var/cache/conftool/dbconfig/20220416-211044-ladsgroup.json
21:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
21:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
21:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24714 and previous config saved to /var/cache/conftool/dbconfig/20220416-211037-ladsgroup.json
21:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24713 and previous config saved to /var/cache/conftool/dbconfig/20220416-210403-ladsgroup.json
20:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P24712 and previous config saved to /var/cache/conftool/dbconfig/20220416-205906-ladsgroup.json
20:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24711 and previous config saved to /var/cache/conftool/dbconfig/20220416-205531-ladsgroup.json
20:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24710 and previous config saved to /var/cache/conftool/dbconfig/20220416-205234-ladsgroup.json
20:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
20:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
20:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24709 and previous config saved to /var/cache/conftool/dbconfig/20220416-205227-ladsgroup.json
20:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P24708 and previous config saved to /var/cache/conftool/dbconfig/20220416-204147-ladsgroup.json
20:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
20:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
20:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P24707 and previous config saved to /var/cache/conftool/dbconfig/20220416-204138-ladsgroup.json
20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24706 and previous config saved to /var/cache/conftool/dbconfig/20220416-204026-ladsgroup.json
20:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P24705 and previous config saved to /var/cache/conftool/dbconfig/20220416-203722-ladsgroup.json
20:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P24704 and previous config saved to /var/cache/conftool/dbconfig/20220416-202633-ladsgroup.json
20:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24703 and previous config saved to /var/cache/conftool/dbconfig/20220416-202521-ladsgroup.json
20:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P24702 and previous config saved to /var/cache/conftool/dbconfig/20220416-202217-ladsgroup.json
20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24701 and previous config saved to /var/cache/conftool/dbconfig/20220416-201323-ladsgroup.json
20:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
20:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24700 and previous config saved to /var/cache/conftool/dbconfig/20220416-201315-ladsgroup.json
20:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P24699 and previous config saved to /var/cache/conftool/dbconfig/20220416-201128-ladsgroup.json
20:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24698 and previous config saved to /var/cache/conftool/dbconfig/20220416-200711-ladsgroup.json
19:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24697 and previous config saved to /var/cache/conftool/dbconfig/20220416-195810-ladsgroup.json
19:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P24696 and previous config saved to /var/cache/conftool/dbconfig/20220416-195623-ladsgroup.json
19:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
19:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
19:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
19:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
19:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
19:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
19:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
19:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
19:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P24695 and previous config saved to /var/cache/conftool/dbconfig/20220416-194557-ladsgroup.json
19:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24694 and previous config saved to /var/cache/conftool/dbconfig/20220416-194305-ladsgroup.json
19:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P24693 and previous config saved to /var/cache/conftool/dbconfig/20220416-193901-ladsgroup.json
19:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
19:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P24692 and previous config saved to /var/cache/conftool/dbconfig/20220416-193052-ladsgroup.json
19:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24691 and previous config saved to /var/cache/conftool/dbconfig/20220416-192800-ladsgroup.json
19:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24690 and previous config saved to /var/cache/conftool/dbconfig/20220416-191602-ladsgroup.json
19:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
19:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
19:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24689 and previous config saved to /var/cache/conftool/dbconfig/20220416-191554-ladsgroup.json
19:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P24688 and previous config saved to /var/cache/conftool/dbconfig/20220416-191546-ladsgroup.json
19:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24687 and previous config saved to /var/cache/conftool/dbconfig/20220416-190049-ladsgroup.json
19:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P24686 and previous config saved to /var/cache/conftool/dbconfig/20220416-190041-ladsgroup.json
18:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
18:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24685 and previous config saved to /var/cache/conftool/dbconfig/20220416-184537-ladsgroup.json
18:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24684 and previous config saved to /var/cache/conftool/dbconfig/20220416-183032-ladsgroup.json
18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24683 and previous config saved to /var/cache/conftool/dbconfig/20220416-182606-ladsgroup.json
18:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
18:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
18:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
18:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
18:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
18:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
18:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
18:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
18:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P24682 and previous config saved to /var/cache/conftool/dbconfig/20220416-180027-ladsgroup.json
18:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
18:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
17:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
17:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
17:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
17:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
17:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
17:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
17:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
17:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
17:10 cwhite: drop deferred email to tools.libraryupgrader on mx1001
00:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P24681 and previous config saved to /var/cache/conftool/dbconfig/20220416-003538-ladsgroup.json
00:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P24680 and previous config saved to /var/cache/conftool/dbconfig/20220416-002033-ladsgroup.json
00:19 cmooney@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1071.eqiad.wmnet with OS stretch
00:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P24679 and previous config saved to /var/cache/conftool/dbconfig/20220416-000528-ladsgroup.json

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s