Incidents/20151005-MediaWiki

The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Summary

Typo in MediaWiki config change took all of the wikis down for around a minute.

Timeline

Times are approximate and based on Alex's IRC logs.

23:37:35 Alex merges 243728, to change mw:Extension:WikidataPageBanner config to include all namespaces on ruwikivoyage
- The change is not ideal, but looks harmless enough and passes automated checks (e.g. syntax)...
23:37:52 sync-file starts
23:38:09 sync-file finishes
23:38:34 First report of fatals in #wikimedia-operations from MaxSem, users begin to notice in other channels
23:38:57 Alex resets on tin, starts sync-file again
23:39:14 sync-file finishes

Conclusions

There are exactly two 'r's in array.
Adding an extra r (arrray()) will form a valid PHP function name so passes a PHP syntax check.

Actionables

Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.

mediawiki-config needs better test coverage - task T114725