Portal:Toolforge/Admin/Runbooks/k8s-haproxy

The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

These tests are defined here: https://gerrit.wikimedia.org/g/operations/puppet/+/c85d36363c69c1d01092726af556812138475857/modules/profile/manifests/toolforge/k8s/haproxy.pp#56

These checks verify the behavior of the toolforge web proxy. There are two different endpoints tested:

https://admin.toolforge.org/healthz

This tests a known, minimal k8s webservice that when working properly returns 'ok' at the healthz route.

When this alert fires it is probably because the tool has crashed. Restart on a toolforge bastion with:

$ sudo su -
# become admin
# webservice stop  # and now wait a minute, just doing a quick restart seems insufficient
# webservice start
# tail -f error.log # should be fairly quiet

https://this-tool-does-not-exist.toolforge.org/

As explained by the endpoint, this is a check of the proxy behavior when accessing a non-existent tool. It ought fall through to the 404 handler and report a useful "this tool does not exist" message.

As of 2022-09-02 this test is known to be flaky due to the 404 handler being a bit broken. A (apparently insufficient) attempt to fix this is https://gerrit.wikimedia.org/r/c/operations/puppet/+/826779/