Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady

Toolforge Admin

• Toolforge user docs
• Toolforge admin docs
• System overview
• Reference: all WMCS-owned services

Admin tasks and tools

• List of admin tasks
• Toolforge roots and Toolforge admins
• ToolsDB
• OpenStack Browser
• Infrastructure tools

Failover

• Toolschecker
• Redis
• Prometheus

Kubernetes

• Certificates
• Deploying
• Docker-registry
• Jobs framework
• Labels
• Networking and ingress
• Pod tracing

Kubernetes services

• Jobs framework
• Build service
• Envvars service
• Harbor

Packaging

• .deb packages

Runbooks and emergency guides

• Portal:Toolforge/Admin/Runbooks
• Incident response process

Other

• Exim and Email operations

[]

The procedures in this runbook require admin permissions to complete.

The ToolforgeKubernetesCapacity alert fires when a Toolforge Kubernetes node is marked as not ready. A paging alert also fires when at least 5 nodes are marked as not ready.

Debugging

On a bastion run as your own user:

$ kubectl sudo get node
$ kubectl sudo describe node <node>

Related information

• Portal:Toolforge/Admin/Kubernetes