XHGui/Runbook

Wikimedia infrastructure

Data centers

Networking

Global traffic routing

MediaWiki SRE

MediaWiki Engineering

Multimedia

Data Engineering

Search

SRE Data Persistence

SRE Infra Foundations

SRE Observability

• Arc Lamp
• Distributed tracing
  • Jaeger
• Logs
• MediaWiki UDP logging
• OpenSearch Dashboards (frontend)
• Logstash (backend)
• Prometheus
• performance.wikimedia.org
• Thanos
• Statsd
• Graphite
• Grafana
• Alertmanager
• Klaxon
• Wikimediastatus.net
• XHGui

Wikimedia Performance

Event Platform

Release Engineering

[]

This is the internal runbook for XHGui service.

Hosts

Hosts as of July 2023:

• webperf
• deployment-webperf21 (Beta cluster)

Provisioned by the Puppet role::webperf which installs the XHGui web app at https://performance.wikimedia.org/xhgui/ from the operations/software/xhgui repository (this is a mirror of the upstream perftools/xhgui repo).

Public web access is provided via an Apache proxy from the webperf host that serves performance.wikimedia.org.

Meta

• Public URL:
  • Production: https://performance.wikimedia.org/xhgui/
  • Beta cluster: https://performance.wikimedia.beta.wmflabs.org/xhgui/
• User documentation: XHGui
• Database: m2

Infrastructure diagram.

Upgrade XHGui

You will need:

• a local checkout of operations/software/xhgui.git, which is our mirror.
  • including a remote for upstream. git remote add upstream git@github.com:perftools/xhgui.git
  • including submodules checked out. git submodule update --init --recursive
• a local checkout of performance/docroot.git, which stores the https://performance.wikimedia.org/ website
  • including submodules checked out. git submodule update --init --recursive
• commit access to operations/software/xhgui.git and performance/* repositories (granted to Gerrit group "performance", per access).

Update our mirror

1. Check out our mirror.
2. Merge in the new release tag from upstream.
3. Push to Gerrit (fast-forward only).

$ cd xhgui/
xhgui$ git checkout wmf_deploy && git remote update && git reset --hard origin/wmf_deploy
xhgui$ git merge 0.12.0
xhgui$ git push origin HEAD:wmf_deploy

Update dependencies

To update our dependencies in performance/xhgui-vendor, follow steps in README.

Then, in your local xhgui checkout:

1. Fast-forward the vendor submodule.
2. Commit the submodule update.
3. Push to Gerrit (fast-forward only).

$ cd xhgui/

xhgui$ cd vendor/
xhgui/vendor$ git pull
xhgui/vendor$ cd ..

xhgui$ git add vendor && git commit
xhgui$ git push origin HEAD:wmf_deploy

Upgrade our install

To deploy an XHGui upgrade, update the submodule pointer in performance/docroot.git

1. Fast-forward the xhgui submodule.
2. Commit the submodule update.
3. Push to Gerrit for review.

$ cd performance.wikimedia.org
performance.wikimedia.org$ cd xhgui/
performance.wikimedia.org/xhgui$ git pull
performance.wikimedia.org/xhgui$ cd ..

performance.wikimedia.org$ git add vendor && git commit
performance.wikimedia.org$ git review

Test on Beta Cluster:

1. ssh deployment-webperf21.deployment-prep.eqiad1.wikimedia.cloud
2. Copy the "Download > Anonymous HTTP > Cherry pick" command from Gerrit for your docroot patch and run it as www-data on the beta server:
3. cd /srv/org/wikimedia/performance$ sudo -u www-data sh -c 'git fetch … && git cherry-pick FETCH_HEAD'

Deploy to production:

Once the performance/docroot patch is reviewed and merged, Puppet will automatically apply it to production webperf* hosts within 30 minutes.