Metrics Platform/MPIC/Administration

This page is currently a draft.
Material may not yet be complete, information may presently be omitted, and certain parts of the content may be subject to radical, rapid alteration. More information pertaining to this may be available on the talk page.

Metrics Platform Instrument Configuration (MPIC) is an instrument and experiment management system. The application enables product teams to set up, manage, and configure various instruments for data collection and experimentation. This includes scheduling, designating metadata, setting sample rates, and targeting deployment environments.

Service details

mpic-next (staging)

Attribute	Value
Owner	Data Products
Kubernetes Cluster	dse-k8s-eqiad
Kubernetes Namespace	mpic-next
Chart	https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/mpic/
Helmfiles	https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/dse-k8s-services/mpic-next/
Docker image	https://docker-registry.wikimedia.org/repos/data-engineering/mpic/tags/
Internal service DNS	mpic-next.svc.eqiad.wmnet
Public service URL	https://mpic-next.wikimedia.org
Logs	TBD
Metrics	https://grafana.wikimedia.org/d/ee2057f3-eb34-45a7-a48b-489e3ff0b2ec/mpic?orgId=1
Monitors	TBD
Application documentation	https://wikitech.wikimedia.org/wiki/Metrics_Platform/MPIC
Paging	false
Deployment Phabricator ticket	https://phabricator.wikimedia.org/T361335

mpic (production)

Attribute	Value
Owner	Data Products
Kubernetes Cluster	dse-k8s-eqiad
Kubernetes Namespace	mpic
Chart	https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/mpic/
Helmfiles	https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/dse-k8s-services/mpic
Docker image	https://docker-registry.wikimedia.org/repos/data-engineering/mpic/tags
Internal service DNS	mpic.svc.eqiad.wmnet
Public service URL	https://mpic.wikimedia.org
Logs	TBD
Metrics	https://grafana.wikimedia.org/d/ee2057f3-eb34-45a7-a48b-489e3ff0b2ec/mpic?orgId=1&var-service=mpic
Monitors	TBD
Application documentation	https://wikitech.wikimedia.org/wiki/Metrics_Platform/MPIC
Paging	false
Deployment Phabricator ticket	https://phabricator.wikimedia.org/T361335

Deployment

Note that:

• mpic-next is the staging environment
• mpic is the production environment

# Do the following once you have prepared the change and the change has been reviewed and merged
ssh deployment.[eqiad|codfw].wmnet
cd /srv/deployment-charts
git log -n 1
# Check that the change has been pulled
cd helmfile/dse-k8s-services/<environment>
# Check changes and deploy
helmfile -e dse-k8s-eqiad diff
helmfile -e dse-k8s-eqiad -i apply

Troubleshooting

Note that:

• mpic-next is the staging environment
• mpic is the production environment
• mpic-staging is the container name for staging environment (mpic-next)
• mpic-production is the container name for production environment (mpic)

Check pods

$ ssh deployment.[eqiad|codfw].wmnet
$ kube_env <environment> dse-k8s-eqiad
$ kubectl get pods -w
NAME                            READY   STATUS   RESTARTS   AGE
mpic-staging-57d4bbb6bf-z2p5k   1/2     Error    0          10s
mpic-staging-57d4bbb6bf-z2p5k   1/2     Error    1 (1s ago)   10s
mpic-staging-57d4bbb6bf-z2p5k   1/2     CrashLoopBackOff   1 (1s ago)   11s

See logs

$ ssh deployment.[eqiad|codfw].wmnet
$ kube_env <environment> dse-k8s-eqiad
$ kubectl get pods -w
$ kubectl logs -f <pod_name> -c <container_name>
{"level":50,"time":1714483575453,"pid":1,"hostname":"mpic-staging-66554859c4-p9m28","msg":"Error while setting an env variable. There is no value for SAL_PASSWORD"}
{"level":30,"time":1714483575592,"pid":1,"hostname":"mpic-staging-66554859c4-p9m28","msg":"Connected to the database at an-mariadb1001.eqiad.wmnet:mpic_staging"}
{"level":30,"time":1714483575600,"pid":1,"hostname":"mpic-staging-66554859c4-p9m28","msg":"mpic has started listening on port 8080"}