Performance/Runbooks/Analyze memcached
< Performance | Runbooks
memkeys
This is like "top" but for memcached traffic. Real-world examples:
- Memcached incident in 2022, T310532#8004500
- Traffic audit in 2021, T264604#7175809
- Traffic audit in 2017, T74024
# from an appserver, see local requests to the "on-host tier" instance
mw0011$ sudo memkeys -i lo -p 11210
# from an appserver, see outgoing requests to the memcached cluster
mw0011$ sudo memkeys -i lo -p 11213
# from a memcached server, see all incoming requests to that server
mc0032$ sudo memkeys -i eth0
mc0046$ sudo memkeys -i ens2f0np0
Example output:
# memcached key calls objsize req/sec
# WANCache:global:rdbms-server-readonly:deployment-db07:enwiki:ǀ#ǀv 3581 53 12.88
# WANCache:global:centralauth-user:44df0d9a65e7d3fad8664f4b2dc2ǀ#ǀv 2973 1178 10.40
# WANCache:enwiki:messages:en:hash:v1ǀ#ǀv 2954 124 10.19
Upstream:
- https://github.com/bmatheny/memkeys
- https://gitlab.wikimedia.org/repos/sre/memkeys - WMF debian package
cachedump
Inspied by @elukey via T203786#4999052.
krinkle@mc1048:~$ echo "stats cachedump 1 0" | nc -q 1 localhost 11211 > dump-01.txt echo "stats cachedump 16 0" | nc -q 1 localhost 11211 > dump-16.txt echo "stats cachedump 26 0" | nc -q 1 localhost 11211 > dump-25.txt echo "stats cachedump 32 0" | nc -q 1 localhost 11211 > dump-16.txt
$ head cache-01.txt ITEM WANCache:global:SqlBlobStore-blob:commonswiki:tt%3A429993772|#|v [756 b; 1660458645 s] ITEM WANCache:global:revision-row-1.29:frwiki:6035826:144772360|#|v [790 b; 1660264028 s] ITEM WANCache:commonswiki:MessageBlobStore:ext.visualEditor.language:nl:2b6785d…|#|v [724 b; 1660719066 s]
$ cat dump-*.txt | fgrep WANCache | cut -d " " -f 2| cut -d ':' -f3 | sort | uniq -c | sort -rn 18669 SqlBlobStore-blob 8855 filerepo-file-foreign-description 1735 preprocess-hash 596 getExtendedMetadata 254 MessageBlobStore
Upstream: