Swift/Preliminary test plan

The "Swift" project is Current as of 2012-04-01. Owner: Bhartshorne. See also RT:1384

This page documents a preliminary test plan to get a basic handle on how swift performs, as well as a battery of functional tests to verify swift is working correctly.

Background information

Some statistics from ms5, to be used as a starting point for the tests

• How many images per second does ms5 serve? about 110/s at peak (from ganglia
• How many requests per second does ms5 send back to the image scalers? about 50/s at peak (from ganglia
• How much storage does ms5 have? 10T, 8T used. (ms7: 27T, 18T used)
• What is the rate of disk consumption on ms5? Historic: 0.3T/mo. Current: 1.2T/mo (ms7: 1T/mo)
• How long does it take to request 1000 images on ms5 (sequentially)?
  • note - this test is performed during normal load, so it's actually 1000 images plus regular load
• How long does it take to request 1000 images on ms5 (with 20 clients)?
  • note - this test is performed during normal load, so it's actually 1000 images plus regular load

Test platforms

First iteration

three machines using high performance misc servers

• X CPU
• Y RAM
• 2 disks

two machines are configured as proxy machines, three as storage nodes (the two proxy servers are also storage nodes)

Second iteration (proposed)

separate proxy nodes from storage nodes - five high performance misc servers (two proxy, three storage)

production config (proposed)

• three proxy nodes - high performance misc
• 6 storage nodes - ES hardware (grow to 10 over 1yr)

Tests

speed

a list of 30,000 images (pulled from a 3min tcpdump off ms5) is at fenari:~ben/swift/list-of-images. This can serve as input to performance tests.

(30,000 chosen because peak load on ms5 is ~100qps == 6000qpm. 5 minutes =~ 30k.)

• How many images per second can swift serve?
• How long does it take to request 30000 images on swift (sequentially)?
• How long does it take to request 30000 images on swift (with 20 clients)?
• How does swift maintenance (rebalancing, etc) affect performance? (fail out a disk and retest 1000 image retrieval)

functional against swift directly

• upload file
• retrieve file
• rename file
• delete file

functional through mediawiki - thumbnails

• request a thumb that exists on swift
• request a thumb that doesn't exist on swift (but whose original does exist), watch it get created and stored
  • re-request, verify it comes from local disk
• request a thumb for an original image that doesn't exist, verify correct error handling
• purge the image cache, verify the image gets deleted from swift

functional through mediawiki - originals

• upload file
• upload new version of file
• revert to previous version of file
• rename file
• delete file