Jump to content

OCG/Deployments

From Wikitech
< OCG
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Planned deployments, linked from Deployments. For a list of past deployments, look for 'ocg' in Server Admin Log.

See OCG#Deploying changes to learn how to deploy a new version of OCG.

2016-10-03

ocg-collection:

phab:T147211: Add en.wiktionary.org to blacklist temporarily to prevent DoS.

Deployed new service (d0f8e126f5d1ad53358c7c7380ca02f41c33ead6):

  • phab:T147211: Allow administrators to blacklist renders of certain metabooks.

Deployed new latexer (45c30840d22a7590cee904aff5f2e272c75d3bdb):

2016-08-17

ocg-collection:

Deployed new bundler (40ba27f770480cb1642ed58ef18b986ec67b0daf):

  • Remove rest.wikimedia.org, the server is being shut down Sept 1. (phab:T133001)

Deployed new service (51d2eb6d47dfc5225d8d5bc45444e81cc233b1c7):

  • Add metrics for number of items in collections.

Deployed new texter (63333e2c2d04f00c379ac35cac9d85af7096e767):

  • Add option to emit `blessed`-style markup.

2016-05-10

ocg-collection:

Deployed new service (ac2dff089f66fd5f6d50d7e7cd046cfd5558ac59):

  • Update npm dependencies; switch to `babybird` Promise implementation.

Deployed new bundler (e33963029ff99f63a54fae78515b742375d88a3c):

  • Update npm dependencies; switch to `babybird` Promise implementation.

Deployed new latexer (544a26ae19eb0e0c0ae9bea98e24630b8ca7982b):

  • Update npm dependencies; switch to `babybird` Promise implementation.

Deployed new texter (df26683c689eb282dba55b0f287fd71307c4a2e2):

  • Update npm dependencies; switch to `babybird` Promise implementation.

Dependencies were rebuilt.

The improved Promise implementation should fix the recursive nextTick issues encountered in phab:T120079.

2016-05-01

ocg-collection:

Deployed new bundler (d1b9c6b1e73766d056a3a80773da1e2cd38bd851):

  • Fix --domainname option when bundling a collection from the command line.

Deployed new latexer (77b0b9d45bf15021a7b2e79a8bee3f528e2e5943):

  • Add sawNonSpace internal flag to avoid problems when forcing a newline.
  • Skip images inside headings.

2016-04-27

ocg-collection:

Deployed new service (fc936c8ad7893d606ff67e5fa2fe4f62e755e5fe):

  • Allow decommissioning OCG hosts. phab:T120077
  • Periodically check for configuration file changes. (Not enabled in default configuration.)
  • Improve graceful shutdown.

2016-04-25

ocg-collection:

Deployed new service (4442920b712dad67e3ca7c7bf6af4a37a794405d):

  • Update README.

Deployed new bundler (c4f4c4b212aada9646dbc113b6db352f51408f75):

  • Don't fallback to rest.wikimedia.org.
  • Use domain for status messages if prefix is not available.
  • Resolve titles even when $wgArticlePath involves a query string. phab:T124762
  • Send Accept header to Parsoid/RESTBase to get data-mw attributes inline. phab:T130685

Deployed new latexer (3ff99a570acf7b8f3a8af3c47f02b02ac5aad01c):

  • Update README.

Deployed new texter (6b6ef440cb03c16d79c76e123044ee8e48b9c6b8):

  • Use released version 0.3.3. (No code changes.)

2016-03-17

ocg-collection:

Deployed new bundler (1fec608a4d55e7bddb115122dd0db0fd4bc88570):

  • Use underscores in titles sent to the RESTBase / Parsoid API.

2016-01-26 - 2016-01-27

Originally planned for 2016-01-20, deferred to 2016-01-25, then phab:T124720 prevented actual deploy to labs until ~9am PST 2016-01-26. However, the redis server was down in labs (phab:T124791,phab:T124677). That was fixed ~10am PST. Deployed to production 2016-01-27 in our usual deploy window.

  • Deployed to BETA: 2016-01-26
  • Deployed to PRODUCTION: 2016-01-27

ocg-collection:

Rebuilt dependencies.

Deployed new service (8a9626602a907ef6c2cb825119292933245efede):

  • Update dependencies; delint code.

Deployed new bundler (6491a016302de2db86a8ee14238ce5f25707df74):

  • Use body_only instead of bodyOnly in REST v1/Parsoid v3 API.
  • New --fetch-modules option (turned off by default). phab:T114788
  • Add username to the User-Agent header.
  • Remove most references to parsoid-lb.eqiad.wikimedia.org. phab:T110474
  • Handle protocol-relative API URLs.
  • Update dependencies; delint code.

Deployed new latexer (28793564325d1f5c941ce2d3be0b6cd288ae714f):

  • Update dependencies; delint code.

Deployed new texter (73f7b86871abff978c818ce3941c52072b0e53f5):

  • Update dependencies; delint code.

2015-09-16

ocg-collection:

Deployed new bundler (f673740b2a61b0bda64b0e58f7b7e596e3e7de4d):

  • Released version 1.3.0
  • Work around bug with --domain CLI option.

Deployed new latexer (2c2727487be7af4d55c78982a8891b6dd1ee01d2):

  • Released version 0.3.1.

Deployed new texter (3d4ab2dcdb3ab39cc781b757792a1ee8419fae6d):

  • Work around bug with --domain CLI option.
  • Improve layout of preformatted content.
  • Released version 0.3.2.

2015-09-14

ocg-collection:

Deployed new bundler (8d8cd3ec79367fa086a203768971e5aaa26811a8):

  • Add User-Agent header to API requests.
  • task T110782: Add Parsoid v3 API support.
  • Remove the (broken) --api option alias.

Deployed new texter (de5a441228dc6e6e8eafd68f198ea6fa33de6049):

  • Support "standalone mode" for single-article conversion.

2015-06-23 pm

ocg-collection:

Rebuilt dependencies on the wmf-deploy branch again, this time including the required request package.

2015-06-23 am (reverted)

Attempted to deploy d7c698d5bf730d34057945e912ac75dc542dd788 to production. Found that labs machines have native Ubuntu node-* packages installed which are not installed in production. This made production fail -- the most immediate cause was that the npm request package was missing from the wmf-deploy repo (due to an issue with the npm prune step of make production), but it wasn't noticed during the beta canary deploy because deployment-pdf01 had the Ubuntu node-request package installed.

Reverted production to ca4f64852de5b1de782b292b50038fbd2dd84266 (2015-05-20 deploy).

2015-06-22 (beta only)

ocg-collection:

Deployed new bundler (fe21e6f1577f571ea83226ba04dbe0285e50e710):

  • Update rimraf dependency.
  • Update sqlite3 dependency for compatibility with node > 0.10.
  • Convert ocg-bundler to make use of the https addresses.
  • task T75130: Add "AttributionRequired" metadata to OCG bundle.
  • Parsoid v2 API uses `body` as a query parameter, not `bodyOnly`.
  • Use SPDX license.

Deployed new latexer (dd0e83952ec12d0ca675b63396424b3a4330e1e6):

  • Update sqlite3 dependency for compatibility with node > 0.10.
  • task T99908: Use Devanagari script for Bhojpuri wikipedia.
  • Use SPDX license.

Deployed new service (2941c3dc0f25c654d7ddab1b82ecb726728551e1):

  • Update hiredis to latest version.
  • Update bunyan, bunyan-syslog, and gelf-stream.
  • Update node redis to support redis 2.8.x, which has been installed in beta/labs.
  • Update rimraf and mv dependency.
  • Use SPDX license.

Deployed new texter (bb897807e5ffa62788f8036bc58d31bf78b079f5):

  • Update sqlite3 dependency for compatibility with node > 0.10.
  • Use SPDX license.

Repeat of the earlier deploy attempt, after upgrading dependencies to fix redis issue. A patch to the Collection extension ( https://gerrit.wikimedia.org/r/220069 ) was also necessary to fix labs.

2015-06-22 (cancelled)

ocg-collection:

Deployed new bundler (0d70ff85138da5cdd5ec0147f6eaba5bb8f49a91):

  • Convert ocg-bundler to make use of the https addresses
  • task T75130: Add "AttributionRequired" metadata to OCG bundle.
  • Parsoid v2 API uses `body` as a query parameter, not `bodyOnly`.
  • Use SPDX license.

Deployed new latexer (9d81c73ae4bc1555b6f513639a025c5f5f09ffd6):

  • task T99908: Use Devanagari script for Bhojpuri wikipedia.
  • Use SPDX license.

Deployed new service (65ce1bb81bff63b810a9b135811edb9b5f147e3d):

  • Use SPDX license.

Deployed new texter (cbea03456f0c53212bb7d228bee1f09fcb88686e):

  • Use SPDX license.

This deploy was cancelled after being deployed to beta, since beta ran into redis problems. Redis apparently was upgraded to 2.8.x, which made node-redis unhappy.

2015-05-20

ocg-collection:

Deployed new latexer (e08a3d09a547d19b9c4b2a630851505f1ed5146d):

  • task T92817: improve text rendering for nepali languages.

2015-05-13

ocg-collection:

Deployed new bundler (24365a9154a86c6870e89b2256ee5bc1d88f5a49):

  • Add --php-api option for local debugging.
  • Fix file revision lookup failures.
  • task T93292: Allow bundler to fetch parsed articles from RESTBase.
  • Switch from es6-shim to core-js; update prfun and jshint.

Deployed new latexer (efae1a449bfaf2fa925f1968101fafedf87edb48):

  • Switch from es6-shim to core-js; update prfun and jshint.

Deployed new service (efa690d717ae257879a4dfbdd7ecc9227b586bed):

  • Remove deprecated download command.
  • Add script to allow date-based cache cleaning.
  • Update docs and config options to allow use of RESTBase.
  • Switch from es6-shim to core-js; update prfun and jshint.

Deployed new texter (e325556133e9ece9af20631d91635574a0b6f8d0):

  • Switch from es6-shim to core-js; update prfun and jshint.

2015-03-23

ocg-collection:

Deployed new latexer (39534c0e9f0a48b6c3565fe8b50f437103e0d218):

  • task T93008: use {{DISPLAYTITLE}} or subpage title where appropriate.

2014-12-11

ocg-collection:

Deployed new service (f6f3788da146e92dce8dcb11a14479c8b117f2be):

  • Ensure we exit when `clear-host-cache` is done.
  • Don't swallow output from helper scripts unless `--quiet` is given.
  • Separate out `download` and `download_file` commands.

2014-12-03

ocg-collection:

Deployed new service (4952462b0045d0f22ce31eaa80d671e9b3dc9350):

  • Don't halt gc due to readdir after clean; more verbose error reporting.
  • Ensure that directories are initialized in GC thread.
  • Fix GC of failed jobs.
  • Fix job progress status indication.
  • Record which host picks up a job.
  • Add script to allow host decommissioning.
  • Be sure that `console.log` appears on console output.

Deployed new bundler (8565c79549e107ffa10fa817d00348b85ba9dbb9):

  • Hook up npm-travis.

2014-12-01

ocg-collection:

Deployed new service (3bcfaa00595c8b331175577286733030328d146b):

  • Limit memory usage during directory clean up.
  • Don't halt gc if a directory is bad.

2014-11-26

ocg-collection:

Deployed new service (447940eba611ef65f9d251b86c63345c26b7adb5):

  • Don't use the writer as part of the renderTempDir.
  • Validate 'writer' parameter before creating a job object.
  • Remove stringified metabook from most log messages.

2014-11-10

ocg-collection:

Deployed new service (2b2816081120a5e65f5bf80c294b2a85a1de38f8):

  • Double cache lifetime of successful renders, to 4 days.
  • Send SIGKILL if SIGTERM doesn't do the job (after execution timeout).

Deployed new bundler (a93dd0c9935cb6db56da4f6289d66adbdc0710d3):

  • Better debugging if article name resolution fails.

Deployed new latexer (c4c7f7515ea337f05c2d4d851029ba90e469a8f8):

  • Add `--helper-exec-limit` option to protect against runaway image helpers.

2014-11-03

ocg-collection:

Deployed new service (379cea3f9b3678bb83963296fdb55e5b3e8887fa):

  • Allow prevention of retries of failed jobs.
  • Update dependencies.
  • Add _version API endpoint.
  • Switch from winston to bunyan for logging.

Deployed new bundler (c9bf3d430721dfd2553a49dbe31790113916e006):

  • Don't create pipes for P.spawn (avoid blocking unexpectedly).
  • Update dependencies.

Deployed new latexer (aad4dfbab9e6d6c8d05ec82679843541dfd504ca):

  • Prevent 100% CPU hang on certain input text.
  • Don't create pipes for P.spawn (avoid blocking unexpectedly).
  • Update dependencies.

Deployed new texter (8712aeb6e6490b558a22e964ec1bdf995cfd9ca0):

  • Don't create pipes for P.spawn (avoid blocking unexpectedly).
  • Update dependencies.

2014-10-27

ocg-collection:

Deployed new service (fe36c9cf355955870c8edb8eac216821899ce970):

  • Set TMPDIR for the renderer (don't clutter /tmp).

Deployed new bundler (7851aa2d51265eefdef0cf0edf012e7b6e418c3c):

  • Suppress console output during zip (avoid a possible hang).

Deployed new latexer (bd96f76038f9df8ece71e29ea2c1fbf005605077):

  • Suppress console output during unzip (avoid a possible hang).

2014-10-22

ocg-collection:

Deployed new service (f0f6951d09e570ef114567a0ca6e3180af0553d9):

  • Force images to be 600x600 pixels maximum.

Deployed new bundler (7296508d17f6634490cf079cb0fe03c175ff4ace):

  • Fix "image size too big" check.

This was User:Arlolra's first OCG deploy.

2014-10-21

ocg-collection:

Deployed new service (6a1b6a17a4d6e2f7d4b8c6bd0d6f033d425fd886):

  • Update bug URL for the package.
  • Factor out configuration file parsing and logging setup.

Deployed new bundler (f25c1956333a124e4cffec8a3ee52a2f4c5f76a3):

  • Add ability to limit the number of simultaneous requests for small wikis (bugzilla:71895).

Deployed new latexer (dda888e5f64008c71cc37221f942a404e3a32107):

Prepared 2014-10-9, Deployed 2014-10-20

ocg-collection:

Deployed new service (c4c7fbfb743306ad0dd7dfce1475ded536f68f97):

  • Log metabook on every request.

Deployed new latexer (820334a8452762f958e3ebf1a5342d9a667a7b0a):

  • Allow lists nested nine level deep (bugzilla:71896).
  • Fix non-curly quotes (`extquotedbl` in output).
  • Don't crash when the page title is RTL.

2014-10-8

ocg-collection:

Deployed new service (160404df14fb3e92b7b5fff3121a13ee55887bfd):

  • Use inline content disposition, instead of attachment.

Deployed new bundler (71100ec7c7b86fc7ded21aa9977dd303213f4281):

  • Command-line prefix guessing tweak.

Deployed new latexer (8097c38e172a0d358694d800feacc0af075d3cc2):

2014-10-7

ocg-collection:

Deployed new service (531bfb404ce39debfc028c3c2ae72e7f1c59d661):

  • Use article title as download filename (bugzilla:71720, bugzilla:71286).
  • The rl writer should be an alias for rdf2latex (fixes "I got plaintext when I tried to download my PDF using an old link").
  • Ignore backwards-compatibility settings field in metabook.

Deployed new bundler (f55bf164145bd1390a565480a87e04cc640deba4):

Deployed new latexer (fdde609f819679eca109d6a49851bf1e70006172):

  • Support Lao, Khmer, Tibetan, Macedonian, Aramaic, and Bosnian.
  • Improve CJK support.
  • Be careful about square brackets inside \item[...].
  • Support djvu images, but only extract first page by default.
  • Use narrower margins.
  • Equations should be flush left and not indented (bugzilla:68838).

2014-10-6 (afternoon)

ocg-collection:

Deployed new service (5fad457ac6536ccf2fb88df02b31215b2d14cb7b):

  • Avoid use of redis `hkeys`; fix OOM during gc.
  • Don't stop gc'ing if a file disappears.

Deployed new bundler (efbeee40948f80532a3e423e4d1813338c206f5f):

  • Add options to limit bundle and image size. (bug 71647)

Deployed new texter (b7a64572c93f11d75cb2c029c13b2a78f74ed480):

  • Fix line wrapping.

2014-10-6 (morning)

GC fixes.

ocg-collection:

Deployed new service (6579c1b1a01e09c6d3806a74b8e3e2152c293aae):

  • Use proper extension for downloaded file (partial fix for bug 71286)
  • Log the total user-experienced render time to statsd
  • Shrink expiry for temp files and postmortem files (to 6 hours and 1 day, respectively)
  • Reduce basic cache expiry time to 2 days (from 3)
  • Bug fix: utime was setting directory times 1000 years in the future.
  • Bug fix: we were only gc'ing the first subdirectory in the output directory.

2014-10-2

Remaining parts of mwlib deconfig: https://gerrit.wikimedia.org/r/164099

2014-10-1

Turned off mwlib PDF: https://gerrit.wikimedia.org/r/163609

ocg-collection:

Deployed new service (e603a4d79badfabdcd9c2cea6ec6606b18dfd116):

  • Reduce default job lifetime from 5 days to 3 days.
  • Disable folder size computation in health check unless explicitly requested. (bug 71260)
  • Improve statistics collection and measure cache effectiveness.
  • Various improvements to the load testing scripts.

Deployed new bundler (352f9e02316b698bce4fe4da42f0791ff5df591b):

  • Use released version 1.1.0.

Deployed new latexer (0e4ec243d6087a940bbb0954697ba79c1235f772):

  • Save and restore stack of inline decorations around environments, \item, etc.

Deployed new texter (07b0cc6f6ca1c13ec8ce3c1e06b665bb314f6653):

  • Minor changes to travis infrastructure.

2014-09-29

Made OCG default renderer in production: https://gerrit.wikimedia.org/r/163608

ocg-collection:

Deployed new service (037f01ca8c9c9beb66ddfca6d1f78c69b59df97f):

  • Fix redis garbage collection of status objects.
  • Fix redis key expiration time (should be seconds, not milliseconds).
  • Improve the gc cleanDir function to handle nested directories.
  • Write output files in subdirectories to avoid having an excessive number of files in one directory.
  • Remove some blocking operations in the backend.
  • Ensure postmortem moves complete before cleanup.

Deployed new latexer (de5b32c9579169067f3dd87ef04675184c278e77):

  • Treat zh language code as if it were zh-Hans (that is, Simplified Chinese)
  • Match language codes case-insensitively.

2014-09-25

ocg-collection:

Deployed new service (2946c8901cbd5fcb8abf7bb8cb63b579066bbe17):

  • Speed up/cache directory size computation in health check (bug 71260).

Deployed new latexer (144bdd2962702abaa468a18c343509a0945c11e4):

  • Support Simplified and Traditional Chinese scripts.
  • Don't crash if images are inside <dt> tags.
  • Fix TeX escaping of [].
  • Fix "There's no line here to end." crasher (<br> after <dd>).
  • Ensure that we switch to the latin font where necessary.
  • Indic language support (fixes for Assamese, Gujarati, Marathi, Oriya, and Punjabi).

2014-09-24

ocg-collection:

ocg-collection changes:

  • Added load testing tool.
  • Set parsoid endpoint for OCG on wikisource.

Deployed new service (34dce8513b9ce0f6c84b22f854f3d732c36c779e):

  • Clean up directory size computation, eliminate a spurious log message.
  • Add force_render option to help page.

Deployed new bundler (90d0700afb5937ec9ec5bf43f37b653dd1559828):

  • Improve Metabook.fromArticles so that it can be used from load testing tool.
  • Handle redirects from subpages (for example, en:Children's_Heritage/D_O_A).

Deployed new latexer (86c1e14305b5109acd68a6bbd3b129692563c195):

  • Don't crash if PNG has bogus resolution information (for example, [[File:Frederick-II-of-Prussia-Coloured-drawing.png]]).
  • Support inline images (like in en:⠦).

2014-09-23

ocg-collection:

Deployed new service (6d1c4a5e00d51ee55eb739ce5469cf7f47a4efa8):

  • Hash all metabook props for the collection id. (Don't cache too aggressively.)

Deployed new bundler (0d8e3664d9347eb6d5c02ec8b3e8db1bb546ca4c):

  • Reduce default image resolution to 150 dpi (bug 68576).
  • Don't crash on images in subpages, or images with invalid titles.
  • Don't crash on bad image metadata (work around bug 71135).

Deployed new latexer (d5ea61db485f6f01554c83abd47a88c2e709e8a0):

  • Fix CJK definition in polyglossia.
  • Add font and language information for additional indic languages.
  • Don't crash if section/chapter title contains a URL.
  • Don't crash if we encounter an uncontained <li>.
  • Don't crash if <div> begins a figure caption.
  • Don't crash on ;#.
  • Don't crash if chapter has no items.
  • Add support for metabook toc option == "auto".
  • Upgrade node-icu-bidi to v0.1.3.

2014-09-18

Beta is still unhappy: see https://bugzilla.wikimedia.org/show_bug.cgi?id=71031

ocg-collection:

Deployed new bundler (021a70cd7680ffe2c814df2c1dd8c94cb565e031)

  • Fix unreliable image streaming (bug 68929)
  • Tighter query for images
  • Use default papersize when bundling

Deployed new latexer (9686d0e937c7ed502353f346419abff84d420448)

  • Convert TIFF images to PNG (bug 70866)
  • Handle animated GIFs (bug 70865)
  • Accept papersize settings from metabook (bug 68836)

Rebuilt all dependencies:

  • easyimage removed
  • node-icu-bidi updated to 0.1.3.

2014-09-15

Deployed to prod, but skipped deploy to beta/labs due to deployment-salt issues.

ocg-collection:

Deployed new service (f00852507bed9f86fc24198db5a474dd1881f392):

  • Log metabook on error.
  • Allow logging extended information from the backends.

Deployed new bundler (55aa6bea33e29053b76b2043d2c96bcb2f4f1964):

  • Save title redirect information.
  • Increase retries of truncated downloads from 3 to 5.
  • Don't crash if a referenced image was deleted or otherwise missing.
  • Redirect more logging through new log framework.
  • Handle missing contributor information.

Deployed new latexer (03c8b04355e31d2b3d2aa80c505215bcd13a2676):

  • Properly escape HTML anchors and id attributes.
  • Pass additional details to logger on XeLaTeX failures.

2014-09-09

ocg-collection:

Rebuilt all dependencies on deployment-pdf01.eqiad.wmflabs using a documented process.

Deployed new service (526b7b1e90d3b32c65e1cc0cdc2cef2d71ffa2b0):

  • Unify logging through the service wrapper
  • Add a human-readable help page to quickly test/check the service
  • Make sure the collection id is specific to the selected backend
  • Ensure that default temp and output directories exist
  • Use /etc/mw-collection-ocg.js configuration file if present.

Deployed new bundler (05a34aef7e31d8073471327b2cbff989a962257d):

  • Use new log and status reporting framework

Deployed new latexer (c784b335525390b1da5d2f64a8503330175f08a3):

  • Use new log and status reporting framework

Deployed new texter (c0360962459362e2de8cd2bbc44e7516a6037f9d):

  • Use new log and status reporting framework
  • Add -T option to CLI to specify a temporary directory
  • Allow rendering from unzipped bundle

2014-08-13

ocg-collection:

  • master: d7d1571d2bbe7da7c85f618f4eec3b0d8749c49e
  • wmf-deploy: faee29a260a96cbc0cdc0d402658b754a7425af9

Deployed new bundler (e12b79121df18158403ca5f0b160fc95883d0ccc):

  • Don't use process.exit on a normal successful exit (avoid truncating files)
  • Sync output directories before bundling (avoid truncating files)
  • Check for truncated image files during download (avoid truncating files)

Deployed new latexer (f90a3141367eddbc72f5a657fe45c5ebe5ee1c98):

  • Allow page breaks between empty sections.
  • Ensure latin font is used for list item bullets.
  • Add Lohit fonts for Indic languages, and use them whenever possible (bug 28206, bug 68922)

Deployed new service (158d50bc9274ace22daaa76927e03c3ec6b1de71):

  • Remove useless configuration variable.
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=OCG/Deployments&oldid=877400"