- Mar 22, 2016, 21:30 UTC: 273 npm packages unpublished — left-pad among them
- Babel, React, Node builds 404 worldwide; a fork in 10 min changes nothing (line-numbers pins 0.0.3)
+ 23:55: npm un-un-publishes 0.0.3 from a backup; 24-hour unpublish rule a week laterOne developer deletes eleven lines of JavaScript, and within minutes Babel, React and Node builds are failing worldwide, hundreds a minute. The function is left-pad. It pads a string with spaces, and by the end of the evening npm does the one thing it says it cannot do: republish a deleted version. How it happens, why it is possible, and who gets the blame.
March 11th. A patent agent at Kik, the messaging app, emails Azer Koçulu: rename your npm package, kik is our brand. Azer says no. An hour later: our trademark lawyers are going to be banging on your door and taking down your accounts. Azer names a price, thirty thousand dollars, and Kik forwards the thread to npm support. March 18th. npm's CEO rules that anyone typing npm install kik expects the messaging app, and hands the name over. Azer asks for everything of his deleted. Two days later he does it himself: two hundred and seventy-three packages. March 22nd, 21:30 UTC. Every build that reaches for left-pad gets a 404. Ten minutes later a stranger republishes it as 1.0.0, and nothing changes, because Babel and Atom load it through a package called line-numbers, which wants 0.0.3 exactly. At 23:03 npm's CTO tweets: un-un-publishing. At 23:55, 0.0.3 is back from a backup. Two and a half hours.
Why can one person do this? Three reasons. Any version on npm is deletable by its author, instantly, with no check of who depends on it. Second, dependencies are transitive: Babel does not know left-pad exists, it knows line-numbers, and the whole tree is resolved live on every install. Third, in 2016 nothing pins that tree by default; lockfiles are opt-in, so every CI run re-asks the internet what eleven lines of code look like. One registry, run by nobody who depends on it. git blame. npm, sixty percent: its own postmortem says unrestricted unpublishing caused the pain, and, quote, we dropped the ball. Kik, twenty-five: a patent agent, not a lawyer, promising lawyers at the door over a name Kik had already decided not to use. The dependency habit, fifteen: everyone who npm-installed eleven lines instead of typing them. Including Kik, whose builds break too. Blast radius: two hundred and seventy-three packages pulled, one of them downloaded two and a half million times a month. The fix takes two and a half hours; the function takes eleven lines. On Hacker News, a post asking whether we have forgotten how to program hits seventeen hundred points, mostly from people whose builds are still red.
Verdict, postmortem: ship it. npm restores from backup, posts the postmortem next day, and within a week ships the rule that still stands: you can unpublish for twenty-four hours, after that you talk to a human. Monday: commit your lockfile, and if a function is eleven lines, it is a paragraph, not a dependency. Send me the incident you are still not allowed to talk about, in the comments, or at the daily diff dot dev.
Verdict: SHIP IT — restored in 2.5 h · postmortem next day · 24-hour unpublish rule in a week
Sources
https://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm
https://blog.npmjs.org/post/141905368000/changes-to-npms-unpublish-policy
https://web.archive.org/web/20160323201305/https://medium.com/@azerbike/i-ve-just-liberated-my-modules-9045c06be67c
https://web.archive.org/web/20170112194445/https://medium.com/@mproberts/a-discussion-about-the-breaking-of-the-internet-3d4d2a83aa4d
https://x.com/seldo/status/712414400808755200
https://registry.npmjs.org/left-pad/0.0.3
https://news.ycombinator.com/item?id=11340510
https://news.ycombinator.com/item?id=11348798
https://news.ycombinator.com/item?id=11349870
https://news.ycombinator.com/item?id=11382885
https://www.theregister.com/2016/03/23/npm_left_pad_chaos/
And that's the diff for today. I'm Niko from Axrisi. Merge responsibly.
YouTube · thedailydiff.dev · forward this to the intern who deployed on Friday.