- Jul 27–31, 2012: RLP code copied by hand to 7 of 8 SMARS servers; no second reviewer, no written procedure
- Aug 1, 08:01 ET: 97 "Power Peg disabled" e-mails before the open; not designed as alerts, not read
- Aug 1, 09:30–10:15: 212 orders become 4M executions in 154 stocks; $3.5B long, $3.15B short; $460M lost
+ Oct 16, 2013: the SEC writes the postmortem — $12M penalty, first Market Access Rule case; Knight is already part of GETCOA trading firm deploys new code to seven of eight servers, and the eighth buys and sells nearly seven billion dollars of stock in forty-five minutes, which is a lot, even for a Wednesday. August 1st, 2012. Knight Capital, a tenth of all U.S. stock trading, reports a 440 million dollar loss. The SEC later counts more than 460, fines Knight twelve million, and writes the postmortem itself: ten pages, zero adjectives. How it happens, why it is possible, and who actually gets the blame.
Timeline. July 27th. The New York Stock Exchange launches a retail liquidity program on August 1st, so Knight copies new code onto SMARS, its order router, a few servers a day, by hand. One technician, eight servers, seven copies. Nobody checks; no rule says anyone has to. 8:01 a.m. An internal system starts emailing staff an error that reads: Power Peg disabled. Ninety-seven emails before the open. They are not designed as alerts, so nobody reads them, which is a design decision with a price tag. 9:30, the market opens. Seven servers run the new code. The eighth sees the same flag and runs Power Peg instead: code Knight retired in 2003 and never deleted. It sends child orders for 212 customer orders, and never stops. Four million executions. 397 million shares.
A holding account swells past its two-million-dollar limit, which is wired to nothing. The engineers, in a live market, uninstall the new code from the seven good servers, which switches Power Peg on everywhere. Around 10:15 it stops: three and a half billion long, three point one five billion short.
Mechanism. One: Power Peg's stop condition, the counter that says the parent order is filled, is moved in 2005 and never retested, so the retired code loops forever. Two: the new feature reuses Power Peg's old flag, so the same yes means two things on two servers. Three: no kill switch. Nothing compares orders out against orders in; nothing halts SMARS. The risk monitor is a screen watched by humans; it does not show the limits.
git blame: a flag reused instead of a new one, dead code callable for nine years, and a manual deploy with no second pair of eyes. Not the technician. The SEC order never names him, and neither will I.
Blast radius: 460 million dollars, roughly ten million a minute. Shares down 63 percent to $2.58, a 400 million dollar rescue at $1.50 a share, a merger with GETCO by Christmas. Hacker News, Thursday: how did it run 45 minutes without a human stepping in? It didn't. The humans made it worse.
Verdict, postmortem: revert. Knight never writes its own postmortem; the regulator does, fourteen months later, and the fix is a fine and a merger. Monday: new feature, new flag; dead code gets deleted, not disabled; anything that sends orders gets an off switch. Send me the incident you are still not allowed to talk about, in the comments, or at thedailydiff.dev.
Verdict: REVERT — no postmortem, no kill switch; the fix was a fine and a merger
Primary sources
SEC Administrative Order, Release No. 34-70694 (Oct 16, 2013), In the Matter of Knight Capital Americas LLC — https://www.sec.gov/litigation/admin/2013/34-70694.pdf
SEC press release 2013-222, "SEC Charges Knight Capital With Violations of Market Access Rule" — https://www.sec.gov/newsroom/press-releases/2013-222
Knight Capital Group 8-K, Aug 2, 2012 — the $440M statement — https://www.sec.gov/Archives/edgar/data/1060749/000119312512332176/d391111dex991.htm
Knight Capital Group 8-K, Aug 6, 2012 — the $400M convertible preferred — https://www.sec.gov/Archives/edgar/data/1060749/000119312512336167/d392288d8k.htm
Knight / GETCO merger announcement, Dec 19, 2012 — https://www.sec.gov/Archives/edgar/data/1060749/000089882212000673/ex991.htm
Press
NYT DealBook, "Knight Capital Says Trading Glitch Cost It $440 Million" (Aug 2, 2012) — https://archive.nytimes.com/dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/
Hacker News, Aug 2, 2012 (73 points, 91 comments) — https://news.ycombinator.com/item?id=4329101
Hacker News, "The $440M software error at Knight Capital" (294 points) — https://news.ycombinator.com/item?id=31239033
And that's the diff for today. I'm Niko from Axrisi. Merge responsibly.
YouTube · thedailydiff.dev · forward this to whoever owns your feature flags.