- SYD ↔ MEL cross-wired: one source eachAn engineer in Melbourne powers a timing chassis back on at ten to three in the morning, and by breakfast Australia's biggest mobile network agrees it is November two thousand six. Telstra's outside report says the date left one GPS card and the whole network voted for it. Nearly nine million customers lose calls, texts or data, and Victoria's regional trains stop for the day. How it happened, why one card outranked the national clock, and who gets the blame.
Twenty ten. The design is textbook. Three national time sources feed two servers, those feed three below, and every node takes time only from above. The auditors call it fit for purpose. Twenty twenty. New chassis. It can't feed its own lower server, so Sydney and Melbourne get cross-wired and each site drops to one reference. Then everything is switched to peering, where any server may take time from any server it finds. The words timing loop appear in no document. October twenty twenty-five. Melbourne keeps losing Sydney, and its alarms say stratum five, meaning it takes time from its own customers. Nobody files a ticket. Instead the engineers switch on a GPS card that has sat idle for five years. The alarms stop. Melbourne is now stratum one, the rank of the national institute, and nobody reviews it.
Eighth of July, two fifty in the morning. A power supply needs replacing, so the chassis reboots, and the GPS card with it. Its firmware was never updated, so it picks the old epoch and lands one thousand twenty-four weeks in the past. Melbourne is the top of the tree, and downstream starts agreeing. Four a.m., the handover nodes start resetting. Four twenty, an alarm storm, on a platform whose alarms are read in business hours. Four twenty-nine, Telstra notices, because customers are googling Telstra outage. The engineers who did the change are on mandatory rest, and finding the date takes hours. By twelve twelve most customers are back. Why was it possible? NTP has two defences. Lower stratum wins, and outliers get voted out. The GPS card made the broken source the highest rank, and every source that could disagree sat downstream, repeating it. NTP never asks whether a date is plausible, only whether the others agree. The protocol worked. The architecture did not.
git blame. Telstra, fifty-five percent. Nobody owned the clock, so nobody reviewed the change, and two vendor bulletins about this exact bug went unread. Peering mode, twenty-five, for letting servers vote with their own echo. The firmware, fifteen, published and never installed. Five percent to the power supply, for picking ten to three on a Wednesday. Blast radius. Forty-five percent of all calls and data sessions that day, the CEO tells the Senate. The fine on the table is thirty million dollars, for a free update missing from a thirty thousand dollar box. The card had survived the real rollover in twenty nineteen because nobody turned it off. The repair killed it. Verdict, postmortem: SHIP IT, on the response. Telstra hires outside auditors, publishes the report unedited, with the line that it never treated time as critical, and moves all three sites off the old servers before it lands. Monday line: run an NTP trace and count how many of your time sources are one clock wearing three hats.
Send me the incident you're still not allowed to talk about, in the comments, or at thedailydiff.dev.
Verdict: SHIP IT — the audit is public and the clocks moved
Sources
https://www.telstra.com.au/content/dam/tcom/dynamic-media-projects/luke-campbell/TAP-Findings-for-Telstra-Outage.pdf
https://www.telstra.com.au/exchange/what-we-ve-learned-from-the-external-investigation-of-our-july-o
https://www.netnod.se/blog/telstra-outage-night-network-decided-year-was-2006
https://news.ycombinator.com/item?id=49748957
https://www.abc.net.au/news/2026-07-10/telstra-warned-about-vulnerability-before-national-outage/106896906
https://www.theregister.com/networks/2026/07/08/telstra-outage-downs-000-calls-trains-payment-systems/5268265
https://www.theguardian.com/business/2026/jul/17/telstra-missing-software-update-undocumented-design-change-outage
https://www.theguardian.com/australia-news/2026/jul/10/2006-throwback-took-down-telstra-national-phone-networ
https://ia.acs.org.au/article/2026/telstra-outage-blamed-on-known-bug-in-obsolete-server.html
https://theconversation.com/a-timing-glitch-was-behind-telstras-nationwide-outage-it-points-to-a-bigger-vulnerability-287160
And that's the diff for today. I'm Niko from Axrisi. Merge responsibly.
YouTube · thedailydiff.dev · forward this to the intern who deployed on Friday.