Fastly has explained how it managed to
black-hole big chunks of the internet yesterday: a customer triggered a bug.
The customer, Fastly points out in a post titled
Summary of June 8 outage, was blameless. "We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change," wrote Nick Rockwell, the company's senior veep of engineering and infrastructure.
The bug was introduced in a 12 May software deployment and lay dormant until, on 8 June, "a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85 per cent of our network to return errors."
Cue global chaos.
Rockwell's post states that Fastly "detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration. Within 49 minutes, 95 per cent of our network was operating as normal."