Automations
that quietly
broke. Nine production scripts, two years of green dashboards, and the exact moment each one stopped doing what we thought it did.

The first one we found by accident. Olav was rebuilding the weekly revenue dashboard in March 2024, because the old one had a date column that quietly switched from Europe/Oslo to UTC the previous autumn and nobody had complained. He ran the new query alongside the old one. The numbers disagreed by 4.7 percent. Then, because Olav is the sort of engineer who pulls a thread until something comes loose, he ran them against the source system. Three of them disagreed.

None of the dashboards were broken. The automation that fed them was broken. A scheduled job had been running on a stale view since November 2023 — a view that itself depended on a materialization that had silently stopped refreshing because a downstream Snowflake task had been paused during a migration and never resumed. Argo Workflows reported success every morning at 06:15. Datadog showed green. Slack received its daily "ETL completed in 4m11s" message at 06:19 like a clock. The job was doing exactly what it was written to do. It was just doing it on yesterday's yesterday.

What is in this issue is a catalog of nine such cases that we, at Hauk & Berge, found between February 2024 and April 2026. I have asked five other teams to contribute — the field reports in §05 come from Caia (a midsize logistics outfit in Porto), from a developer-tools company in Berlin, and from the SRE guild at a Norwegian regional bank we are not allowed to name. The cases are diverse: a slack-bot that addressed the wrong channel for sixteen months; a feature-flag rollback that silently re-rolled the next morning; a backup job that wrote zero-byte tarballs because of a quoting issue and reported success because tar exited cleanly. None of them paged. All of them were technically successful.

The thesis is small and uncomfortable: success metrics are the wrong invariant. A green check answers "did the job finish?" and not "did the job do the thing?". Most production automations are wired only to the first question. That is the gap we are documenting, and the smell list at the end of this issue (§06) is our attempt — in the spirit of Cantrill's "things you wish you knew" talks — to make the gap easier to spot before it costs anyone €87,000 and a Saturday.

I want to thank the on-call rotation for letting me publish their incident notes verbatim, and the editorial committee of the Quiet Failure Review for tolerating a fourteenth consecutive issue about things that "looked fine."

The Caia report is the one that makes us most uncomfortable, because it could happen to anyone — the renewer printed "renewed: ${domain}" as the last line of a shell function regardless of whether the inner acme-tiny call had returned a non-zero exit code. A retry loop earlier in the script swallowed errors. The certs began to expire one by one in late January; the operations team caught the first one at 04:17 on 30 January 2025, when a customer in Belém reported a TLS error to their support line. Nine more would have expired in the following 11 days.

The Berlin case has a different shape. Their nightly "rollback validator" was supposed to confirm that the targeting rule on a feature flag matched the rule that had been rolled out the previous evening. The validator hashed the rule before the variation was attached, and compared that hash to a stored hash from the previous evening — which had also been computed before the variation was attached. The two hashes always matched. The variation, in fact, had been silently overwritten by a separate scheduled "warmup" job that re-applied the previous week's targeting at 02:14 every morning. It took the team six rollouts to discover this: the flag-state on production was, every morning, one week stale, exactly six days out of date by Monday lunch.

The bank's case is the simplest and the most absurd. A PagerDuty rotation shifter, written in 2019 by a contractor whose name has been scrubbed from the repo, rotated a primary on-call across three rotations. By 2024 two of the rotations had been deleted (the API returned 404, which the script interpreted as "no shift needed") and the third had been reduced to one person, Solveig, who was therefore "rotated" onto herself every Monday. The script ran successfully. The audit log of PagerDuty showed seven weeks of "schedule updated" events that did not, in any meaningful sense, update anything. Solveig was on-call continuously from 11 November to 30 December 2024. She mentioned it in a 1:1, gently.

What the three cases share is not a code defect — the code did precisely what it was written to do. What they share is a missing invariant. None of the systems had a way to ask "did the world look different after I ran than before I ran, in the way I intended?" That question is harder to write than "did I exit 0," but it is the only question that detects this class of failure.