The Problem
Well-run systems still fail — often from risks that never made the tracker
Even after a system has been designed, reviewed, tested, and accepted by experienced teams, failures still surface — often in ways no one on the program had prepared for. The most concerning cases are not failures in implementing known requirements or from poorly built systems. The most concerning cases are those that arise after work is complete per specified requirements: failures resulting from residual behaviors the system still permits.
Residual behaviors arise from interactions, conditions, and decision paths that may have been reviewed in familiar combinations but never considered in unintended combinations. Some of those behaviors are allowed because a needed requirement was never conceived or specified. There are too many latent unintended behaviors to enumerate manually and without a systematic capability to assist with their discovery, they remain hidden.
Across government and industry — energy, aviation, financial services, healthcare, and information technology — a familiar pattern repeats: A system is in place, decisions are made based on how it is understood, and confidence builds around that understanding. Then a behavior surfaces that goes sharply against intent, despite satisfying the documented rules. This behavior surfaces not because those rules were implemented poorly, but because the system still permitted high-risk behavior that was knowable — and therefore manageable — but had never made it into the risk register.
REAL-WORLD EXAMPLES
These are well-documented cases
A line trip in Ohio cascaded into an outage for about 50 million people across the U.S. Northeast and Midwest and Ontario, Canada. Contingency practices, alarms, and neighboring reliability coordinators were already part of grid operations. After the control room alarm processor failed, operators continued to work as if those tools were still healthy, while neighboring operators could not form a timely picture of the same deteriorating system (U.S.-Canada Power System Outage Task Force 2004). Continuing to operate after the tools had failed, without a shared view of the grid, was knowable — but it was never named as a risk anyone had to close.
The aircraft was designed, certified, and in passenger service when it crashed after takeoff, killing 189 people. Design and certification treated the new nose-down function as acceptable under assumptions about how a crew would handle a malfunction. In service, that function could act repeatedly on input from a single sensor (Komite Nasional Keselamatan Transportasi 2019). The documented rules and the as-fielded combination were not the same. What had been assumed in certification was not the behavior the aircraft still permitted.
A family office’s swap positions became a multi-billion-dollar counterparty loss, including about $5.5 billion at Credit Suisse (Credit Suisse Group 2021). Each prime-broker relationship could satisfy that firm’s documented risk rules. What those rules still permitted was a concentrated exposure held across several firms at once, visible to no one as a single risk. That combined counterparty exposure was knowable, but it never reached any one register as a risk in its own right.
A software-controlled radiation machine delivered six fatal overdoses despite appearing to execute ordinary treatment commands. Hardware that had once blocked an unsafe beam had been removed because the software was assumed to prevent it. In service, a rapid command change by the operator plus an internal software state still permitted that beam (Leveson and Turner 1993). The machine satisfied its implemented rules but violated the human operator’s intent. That combination was knowable, but it was never treated as a risk in its own right.
A signed update from a trusted vendor reached SolarWinds Orion customers across U.S. government, critical infrastructure, and private sector networks. A much smaller set of those customers was used for follow-on intrusion. The implant arrived through SolarWinds’ own signed software lifecycle, not through a break in the customer environment (Cybersecurity and Infrastructure Security Agency 2021). The update path was known, authorized, and consistent with the documented trust rules. What those rules still permitted was the same high-privilege path after the vendor’s build process had been compromised. That path was knowable, but it was never treated as a risk the customer had to track.
The Common Thread
What these situations have in common
In each case, the system was already specified, reviewed, and in operation. Documentation existed. Controls existed for the system as it was understood. The people involved were experienced. The failure did not come from a missing program of work; it came from a residual behavior the system still permitted — a combination of interactions, conditions, and decision paths that had been reviewed in familiar form but not considered together as one risk.
The resultant cost is not only the event itself; it is balance sheet and mission volatility, months or years of investigation, and a long interval before anyone can say with confidence what the system had been allowing. That reconstruction is done under scrutiny, and often after the damage is public.
While those leftover conditions were not in view at the time, they were not unknowable. They were knowable unknowns: an undocumented requirement that was never conceived, an assumption that was never examined, or a behavior the system still permitted that had not yet been identified as a risk to close.
What current methods cover — and what they leave open
Most organizations already manage the system as specified:
- They monitor signals.
- They assess known risks.
- They test and verify against documented rules.
- They respond to incidents once they occur.
While these methods are necessary, they operate within the boundaries of what has already been identified. They verify known behaviors and the rules that govern them. They were not designed for the systematic surfacing of behaviors that satisfy those rules but still violate intent.
Costly situations begin when the intent is violated: not beyond the system, but in leftover permission.
What must come into view
Executive decision-makers, chief risk officers, program leaders, chief engineers, and mission-assurance, cyber, and safety leads need a way to see that class of behavior before it has to be reconstructed — and before they have to absorb the loss, the inquiry, and the time it takes to learn what the system had been allowing all along.
Our work is built on the Monterey Phoenix behavior modeling framework, which is used to explore how a system can behave across conditions, not only as it was expected to behave. Firelight Logic applies that foundation through EmergeneeringTM, a structured way to model behavior, follow possible paths, and inspect how different conditions interact. It extends what is already understood by bringing into view behaviors the system can still permit but that have not yet been considered together.
If your organization supports or operates high-stakes systems, your main concern is not whether controls exist for known risks. Your concern is latent behaviors that are knowable but not yet identified as risks to close.
How do we control risks that are not even on the tracker? Our work is built to close that gap.
References
Credit Suisse Group. 2021. Credit Suisse Group Publishes the Report of the Independent External Investigation into Archegos Capital Management (Form 6-K dated July 29, 2021). Special Committee of the Board of Directors.
Cybersecurity and Infrastructure Security Agency. 2021. “Alert AA20-352A: Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations.” Last modified April 15, 2021.
Komite Nasional Keselamatan Transportasi. 2019. Aircraft Accident Investigation Report: PT. Lion Mentari Airlines, Boeing 737-8 (MAX), PK-LQP, Tanjung Karawang, West Java, Republic of Indonesia, 29 October 2018. KNKT.18.10.35.04. Jakarta: KNKT.
Leveson, Nancy G., and Clark S. Turner. 1993. “An Investigation of the Therac-25 Accidents.” Computer 26 (7): 18–41.
U.S.-Canada Power System Outage Task Force. 2004. Final Report on the August 14, 2003 Blackout in the United States and Canada: Causes and Recommendations.