Assessing Common-Mode Failures in Redundant Power Distribution Systems

In August 2011, a data center in Dallas lost power for roughly five hours. The facility had the usual markers of resilience — multiple utility service entrances, backup generators, and UPS systems — yet a single automatic transfer switch failed, and the building went dark anyway. Restoration was slow for a telling reason: as the operator later explained, bypassing a transfer switch "is not a simple fix." The redundancy on the one-line diagram was real. It just did not sit where the failure occurred. That is the uncomfortable pattern behind many critical-infrastructure outages. The Uptime Institute, which tracks major outages across the industry, consistently finds that power remains the single largest cause — and that the failures rarely trace to a shortage of generating capacity. They trace to the distribution, transfer, and control path between the sources and the load, and to the people operating it. Two utility sources, two transformers, two switchgear lineups, and a standby generator can make a facility look impressively resilient on a single-line diagram. Yet a deceptively simple question is still hard to answer: can the loads that matter actually remain energized after the failure of any one relevant component? The answer depends on far more than the number of sources. Resilience follows the entire power path — from the upstream supply through protection, switching, controls, distribution, physical rooms, and the equipment being served. A second feeder does not create meaningful redundancy if both paths converge through an unrecognized common point of failure. Redundancy is a system property, not an equipment count Engineers use concepts such as N+1 or 2N to describe capacity and redundancy arrangements, but the label needs a defined boundary. Two incoming services do not necessarily mean two independent distribution paths to the final load. Even when each service has sufficient capacity, both paths may share a transfer switch, control power supply, communications device, physical space, or operational procedure. A good review begins by identifying what must survive. Is the requirement uninterrupted power to particular loads, rapid restoration within an agreed interval, or only the ability to move noncritical demand after an outage? These are different design objectives. The answers affect the selection of equipment and the acceptance testing that proves the design works. A power system can also be electrically redundant but operationally fragile. Protective devices must coordinate adequately for the available source and configuration; transfer equipment must work in the credible conditions it will encounter; operators must be able to execute the sequence safely; and temporary conditions during maintenance must be understood. A 700-kW critical-load example Imagine a hypothetical facility with 700 kW of coincident critical demand. It has two supply paths, each with equipment nominally capable of supporting the critical demand under the assumed operating conditions. A designer might reasonably describe the upstream arrangement as redundant. Now follow the path downstream. Both supplies terminate at a single automatic transfer assembly, which feeds a common critical-load bus. That transfer assembly also depends on one control-power circuit. The upstream supplies are diverse, but a failed transfer mechanism or lost control supply can leave the downstream loads without the intended alternate path. Nothing in this example means that a transfer switch is inherently inappropriate. Shared transfer equipment is common and may be entirely suitable where the agreed reliability objective allows it. The lesson is narrower and more useful: if a component is common to all paths feeding a load, it must be included in the reliability claim and evaluated against the actual performance objective. Now consider maintenance. The facility plans to replace a section of switchgear. During the work, one supply path is removed from service. The remaining path still supports the load on paper, but a protective-device maintenance bypass, an unavailable tie breaker, or an untested switching sequence might change how the system performs under a fault. The normal-condition one-line is no longer enough; engineers need a validated temporary-condition one-line as well. Worked failure assessment: two 1,000-kW paths, one transfer dependency Here is a concrete expansion of the 700-kW example. Every rating, load, duration, and outcome below is fictional and assumed, supplied to demonstrate an assessment method rather than to describe a facility incident. Source A and Source B each feed a common automatic-transfer assembly, and each has a hypothetical 1,000-kW usable capacity under the modeled operating conditions. The downstream bus serves a constant 700-kW coincident load. Of that, 150 kW is supplied through a short-duration UPS with an assumed 10 minutes of usable autonomy at that load; the remaining 550 kW has no ride-through. A single transfer controller supervises the transfer assembly, and a latent controller fault is present but has gone undetected. At 14:00, Source A becomes unavailable. Source B is healthy and has adequate steady-state capacity, but the transfer controller fails to initiate the permitted transition. The shared downstream bus becomes de-energized. The UPS temporarily carries only its 150-kW protected portion. At minute 10, its assumed usable reserve is exhausted. Authorized personnel restore the bus at minute 35, after verifying the condition, following an approved recovery procedure, and confirming the receiving source and equipment are suitable. The 35-minute restoration is an illustrative assumption, not a prediction of field response. TIME EVENT IN MODELED CASE LOAD WITHOUT SERVICE 0 MIN Source A fails; common transfer controller fails to act; Source B remains available 550 kW (UPS holds 150 kW) 0-10 MIN Short-duration UPS supplies protected portion 550 KW 10-35 MIN UPS reaches assumed usable-energy limit 700 KW 35 MIN Authorized recovery completes; bus restored 0 KW Under the simplifying assumption of fixed demand, the illustrative unmet-load energy is (550 kW × 10/60 h) + (700 kW × 25/60 h) = 383.3 kWh. Equivalently, the 550-kW group loses 35 minutes (320.8 kWh) and the 150-kW protected group loses the final 25 minutes (62.5 kWh). This is an accounting illustration, not a validated availability, cost, battery-discharge or energy-not-served study; real demand, transfer transient behavior, UPS derating and approved operating modes would change the result. The five dependency layers I would examine 1. Electrical capacity. Each expected operating configuration needs an appropriately evaluated load flow, equipment loading review, and assessment of voltage and power quality where relevant. Nameplate totals are a starting point, not proof that a source can accept a particular group of loads. Starting currents, load steps, derating, and equipment duty may govern. 2. Protection and selectivity. Different source configurations can change available fault current and the operation of protective devices. The objective is to understand whether a localized fault can be cleared as intended without unnecessarily removing a larger portion of the system. Detailed studies are required; a blog-level diagram cannot establish selective coordination. 3. Control and communication. Remote supervision, transfer logic, interlocks, sensing, and control power have to be treated as equipment dependencies. An independent protective relay may continue to operate when communications are lost; a remote transfer command may not. The design documents should make that distinction explicit. 4. Physical separation. Two circuits routed through the same vulnerable space or the same cable tray may not offer the desired protection against a localized incident. Fire, water, impact, thermal conditions, and access constraints can defeat pathways simultaneously. Physical diversity needs to be evaluated against plausible hazards, not assumed from different circuit labels. 5. Human and maintenance dependencies. A system that requires a complex manual transfer within minutes should be evaluated against realistic staffing, procedures, switching authority, lockout/tagout requirements, and communication between teams. A second source that nobody can safely place in service is a limited resilience resource. Map the failure, then map the consequence A useful method is to build a dependency table for each essential load group. The rows are credible component failures or planned outages; the columns capture load consequences. For example: Event or condition What to verify Evidence needed Source A unavailable Source B can support the required operating configuration Capacity study and functional transfer test Transfer control power lost Defined equipment response and operator options Control schematic and test procedure Network monitoring lost Protection and local operation remain as designed Interface and failure-mode testing One gear room inaccessible Alternate path remains physically operable, if required Route review and contingency plan Maintenance on tie section Temporary configuration still meets the agreed performance objective Approved switching and outage plan This table does not predict numerical reliability. It does something more immediate: exposes the claims that have not yet been demonstrated. Numerical availability calculations can be useful, but only when failure data, repair assumptions, common-cause effects, and operational boundaries are credible.Cyber-physical dependencies belong in the same review A modern distribution system may include networked metering, supervisory controls, remote access, and integrated alarm platforms. These technologies improve situational awareness but can also create misleading assumptions about independence. A compromised dashboard may not interrupt power, while compromised supervisory control could matter enormously if it can issue commands. The distinction must be established through the actual architecture. NIST SP 800-82 Revision 3 emphasizes OT security with attention to reliability and safety. NIST SP 800-160 Volume 2 also promotes engineering systems that can withstand and recover from adverse conditions. Applied to electrical power, these principles argue for reviewing digital and physical dependencies together rather than allocating one set of drawings to electrical engineers and another to cybersecurity staff. One practical deliverable is a cross-domain dependency register. For each essential load, show its electrical source path, switching devices, control power, communications connections, required operator interventions, and known shared physical locations. When a change order adds a monitoring gateway or combines two control cabinets, the register should be updated. Otherwise, a small integration decision can quietly invalidate an earlier redundancy assumption. Make failure testing part of the design narrative A one-line diagram shows connections. A commissioning plan should show behavior. I would ask the project team to define several credible loss-of-function tests: loss of primary source, loss of a monitoring network, loss of transfer-control indication, restoration after a disturbance, and operation during a planned source outage. Each test needs safety precautions, permitted methods, responsible personnel, and measurable acceptance criteria developed for the specific facility. Tests should not involve live hazardous switching or intentional disruption without an approved engineering and operational procedure. Where full functional testing is unsafe or impractical, analyses, factory tests, simulated inputs, and partial demonstrations can provide supporting evidence—but their limitations should be recorded. An important question for every redundancy review The phrase "we have two sources" should be the beginning of the discussion, not the conclusion. The question worth asking is: Which single component, shared condition, or human action could still take down this supposedly redundant load? Good electrical design is not only about adding capacity or equipment. It is about understanding dependencies so that the facility's performance claim survives the operating conditions for which it was intended. That is where redundancy becomes resilience. References NIST, Guide to Operational Technology (OT) Security, SP 800-82 Rev. 3 (2023) NIST, Developing Cyber-Resilient Systems, SP 800-160 Vol. 2 Rev. 1 (2021) NIST, Cybersecurity Framework 2.0 (2024) DatacenterDynamics, Transfer switch failure causes outage at Colo4 data center (Dallas, August 2011) Uptime Institute, Annual Outage Analysis (power as the leading cause of significant outages)