Google details PLC misconfiguration behind October 19 UK outage

By Amara Osei

Google said an external underground cable fault on October 19 exhausted roughly five minutes of UPS battery runtime at its Waltham Cross data center in Hertfordshire, making workloads in the europe-west2-b availability zone unavailable.

Google’s status update called the cause an “external electrical power disruption.” The Register first reported the outage, while the company supplied the technical explanation. Both descriptions point to a utility failure, but neither explains why the backup generator stayed silent.

What failed at Waltham Cross?

Before the disruption, Google said the site had temporarily used a single-feed 1N configuration for generator load testing. An underground cable then faulted on primary utility feed A. The test conditions met a real grid event, which is an inconvenient form of timing.

Google’s update said the failure involved “a voltage-sensing design flaw and PLC misconfiguration.” The combination prevented the generator plant from receiving an automatic start signal after feed A failed. Static UPS battery banks sustained the critical load as intended until their reserves ran out, causing a loss of power across the data halls.

The PLC had been programmed to check for voltage on the line side of the feeder before commanding a generator start. Utility feed B remained energized upstream, although its incomer feed sat in the open position. The active and standby master PLCs ran identical control logic and sensed live voltage from feed B through shared upstream sensors on that disconnected path.

Both controllers therefore classified the utility source as healthy and available, so neither sent a start command to the generator plant. Backup power was present. The software simply never called it, which is not a reassuring sentence for an auditor.

Google described the site as using a 2N utility-power architecture, with complete duplicate paths for every critical load and a central high-voltage generator bank managed by a dual-redundant Programmable Logic Controller, or PLC. Two independent primary utility feeders, A and B, supplied the facility.

Each primary feeder terminated at a separate primary intake switchboard, and either board could operate in closed loop if the other went offline. A standby utility connection supported manual, campus-wide transfer during maintenance or emergencies. Switching between primary and standby sources used open transitions and key interlocks, preventing parallel connections across providers.

The UPS systems provided approximately five minutes of autonomous runtime. During that bridge, the PLC was designed to detect utility undervoltage, start the diesel generators, synchronize them and transfer the site load. The five-minute battery reserve was a grace period, not a replacement for generation.

The sequence stopped because the controllers saw voltage they believed could carry the load. The standby supply was not an automatic third source; its transfer was a manual operation across the campus. When feed A faulted, feed B’s open incomer prevented transfer, while shared upstream sensors made that separation invisible to the PLCs.

Power was reconnected through the standby utility source, restoring critical load and recharging the UPS batteries. Engineers then began work to bring services back online. The status update did not specify a restoration time or customer count.

What changes after the outage?

Google says it is deploying a communications enhancement intended to improve notification speed for location-impacting incidents. The company describes this as an automated alert-to-communication mechanism. In practice, it should shorten the interval between detecting a site problem and notifying the people who must respond, though Google has announced no completion date.

The disclosed response, however, concerns communications rather than a stated redesign of the voltage-sensing or PLC logic. Google has said how it plans to announce incidents faster, but the update gives no correction schedule for the control flaw.

Google opened its Waltham Cross data center in Hertfordshire in September 2025. It is also reportedly considering a data center in Teesside, UK. More geographic capacity may follow, but duplicating generators will not fix a controller that mistakes an open incomer for available supply.

The communications upgrade is under way without a timetable, while the reported Teesside proposal remains under consideration. The next question for Google’s UK data-center expansion is whether its power controls can match the redundancy of its hardware.