Networks & InfrastructureMicrosoft 365Cloud OutageNetwork FailoverSoftware Defect
Microsoft traces global 365 outage to bug, failover and traffic
Microsoft says a July software defect, a gateway failover and a traffic surge combined to cause the November 13 global 365 outage, which lasted about 24 hours.
Networks & InfrastructureWhy it matters
- The global Microsoft 365 outage began on November 13 and lasted about 24 hours.
- Microsoft operates two independent 365 network gateways, one of which also supports Exchange Online.
- Teams switched to the primary gateway about 15 hours before the outage.
- The triggering defect came from a software update made in July.
- Most users could not access any 365 services during the outage.
The story
Microsoft said the worldwide failure of its 365 services on November 13 lasted about 24 hours and resulted from three interacting problems: an undetected July software defect, a surge in gateway traffic and a network failover.
By Rebecca Stone
The outage took down access to services including Outlook and Teams and also affected Microsoft customers OpenAI and Anthropic. It occurred on the same day as a separate Federal Aviation Administration outage, but the two incidents were unrelated.
What caused the November 13 outage?
Charles Lamanna, Microsoft’s executive vice president of Copilot, agents and platforms, traced the incident to a latent defect carried by a software update made in July. The update initially showed no problems, yet the defect caused servers to become overwhelmed and unresponsive under certain load conditions.
Microsoft’s normal load tests used older, weaker hardware and did not expose the fault. The company said the issue worsened when the software ran on newer systems with more processor cores, raising a question about whether its test environment covered the affected configurations.
That leaves several details outside Microsoft’s public account: the post did not identify the July update or quantify the extra load at which the defect appeared. It also gave no request volumes or server counts with which to test the capacity explanation. No independent validation was described.
How did failover increase congestion?
Microsoft runs two independent network gateways for 365 so that one can take over during a disruption. About 15 hours before the outage, however, Teams failed over from the alternative gateway to the primary one, which also supports Exchange Online. That move increased congestion on the destination gateway.
Microsoft also cited a surge in load on the networking gateway and an unrelated issue from the previous day. The combination, rather than the July defect alone, determined the outcome, Lamanna said, as requests accumulated during the U.S. peak traffic window until servers stopped responding.
Lamanna wrote: “Taken together, the increased load and the reduced server capacity caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the U.S.”
Why did recovery take so long?
Microsoft said restoration took longer than usual because its automated recovery processes were not designed to restore service from such a broad unresponsive state. Most users could not access any 365 services during the outage, making the roughly 24-hour disruption comparable to losing one full conventional business day.
Rollback presented another difficulty. Because engineers had not detected the defect since July, they faced a harder task in reversing the update. Microsoft did not say how many servers or requests were affected, or how many users remained without service after the first recoveries began.
Services were largely restored some 24 hours later, according to Microsoft. That endpoint does not mean every customer had returned to normal at the same moment, because the company described the restoration as largely, rather than complete, without providing a final all-clear time.
Lamanna added: “We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business and operations.”
What remains unresolved?
Microsoft’s explanation identifies the technical sequence but leaves the corrective plan unstated. The company has not said whether it will broaden load testing, alter failover traffic or redesign recovery automation to cover the failure conditions.
With no timetable disclosed for new load testing, failover changes or recovery-automation work, the next U.S. peak-traffic period remains the first practical test of whether the same three-factor sequence can recur.
Also reported
Source: sdxcentral.com
More from Rebecca Stone
Show full bio
Correspondent covering technology and product news in Carrier business, 5G and 6G, fiber and broadband at Telecom Gazette.
521 articles