Incident Report

Summary

On Thursday, 4 September, 2025, customer traffic from any source region to end-users served by the Sofia, Bulgaria POP locations may have experienced network disruptions for a total of 12 hours and 5 minutes. A period of severe impact, lasting 1 hour and 4 minutes, resulted in network congestion, elevated latency, and timeouts. Google Workspace services were impacted during this 1 hour and 4 minute period of severe impact.

We sincerely apologize for this incident, which does not reflect the level of quality and reliability we strive to offer. We are taking immediate steps to improve the platform’s performance and availability.

Root Cause

Google’s production backbone is a global network that enables connectivity for all user-facing traffic via Points of Presence (POPs) or internet exchanges.

The incident was caused by a disruption of multiple fiber paths in the production backbone within Bulgaria which reduced the available network bandwidth for the POP location in Sofia, Bulgaria. Although capacity for POP locations is planned with multiple redundant fiber paths, the remaining capacity for the Sofia, Bulgaria sites to the rest of the backbone could not support the traffic load after the series of fiber failures.

Our network is designed to be resilient and can automatically handle multiple, isolated fiber cuts by rerouting traffic. In this instance, the network successfully handled the initial fiber failures without significant customer impact. However, a subsequent fiber failure overwhelmed the remaining network capacity, which led to the service disruption.

Remediation and Prevention

Google engineers were alerted to the outage via our internal monitoring on Thursday, 4 September, 2025 00:19 US/Pacific and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers first quickly rerouted serving traffic to adjacent locations, restoring connectivity for most services by approximately Thursday, 4 September, 2025 01:12 US/Pacific.

Sufficient fiber connectivity was restored on Thursday, 4 September, 2025 07:52 US/Pacific and Google engineers began the process of reactivating the capacity, and after validation, allowing traffic serving from the Sofia, Bulgaria POP location.

All traffic migrations back to the pre-incident state were completed by Thursday, 4 September, 2025 12:13 US/Pacific.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• To enhance service resilience, we have implemented additional redundancy on the Milan- Sofia fiber path.
• While adding diversity will improve the overall availability of this site, we are also investigating ways to automatically signal fail-over to our external connection partners when internal capacity reduces beyond a critical threshold to satisfy demand. This will more quickly benefit those customers who connect in multiple regions.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

The issue primarily resulted in inaccessibility to Google Workspace services for customers in various locations served by the POP in Sofia, Bulgaria. The impact was first detected at approximately 00:19 US/Pacific on Thursday, 4 September, 2025. Google Workspace users that were routed to the affected POPs would have observed errors and high latency trying to connect to Google services. Additionally, some users encountered DNS resolution issues, including an inability to resolve some publicly accessible Google services.

Google Workspace services were restored by Thursday, 4 September, 2025 01:12 US/Pacific after Google engineers rerouted serving traffic to adjacent locations, restoring connectivity for services. All fiber repairs were completed on Thursday, 4 September, 2025 12:13 US/Pacific.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 4 September 2025 00:08

Incident End: 4 September 2025 01:12

Duration: 1 hour, 4 minutes

GWS- Affected Services and Features:

Potentially all Google Workspace services may have been impacted for users depending on the point of ingress.

Locations:

Turkey, Greece, and Bulgaria among other regions

Description:

Google’s production backbone is a global network that enables connectivity for user-facing traffic via Points of Presence (POPs) or internet exchanges.

A disruption of fiber paths in the production backbone within Bulgaria reduced the available network bandwidth for the POP location in Sofia, Bulgaria. Although capacity for POP locations is planned with multiple redundant fiber paths, the remaining capacity for the Sofia, Bulgaria sites to the rest of the backbone could not support the traffic load after the series of fiber disruptions.

Google engineers mitigated the impact by rerouting serving traffic to adjacent locations, restoring connectivity by approximately Thursday, 4 September, 2025 01:12 US/Pacific.

Customer Impact:

Impacted users would have encountered network errors and been unable to connect to Google Workspace services.

The issue with Google Workspace services has been resolved for all affected users as of Thursday, 2025-09-04 02:18 PDT.

We thank you for your patience while we worked on resolving the issue.

We are experiencing an issue with multiple Google Workspace services beginning on Thursday, 2025-09-04 00:43 PDT. Google Workspace services may not load for impacted users.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2025-09-04 04:00 PDT with current details.

We apologize to all who are affected by the disruption.