Incident Report

Summary

AppSheet experienced elevated latency and intermittent synchronization failures, resulting in application loading timeouts across two distinct windows for users in the EMEA and AMER regions.

Root Cause

Horizontal Pod Autoscaling (HPA) was deleted as we moved from one actuation method to another. The HPA removal resulted in some regions not scaling with customer traffic, which caused resource contention based on single server load. Additionally, cross region failover in Anthos Service Mesh did not work, resulting in a whole region outage rather than requests getting rerouted to a serving region.

Remediation and Prevention

Google was alerted via internal monitoring for elevated latency and service instability starting with the initial impact window on 9 October 2025 at 00:10 US/Pacific. Our engineers were immediately engaged and commenced investigation and mitigation efforts.

The problem was resolved by applying manual mitigations in two distinct phases corresponding to the two impact windows:

• Impact Window 1: (9 October 2025, 00:10 - 02:05 US/Pacific):

  Our engineers identified the initial server overloads caused by the autoscaling failure in the EMEA region. They immediately began manually rerouting customer traffic away from the impacted servers, resolving the first service disruption by 02:05 US/Pacific.

• Impact Window 2: (9 October 2025, 05:25 - 06:20 US/Pacific):

  Upon recurrence of the issue, which affected the EMEA and partial AMER regions, our engineers manually rerouted all affected traffic away from the two regions where the autoscaling failure originated. This action successfully stabilized the environment and ended the incident at 06:20 US/Pacific.

Prevention AIs:

• We added an alert to detect missing HPA, ensuring that our engineers can remediate the issue manually if needed.
• We updated the migration process for our Kubernetes infrastructure to prevent the continuous removal of HPA after migration.
• We are fixing cross region failover routing in Anthos Service Mesh.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

The overarching issue was that AppSheet servers became overloaded in two regions due to a failure in the automatic scaling mechanism. This resulted in high latency and elevated error rates, causing cascading issues with app synchronization and the ability to load the editor.

The impact occurred across two distinct windows, initially affecting users in the EMEA region, and later expanding to include a partial section of the AMER region.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article: https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 9 October 2025 at 00:10

Incident End: 9 October 2025 at 06:20

Total Duration: The incident occurred across two distinct impact windows:

• Impact Window #1 (EMEA): 1 hour, 55 minutes
• Impact Window #2 (EMEA & partial AMER): 55 minutes

Affected Services and Features:

AppSheet - App synchronization and Editor loading

Regions/Zones:

• EMEA
• AMER (partial impact)

Description:

On Thursday, 9 October 2025, some AppSheet users in the EMEA and AMER regions experienced elevated latency, resulting in app synchronization failures and editor loading timeouts. The cumulative duration of impact across both windows was 2 hours, 50 minutes.

Preliminary investigation identified the issue was due to failures in autoscaling in 2 regions which caused servers to overload and cross-region failover did not occur. The issue was mitigated by rerouting traffic away from the impacted regions.

Google will complete a full Incident Report in the following days that will provide a detailed root cause.

Customer Impact:

Customers in the affected regions may have experienced high latency, resulting in errors when syncing data and loading AppSheet applications. The issue was initially observed in the EMEA region and was later observed in both EMEA and parts of the AMER region.

The timeline of the two distinct impact windows is as follows:

• EMEA only: 9 October 2025, from 00:10 to 02:05 US/Pacific.
• EMEA and partial AMER: 9 October 2025, from 05:25 to 06:20 US/Pacific.

The issue with AppSheet in Europe and America is now resolved.

From preliminary investigation the issue was due to resource contention that resulted in our servers not being able to startup during the duration of the issue. The issue was mitigated by rerouting traffic away from the impacted servers.

Some AppSheet users in Europe and America experienced increased latency, resulting in app synchronization failures and editor loading timeouts from Thursday, 2025-10-09 5:25 AM PDT to 2025-10-09 6:20 PDT

We will provide an update by Thursday, 2025-10-09 08:30 PDT with current details.