[{"id":"NrAh49n3sxiih9w4LQy6","number":"8190337203990041963","begin":"2024-11-20T04:45:00+00:00","created":"2024-11-20T16:49:18+00:00","end":"2024-11-20T16:21:00+00:00","modified":"2024-11-20T17:49:37+00:00","external_desc":"**Summary :**\nGoogle Voice group chat users may receive malformed SMS messages.\n**Description :**\nWe are experiencing an issue with Google Voice beginning on Tuesday, 2024-11-19 20:45 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Wednesday, 2024-11-20 10:00 US/Pacific with current details.\n**Customer Symptoms :**\nGoogle Voice group chat users may receive malformed SMS messages.\n**Workaround :**\nNone at this time","updates":[{"created":"2024-11-20T17:49:15+00:00","modified":"2024-11-20T17:49:37+00:00","when":"2024-11-20T17:49:15+00:00","text":"**Summary :** Google Voice group chat users may receive malformed SMS messages.\n**Description :**\nThe issue with Google Voice has been resolved for all affected users as of Wednesday, 2024-11-20 08:21 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-11-20T16:49:18+00:00","modified":"2024-11-20T17:49:15+00:00","when":"2024-11-20T16:49:18+00:00","text":"**Summary :**\nGoogle Voice group chat users may receive malformed SMS messages.\n**Description :**\nWe are experiencing an issue with Google Voice beginning on Tuesday, 2024-11-19 20:45 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Wednesday, 2024-11-20 10:00 US/Pacific with current details.\n**Customer Symptoms :**\nGoogle Voice group chat users may receive malformed SMS messages.\n**Workaround :**\nNone at this time","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-11-20T17:49:15+00:00","modified":"2024-11-20T17:49:37+00:00","when":"2024-11-20T17:49:15+00:00","text":"**Summary :** Google Voice group chat users may receive malformed SMS messages.\n**Description :**\nThe issue with Google Voice has been resolved for all affected users as of Wednesday, 2024-11-20 08:21 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"gh79SrQk6wqonWoxPdCm","service_name":"Google Voice","affected_products":[{"title":"Google Voice","id":"gh79SrQk6wqonWoxPdCm"}],"uri":"incidents/NrAh49n3sxiih9w4LQy6"},{"id":"vU5ZcqrdVSm1zxxY1E5Q","number":"4102671141080167299","begin":"2024-11-19T18:44:00+00:00","created":"2024-11-20T18:16:04+00:00","end":"2024-11-19T20:24:00+00:00","modified":"2024-11-22T17:10:18+00:00","external_desc":"**Summary:**\nGoogle Calendar experienced service issues, preventing some web users from accessing their calendars.\n**Description:**\nOn November 19, 2024, between 10:44 AM and 12:24 PM US/Pacific, a subset of Google Calendar users encountered problems accessing the Calendar web UI.\nWe sincerely apologize for any inconvenience this may have caused our valued customers.\n**Diagnosis:**\nThe issue affected Google Calendar Web users and was triggered when users attempted to transfer a calendar event ownership. Once this happened, one or more users were unable to access any part of their calendar through the website.\nThe issue did not affect the Calendar mobile application\n**Workaround:**\nThere were no workarounds at the time of the issue","updates":[{"created":"2024-11-22T17:10:18+00:00","modified":"2024-11-22T17:10:18+00:00","when":"2024-11-22T17:10:18+00:00","text":"# Incident Report\n## Summary\nOn 19 November 2024 at 10:44 US/Pacific, Google Calendar experienced service disruptions globally for web users due to a misconfigured denial-of-service rule. This resulted in some users being unable to access their calendars for a duration of 1 hour and 40 minutes. Google Calendar’s mobile application users were not impacted.\nTo our Google Calendar users impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nThe incident was triggered by the implementation of a denial-of-service (DoS) rule, intended to mitigate malicious traffic. However, a misconfiguration led to the rule being applied to all traffic instead of traffic to a specific endpoint. Consequently, some users who triggered the rule may have had all of their web traffic to Google Calendar blocked, resulting in a denial of service for those users.\nThe root cause was an incorrect scope for a rule within the DoS configuration. The rule was applied too broadly with the purpose of blocking malicious traffic, and led to unintended blocking of legitimate user traffic.\nInternal monitoring shows a traffic anomaly during the incident, but no alerts were triggered.\n## Remediation and Prevention\nThe issue was escalated by the support team to the engineering team at 12:12 US/Pacific. The on-call Engineer identified the misconfiguration and rolled back the faulty change to mitigate the issue at 12:24 US/Pacific.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n- Ensuring oncall engineers are alerted when unusual amounts of traffic are blocked.\n- Updating processes and documentation around denial-of-service configurations to prevent recurrence.\n- Creating internal trainings on best practices for denial-of-service configurations to prevent recurrence.\n- Investigating validation mechanisms to catch bad configurations.\n## Detailed Description of Impact\nGoogle Calendar:\nOn 19 November 2024 at 10:44 US/Pacific,\n- Google Calendar web users globally were impacted.\n- The primary impact was the inability to access calendar functionalities after transferring an event.\n- The issue was mitigated at 12:24 US/Pacific.","status":"AVAILABLE"},{"created":"2024-11-20T20:58:53+00:00","modified":"2024-11-22T17:10:18+00:00","when":"2024-11-20T20:58:53+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 19 November 2024 10:44\n**Incident End:** 19 November 2024 12:24\n**Duration:** 1 hour, 40 minutes\n**Affected Services and Features:**\nGoogle Calendar\n**Regions/Zones:** Global\n**Description:**\nGoogle Calendar experienced service issues, preventing some web users from accessing their calendars for a duration of 1 hour, 40 minutes. From preliminary analysis, the root cause of the issue was a misconfigured traffic management rule that incorrectly blocked legitimate traffic.\nGoogle engineers mitigated impact by rolling back the misconfigured traffic management rule that was blocking legitimate traffic.\nGoogle will complete a full IR in the following days that will provide a full root cause.\n**Customer Impact:**\nThe issue affected Google Calendar Web users and was triggered when users attempted to transfer calendar event ownership. Once this happened, one or more users were unable to access any part of their calendar through the website.\nCalendar mobile application users were not affected.","status":"AVAILABLE"},{"created":"2024-11-20T18:18:52+00:00","modified":"2024-11-20T20:58:53+00:00","when":"2024-11-20T18:18:52+00:00","text":"The problem with Google Calendar has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. The affected users are unable to access Google Calendar.\nThe issue affected Google Calendar Web users and was triggered when users attempted to transfer a calendar event ownership. Once this happened, one or more users were unable to access any part of their calendar through the website.\nThe issue did not affect the Calendar mobile application","status":"SERVICE_DISRUPTION"},{"created":"2024-11-20T18:16:04+00:00","modified":"2024-11-20T18:18:52+00:00","when":"2024-11-20T18:16:04+00:00","text":"**Summary:**\nGoogle Calendar experienced service issues, preventing some web users from accessing their calendars.\n**Description:**\nOn November 19, 2024, between 10:44 AM and 12:24 PM US/Pacific, a subset of Google Calendar users encountered problems accessing the Calendar web UI.\nWe sincerely apologize for any inconvenience this may have caused our valued customers.\n**Diagnosis:**\nThe issue affected Google Calendar Web users and was triggered when users attempted to transfer a calendar event ownership. Once this happened, one or more users were unable to access any part of their calendar through the website.\nThe issue did not affect the Calendar mobile application\n**Workaround:**\nThere were no workarounds at the time of the issue","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-11-22T17:10:18+00:00","modified":"2024-11-22T17:10:18+00:00","when":"2024-11-22T17:10:18+00:00","text":"# Incident Report\n## Summary\nOn 19 November 2024 at 10:44 US/Pacific, Google Calendar experienced service disruptions globally for web users due to a misconfigured denial-of-service rule. This resulted in some users being unable to access their calendars for a duration of 1 hour and 40 minutes. Google Calendar’s mobile application users were not impacted.\nTo our Google Calendar users impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nThe incident was triggered by the implementation of a denial-of-service (DoS) rule, intended to mitigate malicious traffic. However, a misconfiguration led to the rule being applied to all traffic instead of traffic to a specific endpoint. Consequently, some users who triggered the rule may have had all of their web traffic to Google Calendar blocked, resulting in a denial of service for those users.\nThe root cause was an incorrect scope for a rule within the DoS configuration. The rule was applied too broadly with the purpose of blocking malicious traffic, and led to unintended blocking of legitimate user traffic.\nInternal monitoring shows a traffic anomaly during the incident, but no alerts were triggered.\n## Remediation and Prevention\nThe issue was escalated by the support team to the engineering team at 12:12 US/Pacific. The on-call Engineer identified the misconfiguration and rolled back the faulty change to mitigate the issue at 12:24 US/Pacific.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n- Ensuring oncall engineers are alerted when unusual amounts of traffic are blocked.\n- Updating processes and documentation around denial-of-service configurations to prevent recurrence.\n- Creating internal trainings on best practices for denial-of-service configurations to prevent recurrence.\n- Investigating validation mechanisms to catch bad configurations.\n## Detailed Description of Impact\nGoogle Calendar:\nOn 19 November 2024 at 10:44 US/Pacific,\n- Google Calendar web users globally were impacted.\n- The primary impact was the inability to access calendar functionalities after transferring an event.\n- The issue was mitigated at 12:24 US/Pacific.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"n5hVexuq1PM9onY9qrmo","service_name":"Google Calendar","affected_products":[{"title":"Google Calendar","id":"n5hVexuq1PM9onY9qrmo"}],"uri":"incidents/vU5ZcqrdVSm1zxxY1E5Q"},{"id":"WApwTU6dK1mkDw3kwopX","number":"7948131055633976189","begin":"2024-11-15T21:00:00+00:00","created":"2024-11-16T00:40:16+00:00","end":"2024-11-16T00:35:00+00:00","modified":"2024-11-16T00:59:34+00:00","external_desc":"**Summary:**\nWe are investigating reports of issues with Google Cloud and Google Workspace Support.\n**Description:**\nWe are investigating reports of issues with Google Cloud and Google Workspace Support beginning on Friday, 2024-11-15 13:00 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Friday, 2024-11-15 17:30 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Diagnosis:**\nImpacted customers may experience issues with creating and managing the vector cases. The operations like case creation, read and updates may have been impacted.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-11-16T00:59:34+00:00","modified":"2024-11-16T00:59:34+00:00","when":"2024-11-16T00:59:34+00:00","text":"**Summary:**\nGoogle Cloud and Google Workspace Support experienced issues with case creation and case management.\n**Description:**\nThe issue with Google Cloud and Google Workspace Support has been resolved for all affected customers as of Friday, 2024-11-15 16:35 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-11-16T00:40:16+00:00","modified":"2024-11-16T00:59:34+00:00","when":"2024-11-16T00:40:16+00:00","text":"**Summary:**\nWe are investigating reports of issues with Google Cloud and Google Workspace Support.\n**Description:**\nWe are investigating reports of issues with Google Cloud and Google Workspace Support beginning on Friday, 2024-11-15 13:00 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Friday, 2024-11-15 17:30 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Diagnosis:**\nImpacted customers may experience issues with creating and managing the vector cases. The operations like case creation, read and updates may have been impacted.\n**Workaround:**\nNone at this time.","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-11-16T00:59:34+00:00","modified":"2024-11-16T00:59:34+00:00","when":"2024-11-16T00:59:34+00:00","text":"**Summary:**\nGoogle Cloud and Google Workspace Support experienced issues with case creation and case management.\n**Description:**\nThe issue with Google Cloud and Google Workspace Support has been resolved for all affected customers as of Friday, 2024-11-15 16:35 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"dnz8HVUHoMMcW9wV91LY","service_name":"Google Workspace Support","affected_products":[{"title":"Google Workspace Support","id":"dnz8HVUHoMMcW9wV91LY"}],"uri":"incidents/WApwTU6dK1mkDw3kwopX"},{"id":"f5n51VFCprgtnJFJf48a","number":"7315944190933260836","begin":"2024-11-15T05:59:00+00:00","created":"2024-11-15T06:59:23+00:00","end":"2024-11-15T12:09:00+00:00","modified":"2024-11-20T12:29:52+00:00","external_desc":"**Summary:**\nWe are investigating a potential issue with Google Workspace Support. Case creation may fail with an error.\n**Description:**\nOur engineering team has determined that further investigation is required to mitigate the issue.\nWe will provide an update by Thursday, 2024-11-14 23:30 US/Pacific with current details.\n**Diagnosis:**\nCustomers impacted by this issue may observe errors during case creation.\n**Workaround:**\nIf case creation fails, impacted customers can use the Contact Us Form (CUF) that is automatically generated after the failure.","updates":[{"created":"2024-11-20T12:27:12+00:00","modified":"2024-11-20T12:29:52+00:00","when":"2024-11-20T12:27:12+00:00","text":"A detailed Incident Report has been posted on the Google\nCloud Service Health Dashboard: https://status.cloud.google.com/incidents/NEtbe4S7pMgACeuopMaZ","status":"AVAILABLE"},{"created":"2024-11-15T22:49:57+00:00","modified":"2024-11-20T12:27:12+00:00","when":"2024-11-15T22:49:57+00:00","text":"The Mini Incident Report is available on the Google Cloud Service Health Dashboard :\nhttps://status.cloud.google.com/incidents/NEtbe4S7pMgACeuopMaZ","status":"AVAILABLE"},{"created":"2024-11-15T13:13:52+00:00","modified":"2024-11-15T22:49:57+00:00","when":"2024-11-15T13:13:52+00:00","text":"**Summary:**\nWe are investigating a potential issue with Google Workspace Support. Case creation may fail with an error.\n**Description:**\nThe issue with Google Cloud Support has been resolved for all affected users as of Friday, 2024-11-15 04:36 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"SERVICE_OUTAGE"},{"created":"2024-11-15T12:49:17+00:00","modified":"2024-11-15T13:13:52+00:00","when":"2024-11-15T12:49:17+00:00","text":"**Summary:**\nWe are investigating a potential issue with Google Workspace Support. Case creation may fail with an error.\n**Description:**\nMitigation work is still underway by our engineering team.\nCurrent data indicates that the current mitigation strategy is working and the team continues to see positive results.\nWe will provide more information by Friday, 2024-11-15 05:30 US/Pacific.\n**Diagnosis:**\nCustomers impacted by this issue may observe errors during case creation.\n**Workaround:** If case creation fails, impacted customers can use the Contact Us Form (CUF) that is automatically generated after the failure.","status":"SERVICE_OUTAGE"},{"created":"2024-11-15T10:48:11+00:00","modified":"2024-11-15T12:49:17+00:00","when":"2024-11-15T10:48:11+00:00","text":"**Summary:**\nWe are investigating a potential issue with Google Workspace Support. Case creation may fail with an error.\n**Description:**\nOur engineering team continues to investigate the issue. We will provide an update by Friday, 2024-11-15 05:00 US/Pacific with current details.\n**Diagnosis:**\nCustomers impacted by this issue may observe errors during case creation.\n**Workaround:** If case creation fails, impacted customers can use the Contact Us Form (CUF) that is automatically generated after the failure.","status":"SERVICE_OUTAGE"},{"created":"2024-11-15T08:24:05+00:00","modified":"2024-11-15T10:48:11+00:00","when":"2024-11-15T08:24:05+00:00","text":"**Summary:**\nWe are investigating a potential issue with Google Workspace Support. Case creation may fail with an error.\n**Description:**\nOur engineering team continues to investigate the issue. We will provide an update by Friday, 2024-11-15 03:00 US/Pacific with current details.\n**Diagnosis:**\nCustomers impacted by this issue may observe errors during case creation.\n**Workaround:** If case creation fails, impacted customers can use the Contact Us Form (CUF) that is automatically generated after the failure.","status":"SERVICE_OUTAGE"},{"created":"2024-11-15T07:22:50+00:00","modified":"2024-11-15T08:40:17+00:00","when":"2024-11-15T07:22:50+00:00","text":"**Summary:**\nWe are investigating a potential issue with Google Workspace Support. Case creation may fail with an error.\n**Description:**\nOur engineering team continues to investigate the issue. We will provide an update by Friday, 2024-11-15 01:00 US/Pacific with current details.\n**Diagnosis:**\nCustomers impacted by this issue may observe errors during case creation.\n**Workaround:**\nIf case creation fails, impacted customers can use the Contact Us Form (CUF) that is automatically generated after the failure.","status":"SERVICE_OUTAGE"},{"created":"2024-11-15T06:59:23+00:00","modified":"2024-11-15T08:41:01+00:00","when":"2024-11-15T06:59:23+00:00","text":"**Summary:**\nWe are investigating a potential issue with Google Workspace Support. Case creation may fail with an error.\n**Description:**\nOur engineering team has determined that further investigation is required to mitigate the issue.\nWe will provide an update by Thursday, 2024-11-14 23:30 US/Pacific with current details.\n**Diagnosis:**\nCustomers impacted by this issue may observe errors during case creation.\n**Workaround:**\nIf case creation fails, impacted customers can use the Contact Us Form (CUF) that is automatically generated after the failure.","status":"SERVICE_OUTAGE"}],"most_recent_update":{"created":"2024-11-20T12:27:12+00:00","modified":"2024-11-20T12:29:52+00:00","when":"2024-11-20T12:27:12+00:00","text":"A detailed Incident Report has been posted on the Google\nCloud Service Health Dashboard: https://status.cloud.google.com/incidents/NEtbe4S7pMgACeuopMaZ","status":"AVAILABLE"},"status_impact":"SERVICE_OUTAGE","severity":"high","service_key":"dnz8HVUHoMMcW9wV91LY","service_name":"Google Workspace Support","affected_products":[{"title":"Google Workspace Support","id":"dnz8HVUHoMMcW9wV91LY"}],"uri":"incidents/f5n51VFCprgtnJFJf48a"},{"id":"qjmcFHHibfBDsxAjhFnU","number":"11419436351976551194","begin":"2024-11-08T16:05:00+00:00","created":"2024-11-08T17:17:01+00:00","end":"2024-11-08T17:08:00+00:00","modified":"2024-11-08T17:54:19+00:00","external_desc":"We're investigating reports of an issue with Google Workspace Support. We will provide more information shortly.","updates":[{"created":"2024-11-08T17:54:09+00:00","modified":"2024-11-08T17:54:19+00:00","when":"2024-11-08T17:54:09+00:00","text":"The issue with Google Workspace Support has been resolved for all affected users as of Friday, 2024-11-08 09:08 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-11-08T17:17:01+00:00","modified":"2024-11-08T17:54:09+00:00","when":"2024-11-08T17:17:01+00:00","text":"We're investigating reports of an issue with Google Workspace Support. We will provide more information shortly. Users are facing intermittent delays while creating the support cases. Users are requested to retry the process as the issue is intermittent.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-11-08T17:54:09+00:00","modified":"2024-11-08T17:54:19+00:00","when":"2024-11-08T17:54:09+00:00","text":"The issue with Google Workspace Support has been resolved for all affected users as of Friday, 2024-11-08 09:08 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"dnz8HVUHoMMcW9wV91LY","service_name":"Google Workspace Support","affected_products":[{"title":"Google Workspace Support","id":"dnz8HVUHoMMcW9wV91LY"}],"uri":"incidents/qjmcFHHibfBDsxAjhFnU"},{"id":"MUomphETLLW7et9VPTYa","number":"10045463923569170164","begin":"2024-10-30T12:30:00+00:00","created":"2024-10-30T17:11:08+00:00","end":"2024-10-30T18:46:00+00:00","modified":"2024-11-05T14:25:14+00:00","external_desc":"We're investigating reports of an issue with Gmail. We will provide more information shortly.","updates":[{"created":"2024-11-04T16:53:34+00:00","modified":"2024-11-05T14:25:14+00:00","when":"2024-11-04T16:53:34+00:00","text":"# Incident Report\n## Summary\nOn 29 October 2024 some Gmail web users experienced slow loading, unresponsive or crashed web pages for a duration of 22 hours, 46 minutes\nTo our customers whose businesses were impacted during this outage, we sincerely apologize. This is not the level of quality and reliability we strive to offer you.\n## Root Cause\nBrowser extensions, such as third-party ones, have significant freedom to run their own JavaScript code on web pages they're active on. This means they can modify the page's appearance and content, interact with remote servers, and potentially even disrupt the page's functionality.\nWhile extensions operate in their own isolated environment, they can technically access information from the main web page. However, the reverse isn't true, limiting the website's control over the extension's actions.\nEssentially, extensions have a wide range of capabilities with minimal oversight, potentially leading to unintended consequences or conflicts with the web page.\nA recent update to a third-party browser extension resulted in some Gmail web users who have these extensions installed experiencing performance problems, such as increased latency or unresponsiveness.\n## Remediation and Prevention\nGoogle engineers were alerted to the issue by internal monitoring on 30 October 2024 at 07:04 US/Pacific and immediately started an investigation. The issue was fully mitigated at 11:44 US/Pacific after the third-party mitigated the issue from their end.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n- Implement testing for popular browser extensions to better detect and quickly mitigate similar issues.\n- Improve internal documentation/processes to enable faster response and mitigation time for this type of issue.\n## Detailed Description of Impact\n### Gmail\nCustomers using the Gmail Web UI may have experienced increased latency or unresponsiveness when also using certain third-party extensions.\n**Additional Information for Customers:**\nThis issue only affected the Gmail Web UI. Gmail iOS/Android apps could be used as a workaround during the issue duration.","status":"AVAILABLE"},{"created":"2024-10-31T14:49:11+00:00","modified":"2024-11-04T16:53:46+00:00","when":"2024-10-31T14:49:11+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 29 October, 2024 13:00\n**Incident End:** 30 October, 2024 11:46\n**Duration:** 22 hours, 46 minutes\n**Affected Services and Features:**\nGmail\n**Regions/Zones:** Global\n**Description:**\nSome Gmail users experienced latency, unresponsiveness, and inability to load the Web UI for a duration of 22 hours, 46 minutes. From preliminary analysis, the root cause of the issue was an incompatibility with a third-party add-on following an update to Gmail front end servers.\n**Customer Impact:**\nCustomers using the Gmail Web UI may have experienced increased latency or unresponsiveness when also using certain 3rd party addons.\n**Additional details:**\nThis issue only affected the Gmail Web UI. Gmail iOS/Android apps could be used as a workaround.\nEngineers mitigated the issue by rolling back the recent update to Gmail front end servers.","status":"AVAILABLE"},{"created":"2024-10-30T19:00:21+00:00","modified":"2024-10-31T14:49:11+00:00","when":"2024-10-30T19:00:21+00:00","text":"The issue with Gmail has been resolved for all affected users as of Wednesday, 2024-10-30 11:46 US/Pacific.\nWe will publish an analysis of this incident once we have completed our internal investigation.\nWe thank you for your patience while we worked on resolving the issue.","status":"SERVICE_DISRUPTION"},{"created":"2024-10-30T17:20:16+00:00","modified":"2024-10-30T19:00:21+00:00","when":"2024-10-30T17:20:16+00:00","text":"Gmail users may experience slowness in Gmail UI, including lag and non-responsiveness during loading and sending operations.\nWe are experiencing an issue with Gmail beginning at Wednesday, 2024-10-30 05:30 US/Pacific.\nOur engineering team continues to investigate the issue. We will provide an update by Wednesday, 2024-10-30 12:30 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.","status":"SERVICE_DISRUPTION"},{"created":"2024-10-30T17:11:08+00:00","modified":"2024-10-30T17:20:16+00:00","when":"2024-10-30T17:11:08+00:00","text":"We're investigating reports of an issue with Gmail. We will provide more information shortly. Gmail users may experience slowness in Gmail UI, including lag and non-responsiveness during loading and sending operations.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-11-04T16:53:34+00:00","modified":"2024-11-05T14:25:14+00:00","when":"2024-11-04T16:53:34+00:00","text":"# Incident Report\n## Summary\nOn 29 October 2024 some Gmail web users experienced slow loading, unresponsive or crashed web pages for a duration of 22 hours, 46 minutes\nTo our customers whose businesses were impacted during this outage, we sincerely apologize. This is not the level of quality and reliability we strive to offer you.\n## Root Cause\nBrowser extensions, such as third-party ones, have significant freedom to run their own JavaScript code on web pages they're active on. This means they can modify the page's appearance and content, interact with remote servers, and potentially even disrupt the page's functionality.\nWhile extensions operate in their own isolated environment, they can technically access information from the main web page. However, the reverse isn't true, limiting the website's control over the extension's actions.\nEssentially, extensions have a wide range of capabilities with minimal oversight, potentially leading to unintended consequences or conflicts with the web page.\nA recent update to a third-party browser extension resulted in some Gmail web users who have these extensions installed experiencing performance problems, such as increased latency or unresponsiveness.\n## Remediation and Prevention\nGoogle engineers were alerted to the issue by internal monitoring on 30 October 2024 at 07:04 US/Pacific and immediately started an investigation. The issue was fully mitigated at 11:44 US/Pacific after the third-party mitigated the issue from their end.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n- Implement testing for popular browser extensions to better detect and quickly mitigate similar issues.\n- Improve internal documentation/processes to enable faster response and mitigation time for this type of issue.\n## Detailed Description of Impact\n### Gmail\nCustomers using the Gmail Web UI may have experienced increased latency or unresponsiveness when also using certain third-party extensions.\n**Additional Information for Customers:**\nThis issue only affected the Gmail Web UI. Gmail iOS/Android apps could be used as a workaround during the issue duration.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/MUomphETLLW7et9VPTYa"},{"id":"Pyk5VPCaY9XmmTRqTiEV","number":"1613848227674937289","begin":"2024-10-30T09:46:00+00:00","created":"2024-10-30T11:17:57+00:00","end":"2024-10-30T11:08:00+00:00","modified":"2024-10-30T11:23:03+00:00","external_desc":"**Summary:**\nGemini might be unavailable in France due to \"\"Gemini isn’t currently supported in your country. Stay tuned!\"\" message.\n**Description:**\nWe are experiencing an issue with Gemini. Mitigation work is now underway by our engineering team. We will provide an update by Wednesday, 2024-10-30 05:00 US/Pacific with current details. We apologize to all who are affected by the disruption.\n**Diagnosis:**\nCustomers accessing gemini.google.com in France might see \"Gemini isn’t currently supported in your country. Stay tuned!\" message.\nGemini functionalities such as \"Help me write\" in other Google Workspace products work properly.\n**Workaround:**\nNone right now.","updates":[{"created":"2024-10-30T11:22:48+00:00","modified":"2024-10-30T11:23:03+00:00","when":"2024-10-30T11:22:48+00:00","text":"The problem with Gemini has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.","status":"AVAILABLE"},{"created":"2024-10-30T11:17:57+00:00","modified":"2024-10-30T11:22:48+00:00","when":"2024-10-30T11:17:57+00:00","text":"**Summary:**\nGemini might be unavailable in France due to \"\"Gemini isn’t currently supported in your country. Stay tuned!\"\" message.\n**Description:**\nWe are experiencing an issue with Gemini. Mitigation work is now underway by our engineering team. We will provide an update by Wednesday, 2024-10-30 05:00 US/Pacific with current details. We apologize to all who are affected by the disruption.\n**Diagnosis:**\nCustomers accessing gemini.google.com in France might see \"Gemini isn’t currently supported in your country. Stay tuned!\" message.\nGemini functionalities such as \"Help me write\" in other Google Workspace products work properly.\n**Workaround:**\nNone right now.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-10-30T11:22:48+00:00","modified":"2024-10-30T11:23:03+00:00","when":"2024-10-30T11:22:48+00:00","text":"The problem with Gemini has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"npdyhgECDJ6tB66MxXyo","service_name":"Gemini","affected_products":[{"title":"Gemini","id":"npdyhgECDJ6tB66MxXyo"}],"uri":"incidents/Pyk5VPCaY9XmmTRqTiEV"},{"id":"EDinMUwayovibax78KZJ","number":"4675755073320703464","begin":"2024-10-21T17:38:00+00:00","created":"2024-11-06T20:33:26+00:00","end":"2024-11-07T17:04:00+00:00","modified":"2024-11-11T18:21:40+00:00","external_desc":"**Summary :**\nThere may be issues accessing Google Workspace Extensions (Google Calendar, Google Keep and Google Tasks) from Gemini app\n**Description :**\nWe are experiencing an issue with some Extensions in the Gemini app (gemini.google.com).\nOur engineering team continues to investigate the issue.\nWe will provide an update by Friday, 2024-11-08 00:30 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Diagnosis :**\nUsers may observe error messages when trying to use Calendar, Keep, and Tasks extensions from the Gemini app. This is limited to only enterprise users and does not impact consumer accounts.\n**Workaround :**\nNone at this time.","updates":[{"created":"2024-11-11T18:21:17+00:00","modified":"2024-11-11T18:21:40+00:00","when":"2024-11-11T18:21:17+00:00","text":"# Incident Report\n## Summary\nBetween 21 October 2024 and 07 November 2024, a subset of Gemini enterprise users experienced elevated error rate when attempting to use Calendar, Keep, and Tasks extensions from the Gemini App on Web.\nWe sincerely apologize to our Gemini customers for the disruption. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to prevent such disruptions in the future.\n## Root Cause\nOn 21 October 2024, Google began rolling out a feature to support Calendar, Tasks and Keep extensions on Gemini app ([gemini.google.com](http://gemini.google.com)) for Enterprise Customers that have Workspace extensions in Gemini set to “On.” Due to the gradual rollout process, the feature was enabled for individual domains on different dates, resulting in varying impact timeframes for different customers.\nThe issue was caused by an incorrect configuration settings on the backend server that controls the rollout of the new feature. These settings determine things like how many users can access the feature at once, and how the feature interacts with other parts of the system.The incorrect configuration prevented the Calendar, Keep, and Tasks extensions from being fully enabled for some enterprise users on Gemini Web, despite the extensions being visible.\n## Remediation and Prevention\nGoogle engineers were alerted to the issue via a customer support case on 29 October 2024 at 18:18 US/Pacific. Following initial investigations from support, the issue was escalated to our engineering team on 05 November 2024\\. Upon identifying the issue, engineers promptly disabled the controlling setting on the backend server, gradually enabled the correct configuration setting, and finally re-enabled the controlling setting on the backend server, thereby resolving the issue.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n* Enhanced Monitoring: We are enhancing real-time monitoring of Gemini feature rollouts to ensure prompt detection of any configuration issues.\n* Improved Rollout Sequencing: We are refining the rollout sequence to guarantee that extensions are fully enabled before being made available to customers.\n## Detailed Description of Impact\nBetween 21 October 2024 09:38 US/Pacific and 07 November 2024 09:04 US/Pacific, a subset of users may have observed the following error messages when trying to use Calendar, Keep, and Tasks extensions from the Gemini app. These messages were the responses by Gemini:\n* ‘I can't assist you with that, as I am only a language model.’\n* ‘I can’t access your Google Calendar because of privacy and security reasons, but I can help you find this information\\!’\n---------","status":"AVAILABLE"},{"created":"2024-11-07T20:30:32+00:00","modified":"2024-11-11T18:21:17+00:00","when":"2024-11-07T20:30:32+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 6 November, 2024 10:03\n**Incident End:** 7 November, 2024 9:04\n**Duration:** 23 hours, 1 minute\n**Affected Services and Features:**\nGemini extensions for the following services:\n- Google Calendar\n- Google Keep\n- Google Tasks\n**Regions/Zones:** Global (Impact was limited to Enterprise users, irrespective of location)\n**Description:**\nA subset of customers with Gemini subscriptions experienced Google Calendar, Tasks and Keep extensions not working in Gemini App on Web for a duration of 23 hours and 1 minute. From preliminary analysis, the root cause of the issue is a change related to a Gemini feature launch.\nGoogle engineers adjusted launch configurations in the backend to mitigate the impact. Google will complete a full IR in the following days that will provide a full root cause.\n**Customer Impact:**\nA subset of users may have observed the following error messages when trying to use Calendar, Keep, and Tasks extensions from the Gemini app. These messages were responses by Gemini:\n- ‘I can't assist you with that, as I am only a language model.’\n- ‘I can’t access your Google Calendar because of privacy and security reasons, but I can help you find this information!...’","status":"AVAILABLE"},{"created":"2024-11-07T17:30:06+00:00","modified":"2024-11-07T20:30:32+00:00","when":"2024-11-07T17:30:06+00:00","text":"The issue with Gemini has been resolved for all affected users as of Thursday, 2024-11-07 09:04 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"SERVICE_DISRUPTION"},{"created":"2024-11-06T20:33:26+00:00","modified":"2024-11-07T17:30:06+00:00","when":"2024-11-06T20:33:26+00:00","text":"**Summary :**\nThere may be issues accessing Google Workspace Extensions (Google Calendar, Google Keep and Google Tasks) from Gemini app\n**Description :**\nWe are experiencing an issue with some Extensions in the Gemini app (gemini.google.com).\nOur engineering team continues to investigate the issue.\nWe will provide an update by Friday, 2024-11-08 00:30 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Diagnosis :**\nUsers may observe error messages when trying to use Calendar, Keep, and Tasks extensions from the Gemini app. This is limited to only enterprise users and does not impact consumer accounts.\n**Workaround :**\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-11-11T18:21:17+00:00","modified":"2024-11-11T18:21:40+00:00","when":"2024-11-11T18:21:17+00:00","text":"# Incident Report\n## Summary\nBetween 21 October 2024 and 07 November 2024, a subset of Gemini enterprise users experienced elevated error rate when attempting to use Calendar, Keep, and Tasks extensions from the Gemini App on Web.\nWe sincerely apologize to our Gemini customers for the disruption. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to prevent such disruptions in the future.\n## Root Cause\nOn 21 October 2024, Google began rolling out a feature to support Calendar, Tasks and Keep extensions on Gemini app ([gemini.google.com](http://gemini.google.com)) for Enterprise Customers that have Workspace extensions in Gemini set to “On.” Due to the gradual rollout process, the feature was enabled for individual domains on different dates, resulting in varying impact timeframes for different customers.\nThe issue was caused by an incorrect configuration settings on the backend server that controls the rollout of the new feature. These settings determine things like how many users can access the feature at once, and how the feature interacts with other parts of the system.The incorrect configuration prevented the Calendar, Keep, and Tasks extensions from being fully enabled for some enterprise users on Gemini Web, despite the extensions being visible.\n## Remediation and Prevention\nGoogle engineers were alerted to the issue via a customer support case on 29 October 2024 at 18:18 US/Pacific. Following initial investigations from support, the issue was escalated to our engineering team on 05 November 2024\\. Upon identifying the issue, engineers promptly disabled the controlling setting on the backend server, gradually enabled the correct configuration setting, and finally re-enabled the controlling setting on the backend server, thereby resolving the issue.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n* Enhanced Monitoring: We are enhancing real-time monitoring of Gemini feature rollouts to ensure prompt detection of any configuration issues.\n* Improved Rollout Sequencing: We are refining the rollout sequence to guarantee that extensions are fully enabled before being made available to customers.\n## Detailed Description of Impact\nBetween 21 October 2024 09:38 US/Pacific and 07 November 2024 09:04 US/Pacific, a subset of users may have observed the following error messages when trying to use Calendar, Keep, and Tasks extensions from the Gemini app. These messages were the responses by Gemini:\n* ‘I can't assist you with that, as I am only a language model.’\n* ‘I can’t access your Google Calendar because of privacy and security reasons, but I can help you find this information\\!’\n---------","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"npdyhgECDJ6tB66MxXyo","service_name":"Gemini","affected_products":[{"title":"Gemini","id":"npdyhgECDJ6tB66MxXyo"}],"uri":"incidents/EDinMUwayovibax78KZJ"},{"id":"kLLQnDxxmwSec6Gtagnt","number":"13188155848876887199","begin":"2024-10-18T18:30:00+00:00","created":"2024-10-18T20:04:15+00:00","end":"2024-10-20T00:15:00+00:00","modified":"2024-11-04T17:45:38+00:00","external_desc":"We're investigating reports of an issue with Gmail. We will provide more information shortly.","updates":[{"created":"2024-11-04T17:45:38+00:00","modified":"2024-11-04T17:45:38+00:00","when":"2024-11-04T17:45:38+00:00","text":"# Incident Report\n## Summary\nBeginning on 18 October 2024, users of the Gmail application for iOS experienced increased application crash rates for a duration of 1 day, 5 hours, 45 minutes. To our Gmail for iOS users who were impacted by this issue, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nThe Gmail iOS application uses a Google-wide SSO client library to authenticate all accounts on a given device with a backend authentication server. This includes a lookup of account IDs and their corresponding email addresses. Email addresses are returned from the authentication server normalized to avoid potential formatting conflicts. For example, “TestUser@gmail.com” would be normalized to “testuser@gmail.com.”\nA recent server-side feature release introduced an issue where the backend authentication server began returning non-normalized email addresses. This resulted in a formatting mismatch where the Gmail iOS application was unable to match account IDs to the normalized email addresses stored in memory when the application was initially launched, causing the application to crash.\n## Remediation and Prevention\nGoogle engineers were alerted to the issue via internal monitoring on 18 October 2024 at 11:58 US/Pacific and immediately started an investigation.\nEngineers rolled back two recent updates to identity endpoints used by Gmail iOS to mitigate impact on 19 October 2024 at 17:15 US/Pacific.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n* Make Gmail iOS more resilient to identity endpoint changes\n* Limit these identity changes to in-scope users\n* Improved attribution of future backend production changes\n## Detailed Description of Impact\nBetween 18 October 2024 at 11:30 US/Pacific and 19 October 2024 at 17:15 US/Pacific, approximately 2.6M users of Gmail for iOS experienced foreground crashes on application startup.\nImpact was limited to the Gmail application for the iOS platform. Users did not face any errors while using Gmail in other platforms, including Android, Web interfaces.","status":"AVAILABLE"},{"created":"2024-10-21T20:28:43+00:00","modified":"2024-11-04T17:45:38+00:00","when":"2024-10-21T20:28:43+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 18 October 2024 11:30\n**Incident End:** 19 October 2024 17:15\n**Duration:** 1 day, 5 hours, 45 minutes\n**Affected Services and Features:**\nGmail\n**Regions/Zones:** Global\n**Description:**\nThe Gmail application for iOS experienced an elevated crash rate for a duration of 1 day, 5 hours, 45 minutes.\nFrom preliminary analysis, the root cause of the issue was two changes to identity endpoints used by Gmail iOS that caused it to crash.\nGoogle will complete a full incident report in the following days that will provide a full root cause.\n**Customer Impact:**\nApproximately 1 million Gmail for iOS platform users had foreground crashes on start of the application during this period of impact.\nImpact was limited to the Gmail application for the iOS platform, and users did not face any errors while using Gmail in other platforms, including Android, Web interfaces.","status":"AVAILABLE"},{"created":"2024-10-20T01:58:04+00:00","modified":"2024-10-21T20:28:43+00:00","when":"2024-10-20T01:58:04+00:00","text":"The problem with Gmail has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.\nWe will publish an analysis of this incident once we have completed our internal investigation.","status":"SERVICE_DISRUPTION"},{"created":"2024-10-19T04:52:53+00:00","modified":"2024-10-20T01:58:04+00:00","when":"2024-10-19T04:52:53+00:00","text":"### **Summary:**\nGmail iOS application crashes\n### **Description:**\nWe are experiencing an issue with Gmail for iOS.\nOur engineering team is working on multiple streams of investigation to mitigate the issue at the earliest.\nWe will provide an update by Monday, 2024-10-21 10:00 US/Pacific with current details.\n### **Diagnosis:**\nCustomers using Gmail for iOS may experience application crashes.\n### **Workaround:**\nThe affected users should access Gmail through the web interface.","status":"SERVICE_DISRUPTION"},{"created":"2024-10-19T01:02:26+00:00","modified":"2024-10-19T04:52:53+00:00","when":"2024-10-19T01:02:26+00:00","text":"We are experiencing an issue with Gmail for iOS.\nOur engineering team is working on multiple streams of investigation to mitigate the issue at the earliest.\nWe will provide an update by Friday, 2024-10-18 22:00 US/Pacific with current details.\n**Symptom:**\nCustomers using Gmail for iOS may experience application crashes.","status":"SERVICE_DISRUPTION"},{"created":"2024-10-18T22:52:17+00:00","modified":"2024-10-19T01:02:26+00:00","when":"2024-10-18T22:52:17+00:00","text":"We are experiencing an issue with Gmail for iOS.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Friday, 2024-10-18 18:00 US/Pacific with current details.\n**Symptom:**\nCustomers using Gmail for iOS may experience application crashes.","status":"SERVICE_DISRUPTION"},{"created":"2024-10-18T20:56:42+00:00","modified":"2024-10-18T22:52:17+00:00","when":"2024-10-18T20:56:42+00:00","text":"We are experiencing an issue with Gmail for iOS.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Friday, 2024-10-18 16:00 US/Pacific with current details.\n**Symptom:**\nCustomers using Gmail for iOS may experience application crashes.","status":"SERVICE_INFORMATION"},{"created":"2024-10-18T20:04:15+00:00","modified":"2024-10-18T20:56:42+00:00","when":"2024-10-18T20:04:15+00:00","text":"We're investigating reports of an issue with Gmail. We will provide more information shortly. We are experiencing an issue with Gmail for iOS.\nOur engineering team continues to investigate the issue. Customers using Gmail for iOS may experience application crashes.","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-11-04T17:45:38+00:00","modified":"2024-11-04T17:45:38+00:00","when":"2024-11-04T17:45:38+00:00","text":"# Incident Report\n## Summary\nBeginning on 18 October 2024, users of the Gmail application for iOS experienced increased application crash rates for a duration of 1 day, 5 hours, 45 minutes. To our Gmail for iOS users who were impacted by this issue, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nThe Gmail iOS application uses a Google-wide SSO client library to authenticate all accounts on a given device with a backend authentication server. This includes a lookup of account IDs and their corresponding email addresses. Email addresses are returned from the authentication server normalized to avoid potential formatting conflicts. For example, “TestUser@gmail.com” would be normalized to “testuser@gmail.com.”\nA recent server-side feature release introduced an issue where the backend authentication server began returning non-normalized email addresses. This resulted in a formatting mismatch where the Gmail iOS application was unable to match account IDs to the normalized email addresses stored in memory when the application was initially launched, causing the application to crash.\n## Remediation and Prevention\nGoogle engineers were alerted to the issue via internal monitoring on 18 October 2024 at 11:58 US/Pacific and immediately started an investigation.\nEngineers rolled back two recent updates to identity endpoints used by Gmail iOS to mitigate impact on 19 October 2024 at 17:15 US/Pacific.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n* Make Gmail iOS more resilient to identity endpoint changes\n* Limit these identity changes to in-scope users\n* Improved attribution of future backend production changes\n## Detailed Description of Impact\nBetween 18 October 2024 at 11:30 US/Pacific and 19 October 2024 at 17:15 US/Pacific, approximately 2.6M users of Gmail for iOS experienced foreground crashes on application startup.\nImpact was limited to the Gmail application for the iOS platform. Users did not face any errors while using Gmail in other platforms, including Android, Web interfaces.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/kLLQnDxxmwSec6Gtagnt"},{"id":"xp8QVzRnRpE9m3Gz444C","number":"18293782856697911609","begin":"2024-10-17T02:36:00+00:00","created":"2024-10-17T05:02:31+00:00","end":"2024-10-17T09:45:00+00:00","modified":"2024-10-17T21:25:13+00:00","external_desc":"We are experiencing an issue with Google Workspace Support.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Wednesday, 2024-10-16 23:00 US/Pacific with current details.","updates":[{"created":"2024-10-17T21:24:57+00:00","modified":"2024-10-17T21:25:13+00:00","when":"2024-10-17T21:24:57+00:00","text":"A mini incident report has been posted on the [Google\nCloud Service Health Dashboard](https://status.cloud.google.com/incidents/cyHovCjtDs1RPbZAKsme)","status":"AVAILABLE"},{"created":"2024-10-17T12:49:55+00:00","modified":"2024-10-17T21:24:57+00:00","when":"2024-10-17T12:49:55+00:00","text":"The problem with Google Workspace Support has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.","status":"AVAILABLE"},{"created":"2024-10-17T11:30:17+00:00","modified":"2024-10-17T12:49:55+00:00","when":"2024-10-17T11:30:17+00:00","text":"### **Summary:**\nCase creation may fail intermittently with an error\n### **Description:**\nOur engineering team has determined that further investigation is required to mitigate the issue.\nWe will provide an update by Thursday, 2024-10-17 08:00 US/Pacific with current details.\n### **Diagnosis:**\nCustomers impacted by this issue may see issue with case creation and other case updates.\n### **Workaround:**\nRetrying operations may work. Impacted customers may contact Google Workspace Support through chat in the admin console.","status":"SERVICE_DISRUPTION"},{"created":"2024-10-17T09:27:08+00:00","modified":"2024-10-17T11:30:17+00:00","when":"2024-10-17T09:27:08+00:00","text":"### **Summary:**\nCase creation may fail intermittently with an error\n### **Description:**\nMitigation work is currently underway by our engineering team.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Thursday, 2024-10-17 05:00 US/Pacific.\n### **Diagnosis:**\nCustomers impacted by this issue may see issue with case creation and other case updates\n### **Workaround:**\nRetrying operations may work. Impacted customers may contact Google Workspace Support through chat in the admin console.","status":"SERVICE_DISRUPTION"},{"created":"2024-10-17T07:21:38+00:00","modified":"2024-10-17T09:27:08+00:00","when":"2024-10-17T07:21:38+00:00","text":"### **Summary:**\nCase creation may fail intermittently with an error\n### **Description:**\nWe are experiencing an issue with Google Workspace Support.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Thursday, 2024-10-17 02:30 US/Pacific with current details.\n### **Diagnosis:**\nCustomers impacted by this issue may see issue with case creation and other case updates\n### **Workaround:**\nRetrying operations may work. Impacted customers may contact Google Workspace Support through chat in the admin console.","status":"SERVICE_OUTAGE"},{"created":"2024-10-17T05:42:31+00:00","modified":"2024-10-17T07:21:38+00:00","when":"2024-10-17T05:42:31+00:00","text":"### **Summary:**\nCase creation may fail intermittently with an error\n### **Description:**\nWe are experiencing an issue with Google Workspace Support.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Thursday, 2024-10-17 00:30 US/Pacific with current details.\n### **Diagnosis:**\nCustomers impacted by this issue may see issue with case creation and other case updates\n### **Workaround:**\nRetrying operations may work. Impacted customers may contact Google Workspace Support through chat in the admin console.","status":"SERVICE_OUTAGE"},{"created":"2024-10-17T05:06:07+00:00","modified":"2024-10-17T05:42:31+00:00","when":"2024-10-17T05:06:07+00:00","text":"### **Summary:**\nCase creation may fail intermittently with an error\n### **Description:**\nWe are experiencing an issue with Google Workspace Support.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Wednesday, 2024-10-16 23:00 US/Pacific with current details.\n### **Diagnosis:**\nCustomers impacted by this issue may see issue with case creation and other case updates\n### **Workaround:**\nRetrying operations may work.","status":"SERVICE_OUTAGE"},{"created":"2024-10-17T05:02:31+00:00","modified":"2024-10-17T05:06:07+00:00","when":"2024-10-17T05:02:31+00:00","text":"We are experiencing an issue with Google Workspace Support.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Wednesday, 2024-10-16 23:00 US/Pacific with current details.","status":"SERVICE_OUTAGE"}],"most_recent_update":{"created":"2024-10-17T21:24:57+00:00","modified":"2024-10-17T21:25:13+00:00","when":"2024-10-17T21:24:57+00:00","text":"A mini incident report has been posted on the [Google\nCloud Service Health Dashboard](https://status.cloud.google.com/incidents/cyHovCjtDs1RPbZAKsme)","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"dnz8HVUHoMMcW9wV91LY","service_name":"Google Workspace Support","affected_products":[{"title":"Google Workspace Support","id":"dnz8HVUHoMMcW9wV91LY"}],"uri":"incidents/xp8QVzRnRpE9m3Gz444C"},{"id":"ESTn41B9afCeGkPKBZAR","number":"18003747602384613121","begin":"2024-10-02T21:20:00+00:00","created":"2024-10-02T23:59:19+00:00","end":"2024-10-02T23:12:00+00:00","modified":"2024-10-07T23:49:05+00:00","external_desc":"**Summary:**\nCustomers may experience intermittent issues with support case creation\n**Description:**\nWe are experiencing an intermittent issue with Google Workspace Support case creation beginning on Wednesday, 2024-10-02 14:20 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Wednesday, 2024-10-02 18:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Diagnosis:**\nImpacted customers may receive an unexpected error while attempting case creation.\n**Workaround:**\nRetrying the case creation process may be successful due to the intermittent nature of the issue.","updates":[{"created":"2024-10-07T23:49:05+00:00","modified":"2024-10-07T23:49:05+00:00","when":"2024-10-07T23:49:05+00:00","text":"A detailed incident report has been posted on the Google\nCloud Service Health Dashboard : https://status.cloud.google.com/incidents/MXtAVH2fTnMpaCDx53JN","status":"AVAILABLE"},{"created":"2024-10-03T20:01:37+00:00","modified":"2024-10-07T23:49:05+00:00","when":"2024-10-03T20:01:37+00:00","text":"The Mini Incident Report is available on the Google Cloud Service Health Dashboard : https://status.cloud.google.com/incidents/MXtAVH2fTnMpaCDx53JN","status":"AVAILABLE"},{"created":"2024-10-03T01:53:41+00:00","modified":"2024-10-03T20:01:37+00:00","when":"2024-10-03T01:53:41+00:00","text":"The issue with Google Workspace Support case creation has been resolved for all affected customers as of Wednesday, 2024-10-02 16:12 US/Pacific. We thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-10-03T01:10:44+00:00","modified":"2024-10-03T01:53:41+00:00","when":"2024-10-03T01:10:44+00:00","text":"**Summary:**\nCustomers may experience intermittent issues with support case creation\n**Description:**\nWe experienced an issue where some customers were unable to create cases. This issue has been mitigated as of 16:12 US/Pacific. The support case creation process is now working successfully, but we continue to investigate the underlying cause and monitor our environment for stability. Initial investigations have determined a temporary issue due to an unexpected increase in traffic.\nWe will provide an update by Wednesday, 2024-10-02 20:00 US/Pacific with current details.\n**Diagnosis:**\nImpacted customers may receive an unexpected error while attempting case creation.\n**Workaround:**\nRetrying the case creation process may be successful due to the intermittent nature of the issue.","status":"SERVICE_DISRUPTION"},{"created":"2024-10-02T23:59:19+00:00","modified":"2024-10-03T01:10:44+00:00","when":"2024-10-02T23:59:19+00:00","text":"**Summary:**\nCustomers may experience intermittent issues with support case creation\n**Description:**\nWe are experiencing an intermittent issue with Google Workspace Support case creation beginning on Wednesday, 2024-10-02 14:20 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Wednesday, 2024-10-02 18:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Diagnosis:**\nImpacted customers may receive an unexpected error while attempting case creation.\n**Workaround:**\nRetrying the case creation process may be successful due to the intermittent nature of the issue.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-10-07T23:49:05+00:00","modified":"2024-10-07T23:49:05+00:00","when":"2024-10-07T23:49:05+00:00","text":"A detailed incident report has been posted on the Google\nCloud Service Health Dashboard : https://status.cloud.google.com/incidents/MXtAVH2fTnMpaCDx53JN","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"dnz8HVUHoMMcW9wV91LY","service_name":"Google Workspace Support","affected_products":[{"title":"Google Workspace Support","id":"dnz8HVUHoMMcW9wV91LY"}],"uri":"incidents/ESTn41B9afCeGkPKBZAR"},{"id":"XmBN1zthW43h8LVENBXY","number":"9307109642312185616","begin":"2024-09-16T17:11:00+00:00","created":"2024-09-16T17:49:05+00:00","end":"2024-09-17T04:50:00+00:00","modified":"2024-09-17T05:19:47+00:00","external_desc":"**Summary:**\nGoogle Drive users may experience increased error rates.\n**Description:**\nWe are experiencing an intermittent issue with Google Drive beginning on Monday, 2024-09-16 10:11 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Monday, 2024-09-16 11:45 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Diagnosis:**\nGoogle Drive users may experience increased error rates.\n**Workaround:**\nCustomers may see success in retries.","updates":[{"created":"2024-09-17T05:19:35+00:00","modified":"2024-09-17T05:19:35+00:00","when":"2024-09-17T05:19:35+00:00","text":"The issue with Google Drive has been resolved for all affected users as of Monday, 2024-09-16 21:50 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-09-16T23:51:23+00:00","modified":"2024-09-16T23:51:23+00:00","when":"2024-09-16T23:51:23+00:00","text":"**Summary:**\nGoogle Drive users may experience increased error rates.\n**Description:**\nWe are experiencing an intermittent issue with Google Drive beginning on Monday, 2024-09-16 10:11 US/Pacific.\nImpact has subsided while the engineering team continues to investigate the root cause to ensure the issue does not reoccur.\nOur engineering team is currently implementing the mitigation to further reduce the risk of recurrence. The mitigation is expected to complete by Monday, 2024-09-16 22:30 US/Pacific\nIn the meantime, we are closely monitoring the situation and we will provide an update by Monday, 2024-09-16 23:00 US/Pacific with current details.\n**Diagnosis:**\nGoogle Drive users may experience increased error rates.\n**Workaround:**\nCustomers may see success in retries.","status":"SERVICE_INFORMATION"},{"created":"2024-09-16T20:38:24+00:00","modified":"2024-09-16T20:38:24+00:00","when":"2024-09-16T20:38:24+00:00","text":"**Summary:**\nGoogle Drive users may experience increased error rates.\n**Description:**\nWe are experiencing an intermittent issue with Google Drive beginning on Monday, 2024-09-16 10:11 US/Pacific.\nImpact has subsided while the engineering team continues to investigate the root cause to ensure the issue does not reoccur.\nWe are currently working on implementing a mitigation to further reduce the risk of recurrence. In the meantime, we are closely monitoring the situation and we will provide an update by Monday, 2024-09-16 17:30 US/Pacific with current details.\n**Diagnosis:**\nGoogle Drive users may experience increased error rates.\n**Workaround:**\nCustomers may see success in retries.","status":"SERVICE_INFORMATION"},{"created":"2024-09-16T19:44:21+00:00","modified":"2024-09-16T19:44:21+00:00","when":"2024-09-16T19:44:21+00:00","text":"**Summary:**\nGoogle Drive users may experience increased error rates.\n**Description:**\nWe are experiencing an intermittent issue with Google Drive beginning at Monday, 2024-09-16 10:11 US/Pacific.\nImpact has subsided while the engineering team continues to investigate the root cause to ensure the issue does not reoccur.\nWe will provide an update by Monday, 2024-09-16 13:45 US/Pacific with current details.\n**Diagnosis:**\nGoogle Drive users may experience increased error rates.\n**Workaround:**\nCustomers may see success in retries.","status":"SERVICE_INFORMATION"},{"created":"2024-09-16T18:46:02+00:00","modified":"2024-09-16T18:46:02+00:00","when":"2024-09-16T18:46:02+00:00","text":"**Summary:**\nGoogle Drive users may experience increased error rates.\n**Description:**\nWe are experiencing an intermittent issue with Google Drive beginning at Monday, 2024-09-16 10:11 US/Pacific.\nImpact has subsided while the engineering team continues to investigate the root cause to ensure the issue does not reoccur.\nWe will provide an update by Monday, 2024-09-16 12:45 US/Pacific with current details.\n**Diagnosis:**\nGoogle Drive users may experience increased error rates.\n**Workaround:**\nCustomers may see success in retries.","status":"SERVICE_INFORMATION"},{"created":"2024-09-16T17:49:07+00:00","modified":"2024-09-16T17:49:07+00:00","when":"2024-09-16T17:49:07+00:00","text":"**Summary:**\nGoogle Drive users may experience increased error rates.\n**Description:**\nWe are experiencing an intermittent issue with Google Drive beginning on Monday, 2024-09-16 10:11 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Monday, 2024-09-16 11:45 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Diagnosis:**\nGoogle Drive users may experience increased error rates.\n**Workaround:**\nCustomers may see success in retries.","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-09-17T05:19:35+00:00","modified":"2024-09-17T05:19:35+00:00","when":"2024-09-17T05:19:35+00:00","text":"The issue with Google Drive has been resolved for all affected users as of Monday, 2024-09-16 21:50 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"VHNA7p3Z5p3iakj5sA8V","service_name":"Google Drive","affected_products":[{"title":"Google Drive","id":"VHNA7p3Z5p3iakj5sA8V"}],"uri":"incidents/XmBN1zthW43h8LVENBXY"},{"id":"F5NGEmCZ9EC3gNGCckn6","number":"15088881287033216919","begin":"2024-08-19T18:10:00+00:00","created":"2024-08-19T19:04:17+00:00","end":"2024-08-19T18:30:00+00:00","modified":"2024-08-22T18:51:08+00:00","external_desc":"We're investigating reports of an issue with Gmail. We will provide more information shortly.","updates":[{"created":"2024-08-22T18:51:08+00:00","modified":"2024-08-22T18:51:08+00:00","when":"2024-08-22T18:51:08+00:00","text":"# Incident Report\n## Summary\nOn Monday, 19 August 2024, at 11:10 AM US/Pacific, Gmail experienced an increase in the number of server unavailability (500/502) errors for a duration of 20 minutes. Impacted users may have encountered errors when attempting to access Gmail via IMAP / web / mobile clients.\nTo our affected customers, we sincerely apologize for the inconvenience this disruption may have caused. Our engineers have been focused on the investigation and analysis that followed, in order to ensure this issue does not recur.\n## Root Cause\nOur engineering team has conducted a thorough investigation of the underlying cause and has identified a combination of factors that led to the issue:\n- A previously unknown condition existed in our system that, when triggered, could result in performance slowdowns and errors when the service instances are restarted during high-load situations.\n- A recent change that was introduced to improve the efficiency of our system interacted negatively with an ongoing unrelated canary-testing of another system configuration. This combination resulted in a high load situation.\n- At the same time, a routine rollout of a new software version caused a restart of the canary test area during a peak usage period, further increasing the load.\n## Remediation and Prevention\nGoogle engineers were alerted to the service disruption on Monday, 19 August 2024 at 11:19 AM US/Pacific via our proactive monitoring system and immediately started an investigation. The team quickly concluded that the errors were caused by instance restarts and immediately halted all deployments to mitigate the issue at 11:30 US/Pacific.\nGoogle is committed to preventing a repeat of the issue in the future and is completing the following actions:\n- Fix the latent issue that causes performance slowdowns and system errors when service instances are restarted under high load.\n- Introduce a step to force a system restart under high load as part of our efficiency rollouts process.\nIf your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability.\n## Detailed Description of Impact\nOn Monday, 19 August, 2024 at 11:10 AM US/Pacific, a small percentage of Gmail customers experienced increased server unavailability errors for a duration of 20 minutes. Customers who were impacted experienced Gmail account unavailability errors on their IMAP, web, and mobile app. The impacted customers also experienced Internal server (500) errors along with other in app errors.\n---","status":"AVAILABLE"},{"created":"2024-08-19T23:05:29+00:00","modified":"2024-08-19T23:05:29+00:00","when":"2024-08-19T23:05:29+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 19 August, 2024 11:10\n**Incident End:** 19 August, 2024 11:30\n**Duration:** 20 minutes\n**Affected Services and Features:**\nGmail\n**Regions/Zones:** Global\n**Description:**\nOn 19 August, 2024 11:10 US/Pacific, Gmail experienced increased server unavailability errors for a duration of 20 minutes. From preliminary analysis, the root cause of the issue was resource contention issues on the Gmail backend.\nGoogle will complete a full Incident Report in the following days that will provide a full root cause.\n**Customer Impact:**\nCustomers may have been unable to access Gmail via IMAP, web or mobile clients.","status":"AVAILABLE"},{"created":"2024-08-19T19:17:55+00:00","modified":"2024-08-19T19:17:55+00:00","when":"2024-08-19T19:17:55+00:00","text":"The issue with Gmail has been resolved for all affected users as of Monday, 2024-08-19 12:03 US/Pacific.\nWe will publish an analysis of this incident once we have completed our internal investigation.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-08-19T19:04:18+00:00","modified":"2024-08-19T19:04:18+00:00","when":"2024-08-19T19:04:18+00:00","text":"We're investigating reports of an issue with Gmail. We will provide more information shortly.","status":"SERVICE_OUTAGE"}],"most_recent_update":{"created":"2024-08-22T18:51:08+00:00","modified":"2024-08-22T18:51:08+00:00","when":"2024-08-22T18:51:08+00:00","text":"# Incident Report\n## Summary\nOn Monday, 19 August 2024, at 11:10 AM US/Pacific, Gmail experienced an increase in the number of server unavailability (500/502) errors for a duration of 20 minutes. Impacted users may have encountered errors when attempting to access Gmail via IMAP / web / mobile clients.\nTo our affected customers, we sincerely apologize for the inconvenience this disruption may have caused. Our engineers have been focused on the investigation and analysis that followed, in order to ensure this issue does not recur.\n## Root Cause\nOur engineering team has conducted a thorough investigation of the underlying cause and has identified a combination of factors that led to the issue:\n- A previously unknown condition existed in our system that, when triggered, could result in performance slowdowns and errors when the service instances are restarted during high-load situations.\n- A recent change that was introduced to improve the efficiency of our system interacted negatively with an ongoing unrelated canary-testing of another system configuration. This combination resulted in a high load situation.\n- At the same time, a routine rollout of a new software version caused a restart of the canary test area during a peak usage period, further increasing the load.\n## Remediation and Prevention\nGoogle engineers were alerted to the service disruption on Monday, 19 August 2024 at 11:19 AM US/Pacific via our proactive monitoring system and immediately started an investigation. The team quickly concluded that the errors were caused by instance restarts and immediately halted all deployments to mitigate the issue at 11:30 US/Pacific.\nGoogle is committed to preventing a repeat of the issue in the future and is completing the following actions:\n- Fix the latent issue that causes performance slowdowns and system errors when service instances are restarted under high load.\n- Introduce a step to force a system restart under high load as part of our efficiency rollouts process.\nIf your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability.\n## Detailed Description of Impact\nOn Monday, 19 August, 2024 at 11:10 AM US/Pacific, a small percentage of Gmail customers experienced increased server unavailability errors for a duration of 20 minutes. Customers who were impacted experienced Gmail account unavailability errors on their IMAP, web, and mobile app. The impacted customers also experienced Internal server (500) errors along with other in app errors.\n---","status":"AVAILABLE"},"status_impact":"SERVICE_OUTAGE","severity":"high","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/F5NGEmCZ9EC3gNGCckn6"},{"id":"5AjHWPCTCP5uenZRjuwz","number":"16964296537047791034","begin":"2024-08-12T13:20:00+00:00","created":"2024-08-12T15:04:42+00:00","end":"2024-08-12T15:32:00+00:00","modified":"2024-08-15T21:50:16+00:00","external_desc":"**SUMMARY**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region\n**DESCRIPTION**\nWe have experienced an issue with Gmail, Google Drive, Google Calendar, Google Docs, Google Chat, Google Tasks beginning at Monday, 2024-08-12 06:28 US/Pacific. The issue is now mitigated and our engineering teams are closely monitoring for any residual impact We will provide an update by Monday, 2024-08-12 09:15 US/Pacific with current details. We apologize to all who are affected by the disruption.\n**DIAGNOSIS**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region.\n**WORKAROUND**\nNone at this time.","updates":[{"created":"2024-08-15T21:42:40+00:00","modified":"2024-08-15T21:42:40+00:00","when":"2024-08-15T21:42:40+00:00","text":"## Incident Report\n## Summary\nOn 12 August 2024 at 06:20 US/Pacific, multiple Google Cloud and Google Workspace products experienced connectivity issues in europe-west2 for a duration of 40 minutes. During the time, ingress traffic to europe-west2 and egress traffic from europe-west2 experienced elevated latencies, connection timeouts, and connection failures.\n## Root Cause\nOn 12 August 2024 06:20 US/Pacific, primary and backup power feeds were both lost in a [Google Point of Presence (POP)](https://cloud.google.com/vpc/docs/edge-locations) due to a substation switchgear failure. The affected POP hosts about ⅓ of serving first-layer [Google Front Ends (GFEs)](https://cloud.google.com/docs/security/infrastructure/design#google-frontend-service) located in europe-west2 and some distributed networking equipment for that region. The power loss impacted the following Google products and services that depend on GFEs in that region:\n* Google Cloud APIs, Google Workspace, and other Google services like YouTube,\n* Customer-created global external application and proxy network load balancers, including Cloud CDN\nThe power loss also impacted the following Google Cloud products which depended on impacted networking equipment:\n* Customer-created regional external application, proxy network, and passthrough network load balancers in the europe-west2 region,\n* External protocol forwarding and VM external IP address connectivity for VMs in the europe-west2 region.\n* Google Cloud Interconnect connections in some LHR colocation facilities.\nImpact was limited to situations where either or both of the following was true:\n* Inbound requests or connections were routed into the europe-west2 region of Google’s network, from the Internet, and those requests or connections depended on networking equipment that was offline, or unreachable pending reconvergence.\n* Outbound responses were routed to the Internet, from the europe-west2 region of Google’s network, and those responses depended on networking equipment that was without power.\nThe power outage caused Internet routes advertised by Google to be withdrawn in networks connected to Google’s network. The withdrawn routes were automatically replaced by other Google-advertised routes that didn’t depend on impacted networking equipment. Withdrawing and replacing routes relies on the BGP protocol and its timers, so replacement route convergence is not instantaneous, and overloading in the automatically selected replacement route GFEs extended the duration of the incident.\n## Detailed Description of Impact\n* Google Workspace: _Gmail, Google Calendar, Google Chat, Google Docs, Google Drive, Google Meet and Google Tasks users connecting to Workspace services from the UK region and surrounding areas experienced connectivity issues as described in the next point.\n* GFE-based products and services: _Customers on the Internet experienced a spike of broken connections followed by elevated latencies or HTTP error responses when communicating with GFE-powered Google APIs and services or customer-created global external application and proxy network load balancers. At roughly 06:23 US/Pacific, Google automatically redirected connections to the nearest possible first-layer GFEs with some latency penalty. Unfortunately, some of the nearest possible first-layer GFEs were overloaded until 06:48 when Google engineers made adjustments to more efficiently distribute incoming requests among nearby first-layer GFEs. Depending on the Google API or service or the customer-created global external load balancer, elevated latencies could have persisted until about 08:30 US/Pacific. Elevated latencies also could have applied to customer-created global external load balancers that had Cloud CDN enabled.\n* Regional Google Cloud products and services: _Until replacement routes were in effect, customers on the Internet experienced connection failures to the following GCP resources in the europe-west2 region: * Regional external application, proxy network, and passthrough network load balancers. * External protocol forwarding and VM external IP addresses.\n* Google Cloud Interconnect: _Google Cloud Interconnect connections in some LHR colocation facilities (lhr-zone1-47, lhr-zone1-832, lhr-zone1-2262, lhr-zone1-4885, lhr-zone1-99051 and lhr-zone2-47) remained offline from 06:20 US/Pacific to at least 06:57 US/Pacific, when power was restored.\nAt 06:43 US/Pacific, power was restored to the impacted networking equipment. Google networking equipment was fully operational by 06:57 US/Pacific, and connectivity to GFE-based products and services, regional Google Cloud products and services, and Google Cloud Interconnect resumed shortly thereafter.\n## Remediation and Prevention\nMultiple Google engineering teams were alerted and automated recovery tooling was triggered as expected; however, manual adjustments were required to address subsequent first-layer GFE overload. Google is reviewing automation improvements in tasks that required manual intervention to reduce the duration of future power event impact. Similarly, Google is working to increase Cloud Interconnect control plane resilience and reduce mitigation time through automated reaction to isolation events.\nAdditionally Google's partner who maintains the affected facility power in LHR (London) is conducting a full root cause analysis with the switchboard manufacturer and substation owner(s) involved in supplying power, including follow up as to why stored or generated on-site emergency power did not carry loads.","status":"AVAILABLE"},{"created":"2024-08-12T19:33:02+00:00","modified":"2024-08-12T19:33:02+00:00","when":"2024-08-12T19:33:02+00:00","text":"A mini incident report has been posted to\nhttps://status.cloud.google.com/incidents/ETJGhvY9Xaktw7tgi8dF","status":"AVAILABLE"},{"created":"2024-08-12T15:39:36+00:00","modified":"2024-08-12T15:39:36+00:00","when":"2024-08-12T15:39:36+00:00","text":"The issue with Gmail, Google Calendar, Google Chat, Google Docs, Google Drive, Google Tasks has been resolved for all affected users as of Monday, 2024-08-12 08:35 US/Pacific.\nDuring the issue, users connecting to Workspace services from the UK region may have experienced connectivity issues.\nWe will publish an analysis of this incident once we have completed our internal investigation.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-08-12T15:16:40+00:00","modified":"2024-08-12T15:16:40+00:00","when":"2024-08-12T15:16:40+00:00","text":"**SUMMARY**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region\n**DESCRIPTION**\nWe have experienced an issue with Gmail, Google Drive, Google Calendar, Google Docs, Google Chat, Google Tasks beginning at Monday, 2024-08-12 06:28 US/Pacific. The issue is now mitigated and our engineering teams are closely monitoring for any residual impact We will provide an update by Monday, 2024-08-12 09:15 US/Pacific with current details. We apologize to all who are affected by the disruption.\n**DIAGNOSIS**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region.\n**WORKAROUND**\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-08-15T21:42:40+00:00","modified":"2024-08-15T21:42:40+00:00","when":"2024-08-15T21:42:40+00:00","text":"## Incident Report\n## Summary\nOn 12 August 2024 at 06:20 US/Pacific, multiple Google Cloud and Google Workspace products experienced connectivity issues in europe-west2 for a duration of 40 minutes. During the time, ingress traffic to europe-west2 and egress traffic from europe-west2 experienced elevated latencies, connection timeouts, and connection failures.\n## Root Cause\nOn 12 August 2024 06:20 US/Pacific, primary and backup power feeds were both lost in a [Google Point of Presence (POP)](https://cloud.google.com/vpc/docs/edge-locations) due to a substation switchgear failure. The affected POP hosts about ⅓ of serving first-layer [Google Front Ends (GFEs)](https://cloud.google.com/docs/security/infrastructure/design#google-frontend-service) located in europe-west2 and some distributed networking equipment for that region. The power loss impacted the following Google products and services that depend on GFEs in that region:\n* Google Cloud APIs, Google Workspace, and other Google services like YouTube,\n* Customer-created global external application and proxy network load balancers, including Cloud CDN\nThe power loss also impacted the following Google Cloud products which depended on impacted networking equipment:\n* Customer-created regional external application, proxy network, and passthrough network load balancers in the europe-west2 region,\n* External protocol forwarding and VM external IP address connectivity for VMs in the europe-west2 region.\n* Google Cloud Interconnect connections in some LHR colocation facilities.\nImpact was limited to situations where either or both of the following was true:\n* Inbound requests or connections were routed into the europe-west2 region of Google’s network, from the Internet, and those requests or connections depended on networking equipment that was offline, or unreachable pending reconvergence.\n* Outbound responses were routed to the Internet, from the europe-west2 region of Google’s network, and those responses depended on networking equipment that was without power.\nThe power outage caused Internet routes advertised by Google to be withdrawn in networks connected to Google’s network. The withdrawn routes were automatically replaced by other Google-advertised routes that didn’t depend on impacted networking equipment. Withdrawing and replacing routes relies on the BGP protocol and its timers, so replacement route convergence is not instantaneous, and overloading in the automatically selected replacement route GFEs extended the duration of the incident.\n## Detailed Description of Impact\n* Google Workspace: _Gmail, Google Calendar, Google Chat, Google Docs, Google Drive, Google Meet and Google Tasks users connecting to Workspace services from the UK region and surrounding areas experienced connectivity issues as described in the next point.\n* GFE-based products and services: _Customers on the Internet experienced a spike of broken connections followed by elevated latencies or HTTP error responses when communicating with GFE-powered Google APIs and services or customer-created global external application and proxy network load balancers. At roughly 06:23 US/Pacific, Google automatically redirected connections to the nearest possible first-layer GFEs with some latency penalty. Unfortunately, some of the nearest possible first-layer GFEs were overloaded until 06:48 when Google engineers made adjustments to more efficiently distribute incoming requests among nearby first-layer GFEs. Depending on the Google API or service or the customer-created global external load balancer, elevated latencies could have persisted until about 08:30 US/Pacific. Elevated latencies also could have applied to customer-created global external load balancers that had Cloud CDN enabled.\n* Regional Google Cloud products and services: _Until replacement routes were in effect, customers on the Internet experienced connection failures to the following GCP resources in the europe-west2 region: * Regional external application, proxy network, and passthrough network load balancers. * External protocol forwarding and VM external IP addresses.\n* Google Cloud Interconnect: _Google Cloud Interconnect connections in some LHR colocation facilities (lhr-zone1-47, lhr-zone1-832, lhr-zone1-2262, lhr-zone1-4885, lhr-zone1-99051 and lhr-zone2-47) remained offline from 06:20 US/Pacific to at least 06:57 US/Pacific, when power was restored.\nAt 06:43 US/Pacific, power was restored to the impacted networking equipment. Google networking equipment was fully operational by 06:57 US/Pacific, and connectivity to GFE-based products and services, regional Google Cloud products and services, and Google Cloud Interconnect resumed shortly thereafter.\n## Remediation and Prevention\nMultiple Google engineering teams were alerted and automated recovery tooling was triggered as expected; however, manual adjustments were required to address subsequent first-layer GFE overload. Google is reviewing automation improvements in tasks that required manual intervention to reduce the duration of future power event impact. Similarly, Google is working to increase Cloud Interconnect control plane resilience and reduce mitigation time through automated reaction to isolation events.\nAdditionally Google's partner who maintains the affected facility power in LHR (London) is conducting a full root cause analysis with the switchboard manufacturer and substation owner(s) involved in supplying power, including follow up as to why stored or generated on-site emergency power did not carry loads.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"wNHuVFtZEWU5Mmj2eRCK","service_name":"Google Docs","affected_products":[{"title":"Google Docs","id":"wNHuVFtZEWU5Mmj2eRCK"}],"uri":"incidents/5AjHWPCTCP5uenZRjuwz"},{"id":"r6q8Wfdmf8PPJtdjQ8uH","number":"2450934655095805328","begin":"2024-08-12T13:20:00+00:00","created":"2024-08-12T15:03:39+00:00","end":"2024-08-12T15:32:00+00:00","modified":"2024-08-15T21:50:05+00:00","external_desc":"**SUMMARY**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region\n**DESCRIPTION**\nWe have experienced an issue with Gmail, Google Drive, Google Calendar, Google Docs, Google Chat, Google Tasks beginning at Monday, 2024-08-12 06:28 US/Pacific. The issue is now mitigated and our engineering teams are closely monitoring for any residual impact We will provide an update by Monday, 2024-08-12 09:15 US/Pacific with current details. We apologize to all who are affected by the disruption.\n**DIAGNOSIS**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region.\n**WORKAROUND**\nNone at this time.\nPlease refer to https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz for further updates on this issue.","updates":[{"created":"2024-08-15T21:49:55+00:00","modified":"2024-08-15T21:49:55+00:00","when":"2024-08-15T21:49:55+00:00","text":"A Full incident Report is published in https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz","status":"AVAILABLE"},{"created":"2024-08-12T19:29:59+00:00","modified":"2024-08-12T19:29:59+00:00","when":"2024-08-12T19:29:59+00:00","text":"A mini incident report has been posted to\nhttps://status.cloud.google.com/incidents/ETJGhvY9Xaktw7tgi8dF","status":"AVAILABLE"},{"created":"2024-08-12T15:41:54+00:00","modified":"2024-08-12T15:41:54+00:00","when":"2024-08-12T15:41:54+00:00","text":"The issue with Gmail, Google Calendar, Google Chat, Google Docs, Google Drive, Google Tasks has been resolved for all affected users as of Monday, 2024-08-12 08:35 US/Pacific.\nDuring the issue, users connecting to Workspace services from the UK region may have experienced connectivity issues.\nWe will publish an analysis of this incident once we have completed our internal investigation.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-08-12T15:03:39+00:00","modified":"2024-08-12T15:19:09+00:00","when":"2024-08-12T15:03:39+00:00","text":"**SUMMARY**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region\n**DESCRIPTION**\nWe have experienced an issue with Gmail, Google Drive, Google Calendar, Google Docs, Google Chat, Google Tasks beginning at Monday, 2024-08-12 06:28 US/Pacific. The issue is now mitigated and our engineering teams are closely monitoring for any residual impact We will provide an update by Monday, 2024-08-12 09:15 US/Pacific with current details. We apologize to all who are affected by the disruption.\n**DIAGNOSIS**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region.\n**WORKAROUND**\nNone at this time.\nPlease refer to https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz for further updates on this issue.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-08-15T21:49:55+00:00","modified":"2024-08-15T21:49:55+00:00","when":"2024-08-15T21:49:55+00:00","text":"A Full incident Report is published in https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"n5hVexuq1PM9onY9qrmo","service_name":"Google Calendar","affected_products":[{"title":"Google Calendar","id":"n5hVexuq1PM9onY9qrmo"}],"uri":"incidents/r6q8Wfdmf8PPJtdjQ8uH"},{"id":"afFHu9w8zAYWoVbpkRMT","number":"17776955237206687686","begin":"2024-08-12T13:20:00+00:00","created":"2024-08-12T15:19:55+00:00","end":"2024-08-12T15:32:00+00:00","modified":"2024-08-15T21:51:02+00:00","external_desc":"**SUMMARY**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region\n**DESCRIPTION**\nWe have experienced an issue with Gmail, Google Drive, Google Calendar, Google Docs, Google Chat, Google Tasks beginning at Monday, 2024-08-12 06:28 US/Pacific. The issue is now mitigated and our engineering teams are closely monitoring for any residual impact We will provide an update by Monday, 2024-08-12 09:15 US/Pacific with current details. We apologize to all who are affected by the disruption.\n**DIAGNOSIS**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region.\n**WORKAROUND**\nNone at this time.\nPlease refer to https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz for further updates on this issue.","updates":[{"created":"2024-08-15T21:47:56+00:00","modified":"2024-08-15T21:47:56+00:00","when":"2024-08-15T21:47:56+00:00","text":"A Full incident Report is published in https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz","status":"AVAILABLE"},{"created":"2024-08-12T19:31:16+00:00","modified":"2024-08-12T19:31:16+00:00","when":"2024-08-12T19:31:16+00:00","text":"A mini incident report has been posted to\nhttps://status.cloud.google.com/incidents/ETJGhvY9Xaktw7tgi8dF","status":"AVAILABLE"},{"created":"2024-08-12T15:41:22+00:00","modified":"2024-08-12T15:41:22+00:00","when":"2024-08-12T15:41:22+00:00","text":"The issue with Gmail, Google Calendar, Google Chat, Google Docs, Google Drive, Google Tasks has been resolved for all affected users as of Monday, 2024-08-12 08:35 US/Pacific.\nDuring the issue, users connecting to Workspace services from the UK region may have experienced connectivity issues.\nWe will publish an analysis of this incident once we have completed our internal investigation.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-08-12T15:19:56+00:00","modified":"2024-08-12T15:19:56+00:00","when":"2024-08-12T15:19:56+00:00","text":"**SUMMARY**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region\n**DESCRIPTION**\nWe have experienced an issue with Gmail, Google Drive, Google Calendar, Google Docs, Google Chat, Google Tasks beginning at Monday, 2024-08-12 06:28 US/Pacific. The issue is now mitigated and our engineering teams are closely monitoring for any residual impact We will provide an update by Monday, 2024-08-12 09:15 US/Pacific with current details. We apologize to all who are affected by the disruption.\n**DIAGNOSIS**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region.\n**WORKAROUND**\nNone at this time.\nPlease refer to https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz for further updates on this issue.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-08-15T21:47:56+00:00","modified":"2024-08-15T21:47:56+00:00","when":"2024-08-15T21:47:56+00:00","text":"A Full incident Report is published in https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"Vups7kniH52V5x4UX1gM","service_name":"Google Tasks","affected_products":[{"title":"Google Tasks","id":"Vups7kniH52V5x4UX1gM"}],"uri":"incidents/afFHu9w8zAYWoVbpkRMT"},{"id":"FK2vgw8XkKt77kspGxjo","number":"9822559119539518235","begin":"2024-08-12T13:20:00+00:00","created":"2024-08-12T15:02:19+00:00","end":"2024-08-12T15:32:00+00:00","modified":"2024-08-15T21:50:26+00:00","external_desc":"**SUMMARY**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region\n**DESCRIPTION**\nWe have experienced an issue with Gmail, Google Drive, Google Calendar, Google Docs, Google Chat, Google Tasks beginning at Monday, 2024-08-12 06:28 US/Pacific. The issue is now mitigated and our engineering teams are closely monitoring for any residual impact We will provide an update by Monday, 2024-08-12 09:15 US/Pacific with current details. We apologize to all who are affected by the disruption.\n**DIAGNOSIS**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region.\n**WORKAROUND**\nNone at this time.\nPlease refer to https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz for further updates on this issue.","updates":[{"created":"2024-08-15T21:45:55+00:00","modified":"2024-08-15T21:45:55+00:00","when":"2024-08-15T21:45:55+00:00","text":"A Full incident Report is published in https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz","status":"AVAILABLE"},{"created":"2024-08-12T19:32:34+00:00","modified":"2024-08-12T19:32:34+00:00","when":"2024-08-12T19:32:34+00:00","text":"A mini incident report has been posted to\nhttps://status.cloud.google.com/incidents/ETJGhvY9Xaktw7tgi8dF","status":"AVAILABLE"},{"created":"2024-08-12T15:40:23+00:00","modified":"2024-08-12T15:40:23+00:00","when":"2024-08-12T15:40:23+00:00","text":"The issue with Gmail, Google Calendar, Google Chat, Google Docs, Google Drive, Google Tasks has been resolved for all affected users as of Monday, 2024-08-12 08:35 US/Pacific.\nDuring the issue, users connecting to Workspace services from the UK region may have experienced connectivity issues.\nWe will publish an analysis of this incident once we have completed our internal investigation.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-08-12T15:18:28+00:00","modified":"2024-08-12T15:18:28+00:00","when":"2024-08-12T15:18:28+00:00","text":"**SUMMARY**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region\n**DESCRIPTION**\nWe have experienced an issue with Gmail, Google Drive, Google Calendar, Google Docs, Google Chat, Google Tasks beginning at Monday, 2024-08-12 06:28 US/Pacific. The issue is now mitigated and our engineering teams are closely monitoring for any residual impact We will provide an update by Monday, 2024-08-12 09:15 US/Pacific with current details. We apologize to all who are affected by the disruption.\n**DIAGNOSIS**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region.\n**WORKAROUND**\nNone at this time.\nPlease refer to https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz for further updates on this issue.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-08-15T21:45:55+00:00","modified":"2024-08-15T21:45:55+00:00","when":"2024-08-15T21:45:55+00:00","text":"A Full incident Report is published in https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/FK2vgw8XkKt77kspGxjo"},{"id":"Tf4iYUqvRiCtpqyoRPm9","number":"11249187714608568293","begin":"2024-08-12T13:20:00+00:00","created":"2024-08-12T15:04:02+00:00","end":"2024-08-12T15:32:00+00:00","modified":"2024-08-15T21:50:38+00:00","external_desc":"**SUMMARY**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region\n**DESCRIPTION**\nWe have experienced an issue with Gmail, Google Drive, Google Calendar, Google Docs, Google Chat, Google Tasks beginning at Monday, 2024-08-12 06:28 US/Pacific. The issue is now mitigated and our engineering teams are closely monitoring for any residual impact We will provide an update by Monday, 2024-08-12 09:15 US/Pacific with current details. We apologize to all who are affected by the disruption.\n**DIAGNOSIS**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region.\n**WORKAROUND**\nNone at this time.\nPlease refer to https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz for further updates on this issue.","updates":[{"created":"2024-08-15T21:46:28+00:00","modified":"2024-08-15T21:46:28+00:00","when":"2024-08-15T21:46:28+00:00","text":"A Full incident Report is published in https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz","status":"AVAILABLE"},{"created":"2024-08-12T19:31:46+00:00","modified":"2024-08-12T19:31:46+00:00","when":"2024-08-12T19:31:46+00:00","text":"A mini incident report has been posted to\nhttps://status.cloud.google.com/incidents/ETJGhvY9Xaktw7tgi8dF","status":"AVAILABLE"},{"created":"2024-08-12T15:40:52+00:00","modified":"2024-08-12T15:40:52+00:00","when":"2024-08-12T15:40:52+00:00","text":"The issue with Gmail, Google Calendar, Google Chat, Google Docs, Google Drive, Google Tasks has been resolved for all affected users as of Monday, 2024-08-12 08:35 US/Pacific.\nDuring the issue, users connecting to Workspace services from the UK region may have experienced connectivity issues.\nWe will publish an analysis of this incident once we have completed our internal investigation.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-08-12T15:18:49+00:00","modified":"2024-08-12T15:18:49+00:00","when":"2024-08-12T15:18:49+00:00","text":"**SUMMARY**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region\n**DESCRIPTION**\nWe have experienced an issue with Gmail, Google Drive, Google Calendar, Google Docs, Google Chat, Google Tasks beginning at Monday, 2024-08-12 06:28 US/Pacific. The issue is now mitigated and our engineering teams are closely monitoring for any residual impact We will provide an update by Monday, 2024-08-12 09:15 US/Pacific with current details. We apologize to all who are affected by the disruption.\n**DIAGNOSIS**\nMultiple Workspace services experienced brief connectivity issues for users connecting from the UK region.\n**WORKAROUND**\nNone at this time.\nPlease refer to https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz for further updates on this issue.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-08-15T21:46:28+00:00","modified":"2024-08-15T21:46:28+00:00","when":"2024-08-15T21:46:28+00:00","text":"A Full incident Report is published in https://www.google.com/appsstatus/dashboard/incidents/5AjHWPCTCP5uenZRjuwz","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"VHNA7p3Z5p3iakj5sA8V","service_name":"Google Drive","affected_products":[{"title":"Google Drive","id":"VHNA7p3Z5p3iakj5sA8V"}],"uri":"incidents/Tf4iYUqvRiCtpqyoRPm9"},{"id":"s9TB8tK4bRafntzYHwHM","number":"1493558468576204316","begin":"2024-08-08T19:06:00+00:00","created":"2024-08-08T20:44:26+00:00","end":"2024-08-08T23:16:00+00:00","modified":"2024-08-14T21:42:36+00:00","external_desc":"**Summary:**\nGoogle Drive experiencing service degradation\n**Description:**\nWe are experiencing an issue with Google Drive beginning on Thursday, 2024-08-08 12:15 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Thursday, 2024-08-08 15:00 US/Pacific with current details.\n**Customer Symptoms:**\nGoogle Drive users may see degraded experience while performing upload operation.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-08-14T21:42:30+00:00","modified":"2024-08-14T21:42:30+00:00","when":"2024-08-14T21:42:30+00:00","text":"# Incident Report\n## Summary\nOn 8 August 2024, Gmail and Google Drive experienced service degradation globally for a duration of 4 hours and 10 minutes between 12:06 and 16:16 US/Pacific. During the incident, affected users experienced issues with email attachment and delivery functionalities in Gmail, and upload operations in Google Drive.\nTo our Gmail and Google Drive customers, we apologize for the impact this service disruption had on your organization. We have completed an internal investigation and are taking immediate steps to improve the quality and reliability of our services.\n## Root Cause\nAs part of a standard data operation, we restored a large volume of data into an internal Bigtable database. This operation resulted in Bigtable servers being overloaded.\nDuring the restore operation, Bigtable identified certain tablets as ‘unloadable’ and added the corresponding entries into a [1]Chubby file, as expected. As the file size grew, it exceeded the memory limit allocated for Chubby, resulting in Bigtable being unable to write to the file.\nThe cumulative impact of these events led to a subset of Bigtable instances entering a degraded state, impacting read/write operations. These instances store internal data for Gmail and Google Drive, which consequently affected the performance of those services.\n[1] - https://research.google/pubs/the-chubby-lock-service-for-loosely-coupled-distributed-systems\n## Remediation and Prevention\nGoogle engineers were alerted to the issue on Thursday, 8 Aug 2024 at 11:58 US/Pacific via our monitoring system. Once the nature and scope of the issue were understood, our engineers devised and executed a multi-pronged approach to address the root cause and mitigate impact. The Chubby quota was increased to address the memory issue, while traffic from the affected instances was re-routed to avoid further impact. To ensure that any degraded instances were addressed, Bigtable master servers for the affected instances were successfully restarted. Impact was fully mitigated by 16:16 US/Pacific.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\nWe are committed to preventing a repeat of this issue in the future and are completing the following actions:\n* We have completed a deep analysis of our Bigtable instances and ensured that the Chubby space quota limits and configurations are optimal across all instances.\n* We are working on enhancing our monitoring for Chubby quota usage in Bigtable to enable early detection and prevention of potential issues.\n* We will establish clear guidelines on the recommended volume of data that can be safely restored at any given time, minimizing the risk of service disruptions.\n* We have enhanced the logic for diverting traffic from Bigtable clusters to ensure smoother transitions and minimal impact on users.\n## Detailed Description of Impact\nOn Thursday, 8 August 2024 from 12:06 to 16:16 US/Pacific, Gmail and Google Drive experienced service degradation for a duration of 4 hours and 10 minutes.\n#### Gmail\nAffected users experienced issues with attachment functionality wherein they were unable to send emails or save drafts containing attachments. There was no impact to emails sent without attachments.\n#### Google Drive\nAffected users may have observed degraded performance while performing upload operations.","status":"AVAILABLE"},{"created":"2024-08-09T09:00:35+00:00","modified":"2024-08-09T09:00:35+00:00","when":"2024-08-09T09:00:35+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 8 Aug, 2024 12:06\n**Incident End:** 8 Aug, 2024 16:16\n**Duration:** 4 hours, 10 minutes\n**Affected Services and Features:** Gmail and Google Drive\n**Regions/Zones:** Global\n**Description:** Gmail and Google Drive saw service degradation for a duration of 4 hours 10 minutes. From preliminary analysis, the root cause was a latent misconfiguration of our lock-service infrastructure that impacted a critical storage layer during a high-load scenario.\nGoogle will complete a full IR in the following days that will provide a full root cause.\n**Customer Impact:**\n* Gmail users experienced issues with attachment functionality. Users may be unable to send emails or save drafts containing attachments. Sending emails without attachments remains functional.\n* Google Drive users may have observed degraded experience while performing upload operations.","status":"AVAILABLE"},{"created":"2024-08-09T00:10:31+00:00","modified":"2024-08-09T00:10:31+00:00","when":"2024-08-09T00:10:31+00:00","text":"**Summary:**\nGoogle Drive experienced service degradation\n**Description:**\nThe recent fix implemented by our engineering teams has been successful in fully resolving the issue. We will be publishing a Preliminary Incident Report(IR) shortly which will provide additional details on root cause.","status":"AVAILABLE"},{"created":"2024-08-08T23:45:15+00:00","modified":"2024-08-08T23:45:15+00:00","when":"2024-08-08T23:45:15+00:00","text":"**Summary:**\nGoogle Drive is experiencing service degradation\n**Description:**\nWe continue to see a reduction in observed error rates based on the most recent fix our engineering teams have made. We are continuing to monitor the overall performance and stability of the Workspace impacted products and will provide more information by Thursday, 2024-08-08 18:00 US/Pacific.\nWe appreciate your patience as we continue our efforts to restore full service.\n**Customer Symptoms:**\nGoogle Drive users may see degraded experience while performing upload operations.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-08-08T22:08:05+00:00","modified":"2024-08-08T22:08:05+00:00","when":"2024-08-08T22:08:05+00:00","text":"**Summary:**\nGoogle Drive is experiencing service degradation\n**Description:**\nOur engineers are actively implementing several fixes, and we’re seeing a decrease in error rates. We understand the impact this has had, and we want to assure you that we’re committed to resolving fully, and quickly. We appreciate your patience as we continue our efforts to restore full service.\nWe do not have an ETA for full mitigation at this point.\nWe will provide more information by Thursday, 2024-08-08 16:30 US/Pacific.\n**Customer Symptoms:**\nGoogle Drive users may see degraded experience while performing upload operations.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-08-08T20:44:27+00:00","modified":"2024-08-08T20:44:27+00:00","when":"2024-08-08T20:44:27+00:00","text":"**Summary:**\nGoogle Drive experiencing service degradation\n**Description:**\nWe are experiencing an issue with Google Drive beginning on Thursday, 2024-08-08 12:15 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Thursday, 2024-08-08 15:00 US/Pacific with current details.\n**Customer Symptoms:**\nGoogle Drive users may see degraded experience while performing upload operation.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-08-14T21:42:30+00:00","modified":"2024-08-14T21:42:30+00:00","when":"2024-08-14T21:42:30+00:00","text":"# Incident Report\n## Summary\nOn 8 August 2024, Gmail and Google Drive experienced service degradation globally for a duration of 4 hours and 10 minutes between 12:06 and 16:16 US/Pacific. During the incident, affected users experienced issues with email attachment and delivery functionalities in Gmail, and upload operations in Google Drive.\nTo our Gmail and Google Drive customers, we apologize for the impact this service disruption had on your organization. We have completed an internal investigation and are taking immediate steps to improve the quality and reliability of our services.\n## Root Cause\nAs part of a standard data operation, we restored a large volume of data into an internal Bigtable database. This operation resulted in Bigtable servers being overloaded.\nDuring the restore operation, Bigtable identified certain tablets as ‘unloadable’ and added the corresponding entries into a [1]Chubby file, as expected. As the file size grew, it exceeded the memory limit allocated for Chubby, resulting in Bigtable being unable to write to the file.\nThe cumulative impact of these events led to a subset of Bigtable instances entering a degraded state, impacting read/write operations. These instances store internal data for Gmail and Google Drive, which consequently affected the performance of those services.\n[1] - https://research.google/pubs/the-chubby-lock-service-for-loosely-coupled-distributed-systems\n## Remediation and Prevention\nGoogle engineers were alerted to the issue on Thursday, 8 Aug 2024 at 11:58 US/Pacific via our monitoring system. Once the nature and scope of the issue were understood, our engineers devised and executed a multi-pronged approach to address the root cause and mitigate impact. The Chubby quota was increased to address the memory issue, while traffic from the affected instances was re-routed to avoid further impact. To ensure that any degraded instances were addressed, Bigtable master servers for the affected instances were successfully restarted. Impact was fully mitigated by 16:16 US/Pacific.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\nWe are committed to preventing a repeat of this issue in the future and are completing the following actions:\n* We have completed a deep analysis of our Bigtable instances and ensured that the Chubby space quota limits and configurations are optimal across all instances.\n* We are working on enhancing our monitoring for Chubby quota usage in Bigtable to enable early detection and prevention of potential issues.\n* We will establish clear guidelines on the recommended volume of data that can be safely restored at any given time, minimizing the risk of service disruptions.\n* We have enhanced the logic for diverting traffic from Bigtable clusters to ensure smoother transitions and minimal impact on users.\n## Detailed Description of Impact\nOn Thursday, 8 August 2024 from 12:06 to 16:16 US/Pacific, Gmail and Google Drive experienced service degradation for a duration of 4 hours and 10 minutes.\n#### Gmail\nAffected users experienced issues with attachment functionality wherein they were unable to send emails or save drafts containing attachments. There was no impact to emails sent without attachments.\n#### Google Drive\nAffected users may have observed degraded performance while performing upload operations.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"VHNA7p3Z5p3iakj5sA8V","service_name":"Google Drive","affected_products":[{"title":"Google Drive","id":"VHNA7p3Z5p3iakj5sA8V"}],"uri":"incidents/s9TB8tK4bRafntzYHwHM"},{"id":"G4vfJGa9LyWQjK4hiiHA","number":"6419236895627715759","begin":"2024-08-08T19:06:00+00:00","created":"2024-08-08T20:43:23+00:00","end":"2024-08-08T23:16:00+00:00","modified":"2024-08-14T21:39:52+00:00","external_desc":"**Summary:**\nGmail experiencing service degradation\n**Description:**\nWe are experiencing an issue with Gmail beginning on Thursday, 2024-08-08 12:15 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Thursday, 2024-08-08 15:00 US/Pacific with current details.\n**Customer Symptoms:**\nGmail: Users trying to send emails or save drafts with attachments would experience failures.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-08-14T21:39:52+00:00","modified":"2024-08-14T21:39:52+00:00","when":"2024-08-14T21:39:52+00:00","text":"# Incident Report\n## Summary\nOn 8 August 2024, Gmail and Google Drive experienced service degradation globally for a duration of 4 hours and 10 minutes between 12:06 and 16:16 US/Pacific. During the incident, affected users experienced issues with email attachment and delivery functionalities in Gmail, and upload operations in Google Drive.\nTo our Gmail and Google Drive customers, we apologize for the impact this service disruption had on your organization. We have completed an internal investigation and are taking immediate steps to improve the quality and reliability of our services.\n## Root Cause\nAs part of a standard data operation, we restored a large volume of data into an internal Bigtable database. This operation resulted in Bigtable servers being overloaded.\nDuring the restore operation, Bigtable identified certain tablets as ‘unloadable’ and added the corresponding entries into a [1]Chubby file, as expected. As the file size grew, it exceeded the memory limit allocated for Chubby, resulting in Bigtable being unable to write to the file.\nThe cumulative impact of these events led to a subset of Bigtable instances entering a degraded state, impacting read/write operations. These instances store internal data for Gmail and Google Drive, which consequently affected the performance of those services.\n[1] - https://research.google/pubs/the-chubby-lock-service-for-loosely-coupled-distributed-systems\n## Remediation and Prevention\nGoogle engineers were alerted to the issue on Thursday, 8 Aug 2024 at 11:58 US/Pacific via our monitoring system. Once the nature and scope of the issue were understood, our engineers devised and executed a multi-pronged approach to address the root cause and mitigate impact. The Chubby quota was increased to address the memory issue, while traffic from the affected instances was re-routed to avoid further impact. To ensure that any degraded instances were addressed, Bigtable master servers for the affected instances were successfully restarted. Impact was fully mitigated by 16:16 US/Pacific.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\nWe are committed to preventing a repeat of this issue in the future and are completing the following actions:\n* We have completed a deep analysis of our Bigtable instances and ensured that the Chubby space quota limits and configurations are optimal across all instances.\n* We are working on enhancing our monitoring for Chubby quota usage in Bigtable to enable early detection and prevention of potential issues.\n* We will establish clear guidelines on the recommended volume of data that can be safely restored at any given time, minimizing the risk of service disruptions.\n* We have enhanced the logic for diverting traffic from Bigtable clusters to ensure smoother transitions and minimal impact on users.\n## Detailed Description of Impact\nOn Thursday, 8 August 2024 from 12:06 to 16:16 US/Pacific, Gmail and Google Drive experienced service degradation for a duration of 4 hours and 10 minutes.\n#### Gmail\nAffected users experienced issues with attachment functionality wherein they were unable to send emails or save drafts containing attachments. There was no impact to emails sent without attachments.\n#### Google Drive\nAffected users may have observed degraded performance while performing upload operations.","status":"AVAILABLE"},{"created":"2024-08-09T08:59:06+00:00","modified":"2024-08-09T08:59:06+00:00","when":"2024-08-09T08:59:06+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 8 Aug, 2024 12:06\n**Incident End:** 8 Aug, 2024 16:16\n**Duration:** 4 hours, 10 minutes\n**Affected Services and Features:** Gmail and Google Drive\n**Regions/Zones:** Global\n**Description:** Gmail and Google Drive saw service degradation for a duration of 4 hours 10 minutes. From preliminary analysis, the root cause was a latent misconfiguration of our lock-service infrastructure that impacted a critical storage layer during a high-load scenario.\nGoogle will complete a full IR in the following days that will provide a full root cause.\n**Customer Impact:**\n* Gmail users experienced issues with attachment functionality. Users may be unable to send emails or save drafts containing attachments. Sending emails without attachments remains functional.\n* Google Drive users may have observed degraded experience while performing upload operations.","status":"AVAILABLE"},{"created":"2024-08-09T00:11:43+00:00","modified":"2024-08-09T00:11:43+00:00","when":"2024-08-09T00:11:43+00:00","text":"**Summary:**\nGmail experienced service degradation\n**Description:**\nThe recent fix implemented by our engineering teams has been successful in fully resolving the issue. We will be publishing a Preliminary Incident Report(IR) shortly which will provide additional details on root cause.","status":"AVAILABLE"},{"created":"2024-08-08T23:44:02+00:00","modified":"2024-08-08T23:44:02+00:00","when":"2024-08-08T23:44:02+00:00","text":"**Summary:**\nGmail is experiencing service degradation\n**Description:**\nWe continue to see a reduction in observed error rates based on the most recent fix our engineering teams have made. We are continuing to monitor the overall performance and stability of the Workspace impacted products and will provide more information by Thursday, 2024-08-08 18:00 US/Pacific.\nWe appreciate your patience as we continue our efforts to restore full service.\n**Customer Symptoms:**\nGmail is experiencing issues with attachment functionality. Users may be unable to send emails or save drafts containing attachments. Sending emails without attachments remains functional. Receiving emails with attachments remains unaffected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-08-08T22:07:08+00:00","modified":"2024-08-08T22:07:08+00:00","when":"2024-08-08T22:07:08+00:00","text":"**Summary:**\nGmail is experiencing service degradation\n**Description:**\nOur engineers are actively implementing several fixes, and we’re seeing a decrease in error rates. We understand the impact this has had, and we want to assure you that we’re committed to resolving fully, and quickly. We appreciate your patience as we continue our efforts to restore full service.\nWe do not have an ETA for full mitigation at this point.\nWe will provide more information by Thursday, 2024-08-08 16:30 US/Pacific.\n**Customer Symptoms:**\nGmail is experiencing issues with attachment functionality. Users may be unable to send emails or save drafts containing attachments. Sending emails without attachments remains functional. Receiving emails with attachments remains unaffected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-08-08T20:43:23+00:00","modified":"2024-08-08T20:43:23+00:00","when":"2024-08-08T20:43:23+00:00","text":"**Summary:**\nGmail experiencing service degradation\n**Description:**\nWe are experiencing an issue with Gmail beginning on Thursday, 2024-08-08 12:15 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Thursday, 2024-08-08 15:00 US/Pacific with current details.\n**Customer Symptoms:**\nGmail: Users trying to send emails or save drafts with attachments would experience failures.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-08-14T21:39:52+00:00","modified":"2024-08-14T21:39:52+00:00","when":"2024-08-14T21:39:52+00:00","text":"# Incident Report\n## Summary\nOn 8 August 2024, Gmail and Google Drive experienced service degradation globally for a duration of 4 hours and 10 minutes between 12:06 and 16:16 US/Pacific. During the incident, affected users experienced issues with email attachment and delivery functionalities in Gmail, and upload operations in Google Drive.\nTo our Gmail and Google Drive customers, we apologize for the impact this service disruption had on your organization. We have completed an internal investigation and are taking immediate steps to improve the quality and reliability of our services.\n## Root Cause\nAs part of a standard data operation, we restored a large volume of data into an internal Bigtable database. This operation resulted in Bigtable servers being overloaded.\nDuring the restore operation, Bigtable identified certain tablets as ‘unloadable’ and added the corresponding entries into a [1]Chubby file, as expected. As the file size grew, it exceeded the memory limit allocated for Chubby, resulting in Bigtable being unable to write to the file.\nThe cumulative impact of these events led to a subset of Bigtable instances entering a degraded state, impacting read/write operations. These instances store internal data for Gmail and Google Drive, which consequently affected the performance of those services.\n[1] - https://research.google/pubs/the-chubby-lock-service-for-loosely-coupled-distributed-systems\n## Remediation and Prevention\nGoogle engineers were alerted to the issue on Thursday, 8 Aug 2024 at 11:58 US/Pacific via our monitoring system. Once the nature and scope of the issue were understood, our engineers devised and executed a multi-pronged approach to address the root cause and mitigate impact. The Chubby quota was increased to address the memory issue, while traffic from the affected instances was re-routed to avoid further impact. To ensure that any degraded instances were addressed, Bigtable master servers for the affected instances were successfully restarted. Impact was fully mitigated by 16:16 US/Pacific.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\nWe are committed to preventing a repeat of this issue in the future and are completing the following actions:\n* We have completed a deep analysis of our Bigtable instances and ensured that the Chubby space quota limits and configurations are optimal across all instances.\n* We are working on enhancing our monitoring for Chubby quota usage in Bigtable to enable early detection and prevention of potential issues.\n* We will establish clear guidelines on the recommended volume of data that can be safely restored at any given time, minimizing the risk of service disruptions.\n* We have enhanced the logic for diverting traffic from Bigtable clusters to ensure smoother transitions and minimal impact on users.\n## Detailed Description of Impact\nOn Thursday, 8 August 2024 from 12:06 to 16:16 US/Pacific, Gmail and Google Drive experienced service degradation for a duration of 4 hours and 10 minutes.\n#### Gmail\nAffected users experienced issues with attachment functionality wherein they were unable to send emails or save drafts containing attachments. There was no impact to emails sent without attachments.\n#### Google Drive\nAffected users may have observed degraded performance while performing upload operations.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/G4vfJGa9LyWQjK4hiiHA"},{"id":"sowQNioouYdVq6hNkZti","number":"13750020100651141057","begin":"2024-08-06T15:00:00+00:00","created":"2024-08-07T05:06:42+00:00","end":"2024-08-07T10:29:00+00:00","modified":"2024-08-15T19:46:47+00:00","external_desc":"### **Summary:**\nChromeOS sign in issues affecting multiple customers throughout the globe\n### **Description:**\nWe are experiencing an issue with ChromeOS.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Tuesday, 2024-08-06 23:30 US/Pacific with current details.\n### **Diagnosis:**\nDuring the full sign in process ChromeOS devices complete the sign in process (even Multi Factor Authentication (MFA) if applicable) but are eventually brought back to the login page.\nThe issue is affecting multiple ChromeOS versions\n### **Workaround:**\nNone at this time.","updates":[{"created":"2024-08-15T19:46:47+00:00","modified":"2024-08-15T19:46:47+00:00","when":"2024-08-15T19:46:47+00:00","text":"# Incident Report\n## Summary\nOn 06 August at 08:00 US/Pacific, Chrome OS devices experienced issues with logins globally for a duration of 19 hours 29 minutes. During this time, users on ChromeOS devices were able to complete the full sign-in process, including any Multi-factor Authentication when required. However, some users were immediately redirected back to the login page after logging in.\n## Root Cause\nGoogle’s internal authentication system uses an HTTP endpoint microservice that provides clients with a list of accounts currently in the authentication cookies. This endpoint is mainly used for various account switcher user interfaces, providing a list of accounts the current session has access to.\nThe root cause of the issue is an incorrect configuration made to the Google authentication service, which resulted in a malformed response to the queries from this microservice in a particular condition. As a result, Chrome OS was unable to get the account details from the internal Google authentication service, which caused the sign-in failures.\n## Remediation and Prevention\nGoogle engineers were alerted to the outage via a support case on Tuesday, 06 August at 15:51 US/Pacific, and immediately started an investigation. Once the nature and scope of the issue became clear, the configuration change was reverted and Google engineers ensured that sign-in worked again.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n* Migrate the endpoints used in Google authentication systems to a different format so that the issue is not triggered\n* Expand the integration testing environment to include more clients including Google Chrome so that all scenarios are identified\n* Include additional control mechanisms to ensure that configuration changes are approved appropriately\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\n## Detailed Description of Impact\n* On devices which were already configured, users were unable to add new users from the login screen.\n* Users were able to successfully authenticate however, and were sent back to the login screen before starting a session.\n* Users were unable to setup the first user account on their device (the device owner), resulting in the devices being reset to factory defaults","status":"AVAILABLE"},{"created":"2024-08-07T17:46:01+00:00","modified":"2024-08-07T17:46:01+00:00","when":"2024-08-07T17:46:01+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 2024-08-06 08:00\n**Incident End:** 2024-08-07 03:29\n**Duration:** 19 hours 29 minutes\n**Affected Services and Features:**\nChrome OS\n**Regions/Zones:** Global\n**Description:**\nChrome OS devices experienced issues with login due to a recent change to a backend identity service, causing incorrect parsing of account details. Google will complete a full IR in the following days that will provide a full root cause.\n**Customer Impact:**\n- During the full sign in process ChromeOS devices completed the sign in (even Multi Factor Authentication (MFA) if it was applicable) but were eventually being brought back to the login page","status":"AVAILABLE"},{"created":"2024-08-07T10:44:39+00:00","modified":"2024-08-07T10:44:39+00:00","when":"2024-08-07T10:44:39+00:00","text":"The problem with ChromeOS has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.","status":"AVAILABLE"},{"created":"2024-08-07T08:00:24+00:00","modified":"2024-08-07T09:37:48+00:00","when":"2024-08-07T08:00:24+00:00","text":"### **Summary:**\nChromeOS sign in issues affecting multiple customers throughout the globe\n### **Description:**\nWe are experiencing an issue with ChromeOS.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Wednesday, 2024-08-07 06:00 US/Pacific with current details.\n### **Diagnosis:**\nDuring the full sign in process ChromeOS devices complete the sign in process (even Multi Factor Authentication (MFA) if applicable) but are eventually brought back to the login page.\nThe issue is affecting ChromeOS versions 120 to 128 (potentially more versions).\n### **Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-08-07T06:01:47+00:00","modified":"2024-08-07T06:04:26+00:00","when":"2024-08-07T06:01:47+00:00","text":"### **Summary:**\nChromeOS sign in issues affecting multiple customers throughout the globe\n### **Description:**\nWe are experiencing an issue with ChromeOS.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Wednesday, 2024-08-07 01:30 US/Pacific with current details.\n### **Diagnosis:**\nDuring the full sign in process ChromeOS devices complete the sign in process (even Multi Factor Authentication (MFA) if applicable) but are eventually brought back to the login page.\nThe issue is affecting multiple ChromeOS versions\n### **Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-08-07T05:06:42+00:00","modified":"2024-08-07T05:06:42+00:00","when":"2024-08-07T05:06:42+00:00","text":"### **Summary:**\nChromeOS sign in issues affecting multiple customers throughout the globe\n### **Description:**\nWe are experiencing an issue with ChromeOS.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Tuesday, 2024-08-06 23:30 US/Pacific with current details.\n### **Diagnosis:**\nDuring the full sign in process ChromeOS devices complete the sign in process (even Multi Factor Authentication (MFA) if applicable) but are eventually brought back to the login page.\nThe issue is affecting multiple ChromeOS versions\n### **Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-08-15T19:46:47+00:00","modified":"2024-08-15T19:46:47+00:00","when":"2024-08-15T19:46:47+00:00","text":"# Incident Report\n## Summary\nOn 06 August at 08:00 US/Pacific, Chrome OS devices experienced issues with logins globally for a duration of 19 hours 29 minutes. During this time, users on ChromeOS devices were able to complete the full sign-in process, including any Multi-factor Authentication when required. However, some users were immediately redirected back to the login page after logging in.\n## Root Cause\nGoogle’s internal authentication system uses an HTTP endpoint microservice that provides clients with a list of accounts currently in the authentication cookies. This endpoint is mainly used for various account switcher user interfaces, providing a list of accounts the current session has access to.\nThe root cause of the issue is an incorrect configuration made to the Google authentication service, which resulted in a malformed response to the queries from this microservice in a particular condition. As a result, Chrome OS was unable to get the account details from the internal Google authentication service, which caused the sign-in failures.\n## Remediation and Prevention\nGoogle engineers were alerted to the outage via a support case on Tuesday, 06 August at 15:51 US/Pacific, and immediately started an investigation. Once the nature and scope of the issue became clear, the configuration change was reverted and Google engineers ensured that sign-in worked again.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n* Migrate the endpoints used in Google authentication systems to a different format so that the issue is not triggered\n* Expand the integration testing environment to include more clients including Google Chrome so that all scenarios are identified\n* Include additional control mechanisms to ensure that configuration changes are approved appropriately\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\n## Detailed Description of Impact\n* On devices which were already configured, users were unable to add new users from the login screen.\n* Users were able to successfully authenticate however, and were sent back to the login screen before starting a session.\n* Users were unable to setup the first user account on their device (the device owner), resulting in the devices being reset to factory defaults","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"RjKbsvuGeE7M1pvBHmvX","service_name":"ChromeOS","affected_products":[{"title":"ChromeOS","id":"RjKbsvuGeE7M1pvBHmvX"}],"uri":"incidents/sowQNioouYdVq6hNkZti"},{"id":"4XQNFXAa34SoEbbetAka","number":"5855443188559699494","begin":"2024-07-30T14:05:00+00:00","created":"2024-07-30T16:51:18+00:00","end":"2024-07-30T17:10:00+00:00","modified":"2024-08-02T16:18:37+00:00","external_desc":"**Summary:**\nA subset of Gmail users might be experiencing delayed delivery of their emails.\n**Description:**\nWe are experiencing an intermittent issue with Gmail beginning on Tuesday, 2024-07-30 07:05 US/Pacific.\nThis mainly caused delayed email delivery and a small number of users receiving errors when sending/receiving emails.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Tuesday, 2024-07-30 11:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Diagnosis:**\nSome customers might see a yellow banner with the \"Unable to reach Gmail---Trying to reconnect\" message.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-08-02T16:18:37+00:00","modified":"2024-08-02T16:18:37+00:00","when":"2024-08-02T16:18:37+00:00","text":"# Incident Report\n## Summary\nOn Tuesday, 30 July 2024, Gmail experienced increased error rates and delivery delays for a duration of 3 hours and 5 minutes.\nTo our Gmail users who were impacted by these issues, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nGoogle engineers have determined that the increased error rates and delivery delays were due to unrelated root causes.\n**Email Delivery Delays**\nOn Tuesday, 30 July 2024, at 07:05 US/Pacific, a series of fiberlines which carry Gmail in-flight traffic were unintentionally disabled between Google's US and European data centers as part of routine maintenance.\n**Increased Error Rates**\nOn Tuesday, 30 July 2024, at 08:00 US/Pacific, a non-critical process responsible for batch data processing started generating an increased volume of traffic which overloaded Gmail’s back-end system.\n## Remediation and Prevention\n**Email Delivery Delays**\nGoogle engineers were alerted of the incident through automated alert systems on Tuesday, 30 July 2024 at 07:14 US/Pacific and immediately began an investigation. By 07:34 US/Pacific, engineers performed a change to mitigate congestion/throttling which rerouted traffic to available servers for processing.\n**Increased Error Rates**\nGoogle engineers were alerted of the incident through automated alert systems on Tuesday, 30 July 2024 at 08:00 US/Pacific and immediately began an investigation. The issue self-resolved on Tuesday, 30 July 2024 at 10:10 US/Pacific, and errors were trending down to the baseline. The engineers identified and made the necessary changes to avoid a repeat of the issue at 10:30 US/Pacific.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n- Audit and test our traffic priority labels and storage system behavior to ensure that background traffic cannot overload the system.\n- Audit/test error propagation to make sure the routing layer responds adequately to error conditions.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\n## Detailed Description of Impact\n**Email Delivery Delays**\nOn July 30, 2024 07:05 to 08:00 US/Pacific, some emails were delayed between 10-20 minutes.\n**Increased Error Rates**\nOn July 30, 2024 07:46 to 10:10 US/Pacific, some users received potentially visible errors:\n- An error message banner with a valid retry button for users to manually retry the next Gmail web server - “Oops… the system encountered a problem (#2014)”","status":"AVAILABLE"},{"created":"2024-07-30T20:40:06+00:00","modified":"2024-08-02T03:54:41+00:00","when":"2024-07-30T20:40:06+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 30 July, 2023 07:05\n**Incident End:** 30 July, 2023 10:10\n**Duration:** 3 hours, 5 minutes\n**Affected Services and Features:**\nGoogle Gmail\n**Regions/Zones:** Global\n**Description:**\nGmail experienced an overload scenario leading to delayed email delivery as well as serving user visible errors \"Unable to reach Gmail---Trying to reconnect\".\nFrom the preliminary analysis, our teams identified a potential root cause and blocked traffic contributing towards the overload.\n**Customer Impact:**\n- Approximately 40M emails, sent by approximately 2.6M unique senders, were delayed by 10 minutes or more.\n- At peak, 70k unique Gmail users received error messages. e.g. \"Unable to reach Gmail---Trying to reconnect\"\n**Additional details:**\nOur engineering team is making the required changes to prevent such issues.\nWe apologize to all who were affected by the disruption.","status":"AVAILABLE"},{"created":"2024-07-30T17:49:53+00:00","modified":"2024-07-30T20:37:01+00:00","when":"2024-07-30T17:49:53+00:00","text":"**Status: RESOLVED**\nThe issue with Gmail has been resolved for all affected users as of Tuesday, 2024-07-30 10:00 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-07-30T16:51:19+00:00","modified":"2024-07-30T20:37:25+00:00","when":"2024-07-30T16:51:19+00:00","text":"**Summary:**\nA subset of Gmail users might be experiencing delayed delivery of their emails.\n**Description:**\nWe are experiencing an intermittent issue with Gmail beginning on Tuesday, 2024-07-30 07:05 US/Pacific.\nThis mainly caused delayed email delivery and a small number of users receiving errors when sending/receiving emails.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Tuesday, 2024-07-30 11:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Diagnosis:**\nSome customers might see a yellow banner with the \"Unable to reach Gmail---Trying to reconnect\" message.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-08-02T16:18:37+00:00","modified":"2024-08-02T16:18:37+00:00","when":"2024-08-02T16:18:37+00:00","text":"# Incident Report\n## Summary\nOn Tuesday, 30 July 2024, Gmail experienced increased error rates and delivery delays for a duration of 3 hours and 5 minutes.\nTo our Gmail users who were impacted by these issues, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nGoogle engineers have determined that the increased error rates and delivery delays were due to unrelated root causes.\n**Email Delivery Delays**\nOn Tuesday, 30 July 2024, at 07:05 US/Pacific, a series of fiberlines which carry Gmail in-flight traffic were unintentionally disabled between Google's US and European data centers as part of routine maintenance.\n**Increased Error Rates**\nOn Tuesday, 30 July 2024, at 08:00 US/Pacific, a non-critical process responsible for batch data processing started generating an increased volume of traffic which overloaded Gmail’s back-end system.\n## Remediation and Prevention\n**Email Delivery Delays**\nGoogle engineers were alerted of the incident through automated alert systems on Tuesday, 30 July 2024 at 07:14 US/Pacific and immediately began an investigation. By 07:34 US/Pacific, engineers performed a change to mitigate congestion/throttling which rerouted traffic to available servers for processing.\n**Increased Error Rates**\nGoogle engineers were alerted of the incident through automated alert systems on Tuesday, 30 July 2024 at 08:00 US/Pacific and immediately began an investigation. The issue self-resolved on Tuesday, 30 July 2024 at 10:10 US/Pacific, and errors were trending down to the baseline. The engineers identified and made the necessary changes to avoid a repeat of the issue at 10:30 US/Pacific.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n- Audit and test our traffic priority labels and storage system behavior to ensure that background traffic cannot overload the system.\n- Audit/test error propagation to make sure the routing layer responds adequately to error conditions.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\n## Detailed Description of Impact\n**Email Delivery Delays**\nOn July 30, 2024 07:05 to 08:00 US/Pacific, some emails were delayed between 10-20 minutes.\n**Increased Error Rates**\nOn July 30, 2024 07:46 to 10:10 US/Pacific, some users received potentially visible errors:\n- An error message banner with a valid retry button for users to manually retry the next Gmail web server - “Oops… the system encountered a problem (#2014)”","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/4XQNFXAa34SoEbbetAka"},{"id":"bRXnorYZuuVLDDqctj87","number":"9738923281992756915","begin":"2024-07-24T22:36:00+00:00","created":"2024-07-25T12:04:20+00:00","end":"2024-07-25T16:27:00+00:00","modified":"2024-07-30T15:02:49+00:00","external_desc":"SUMMARY: Chrome Browser Password Manager Issue\nWe are experiencing an issue with Chrome Browser where impacted users may be unable to find or save passwords in Chrome's Password Manager. This impacts users on M127 version of Chrome Browser.\nOur Engineering team has identified the cause of the issue and are actively working on deploying a fix for all users.\nWe will continue to monitor the situation as the fix is rolled out to ensure a smooth resolution.\nWe will provide more information by Thursday, 2024-07-25 09:45 US/Pacific.\nWORKAROUND:\nUsers currently impacted by this can try the following:\nLaunch Chrome with \" --enable-features=SkipUndecryptablePasswords\" command line flag.\n1. Locate Chrome shortcut on desktop. If you don't have it, go to chrome://settings/manageProfile and at the bottom enable \"Create desktop shortcut\".\n2. Fully exit Chrome\n3. Go to desktop and right click the shortcut then click Properties.\n4. To the target field paste the following string to the end \" --enable-features=SkipUndecryptablePasswords\".\n5. Close the dialog with OK and launch Chrome using this shortcut.","updates":[{"created":"2024-07-30T14:58:18+00:00","modified":"2024-07-30T15:02:49+00:00","when":"2024-07-30T14:58:18+00:00","text":"Based on post incident analysis, we are downgrading the severity of this incident to ‘Service information’, as the issue affected significantly fewer users than initially thought.","status":"AVAILABLE"},{"created":"2024-07-26T08:58:22+00:00","modified":"2024-07-29T11:04:19+00:00","when":"2024-07-26T08:58:22+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 24 July, 2024 15:36\n**Incident End:** 25 July, 2024 09:27\n**Duration:** 17 hours, 51 minutes\n**Affected Services and Features:**\nChrome Browser\n**Regions/Zones:** Global\n**Description:**\nA significant number of users were unable to find or save passwords in the password manager for Chrome browser, for a period of 17 hours, 51 minutes.\nFrom the preliminary analysis the root cause of the issue is a change in product behavior without proper feature guard. Google engineers mitigated the issue by deploying a fix.\n**Customer Impact:**\nImpacted users were unable to find passwords in Chrome's password manager. Users can save passwords, however it was not visible to them. The impact was limited to the M127 version of Chrome Browser on the Windows platform.\nApproximately 2% of users out of the 25% of the entire user base where the configuration change was rolled out, experienced this issue.\n**Additional details:**\nWhile an interim workaround was provided during the incident, after the fix was fully rolled out, users are being advised to restart their Chrome browser to ensure that the fix takes effect.","status":"AVAILABLE"},{"created":"2024-07-25T16:30:52+00:00","modified":"2024-07-25T16:30:52+00:00","when":"2024-07-25T16:30:52+00:00","text":"The issue with Chrome Browser has been mitigated for all affected users as of Thursday, 2024-07-25 09:27 US/Pacific.\nUsers are advised to restart their Chrome Browser to ensure that the fix takes effect.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-07-25T12:04:20+00:00","modified":"2024-07-30T14:58:26+00:00","when":"2024-07-25T12:04:20+00:00","text":"SUMMARY: Chrome Browser Password Manager Issue\nWe are experiencing an issue with Chrome Browser where impacted users may be unable to find or save passwords in Chrome's Password Manager. This impacts users on M127 version of Chrome Browser.\nOur Engineering team has identified the cause of the issue and are actively working on deploying a fix for all users.\nWe will continue to monitor the situation as the fix is rolled out to ensure a smooth resolution.\nWe will provide more information by Thursday, 2024-07-25 09:45 US/Pacific.\nWORKAROUND:\nUsers currently impacted by this can try the following:\nLaunch Chrome with \" --enable-features=SkipUndecryptablePasswords\" command line flag.\n1. Locate Chrome shortcut on desktop. If you don't have it, go to chrome://settings/manageProfile and at the bottom enable \"Create desktop shortcut\".\n2. Fully exit Chrome\n3. Go to desktop and right click the shortcut then click Properties.\n4. To the target field paste the following string to the end \" --enable-features=SkipUndecryptablePasswords\".\n5. Close the dialog with OK and launch Chrome using this shortcut.","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-07-30T14:58:18+00:00","modified":"2024-07-30T15:02:49+00:00","when":"2024-07-30T14:58:18+00:00","text":"Based on post incident analysis, we are downgrading the severity of this incident to ‘Service information’, as the issue affected significantly fewer users than initially thought.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"sfPUGuj9d6b63iYijuYV","service_name":"Chrome Browser","affected_products":[{"title":"Chrome Browser","id":"sfPUGuj9d6b63iYijuYV"}],"uri":"incidents/bRXnorYZuuVLDDqctj87"},{"id":"zZeXPJKVA1NWYXYJvAmW","number":"8291830681022331576","begin":"2024-07-23T07:00:00+00:00","created":"2024-07-26T23:05:45+00:00","end":"2024-07-25T23:00:00+00:00","modified":"2024-07-26T23:08:19+00:00","external_desc":"**Summary:**\nPoly Studio X devices experienced unexpected deprovisioning and might require manual re-enrollment.\n**Description:**\nPoly Studio X devices running Google Meet experienced unexpected deprovisioning between Tuesday, 2024-11-23 to Thursday, 2024-11-25.\nGoogle engineers rolled out a software fix to mitigate the issue, however if the device had already got impacted and got to the activation code screen, they would require manual re-enrollment. Please refer to the workaround section for more information.\n**Diagnosis:**\nThe affected customer may encounter Poly Studio X devices offline/deprovisioned.\n**Workaround:**\n**Customer Action required**: Each affected device has to be [manually re-enrolled in the Admin Console](https://support.google.com/a/answer/6109380?hl=en) using the displayed activation code, as part of the “Enroll your device (non-ChromeOS devices) steps. Following the re-enrollment, the devices will be reconfigured to restore their previous settings and its functionality.","updates":[{"created":"2024-07-26T23:07:51+00:00","modified":"2024-07-26T23:07:51+00:00","when":"2024-07-26T23:07:51+00:00","text":"**Summary:**\nPoly Studio X devices experienced unexpected deprovisioning and might require manual re-enrollment.\n**Description:**\nPoly Studio X devices running Google Meet experienced unexpected deprovisioning between Tuesday, 2024-11-23 to Thursday, 2024-11-25.\nGoogle engineers rolled out a software fix to mitigate the issue, however if the device had already got impacted and got to the activation code screen, they would require manual re-enrollment. Please refer to the workaround section for more information.\n**Diagnosis:**\nThe affected customer may encounter Poly Studio X devices offline/deprovisioned.\n**Workaround:**\n**Customer Action required**: Each affected device has to be [manually re-enrolled in the Admin Console](https://support.google.com/a/answer/6109380?hl=en) using the displayed activation code, as part of the “Enroll your device (non-ChromeOS devices) steps. Following the re-enrollment, the devices will be reconfigured to restore their previous settings and its functionality.","status":"AVAILABLE"},{"created":"2024-07-26T23:05:46+00:00","modified":"2024-07-26T23:05:46+00:00","when":"2024-07-26T23:05:46+00:00","text":"**Summary:**\nPoly Studio X devices experienced unexpected deprovisioning and might require manual re-enrollment.\n**Description:**\nPoly Studio X devices running Google Meet experienced unexpected deprovisioning between Tuesday, 2024-11-23 to Thursday, 2024-11-25.\nGoogle engineers rolled out a software fix to mitigate the issue, however if the device had already got impacted and got to the activation code screen, they would require manual re-enrollment. Please refer to the workaround section for more information.\n**Diagnosis:**\nThe affected customer may encounter Poly Studio X devices offline/deprovisioned.\n**Workaround:**\n**Customer Action required**: Each affected device has to be [manually re-enrolled in the Admin Console](https://support.google.com/a/answer/6109380?hl=en) using the displayed activation code, as part of the “Enroll your device (non-ChromeOS devices) steps. Following the re-enrollment, the devices will be reconfigured to restore their previous settings and its functionality.","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-07-26T23:07:51+00:00","modified":"2024-07-26T23:07:51+00:00","when":"2024-07-26T23:07:51+00:00","text":"**Summary:**\nPoly Studio X devices experienced unexpected deprovisioning and might require manual re-enrollment.\n**Description:**\nPoly Studio X devices running Google Meet experienced unexpected deprovisioning between Tuesday, 2024-11-23 to Thursday, 2024-11-25.\nGoogle engineers rolled out a software fix to mitigate the issue, however if the device had already got impacted and got to the activation code screen, they would require manual re-enrollment. Please refer to the workaround section for more information.\n**Diagnosis:**\nThe affected customer may encounter Poly Studio X devices offline/deprovisioned.\n**Workaround:**\n**Customer Action required**: Each affected device has to be [manually re-enrolled in the Admin Console](https://support.google.com/a/answer/6109380?hl=en) using the displayed activation code, as part of the “Enroll your device (non-ChromeOS devices) steps. Following the re-enrollment, the devices will be reconfigured to restore their previous settings and its functionality.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"sUH4BQXzYXma7NiS94Hi","service_name":"Google Meet","affected_products":[{"title":"Google Meet","id":"sUH4BQXzYXma7NiS94Hi"}],"uri":"incidents/zZeXPJKVA1NWYXYJvAmW"},{"id":"mmgZJpHH9hjsL7qHuDmQ","number":"4221494310817348178","begin":"2024-07-16T17:40:00+00:00","created":"2024-07-16T22:34:07+00:00","end":"2024-07-16T18:34:00+00:00","modified":"2024-07-16T22:35:17+00:00","external_desc":"**Summary:**\nWe have received reports of an issue with Gmail.\n**Description:**\nUpon further investigation, our engineering teams believe that the scope is very limited and/or no customers were impacted.\nIf you have questions or feel that you may be impacted, please open a case with the Support Team and we will work with you until the issue is resolved. No further updates will be provided here.","updates":[{"created":"2024-07-16T22:34:44+00:00","modified":"2024-07-16T22:34:44+00:00","when":"2024-07-16T22:34:44+00:00","text":"**Summary:**\nWe have received reports of an issue with Gmail.\n**Description:**\nUpon further investigation, our engineering teams believe that the scope is very limited and/or no customers were impacted.\nIf you have questions or feel that you may be impacted, please open a case with the Support Team and we will work with you until the issue is resolved. No further updates will be provided here.","status":"AVAILABLE"},{"created":"2024-07-16T22:34:09+00:00","modified":"2024-07-16T22:34:09+00:00","when":"2024-07-16T22:34:09+00:00","text":"**Summary:**\nWe have received reports of an issue with Gmail.\n**Description:**\nUpon further investigation, our engineering teams believe that the scope is very limited and/or no customers were impacted.\nIf you have questions or feel that you may be impacted, please open a case with the Support Team and we will work with you until the issue is resolved. No further updates will be provided here.","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-07-16T22:34:44+00:00","modified":"2024-07-16T22:34:44+00:00","when":"2024-07-16T22:34:44+00:00","text":"**Summary:**\nWe have received reports of an issue with Gmail.\n**Description:**\nUpon further investigation, our engineering teams believe that the scope is very limited and/or no customers were impacted.\nIf you have questions or feel that you may be impacted, please open a case with the Support Team and we will work with you until the issue is resolved. No further updates will be provided here.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/mmgZJpHH9hjsL7qHuDmQ"},{"id":"w7jWXXKKSYG2m4Yjd6Sb","number":"4469121945187482761","begin":"2024-06-17T16:06:00+00:00","created":"2024-06-17T20:04:02+00:00","end":"2024-06-17T22:46:00+00:00","modified":"2024-06-21T16:18:25+00:00","external_desc":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nWe are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.\nOur engineering team has identified the root cause of the issue and are currently working on a mitigation.\nWe will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\n- Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.\n- Some Google Meet users are having issues joining the meetings.\n- Some Google Drive users are experiencing elevated latencies and error rates\n- Some Google Docs users are experiencing issues where they see \"Name loading\" in comments.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-06-21T16:18:23+00:00","modified":"2024-06-21T16:18:23+00:00","when":"2024-06-21T16:18:23+00:00","text":"The full incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},{"created":"2024-06-18T17:58:01+00:00","modified":"2024-06-18T17:58:01+00:00","when":"2024-06-18T17:58:01+00:00","text":"A mini incident report has been posted on the Google\nWorkspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},{"created":"2024-06-17T22:51:15+00:00","modified":"2024-06-17T22:51:15+00:00","when":"2024-06-17T22:51:15+00:00","text":"The issue with Google Chat is mitigated. Please find the details in the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},{"created":"2024-06-17T21:37:32+00:00","modified":"2024-06-17T21:37:32+00:00","when":"2024-06-17T21:37:32+00:00","text":"We are aware of an ongoing issue. Our engineers are working towards mitigation.\nPlease find more updates in the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"SERVICE_DISRUPTION"},{"created":"2024-06-17T20:59:18+00:00","modified":"2024-06-17T20:59:18+00:00","when":"2024-06-17T20:59:18+00:00","text":"We are aware of an ongoing issue. Our engineers are working towards mitigation.\nPlease find more updates in the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"SERVICE_DISRUPTION"},{"created":"2024-06-17T20:04:03+00:00","modified":"2024-06-17T20:04:03+00:00","when":"2024-06-17T20:04:03+00:00","text":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nWe are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.\nOur engineering team has identified the root cause of the issue and are currently working on a mitigation.\nWe will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\n- Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.\n- Some Google Meet users are having issues joining the meetings.\n- Some Google Drive users are experiencing elevated latencies and error rates\n- Some Google Docs users are experiencing issues where they see \"Name loading\" in comments.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-06-21T16:18:23+00:00","modified":"2024-06-21T16:18:23+00:00","when":"2024-06-21T16:18:23+00:00","text":"The full incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"Ht9Vx5PFzDPHYvrnrvSk","service_name":"Google Chat","affected_products":[{"title":"Google Chat","id":"Ht9Vx5PFzDPHYvrnrvSk"}],"uri":"incidents/w7jWXXKKSYG2m4Yjd6Sb"},{"id":"B7FqFhqQ4F5AbfNpuGbT","number":"2157661286643548200","begin":"2024-06-17T16:06:00+00:00","created":"2024-06-17T20:06:28+00:00","end":"2024-06-17T22:46:00+00:00","modified":"2024-06-21T16:14:06+00:00","external_desc":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nWe are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.\nOur engineering team has identified the root cause of the issue and are currently working on a mitigation.\nWe will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\n- Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.\n- Some Google Meet users are having issues joining the meetings.\n- Some Google Drive users are experiencing elevated latencies and error rates\n- Some Google Docs users are experiencing issues where they see \"Name loading\" in comments.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-06-21T16:13:52+00:00","modified":"2024-06-21T16:13:52+00:00","when":"2024-06-21T16:13:52+00:00","text":"# Incident Report\n## Summary\nOn 17 June 2024, a subset of our customers using Google Workspace products (Google Chat, Google Meet, Google Drive, Google Calendar, Google Cloud Directory and Google Docs) experienced elevated error rates and latency of varying impact between the period of 6 hours and 40 minutes between 09:06 and 15:46 US/Pacific.\nTo our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nThe root cause is a latent issue that caused contention in getting access in a cache used for profile information. The issue was masked by overprovisioned quotas for some of the tables and only surfaced when quota adjustments were made to match the traffic patterns.\n## Remediation and Prevention\nGoogle engineers were alerted to the issue from our internal monitoring at 09:21 US/Pacific on 17 June 2024 and immediately started working on the incident. Once the nature and scope of the issue was identified, our engineers mitigated the issue by increasing quotas and disabling the suspected feature in the key/value serving system.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions :\n* Change the underlying design that caused the contention in the cache so that the issue does recur.\n* Implement mechanisms that facilitate quicker application of changes and detection of contention, to allow faster remediation.\n* Update Google Chat to be more resilient to profile information unavailability.\n## Detailed Description of Impact\nOn Monday, 17 June 2024, six of our Workspace products were impacted by varying degrees between 09:06 and 15:46 US/Pacific. The details and timeframes for the impact to each product have been outlined below.\n**Google Calendar :** Impacted users saw email addresses instead of names while using Calendar for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024.\n**Google Chat :** Impacted users experienced intermittent issues loading chat, opening chat rooms, and sending messages for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024.\n**Google Meet :** Impacted users sparsely experienced latency or failures while joining scheduled meetings over a period of 4 hours, between 11:00 to 15:00 US/Pacific on 17 June 2024.\n**Google Docs :** Impacted users encountered the error \"Name loading\" in comments or other areas where authors’ names were expected for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024.\n**Google Drive :** Impacted users experienced elevated errors/latency while using third party APIs, for a duration of 6 hours 30 minutes from 09:06 to 15:36 US/Pacific on 17 June 2024.\n**Google Cloud Directory :** Affected customers were unable to access the “User” section in the admin console. The initial dashboard displayed the error \"Data can't be fetched due to system” for a period of 6 hours 11 minutes from 09:05 to 15:16 US/Pacific on 17 June 2024.","status":"AVAILABLE"},{"created":"2024-06-18T17:54:31+00:00","modified":"2024-06-18T17:54:31+00:00","when":"2024-06-18T17:54:31+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 17 June, 2024 09:23\n**Incident End:** 17 June, 2024 15:46\n**Duration:** 6 hours, 23 minutes\n**Affected Services and Features:** Google Calendar, Google Chat, Google Docs, Google Drive \u0026 Google Meet.\n**Regions/Zones:** Global\n**Description:**\nMultiple Google Workspace products experienced elevated error rates and latency. From preliminary analysis, the root cause is a latent issue that caused lock contention in a cache used for profile information. The issue was masked by overprovisioned quotas for some of the tables and only surfaced when quota adjustments were made to match the traffic patterns. Our engineers mitigated the issue by increasing quotas and disabling the suspected feature in the key/value serving system.\nGoogle will complete a full Incident Report in the following days that will provide a full root cause.\n**Customer Impact:**\n* ***Google Chat*** - Affected users would have experienced intermittent latency and errors while loading chats, accessing the chat spaces and sending messages.\n* ***Google Meet*** - Affected users would have experienced latency or failures while joining meetings.\n* ***Google Drive*** - Affected users would have experienced latency/errors while using the third party API.\n* ***Google Docs*** - Affected users would have experienced latency loading comments on the document. Some comments would have author names missing.\n* ***Google Calendar*** - Affected users would have experienced elevated error rates while using Calendar.\n------","status":"AVAILABLE"},{"created":"2024-06-17T22:46:04+00:00","modified":"2024-06-17T22:46:04+00:00","when":"2024-06-17T22:46:04+00:00","text":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nThe issue with Google Calendar, Google Chat, Google Docs, Google Drive, Google Meet has been resolved for all affected users as of Monday, 2024-06-17 15:27 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-06-17T21:34:49+00:00","modified":"2024-06-17T21:34:49+00:00","when":"2024-06-17T21:34:49+00:00","text":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nThe mitigation applied previously did not yield results as expected.\nOur Engineering team continues to investigate and ascertain a mitigation strategy to fix all known issues\nWe will provide more information by Monday, 2024-06-17 16:30 US/Pacific.\n**Customer Symptoms:**\n- Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.\n- Some Google Meet users are having issues joining the meetings.\n- Some Google Drive users are experiencing elevated latencies and error rates\n- Some Google Docs users are experiencing issues where they see \"Name loading\" in comments.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-06-17T20:52:10+00:00","modified":"2024-06-17T20:52:10+00:00","when":"2024-06-17T20:52:10+00:00","text":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nMitigation work is currently underway by our engineering team.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Monday, 2024-06-17 14:45 US/Pacific.\n**Customer Symptoms:**\n- Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.\n- Some Google Meet users are having issues joining the meetings.\n- Some Google Drive users are experiencing elevated latencies and error rates\n- Some Google Docs users are experiencing issues where they see \"Name loading\" in comments.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-06-17T20:06:28+00:00","modified":"2024-06-17T20:06:28+00:00","when":"2024-06-17T20:06:28+00:00","text":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nWe are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.\nOur engineering team has identified the root cause of the issue and are currently working on a mitigation.\nWe will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\n- Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.\n- Some Google Meet users are having issues joining the meetings.\n- Some Google Drive users are experiencing elevated latencies and error rates\n- Some Google Docs users are experiencing issues where they see \"Name loading\" in comments.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-06-21T16:13:52+00:00","modified":"2024-06-21T16:13:52+00:00","when":"2024-06-21T16:13:52+00:00","text":"# Incident Report\n## Summary\nOn 17 June 2024, a subset of our customers using Google Workspace products (Google Chat, Google Meet, Google Drive, Google Calendar, Google Cloud Directory and Google Docs) experienced elevated error rates and latency of varying impact between the period of 6 hours and 40 minutes between 09:06 and 15:46 US/Pacific.\nTo our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nThe root cause is a latent issue that caused contention in getting access in a cache used for profile information. The issue was masked by overprovisioned quotas for some of the tables and only surfaced when quota adjustments were made to match the traffic patterns.\n## Remediation and Prevention\nGoogle engineers were alerted to the issue from our internal monitoring at 09:21 US/Pacific on 17 June 2024 and immediately started working on the incident. Once the nature and scope of the issue was identified, our engineers mitigated the issue by increasing quotas and disabling the suspected feature in the key/value serving system.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions :\n* Change the underlying design that caused the contention in the cache so that the issue does recur.\n* Implement mechanisms that facilitate quicker application of changes and detection of contention, to allow faster remediation.\n* Update Google Chat to be more resilient to profile information unavailability.\n## Detailed Description of Impact\nOn Monday, 17 June 2024, six of our Workspace products were impacted by varying degrees between 09:06 and 15:46 US/Pacific. The details and timeframes for the impact to each product have been outlined below.\n**Google Calendar :** Impacted users saw email addresses instead of names while using Calendar for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024.\n**Google Chat :** Impacted users experienced intermittent issues loading chat, opening chat rooms, and sending messages for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024.\n**Google Meet :** Impacted users sparsely experienced latency or failures while joining scheduled meetings over a period of 4 hours, between 11:00 to 15:00 US/Pacific on 17 June 2024.\n**Google Docs :** Impacted users encountered the error \"Name loading\" in comments or other areas where authors’ names were expected for a duration of 6 hours 21 minutes from 09:06 to 15:27 US/Pacific on 17 June 2024.\n**Google Drive :** Impacted users experienced elevated errors/latency while using third party APIs, for a duration of 6 hours 30 minutes from 09:06 to 15:36 US/Pacific on 17 June 2024.\n**Google Cloud Directory :** Affected customers were unable to access the “User” section in the admin console. The initial dashboard displayed the error \"Data can't be fetched due to system” for a period of 6 hours 11 minutes from 09:05 to 15:16 US/Pacific on 17 June 2024.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"wNHuVFtZEWU5Mmj2eRCK","service_name":"Google Docs","affected_products":[{"title":"Google Docs","id":"wNHuVFtZEWU5Mmj2eRCK"}],"uri":"incidents/B7FqFhqQ4F5AbfNpuGbT"},{"id":"qyyQHqGNLgxwoY4sxNzc","number":"7648634937827532717","begin":"2024-06-17T16:06:00+00:00","created":"2024-06-17T20:04:56+00:00","end":"2024-06-17T22:46:00+00:00","modified":"2024-06-21T16:18:52+00:00","external_desc":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nWe are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.\nOur engineering team has identified the root cause of the issue and are currently working on a mitigation.\nWe will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\n- Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.\n- Some Google Meet users are having issues joining the meetings.\n- Some Google Drive users are experiencing elevated latencies and error rates\n- Some Google Docs users are experiencing issues where they see \"Name loading\" in comments.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-06-21T16:18:50+00:00","modified":"2024-06-21T16:18:50+00:00","when":"2024-06-21T16:18:50+00:00","text":"The full incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},{"created":"2024-06-18T17:57:08+00:00","modified":"2024-06-18T17:57:08+00:00","when":"2024-06-18T17:57:08+00:00","text":"A mini incident report has been posted on the Google\nWorkspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},{"created":"2024-06-17T22:50:13+00:00","modified":"2024-06-17T22:50:13+00:00","when":"2024-06-17T22:50:13+00:00","text":"The issue with Google Drive is mitigated. Please find the details in the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},{"created":"2024-06-17T21:37:10+00:00","modified":"2024-06-17T21:37:10+00:00","when":"2024-06-17T21:37:10+00:00","text":"We are aware of an ongoing issue. Our engineers are working towards mitigation.\nPlease find more updates in the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"SERVICE_DISRUPTION"},{"created":"2024-06-17T20:58:39+00:00","modified":"2024-06-17T20:58:39+00:00","when":"2024-06-17T20:58:39+00:00","text":"We are aware of an ongoing issue. Our engineers are working towards mitigation.\nPlease find more updates in the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"SERVICE_DISRUPTION"},{"created":"2024-06-17T20:04:57+00:00","modified":"2024-06-17T20:04:57+00:00","when":"2024-06-17T20:04:57+00:00","text":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nWe are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.\nOur engineering team has identified the root cause of the issue and are currently working on a mitigation.\nWe will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\n- Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.\n- Some Google Meet users are having issues joining the meetings.\n- Some Google Drive users are experiencing elevated latencies and error rates\n- Some Google Docs users are experiencing issues where they see \"Name loading\" in comments.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-06-21T16:18:50+00:00","modified":"2024-06-21T16:18:50+00:00","when":"2024-06-21T16:18:50+00:00","text":"The full incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"VHNA7p3Z5p3iakj5sA8V","service_name":"Google Drive","affected_products":[{"title":"Google Drive","id":"VHNA7p3Z5p3iakj5sA8V"}],"uri":"incidents/qyyQHqGNLgxwoY4sxNzc"},{"id":"mPtQiYbz95AjCD8FiiKK","number":"6780538630212666025","begin":"2024-06-17T16:06:00+00:00","created":"2024-06-17T20:05:25+00:00","end":"2024-06-17T22:46:00+00:00","modified":"2024-06-21T16:17:56+00:00","external_desc":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nWe are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.\nOur engineering team has identified the root cause of the issue and are currently working on a mitigation.\nWe will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\n- Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.\n- Some Google Meet users are having issues joining the meetings.\n- Some Google Drive users are experiencing elevated latencies and error rates\n- Some Google Docs users are experiencing issues where they see \"Name loading\" in comments.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-06-21T16:17:51+00:00","modified":"2024-06-21T16:17:51+00:00","when":"2024-06-21T16:17:51+00:00","text":"The full incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},{"created":"2024-06-18T17:56:47+00:00","modified":"2024-06-18T17:56:47+00:00","when":"2024-06-18T17:56:47+00:00","text":"A mini incident report has been posted on the Google\nWorkspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},{"created":"2024-06-17T22:49:19+00:00","modified":"2024-06-17T22:49:19+00:00","when":"2024-06-17T22:49:19+00:00","text":"The issue with Google Calendar is mitigated. Please find the details in the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},{"created":"2024-06-17T21:36:42+00:00","modified":"2024-06-17T21:36:42+00:00","when":"2024-06-17T21:36:42+00:00","text":"We are aware of an ongoing issue. Our engineers are working towards mitigation.\nPlease find more updates in the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"SERVICE_DISRUPTION"},{"created":"2024-06-17T20:58:00+00:00","modified":"2024-06-17T20:58:00+00:00","when":"2024-06-17T20:58:00+00:00","text":"We are aware of an ongoing issue. Our engineers are working towards mitigation.\nPlease find more updates in the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"SERVICE_DISRUPTION"},{"created":"2024-06-17T20:05:25+00:00","modified":"2024-06-17T20:05:25+00:00","when":"2024-06-17T20:05:25+00:00","text":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nWe are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.\nOur engineering team has identified the root cause of the issue and are currently working on a mitigation.\nWe will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\n- Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.\n- Some Google Meet users are having issues joining the meetings.\n- Some Google Drive users are experiencing elevated latencies and error rates\n- Some Google Docs users are experiencing issues where they see \"Name loading\" in comments.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-06-21T16:17:51+00:00","modified":"2024-06-21T16:17:51+00:00","when":"2024-06-21T16:17:51+00:00","text":"The full incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"n5hVexuq1PM9onY9qrmo","service_name":"Google Calendar","affected_products":[{"title":"Google Calendar","id":"n5hVexuq1PM9onY9qrmo"}],"uri":"incidents/mPtQiYbz95AjCD8FiiKK"},{"id":"eo6f2ck36qQ2jHjjZmpc","number":"14966795240874101722","begin":"2024-06-17T16:06:00+00:00","created":"2024-06-17T20:05:48+00:00","end":"2024-06-17T22:46:00+00:00","modified":"2024-06-21T16:19:15+00:00","external_desc":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nWe are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.\nOur engineering team has identified the root cause of the issue and are currently working on a mitigation.\nWe will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\n- Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.\n- Some Google Meet users are having issues joining the meetings.\n- Some Google Drive users are experiencing elevated latencies and error rates\n- Some Google Docs users are experiencing issues where they see \"Name loading\" in comments.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-06-21T16:19:13+00:00","modified":"2024-06-21T16:19:13+00:00","when":"2024-06-21T16:19:13+00:00","text":"The full incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},{"created":"2024-06-18T17:56:09+00:00","modified":"2024-06-18T17:56:09+00:00","when":"2024-06-18T17:56:09+00:00","text":"A mini incident report has been posted on the Google\nWorkspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},{"created":"2024-06-17T22:48:33+00:00","modified":"2024-06-17T22:48:33+00:00","when":"2024-06-17T22:48:33+00:00","text":"The issue with Google Meet is mitigated. Please find the details in the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},{"created":"2024-06-17T21:36:21+00:00","modified":"2024-06-17T21:36:21+00:00","when":"2024-06-17T21:36:21+00:00","text":"We are aware of an ongoing issue. Our engineers are working towards mitigation.\nPlease find more updates in the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"SERVICE_DISRUPTION"},{"created":"2024-06-17T20:57:30+00:00","modified":"2024-06-17T20:57:30+00:00","when":"2024-06-17T20:57:30+00:00","text":"We are aware of an ongoing issue. Our engineers are working towards mitigation.\nPlease find more updates in the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"SERVICE_DISRUPTION"},{"created":"2024-06-17T20:05:49+00:00","modified":"2024-06-17T20:05:49+00:00","when":"2024-06-17T20:05:49+00:00","text":"**Summary:**\nMultiple Workspace products are experiencing elevated error rates and latency\n**Description:**\nWe are experiencing an issue with Google Chat, Google Drive, Google Docs, Google Calendar, Google Meet beginning at Monday, 2024-06-17 09:00 US/Pacific.\nOur engineering team has identified the root cause of the issue and are currently working on a mitigation.\nWe will provide an update by Monday, 2024-06-17 14:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\n- Google Chat users may experience high latency, unable to load chat, open chat rooms, or send messages.\n- Some Google Meet users are having issues joining the meetings.\n- Some Google Drive users are experiencing elevated latencies and error rates\n- Some Google Docs users are experiencing issues where they see \"Name loading\" in comments.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-06-21T16:19:13+00:00","modified":"2024-06-21T16:19:13+00:00","when":"2024-06-21T16:19:13+00:00","text":"The full incident report has been posted on the Google Workspace Status Dashboard at https://www.google.com/appsstatus/dashboard/incidents/B7FqFhqQ4F5AbfNpuGbT","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"sUH4BQXzYXma7NiS94Hi","service_name":"Google Meet","affected_products":[{"title":"Google Meet","id":"sUH4BQXzYXma7NiS94Hi"}],"uri":"incidents/eo6f2ck36qQ2jHjjZmpc"},{"id":"QNFwydEugqSm2e4UScci","number":"1981048600274641635","begin":"2024-06-11T13:04:00+00:00","created":"2024-06-11T15:18:29+00:00","end":"2024-06-11T13:41:00+00:00","modified":"2024-06-13T18:34:01+00:00","external_desc":"**Summary:**\nGlobal: Elevated App Script Failures\n**Description:**\nWe experienced an issue with Apps Script beginning at Tuesday, 2024-06-11 06:04 to 06:41 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Tuesday, 2024-06-11 08:40 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Diagnosis:**\nCustomers impacted by this issue may have observed 500 errors and were unable to load / run App Scripts.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-06-13T18:34:01+00:00","modified":"2024-06-13T18:34:01+00:00","when":"2024-06-13T18:34:01+00:00","text":"# Incident Report\n## Summary\nOn Tuesday, 11 June 2024, Apps Script experienced increased error rates and users may have experienced intermittent issues in script creation and execution for a duration of 37 minutes. In addition to this, any of the Workspace services that internally use Apps Script may have encountered issues during the period of impact. We sincerely apologize to all Apps Script users. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nOne of the scripts received an unexpectedly high number of the queries per second (QPS) which resulted in hotspotting for our database service (a database hotspot is created because a single server receives most or all of the writes, which overloads that one server). This led to deadline exceeded errors, thereby increasing the error rate in our remote procedure calls (RPCs) and leading to the script creation and execution issues.\n## Remediation and Prevention\nGoogle engineers were alerted to the outage by the internal monitoring systems that sent multiple alerts starting 06:25 US/Pacific (approximately 15 mins after issue started) and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers first quickly enforced limits on read actions for the affected script and then blocked the script which led to returning database hotspotting and error rate in our RPCs back to their original values.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n- Enable caching on read replicas for App Info to reduce the number of requests being sent to our database service.\n- Adding limits and alerts for all RPCs to ensure a limit on the QPS that one add on can send.\n- Reduce the number of read replicas to avoid loading App Info on multiple replicas.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\n## Detailed Description of Impact\nOn Tuesday, 11 June 2024 from 06:04 US/Pacific to 06:41 US/Pacific Apps Script experienced increased error rates (error rate in our RPCs increased to ~20% whilst the expected error rate is ~1.3%.\n### Apps Script\nAffected users experienced intermittent issues in script creation and execution for a duration of their app script applications receiving DEADLINE_EXCEEDED error status.","status":"AVAILABLE"},{"created":"2024-06-11T17:48:02+00:00","modified":"2024-06-11T17:48:02+00:00","when":"2024-06-11T17:48:02+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 11 June 2024 06:04\n**Incident End:** 11 June 2024 06:41\n**Duration:** 37 minutes\n**Affected Services and Features:**\nApps Script\n**Regions/Zones:** Global\n**Description:**\nApps Script experienced increased error rates for a duration of 37 minutes. From preliminary analysis, the root cause of the issue was due to an unexpected increase in traffic related to a third-party add-on that resulted in resource contention.\nEngineers temporarily blocked the third-party add-on that was the cause of the increased traffic to mitigate the issue.\n**Customer Impact:**\nCustomers may have received errors when attempting to load or run Apps Scripts.","status":"AVAILABLE"},{"created":"2024-06-11T15:20:54+00:00","modified":"2024-06-11T15:20:54+00:00","when":"2024-06-11T15:20:54+00:00","text":"The problem with Apps Script has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.","status":"AVAILABLE"},{"created":"2024-06-11T15:18:29+00:00","modified":"2024-06-11T15:18:29+00:00","when":"2024-06-11T15:18:29+00:00","text":"**Summary:**\nGlobal: Elevated App Script Failures\n**Description:**\nWe experienced an issue with Apps Script beginning at Tuesday, 2024-06-11 06:04 to 06:41 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Tuesday, 2024-06-11 08:40 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Diagnosis:**\nCustomers impacted by this issue may have observed 500 errors and were unable to load / run App Scripts.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-06-13T18:34:01+00:00","modified":"2024-06-13T18:34:01+00:00","when":"2024-06-13T18:34:01+00:00","text":"# Incident Report\n## Summary\nOn Tuesday, 11 June 2024, Apps Script experienced increased error rates and users may have experienced intermittent issues in script creation and execution for a duration of 37 minutes. In addition to this, any of the Workspace services that internally use Apps Script may have encountered issues during the period of impact. We sincerely apologize to all Apps Script users. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nOne of the scripts received an unexpectedly high number of the queries per second (QPS) which resulted in hotspotting for our database service (a database hotspot is created because a single server receives most or all of the writes, which overloads that one server). This led to deadline exceeded errors, thereby increasing the error rate in our remote procedure calls (RPCs) and leading to the script creation and execution issues.\n## Remediation and Prevention\nGoogle engineers were alerted to the outage by the internal monitoring systems that sent multiple alerts starting 06:25 US/Pacific (approximately 15 mins after issue started) and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers first quickly enforced limits on read actions for the affected script and then blocked the script which led to returning database hotspotting and error rate in our RPCs back to their original values.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n- Enable caching on read replicas for App Info to reduce the number of requests being sent to our database service.\n- Adding limits and alerts for all RPCs to ensure a limit on the QPS that one add on can send.\n- Reduce the number of read replicas to avoid loading App Info on multiple replicas.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\n## Detailed Description of Impact\nOn Tuesday, 11 June 2024 from 06:04 US/Pacific to 06:41 US/Pacific Apps Script experienced increased error rates (error rate in our RPCs increased to ~20% whilst the expected error rate is ~1.3%.\n### Apps Script\nAffected users experienced intermittent issues in script creation and execution for a duration of their app script applications receiving DEADLINE_EXCEEDED error status.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"tjQKAokSTcX1h4huHNF2","service_name":"Apps Script","affected_products":[{"title":"Apps Script","id":"tjQKAokSTcX1h4huHNF2"}],"uri":"incidents/QNFwydEugqSm2e4UScci"},{"id":"uKjV4s9GxApbBd95AwW4","number":"12881546115193758986","begin":"2024-06-03T18:36:00+00:00","created":"2024-06-03T20:49:05+00:00","end":"2024-06-04T23:00:00+00:00","modified":"2024-06-05T18:00:38+00:00","external_desc":"**Summary:**\nChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.\n**Description:**\nMitigation work is currently underway by our engineering team.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Monday, 2024-06-03 17:00 US/Pacific.\n**Diagnosis:**\nAffected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.\n**Workaround:**\nNone at this time...","updates":[{"created":"2024-06-05T18:00:38+00:00","modified":"2024-06-05T18:00:38+00:00","when":"2024-06-05T18:00:38+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 3 June 2024 11:36\n**Incident End:** 4 June 2024 16:00\n**Duration:** 1 Day, 4 Hours and 24 Minutes\n**Affected Services and Features:**\nChromeOS\n**Regions/Zones:** Global\n**Description:**\nA subset of Google ChromeOS users experienced persistent sign-in failures when attempting to access their locked devices on ChromeOS version 125 using ephemeral mode for the period of 1 Day, 4 Hours and 24 Minutes.\nBased on the preliminary analysis the root cause of the sign-in failure was a conflict between a recent local password feature release and the ephemeral mode authentication process on ChromeOS. This conflict inadvertently terminated the authentication session prematurely, preventing users from logging into their devices.\nTo mitigate the issue and to restore normal functionality, Google engineers manually disabled the problematic feature flag. Subsequent verification confirmed successful login attempts for the affected users.\n**Customer Impact:**\nThe impacted users of ChromeOS version M125 with the \"DeviceEphemeralUsersEnabled\" device policy activated would have experienced persistent login failures when attempting to access their locked devices. This issue was specifically observed when utilizing ephemeral mode and prevented users from successfully logging back into their ChromeOS devices.\n**Additional details:**\nThe issue has been mitigated as of 4 June 2024 at 16:00. Google strongly recommends that all impacted users upgrade to ChromeOS version 125.0.6422.140 for enhanced stability.","status":"AVAILABLE"},{"created":"2024-06-05T00:19:12+00:00","modified":"2024-06-05T00:32:50+00:00","when":"2024-06-05T00:19:12+00:00","text":"**Summary**\nChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.\n**Description**\nThe issue with ChromeOS has been resolved for all affected users as of Tuesday, 2024-06-04 16:00 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-06-04T23:29:06+00:00","modified":"2024-06-04T23:29:06+00:00","when":"2024-06-04T23:29:06+00:00","text":"**SUMMARY**\nChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.\n**DESCRIPTION**\nOur engineering team is continuing to work towards mitigation of this issue.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Tuesday, 2024-06-04 19:00 US/Pacific.\n**CUSTOMER SYMPTOMS**\nAffected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.\n**WORKAROUND**\nChromeOS customers to use\nhttps://chromeenterprise.google/intl/en_uk/policies/#AllowScreenLock\nand\nhttps://chromeenterprise.google/intl/en_uk/policies/#LidCloseAction\npolicies to prevent occasional screen locking.","status":"SERVICE_INFORMATION"},{"created":"2024-06-04T19:32:34+00:00","modified":"2024-06-04T19:32:34+00:00","when":"2024-06-04T19:32:34+00:00","text":"**Summary:**\nChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.\n**Description:**\nOur engineering team is continuing to work towards mitigation of this issue.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Tuesday, 2024-06-04 17:00 US/Pacific.\n**Diagnosis:**\nAffected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.\n**Workaround:**\nChromeOS customers to use\nhttps://chromeenterprise.google/intl/en_uk/policies/#AllowScreenLock\nand\nhttps://chromeenterprise.google/intl/en_uk/policies/#LidCloseAction\npolicies to prevent occasional screen locking.","status":"SERVICE_INFORMATION"},{"created":"2024-06-04T15:48:14+00:00","modified":"2024-06-04T15:48:14+00:00","when":"2024-06-04T15:48:14+00:00","text":"**Summary:**\nChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.\n**Description:**\nOur engineering team is continuing to work towards mitigation of this issue.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Tuesday, 2024-06-04 13:00 US/Pacific.\n**Diagnosis:**\nAffected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.\n**Workaround:**\nChromeOS customers to use\nhttps://chromeenterprise.google/intl/en_uk/policies/#AllowScreenLock\nand\nhttps://chromeenterprise.google/intl/en_uk/policies/#LidCloseAction\npolicies to prevent occasional screen locking.","status":"SERVICE_INFORMATION"},{"created":"2024-06-04T10:33:55+00:00","modified":"2024-06-04T10:34:12+00:00","when":"2024-06-04T10:33:55+00:00","text":"**Summary:**\nChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.\n**Description:**\nMitigation work is currently underway by our engineering team.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Tuesday, 2024-06-04 09:00 US/Pacific.\n**Diagnosis:**\nAffected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.\n**Workaround:**\nChromeOS customers to use\nhttps://chromeenterprise.google/intl/en_uk/policies/#AllowScreenLock\nand\nhttps://chromeenterprise.google/intl/en_uk/policies/#LidCloseAction\npolicies to prevent occasional screen locking.","status":"SERVICE_INFORMATION"},{"created":"2024-06-04T01:49:22+00:00","modified":"2024-06-04T01:49:22+00:00","when":"2024-06-04T01:49:22+00:00","text":"**Summary:**\nChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.\n**Description:**\nMitigation work is currently underway by our engineering team.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Tuesday, 2024-06-04 09:00 US/Pacific.\n**Diagnosis:**\nAffected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.\n**Workaround:**\nNone at this time...","status":"SERVICE_INFORMATION"},{"created":"2024-06-03T23:55:00+00:00","modified":"2024-06-03T23:55:00+00:00","when":"2024-06-03T23:55:00+00:00","text":"**Summary:**\nChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.\n**Description:**\nMitigation work is currently underway by our engineering team.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Monday, 2024-06-03 19:00 US/Pacific.\n**Diagnosis:**\nAffected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.\n**Workaround:**\nNone at this time...","status":"SERVICE_INFORMATION"},{"created":"2024-06-03T20:49:07+00:00","modified":"2024-06-03T20:49:07+00:00","when":"2024-06-03T20:49:07+00:00","text":"**Summary:**\nChromeOS: Unable to log in from lock screen when ephemeral mode is enabled on managed devices on M-125.\n**Description:**\nMitigation work is currently underway by our engineering team.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Monday, 2024-06-03 17:00 US/Pacific.\n**Diagnosis:**\nAffected users may be unable to log back in from the lock screen on their ChromeOS device. This issue is affecting ChromeOS version M125 when the device policy DeviceEphemeralUsersEnabled is turned on.\n**Workaround:**\nNone at this time...","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-06-05T18:00:38+00:00","modified":"2024-06-05T18:00:38+00:00","when":"2024-06-05T18:00:38+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 3 June 2024 11:36\n**Incident End:** 4 June 2024 16:00\n**Duration:** 1 Day, 4 Hours and 24 Minutes\n**Affected Services and Features:**\nChromeOS\n**Regions/Zones:** Global\n**Description:**\nA subset of Google ChromeOS users experienced persistent sign-in failures when attempting to access their locked devices on ChromeOS version 125 using ephemeral mode for the period of 1 Day, 4 Hours and 24 Minutes.\nBased on the preliminary analysis the root cause of the sign-in failure was a conflict between a recent local password feature release and the ephemeral mode authentication process on ChromeOS. This conflict inadvertently terminated the authentication session prematurely, preventing users from logging into their devices.\nTo mitigate the issue and to restore normal functionality, Google engineers manually disabled the problematic feature flag. Subsequent verification confirmed successful login attempts for the affected users.\n**Customer Impact:**\nThe impacted users of ChromeOS version M125 with the \"DeviceEphemeralUsersEnabled\" device policy activated would have experienced persistent login failures when attempting to access their locked devices. This issue was specifically observed when utilizing ephemeral mode and prevented users from successfully logging back into their ChromeOS devices.\n**Additional details:**\nThe issue has been mitigated as of 4 June 2024 at 16:00. Google strongly recommends that all impacted users upgrade to ChromeOS version 125.0.6422.140 for enhanced stability.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"RjKbsvuGeE7M1pvBHmvX","service_name":"ChromeOS","affected_products":[{"title":"ChromeOS","id":"RjKbsvuGeE7M1pvBHmvX"}],"uri":"incidents/uKjV4s9GxApbBd95AwW4"},{"id":"u3EwVWVQTWrE8e6gs61F","number":"14826990135981321213","begin":"2024-05-30T14:23:00+00:00","created":"2024-05-30T15:54:15+00:00","end":"2024-05-30T17:11:00+00:00","modified":"2024-05-30T18:01:42+00:00","external_desc":"We're investigating reports of an issue with Google Drive. We will provide more information shortly.","updates":[{"created":"2024-05-30T18:01:18+00:00","modified":"2024-05-30T18:01:18+00:00","when":"2024-05-30T18:01:18+00:00","text":"We have determined that this is an isolated issue that will be corrected as part of our ongoing maintenance activities. The incident with Google Drive was initially triggered by our internal monitoring systems.\nUpon further investigation, our engineering teams has confirmed that the scope is very limited and there was no impact to the users of Google Drive.\nIf you have questions or feel that you may be impacted, please open a case with the Support Team and we will work with you until the issue is resolved.","status":"AVAILABLE"},{"created":"2024-05-30T17:15:52+00:00","modified":"2024-05-30T17:15:52+00:00","when":"2024-05-30T17:15:52+00:00","text":"The problem with Google Drive has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.","status":"AVAILABLE"},{"created":"2024-05-30T17:04:14+00:00","modified":"2024-05-30T18:01:32+00:00","when":"2024-05-30T17:04:14+00:00","text":"We expect to resolve the problem affecting a majority of users of Google Drive at May 30, 2024, 7:30 PM UTC. Please note that this time frame is an estimate and may change.","status":"SERVICE_INFORMATION"},{"created":"2024-05-30T15:54:16+00:00","modified":"2024-05-30T18:01:42+00:00","when":"2024-05-30T15:54:16+00:00","text":"We're investigating reports of an issue with Google Drive. We will provide more information shortly. Customer might see 5XX errors while using Google Drive.","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-05-30T18:01:18+00:00","modified":"2024-05-30T18:01:18+00:00","when":"2024-05-30T18:01:18+00:00","text":"We have determined that this is an isolated issue that will be corrected as part of our ongoing maintenance activities. The incident with Google Drive was initially triggered by our internal monitoring systems.\nUpon further investigation, our engineering teams has confirmed that the scope is very limited and there was no impact to the users of Google Drive.\nIf you have questions or feel that you may be impacted, please open a case with the Support Team and we will work with you until the issue is resolved.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"VHNA7p3Z5p3iakj5sA8V","service_name":"Google Drive","affected_products":[{"title":"Google Drive","id":"VHNA7p3Z5p3iakj5sA8V"}],"uri":"incidents/u3EwVWVQTWrE8e6gs61F"},{"id":"dCH2bReebz1YKwKtg2wR","number":"12003636871983100994","begin":"2024-05-24T23:20:00+00:00","created":"2024-05-25T03:01:57+00:00","end":"2024-05-25T03:20:00+00:00","modified":"2024-05-25T04:17:34+00:00","external_desc":"**For more service information please follow**\nhttps://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG","updates":[{"created":"2024-05-25T04:17:34+00:00","modified":"2024-05-25T04:17:34+00:00","when":"2024-05-25T04:17:34+00:00","text":"**For more service information please follow**\nhttps://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG","status":"AVAILABLE"},{"created":"2024-05-25T03:01:58+00:00","modified":"2024-05-25T03:01:58+00:00","when":"2024-05-25T03:01:58+00:00","text":"**For more service information please follow**\nhttps://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-05-25T04:17:34+00:00","modified":"2024-05-25T04:17:34+00:00","when":"2024-05-25T04:17:34+00:00","text":"**For more service information please follow**\nhttps://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"GsNjeYHsogeRgTiC5coU","service_name":"Google Sheets","affected_products":[{"title":"Google Sheets","id":"GsNjeYHsogeRgTiC5coU"}],"uri":"incidents/dCH2bReebz1YKwKtg2wR"},{"id":"imGVfAXCm8TtxgsbUXZX","number":"11784003295713227186","begin":"2024-05-24T23:20:00+00:00","created":"2024-05-25T03:02:39+00:00","end":"2024-05-25T03:20:00+00:00","modified":"2024-05-25T04:18:08+00:00","external_desc":"**For more service information please follow**\nhttps://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG","updates":[{"created":"2024-05-25T04:18:08+00:00","modified":"2024-05-25T04:18:08+00:00","when":"2024-05-25T04:18:08+00:00","text":"**For more service information please follow**\nhttps://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG","status":"AVAILABLE"},{"created":"2024-05-25T03:02:39+00:00","modified":"2024-05-25T03:02:39+00:00","when":"2024-05-25T03:02:39+00:00","text":"**For more service information please follow**\nhttps://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-05-25T04:18:08+00:00","modified":"2024-05-25T04:18:08+00:00","when":"2024-05-25T04:18:08+00:00","text":"**For more service information please follow**\nhttps://www.google.com/appsstatus/dashboard/incidents/qPG3UXdDaYVhcA9DT8zG","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"wNHuVFtZEWU5Mmj2eRCK","service_name":"Google Docs","affected_products":[{"title":"Google Docs","id":"wNHuVFtZEWU5Mmj2eRCK"}],"uri":"incidents/imGVfAXCm8TtxgsbUXZX"},{"id":"qPG3UXdDaYVhcA9DT8zG","number":"1182938916713078705","begin":"2024-05-24T23:20:00+00:00","created":"2024-05-25T01:27:08+00:00","end":"2024-05-25T03:41:00+00:00","modified":"2024-05-31T00:13:00+00:00","external_desc":"**Summary:**\nGoogle Drive users are unable to load Apps Script UI.\n**Description:**\nWe are experiencing an issue with Google Drive beginning on Friday, 2024-05-24 16:20 US/Pacific.\nOur engineering team has identified a mitigation strategy and is currently rolling it out for all impacted users.\nWe will provide an update by Friday, 2024-05-24 21:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\nCustomers may not be able to load Drive UI, they would encounter error messages while trying to do so.\n.\nThis issue may also impact users trying to access Apps Script UI.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-05-31T00:10:36+00:00","modified":"2024-05-31T00:10:36+00:00","when":"2024-05-31T00:10:36+00:00","text":"# Incident Report\n## Summary\nOn 24 May 2024 at 16:20, Apps Script and Drive Admin Console User Interface (UI) experienced failures in loading the UI. A subset of users experienced elevated errors while accessing the Apps Script editor UI or Drive Admin Console UI through multiple Google Workspace products including Google Docs, Google Drive, Google Sheets for a period of 4 hours, 21 minutes.\n## Root Cause\nIn ongoing efforts to ensure very strict security controls for Google Drive, our engineers performed changes to remove unused system-level access to Google Drive; i.e., stale Access Control List (ACLs) groups that are no longer in use.\nUpon performing the change to remove the unused ACLs groups for Google Drive API, this triggered an inadvertent removal of access groups that are still in use. This deletion of the access groups caused a failure in loading the UI.\nGoogle engineers mitigated the issue quickly by rolling back the change.\n## Remediation and Prevention\nGoogle engineers were alerted to the outage via internal alerts and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers rolled back the changes and all the access removed this change was restored. While the rollback was still in progress Google engineers were able to partially mitigate the issue with a configuration change in a security policy.\nThe following changes will be made to ensure that this type of issue does not happen again in the future: - Augment access control safety checks to prevent the inadvertent removal of in-use access, even in complex access inheritance scenarios. - Review and redesign Drives access groups to remove nested ACLs (and prevent adding new nested ACLs). This will reduce the complexity of our access control graph and enable easier detection of potential problems.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\n## Detailed Description of Impact\nOn 24 May 2024, from 16:20 to 20:41, Apps Script and Drive Admin Console User Interface (UI) experienced failures in loading the UI. As a result, a subset of users of multiple Google Workspace products like Google Docs, Google Drive, Google Sheets experienced elevated errors while accessing the Apps Script editor UI or Drive Admin Console UI through the products.\n**Google Drive**\nA small subset of Google Drive users experienced failure in accessing Drive Admin Console. As a result, the impacted users were unable to manage their shared drives on 24 May 2024 between 16:20 and 20:02.\n**Google Docs, Sheets(Apps script)**\nAppsScript UI was inaccessible for editing but AppsScript execution was unaffected starting at 16:12 to 20:41.","status":"AVAILABLE"},{"created":"2024-05-27T17:58:49+00:00","modified":"2024-05-27T17:58:49+00:00","when":"2024-05-27T17:58:49+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 24 May, 2024 16:20\n**Incident End:** 24 May, 2024 20:41\n**Duration:** 4 hours, 21 minutes\n**Affected Services and Features:** Apps Script usage from Google Drive, Google Docs, Google Sheets\n**Regions/Zones:** Global\n**Description:**\nThe user interface for Apps Script was failing to load for a duration of 4 hours, 21 minutes.\nBased on the preliminary analysis the root cause of the issue is a configuration change. The change to the access control list that inadvertently removed access for some processes, has been updated to normalcy.\nGoogle will complete a full incident report in the following days that will provide more details on the root cause.\n**Customer Impact:**\nThe impacted users were unable to develop, manage and execute their script using the Apps Script editor UI. Users would have encountered similar errors while trying to open Apps Script from any of the workspace products as well. However, there was no impact on other executions that did not originate from UI like executions from published scripts, triggers etc.","status":"AVAILABLE"},{"created":"2024-05-25T04:15:39+00:00","modified":"2024-05-25T04:15:39+00:00","when":"2024-05-25T04:15:39+00:00","text":"**Summary:**\nMultiple Google Workspace products experienced failures in loading Apps Script editor UI.\n**Description:**\nThe issues with Apps Script editor UI for multiple Google Workspace products have been resolved for all affected users as of Friday, 2024-05-24 20:41 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-05-25T03:35:37+00:00","modified":"2024-05-25T03:35:37+00:00","when":"2024-05-25T03:35:37+00:00","text":"**Summary:**\nMultiple Google Workspace products are experiencing failures in loading Apps Script editor UI.\n**Description:**\nWe are experiencing issues with Apps Script editor UI for multiple Google Workspace products beginning on Friday, 2024-05-24 16:20 US/Pacific.\nThe users trying to access the Apps Script UI may experience failures in loading the Apps Script UI.\nOur engineering team has rolled out the fix for all the impacted users related to Apps Script UI issue and we started observing the service recovery.\nWe will provide an update by Friday, 2024-05-24 22:30 US/Pacific with current details.\n**The Google Drive UI service degradation has been resolved as of Friday, 2024-05-24 20:02 US/Pacific.**\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\nMultiple Google workspace users would experience errors or failures while loading the Apps Script UI. As a result, the impacted users are unable to develop, manage and execute their script using the Apps Script editor UI. **However, there is no impact on other execution of deployed scripts.**\n**Workaround:**\nNone at this time.","status":"SERVICE_INFORMATION"},{"created":"2024-05-25T02:58:33+00:00","modified":"2024-05-25T02:59:01+00:00","when":"2024-05-25T02:58:33+00:00","text":"**Summary:**\nMultiple Google Workspace products are experiencing failures in loading Apps Script editor UI.\n**Description:**\nWe are experiencing issues with Apps Script editor UI for multiple Google Workspace products beginning on Friday, 2024-05-24 16:20 US/Pacific.\nThe users trying to access the Apps Script UI may experience failures in loading the Apps Script UI.\nOur engineering team has identified a mitigation strategy and is currently rolling it out for all impacted users.\nWe will provide an update by Friday, 2024-05-24 22:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\nMultiple Google workspace users would experience errors or failures while loading the Apps Script UI. As a result, the impacted users are unable to access or manage their script within the Apps Script. **However, there is no impact on the execution of scripts.**\n**Google Drive** - a small subset of users may experience service degradation with Drive UI.\n**Workaround:**\nNone at this time.","status":"SERVICE_INFORMATION"},{"created":"2024-05-25T01:27:10+00:00","modified":"2024-05-25T01:38:05+00:00","when":"2024-05-25T01:27:10+00:00","text":"**Summary:**\nGoogle Drive users are unable to load Apps Script UI.\n**Description:**\nWe are experiencing an issue with Google Drive beginning on Friday, 2024-05-24 16:20 US/Pacific.\nOur engineering team has identified a mitigation strategy and is currently rolling it out for all impacted users.\nWe will provide an update by Friday, 2024-05-24 21:00 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n**Customer Symptoms:**\nCustomers may not be able to load Drive UI, they would encounter error messages while trying to do so.\n.\nThis issue may also impact users trying to access Apps Script UI.\n**Workaround:**\nNone at this time.","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-05-31T00:10:36+00:00","modified":"2024-05-31T00:10:36+00:00","when":"2024-05-31T00:10:36+00:00","text":"# Incident Report\n## Summary\nOn 24 May 2024 at 16:20, Apps Script and Drive Admin Console User Interface (UI) experienced failures in loading the UI. A subset of users experienced elevated errors while accessing the Apps Script editor UI or Drive Admin Console UI through multiple Google Workspace products including Google Docs, Google Drive, Google Sheets for a period of 4 hours, 21 minutes.\n## Root Cause\nIn ongoing efforts to ensure very strict security controls for Google Drive, our engineers performed changes to remove unused system-level access to Google Drive; i.e., stale Access Control List (ACLs) groups that are no longer in use.\nUpon performing the change to remove the unused ACLs groups for Google Drive API, this triggered an inadvertent removal of access groups that are still in use. This deletion of the access groups caused a failure in loading the UI.\nGoogle engineers mitigated the issue quickly by rolling back the change.\n## Remediation and Prevention\nGoogle engineers were alerted to the outage via internal alerts and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers rolled back the changes and all the access removed this change was restored. While the rollback was still in progress Google engineers were able to partially mitigate the issue with a configuration change in a security policy.\nThe following changes will be made to ensure that this type of issue does not happen again in the future: - Augment access control safety checks to prevent the inadvertent removal of in-use access, even in complex access inheritance scenarios. - Review and redesign Drives access groups to remove nested ACLs (and prevent adding new nested ACLs). This will reduce the complexity of our access control graph and enable easier detection of potential problems.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\n## Detailed Description of Impact\nOn 24 May 2024, from 16:20 to 20:41, Apps Script and Drive Admin Console User Interface (UI) experienced failures in loading the UI. As a result, a subset of users of multiple Google Workspace products like Google Docs, Google Drive, Google Sheets experienced elevated errors while accessing the Apps Script editor UI or Drive Admin Console UI through the products.\n**Google Drive**\nA small subset of Google Drive users experienced failure in accessing Drive Admin Console. As a result, the impacted users were unable to manage their shared drives on 24 May 2024 between 16:20 and 20:02.\n**Google Docs, Sheets(Apps script)**\nAppsScript UI was inaccessible for editing but AppsScript execution was unaffected starting at 16:12 to 20:41.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"VHNA7p3Z5p3iakj5sA8V","service_name":"Google Drive","affected_products":[{"title":"Google Drive","id":"VHNA7p3Z5p3iakj5sA8V"}],"uri":"incidents/qPG3UXdDaYVhcA9DT8zG"},{"id":"iWYUDm5oWtEt2tub3Lht","number":"6348465522355943791","begin":"2024-05-02T15:53:00+00:00","created":"2024-05-02T18:44:02+00:00","end":"2024-05-02T19:02:00+00:00","modified":"2024-05-06T18:56:05+00:00","external_desc":"We're investigating reports of an issue with Google Meet. We will provide more information shortly.","updates":[{"created":"2024-05-03T17:32:58+00:00","modified":"2024-05-06T18:56:05+00:00","when":"2024-05-03T17:32:58+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213\n(All Times US/Pacific)\n**Incident Start:** 2 May 2024 08:53\n**Incident End:** 2 May 2024 12:02\n**Duration:** 3 hours, 9 minutes\n**Affected Services and Features:**\nGoogle Meet Broadcast via YouTube Live\n**Regions/Zones:** Global\n**Description:**\nGoogle Meet experienced streaming playback issues for Meet Broadcast via YouTube Live for 3 hours, 9 minutes. From preliminary analysis, the root cause of the issue was a recent rollout for YouTube Live that inadvertently impacted Meet Broadcast.\nEngineers rolled back the change to mitigate the issue. For active streams that were affected by this issue, the host would need to restart the stream for the mitigation to take effect.\n**Customer Impact:**\n* Users attempting to join a live stream that was using Meet Broadcast via YouTube Live would have been stuck at the message “Waiting for streaming to begin. Please stand by.”\n* Live streams utilizing Meet Live were not affected.","status":"AVAILABLE"},{"created":"2024-05-02T19:07:38+00:00","modified":"2024-05-02T19:07:38+00:00","when":"2024-05-02T19:07:38+00:00","text":"The issue with Google Meet has been resolved for all affected users.\nFor new streams, there is no problem now. For broken streams, the stream host can stop and restart the stream to mitigate the issue.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-05-02T18:47:47+00:00","modified":"2024-05-02T18:47:47+00:00","when":"2024-05-02T18:47:47+00:00","text":"Google Meet customers using Broadcast via YouTube Live are experiencing playback issues beginning at Thursday, 2024-05-02 08:53 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe apologize to all who are affected by the disruption.","status":"SERVICE_DISRUPTION"},{"created":"2024-05-02T18:44:03+00:00","modified":"2024-05-02T18:44:03+00:00","when":"2024-05-02T18:44:03+00:00","text":"We're investigating reports of an issue with Google Meet. We will provide more information shortly.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-05-03T17:32:58+00:00","modified":"2024-05-06T18:56:05+00:00","when":"2024-05-03T17:32:58+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213\n(All Times US/Pacific)\n**Incident Start:** 2 May 2024 08:53\n**Incident End:** 2 May 2024 12:02\n**Duration:** 3 hours, 9 minutes\n**Affected Services and Features:**\nGoogle Meet Broadcast via YouTube Live\n**Regions/Zones:** Global\n**Description:**\nGoogle Meet experienced streaming playback issues for Meet Broadcast via YouTube Live for 3 hours, 9 minutes. From preliminary analysis, the root cause of the issue was a recent rollout for YouTube Live that inadvertently impacted Meet Broadcast.\nEngineers rolled back the change to mitigate the issue. For active streams that were affected by this issue, the host would need to restart the stream for the mitigation to take effect.\n**Customer Impact:**\n* Users attempting to join a live stream that was using Meet Broadcast via YouTube Live would have been stuck at the message “Waiting for streaming to begin. Please stand by.”\n* Live streams utilizing Meet Live were not affected.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"sUH4BQXzYXma7NiS94Hi","service_name":"Google Meet","affected_products":[{"title":"Google Meet","id":"sUH4BQXzYXma7NiS94Hi"}],"uri":"incidents/iWYUDm5oWtEt2tub3Lht"},{"id":"4g4kuywfXf5KTJ4TV9hT","number":"8975044941019072442","begin":"2024-04-30T00:00:00+00:00","created":"2024-05-01T19:02:13+00:00","end":"2024-05-01T22:55:00+00:00","modified":"2024-05-01T23:42:15+00:00","external_desc":"We are aware of an issue affecting Gmail users sending emails to some email addresses. We are currently investigating.","updates":[{"created":"2024-05-01T23:41:56+00:00","modified":"2024-05-01T23:41:56+00:00","when":"2024-05-01T23:41:56+00:00","text":"**Summary:**\nGmail customers are currently unable to send unsigned messages to Yahoo.\n**Description:**\nThe issue has been resolved for all affected users as of Wednesday, 2024-05-01 15:55 US/Pacific.\nThere was a problem external to Google which has now been resolved.\nWe thank you for your patience while we worked on resolving the issue.\n------------------------------------------------------------------------------------------------------------------","status":"AVAILABLE"},{"created":"2024-05-01T21:54:29+00:00","modified":"2024-05-01T21:54:29+00:00","when":"2024-05-01T21:54:29+00:00","text":"**Summary**\nGmail customers are currently unable to send unsigned messages to Yahoo.\n**Description:**\nA small subset of Gmail customers are experiencing difficulty sending messages to Yahoo.\nOur engineering team continues to work on mitigation.\nCustomer messages may be bounced with the below error:\n\"This mail has been blocked because the sender is unauthenticated. Yahoo requires all senders to authenticate with either SPF or DKIM.\"\nIf you're seeing this error message, please consider setting up SPF (https://support.google.com/a/answer/33786) or DKIM (https://support.google.com/a/answer/174124) for your sending domain(s).\nNote that Sender Policy Framework (SPF) and DomainKeys Identified Mail (DKIM) records may take some time to become fully propagated in DNS and therefore the issue may persist for a little while longer after setting these records up.\nWe will provide an update by Wednesday, 2024-05-01 20:30 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\n___________________________________________________________","status":"SERVICE_INFORMATION"},{"created":"2024-05-01T19:41:07+00:00","modified":"2024-05-01T19:41:07+00:00","when":"2024-05-01T19:41:07+00:00","text":"Gmail customers are currently unable to send unsigned messages to Yahoo.\nOur engineering team continues to investigate.\nCustomer messages sent to Yahoo/AOL recipients may be bounced with the error \"This mail has been blocked because the sender is unauthenticated. Yahoo requires all senders to authenticate with either SPF or DKIM.\"\nIf you're seeing this error message, please set up SPF (https://support.google.com/a/answer/33786) or DKIM (https://support.google.com/a/answer/174124) for your sending domain(s). For DKIM, you should set up a record for your own sending domain. You should not rely on the default DKIM key applied by Google (`*.gappssmtp.com`) to avoid this error.\nNote that SPF and DKIM records may take some time to become fully propagated in DNS and therefore the issue may persist for a little while longer after setting these records up.\nWe apologize to all who are affected by the disruption.","status":"SERVICE_INFORMATION"},{"created":"2024-05-01T19:02:13+00:00","modified":"2024-05-01T19:02:13+00:00","when":"2024-05-01T19:02:13+00:00","text":"We are aware of an issue affecting Gmail users sending emails to some email addresses. We are currently investigating.","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-05-01T23:41:56+00:00","modified":"2024-05-01T23:41:56+00:00","when":"2024-05-01T23:41:56+00:00","text":"**Summary:**\nGmail customers are currently unable to send unsigned messages to Yahoo.\n**Description:**\nThe issue has been resolved for all affected users as of Wednesday, 2024-05-01 15:55 US/Pacific.\nThere was a problem external to Google which has now been resolved.\nWe thank you for your patience while we worked on resolving the issue.\n------------------------------------------------------------------------------------------------------------------","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/4g4kuywfXf5KTJ4TV9hT"},{"id":"qdTmzP7VFU6CqfJzBfE9","number":"983917633607522325","begin":"2024-04-24T15:00:00+00:00","created":"2024-04-24T21:14:27+00:00","end":"2024-04-24T21:17:00+00:00","modified":"2024-04-24T21:20:34+00:00","external_desc":"Summary:\nWorkspace Super Admins trying to open unauthenticated support cases for account recovery help using toolbox are encountering 500 errors and unable to open support cases for account recovery.\nWe will provide an update by Wednesday, 2024-04--24 14:45 US/Pacific with current details.\nWorkaround:\nCustomers having issues opening the account recovery cases can use\nhttps://support.google.com/a/contact/recovery_form to contact support until the issue is resolved.","updates":[{"created":"2024-04-24T21:20:25+00:00","modified":"2024-04-24T21:20:25+00:00","when":"2024-04-24T21:20:25+00:00","text":"The issue with users creating unauthenticated support cases for account recovery help using toolbox has been resolved for all affected users as of Wednesday, 2024-04-24 14:17 US/Pacific.\nFrom our preliminary analysis, the root cause of the issue is a recent roll out. Our engineers rolled back to last stable version hence mitigating the issue.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-04-24T21:14:27+00:00","modified":"2024-04-24T21:16:50+00:00","when":"2024-04-24T21:14:27+00:00","text":"Summary:\nWorkspace Super Admins trying to open unauthenticated support cases for account recovery help using toolbox are encountering 500 errors and unable to open support cases for account recovery.\nWe will provide an update by Wednesday, 2024-04--24 14:45 US/Pacific with current details.\nWorkaround:\nCustomers having issues opening the account recovery cases can use\nhttps://support.google.com/a/contact/recovery_form to contact support until the issue is resolved.","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-04-24T21:20:25+00:00","modified":"2024-04-24T21:20:25+00:00","when":"2024-04-24T21:20:25+00:00","text":"The issue with users creating unauthenticated support cases for account recovery help using toolbox has been resolved for all affected users as of Wednesday, 2024-04-24 14:17 US/Pacific.\nFrom our preliminary analysis, the root cause of the issue is a recent roll out. Our engineers rolled back to last stable version hence mitigating the issue.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"dnz8HVUHoMMcW9wV91LY","service_name":"Google Workspace Support","affected_products":[{"title":"Google Workspace Support","id":"dnz8HVUHoMMcW9wV91LY"}],"uri":"incidents/qdTmzP7VFU6CqfJzBfE9"},{"id":"MPBaL9VxFxX3iRgkjPQt","number":"5584609343158321678","begin":"2024-04-22T11:10:00+00:00","created":"2024-04-23T00:52:47+00:00","end":"2024-04-23T16:50:00+00:00","modified":"2024-04-24T16:52:32+00:00","external_desc":"**Summary:**\nGoogle Classroom is experiencing an increased error rate in Brazil (for State of Paraná)\n**Description:**\nWe are experiencing an intermittent issue with Google Classroom.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Tuesday, 2024-04-23 09:00 US/Pacific with current details.\n**Customer Symptoms:**\nA subset of users in the state of Parana, Brazil would encounter intermittent error while accessing Google Classroom. However, the impact is only limited to the state of Parana, no impact to customers to the rest of the world.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-04-24T16:50:54+00:00","modified":"2024-04-24T16:50:54+00:00","when":"2024-04-24T16:50:54+00:00","text":"**Mini Incident Report**\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 22 April 2023 04:10\n**Incident End:** 23 April 2023 09:50\n**Duration:** 1 day, 5 hours, 40 mins\n**Affected Services and Features:** Classroom\n**Regions/Zones:** us-east1\n**Description:**\nGoogle Classroom experienced an increased error rate in Brazil (for the State of Paraná) due to hotspotting (overloading) in the product’s database for a duration of 1 day, 5 hours, 40 mins. From preliminary analysis, the root cause of hotspotting was a software bug, which we’ve fixed.\n**Customer Impact:**\nA subset of users in the state of Paraná, Brazil had encountered intermittent errors while accessing Google Classroom. To the best of our knowledge, no other customers were impacted.","status":"AVAILABLE"},{"created":"2024-04-23T19:48:04+00:00","modified":"2024-04-23T19:48:04+00:00","when":"2024-04-23T19:48:04+00:00","text":"The issue with Classroom has been resolved for all affected users as of Tuesday, 2024-04-23 11:02 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2024-04-23T15:39:53+00:00","modified":"2024-04-23T15:39:53+00:00","when":"2024-04-23T15:39:53+00:00","text":"**Summary:**\nGoogle Classroom is experiencing an increased error rate in Brazil (for State of Paraná)\n**Description:**\nMitigation work is currently underway by our engineering team.\nThe mitigation is expected to complete by Tuesday, 2024-04-23 12:30 US/Pacific.\nWe will provide more information by Tuesday, 2024-04-23 12:45 US/Pacific.\n**Customer Symptoms:**\nA subset of users in the state of Parana, Brazil would encounter intermittent error while accessing Google Classroom. However, the impact is only limited to the state of Parana, no impact to customers to the rest of the world.\n**Workaround:**\nNone at this time.","status":"SERVICE_INFORMATION"},{"created":"2024-04-23T00:52:47+00:00","modified":"2024-04-23T00:52:47+00:00","when":"2024-04-23T00:52:47+00:00","text":"**Summary:**\nGoogle Classroom is experiencing an increased error rate in Brazil (for State of Paraná)\n**Description:**\nWe are experiencing an intermittent issue with Google Classroom.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Tuesday, 2024-04-23 09:00 US/Pacific with current details.\n**Customer Symptoms:**\nA subset of users in the state of Parana, Brazil would encounter intermittent error while accessing Google Classroom. However, the impact is only limited to the state of Parana, no impact to customers to the rest of the world.\n**Workaround:**\nNone at this time.","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2024-04-24T16:50:54+00:00","modified":"2024-04-24T16:50:54+00:00","when":"2024-04-24T16:50:54+00:00","text":"**Mini Incident Report**\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 22 April 2023 04:10\n**Incident End:** 23 April 2023 09:50\n**Duration:** 1 day, 5 hours, 40 mins\n**Affected Services and Features:** Classroom\n**Regions/Zones:** us-east1\n**Description:**\nGoogle Classroom experienced an increased error rate in Brazil (for the State of Paraná) due to hotspotting (overloading) in the product’s database for a duration of 1 day, 5 hours, 40 mins. From preliminary analysis, the root cause of hotspotting was a software bug, which we’ve fixed.\n**Customer Impact:**\nA subset of users in the state of Paraná, Brazil had encountered intermittent errors while accessing Google Classroom. To the best of our knowledge, no other customers were impacted.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"3GoNVrVz48jeNZSyLJMV","service_name":"Classroom","affected_products":[{"title":"Classroom","id":"3GoNVrVz48jeNZSyLJMV"}],"uri":"incidents/MPBaL9VxFxX3iRgkjPQt"},{"id":"kyvpnGVXuGba1rAmcdaV","number":"3730274334862140734","begin":"2024-04-18T21:40:00+00:00","created":"2024-04-19T18:23:52+00:00","end":"2024-04-20T19:23:00+00:00","modified":"2024-04-30T20:49:53+00:00","external_desc":"We're investigating reports of an issue with Google Calendar. We will provide more information shortly.","updates":[{"created":"2024-04-30T20:49:53+00:00","modified":"2024-04-30T20:49:53+00:00","when":"2024-04-30T20:49:53+00:00","text":"An incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/D6dP4yHWc2m93ZhpLYKw","status":"AVAILABLE"},{"created":"2024-04-22T19:14:14+00:00","modified":"2024-04-22T19:14:14+00:00","when":"2024-04-22T19:14:14+00:00","text":"A mini incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E","status":"AVAILABLE"},{"created":"2024-04-20T20:08:02+00:00","modified":"2024-04-20T20:08:02+00:00","when":"2024-04-20T20:08:02+00:00","text":"The problem with Gmail, Google Calendar, Google Chat has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.","status":"AVAILABLE"},{"created":"2024-04-19T22:52:40+00:00","modified":"2024-04-19T22:52:40+00:00","when":"2024-04-19T22:52:40+00:00","text":"Further communication will be provided on the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E","status":"SERVICE_DISRUPTION"},{"created":"2024-04-19T21:06:14+00:00","modified":"2024-04-19T21:06:14+00:00","when":"2024-04-19T21:06:14+00:00","text":"Further communication will be provided on the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E","status":"SERVICE_DISRUPTION"},{"created":"2024-04-19T19:36:41+00:00","modified":"2024-04-19T19:36:41+00:00","when":"2024-04-19T19:36:41+00:00","text":"Our Engineering team has identified the root cause of the issue and rolled back a recent change.\nOur engineers are taking additional repair actions for the rollback to take effect for the impacted customers.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Friday, 2024-04-19 14:00 US/Pacific.","status":"SERVICE_DISRUPTION"},{"created":"2024-04-19T18:33:20+00:00","modified":"2024-04-19T18:33:20+00:00","when":"2024-04-19T18:33:20+00:00","text":"Engineering team potentially identified the root cause and working on mitigation.\nThe mitigation is expected to complete by Friday, 2024-04-19 12:15 US/Pacific.","status":"SERVICE_DISRUPTION"},{"created":"2024-04-19T18:23:52+00:00","modified":"2024-04-19T18:23:52+00:00","when":"2024-04-19T18:23:52+00:00","text":"We're investigating reports of an issue with Google Calendar. We will provide more information shortly. Google Calendar customers may experience \"Contact Sharing\" disabled.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-04-30T20:49:53+00:00","modified":"2024-04-30T20:49:53+00:00","when":"2024-04-30T20:49:53+00:00","text":"An incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/D6dP4yHWc2m93ZhpLYKw","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"n5hVexuq1PM9onY9qrmo","service_name":"Google Calendar","affected_products":[{"title":"Google Calendar","id":"n5hVexuq1PM9onY9qrmo"}],"uri":"incidents/kyvpnGVXuGba1rAmcdaV"},{"id":"D6dP4yHWc2m93ZhpLYKw","number":"656747209480464684","begin":"2024-04-18T21:40:00+00:00","created":"2024-04-19T18:22:43+00:00","end":"2024-04-20T19:23:00+00:00","modified":"2024-05-01T14:17:23+00:00","external_desc":"We're investigating reports of an issue with Google Chat. We will provide more information shortly.","updates":[{"created":"2024-04-30T20:40:16+00:00","modified":"2024-05-01T14:17:23+00:00","when":"2024-04-30T20:40:16+00:00","text":"# Incident Report\n## Summary\nOn 18 April 2024, Gmail, Google Calendar and Google Chat experienced issues with the ‘Contact Sharing’ feature for a duration of 1 day, 21 hours 43 minutes. To our Google Workspace customers whose businesses were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nAn internal code update unintentionally altered the default values of a setting that interacts with the current contact sharing system. This change resulted in a misconfiguration of directory settings, impacting the Contact Sharing feature within Google Workspace services like Gmail, Chat, and Calendar.\n## Remediation and Prevention\nWe were alerted to the outage via a support case on April 19 at 7:12 US/Pacific and promptly initiated an investigation. The investigation confirmed that the issue originated from a code change deployed on April 18 at 14:40 US/Pacific. Upon troubleshooting, we immediately halted and rolled back the code change, preventing a wider impact on Google Workspace services.\nWe then focused on repairing the issue for affected customers.\nWe apologize for the length and severity of this incident. Google is committed to preventing recurrence by implementing the following action items:\n1) Engineering team implemented release controls to prevent code changes that alter the default values from being deployed to the production environment.\n2) Engineering team to add test coverage and verify code rollouts like these so issues like this can be detected early.\n3) Engineering team to set up alerts for unexpected spikes in compilation requests during automated syncs, as this could indicate a potential problem with data changes.\n## Detailed Description of Impact\nOn April 18 at 14:40 US/Pacific an internal update unexpectedly turned off the \"Contact Sharing\" feature for Google Workspace users. This resulted in features like autocompleting email addresses and seeing contact details in Gmail, Chat, and Calendar not working as expected.\nKnown user-facing impacts are as follows:\n### Gmail\nAffected users may not have seen a contact list in Gmail. Sending emails were not affected.\nAutocomplete for individual and group addresses entered by users was affected.\nHovercards, also known as profile information cards that show up when users pointed to or tapped on someone’s profile image did not work.\n### Google Contacts\nImpacted users could not find profile information in Google Contacts.\n### Google Calendar\nSearching for users, and intelligent meeting room suggestions in Google Calendar were affected.","status":"AVAILABLE"},{"created":"2024-04-22T19:13:32+00:00","modified":"2024-04-22T19:13:32+00:00","when":"2024-04-22T19:13:32+00:00","text":"A mini incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E","status":"AVAILABLE"},{"created":"2024-04-20T20:10:33+00:00","modified":"2024-04-20T20:10:33+00:00","when":"2024-04-20T20:10:33+00:00","text":"The problem with Gmail, Google Calendar, Google Chat has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.","status":"AVAILABLE"},{"created":"2024-04-19T22:52:08+00:00","modified":"2024-04-19T22:52:08+00:00","when":"2024-04-19T22:52:08+00:00","text":"Further communication will be provided on the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E","status":"SERVICE_DISRUPTION"},{"created":"2024-04-19T21:05:50+00:00","modified":"2024-04-19T21:05:50+00:00","when":"2024-04-19T21:05:50+00:00","text":"Further communication will be provided on the below link.\nhttps://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E","status":"SERVICE_DISRUPTION"},{"created":"2024-04-19T19:36:25+00:00","modified":"2024-04-19T19:36:25+00:00","when":"2024-04-19T19:36:25+00:00","text":"Our Engineering team has identified the root cause of the issue and rolled back a recent change.\nOur engineers are taking additional repair actions for the rollback to take effect for the impacted customers.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Friday, 2024-04-19 14:00 US/Pacific.","status":"SERVICE_DISRUPTION"},{"created":"2024-04-19T18:33:05+00:00","modified":"2024-04-19T18:33:05+00:00","when":"2024-04-19T18:33:05+00:00","text":"Engineering team potentially identified the root cause and working on mitigation.\nThe mitigation is expected to complete by Friday, 2024-04-19 12:15 US/Pacific.","status":"SERVICE_DISRUPTION"},{"created":"2024-04-19T18:22:43+00:00","modified":"2024-04-19T18:22:43+00:00","when":"2024-04-19T18:22:43+00:00","text":"We're investigating reports of an issue with Google Chat. We will provide more information shortly. Google Chat customers may experience \"Contact Sharing\" disabled.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-04-30T20:40:16+00:00","modified":"2024-05-01T14:17:23+00:00","when":"2024-04-30T20:40:16+00:00","text":"# Incident Report\n## Summary\nOn 18 April 2024, Gmail, Google Calendar and Google Chat experienced issues with the ‘Contact Sharing’ feature for a duration of 1 day, 21 hours 43 minutes. To our Google Workspace customers whose businesses were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nAn internal code update unintentionally altered the default values of a setting that interacts with the current contact sharing system. This change resulted in a misconfiguration of directory settings, impacting the Contact Sharing feature within Google Workspace services like Gmail, Chat, and Calendar.\n## Remediation and Prevention\nWe were alerted to the outage via a support case on April 19 at 7:12 US/Pacific and promptly initiated an investigation. The investigation confirmed that the issue originated from a code change deployed on April 18 at 14:40 US/Pacific. Upon troubleshooting, we immediately halted and rolled back the code change, preventing a wider impact on Google Workspace services.\nWe then focused on repairing the issue for affected customers.\nWe apologize for the length and severity of this incident. Google is committed to preventing recurrence by implementing the following action items:\n1) Engineering team implemented release controls to prevent code changes that alter the default values from being deployed to the production environment.\n2) Engineering team to add test coverage and verify code rollouts like these so issues like this can be detected early.\n3) Engineering team to set up alerts for unexpected spikes in compilation requests during automated syncs, as this could indicate a potential problem with data changes.\n## Detailed Description of Impact\nOn April 18 at 14:40 US/Pacific an internal update unexpectedly turned off the \"Contact Sharing\" feature for Google Workspace users. This resulted in features like autocompleting email addresses and seeing contact details in Gmail, Chat, and Calendar not working as expected.\nKnown user-facing impacts are as follows:\n### Gmail\nAffected users may not have seen a contact list in Gmail. Sending emails were not affected.\nAutocomplete for individual and group addresses entered by users was affected.\nHovercards, also known as profile information cards that show up when users pointed to or tapped on someone’s profile image did not work.\n### Google Contacts\nImpacted users could not find profile information in Google Contacts.\n### Google Calendar\nSearching for users, and intelligent meeting room suggestions in Google Calendar were affected.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"Ht9Vx5PFzDPHYvrnrvSk","service_name":"Google Chat","affected_products":[{"title":"Google Chat","id":"Ht9Vx5PFzDPHYvrnrvSk"}],"uri":"incidents/D6dP4yHWc2m93ZhpLYKw"},{"id":"XKnCyj2ED7njHXsC7a6E","number":"17636209183214466509","begin":"2024-04-18T21:40:00+00:00","created":"2024-04-19T17:39:20+00:00","end":"2024-04-20T19:23:00+00:00","modified":"2024-04-30T20:48:16+00:00","external_desc":"We're investigating reports of an issue with Gmail. We will provide more information shortly.","updates":[{"created":"2024-04-30T20:48:16+00:00","modified":"2024-04-30T20:48:16+00:00","when":"2024-04-30T20:48:16+00:00","text":"An incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/D6dP4yHWc2m93ZhpLYKw","status":"AVAILABLE"},{"created":"2024-04-22T19:11:28+00:00","modified":"2024-04-22T19:11:28+00:00","when":"2024-04-22T19:11:28+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 18 April 2024 14:40\n**Incident End:** 20 April 2024 12:23\n**Duration:** 1 day, 21 hours, 43 minutes\n**Affected Services and Features:**\nGmail, Google Calendar, Google Chat\n**Regions/Zones:** Global\n**Description:**\nGmail, Google Calendar and Google Chat experienced issues with “Contact Sharing” for a duration of 1 day, 21 hours, 43 minutes.\nFrom the preliminary analysis, the root cause of the issue was a rollout that led to an unexpected change to the “Directory Settings” policy causing “Contact Sharing” to be disabled for users.\nGoogle engineers mitigated the issue on 20 April 2024 at 12:23 US/Pacific by reverting to a previous stable release and took additional actions to re-enable \"Contact Sharing\" for all affected users.\nGoogle will complete a full IR in the following days that will provide a detailed root cause.\n**Customer Impact:**\n* Affected users may not have seen a contact list in Gmail, but would still have been able to send mail.\n* Affected users could not find profile information in Google Contacts and other Google services.\n* Autocomplete of the individual and group addresses as users entered them was affected.\n* Searching for users and intelligent meeting room suggestions in Google Calendar was affected.\n* Profile information cards that opened when users pointed to or tapped on someone’s profile image were impacted as well.\n---------","status":"AVAILABLE"},{"created":"2024-04-20T20:11:22+00:00","modified":"2024-04-20T20:11:22+00:00","when":"2024-04-20T20:11:22+00:00","text":"The problem with Gmail, Google Calendar, Google Chat has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.","status":"AVAILABLE"},{"created":"2024-04-19T22:48:42+00:00","modified":"2024-04-19T22:48:42+00:00","when":"2024-04-19T22:48:42+00:00","text":"**Summary:**\nGmail , Chat and Calendar customers experiencing issues with Contact sharing\n**Description:**\nOur Engineering team has identified the root cause of the issue and is deploying the appropriate mitigations.\nThe issue is expected to be resolved by Saturday 2024-04-20 13:00 US/Pacific.\nWe will provide more information by Saturday 2024-04-20 13:00 US/Pacific.\n**Diagnosis:**\nCustomers for Gmail, Chat, Calendar may experience \"Contact Sharing\" disabled. Users may not see an address list in Gmail, but will still be able to send email.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-04-19T20:59:38+00:00","modified":"2024-04-19T21:08:29+00:00","when":"2024-04-19T20:59:38+00:00","text":"**Summary:**\nGmail , Chat and Calendar customers experiencing issues with Contact sharing\n**Description:**\nOur Engineering team has identified the root cause of the issue and rolled back a recent change.\nOur engineers have started to deploy mitigations to affected customers, but will take some time to fully propagate.\nWe do not have an ETA for full resolution at this point.\nWe will provide more information by Friday, 2024-04-19 16:30 US/Pacific.\n**Diagnosis:**\nCustomers for Gmail, Chat, Calendar may experience \"Contact Sharing\" disabled. Users may not see an address list in Gmail, but will still be able to send email.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-04-19T19:36:10+00:00","modified":"2024-04-19T19:36:10+00:00","when":"2024-04-19T19:36:10+00:00","text":"Our Engineering team has identified the root cause of the issue and rolled back a recent change.\nOur engineers are taking additional repair actions for the rollback to take effect for the impacted customers.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Friday, 2024-04-19 14:00 US/Pacific.","status":"SERVICE_DISRUPTION"},{"created":"2024-04-19T17:57:09+00:00","modified":"2024-04-19T18:32:27+00:00","when":"2024-04-19T17:57:09+00:00","text":"Engineering team potentially identified the root cause and working on mitigation.\nThe mitigation is expected to complete by Friday, 2024-04-19 12:15 US/Pacific.","status":"SERVICE_DISRUPTION"},{"created":"2024-04-19T17:39:21+00:00","modified":"2024-04-19T18:21:16+00:00","when":"2024-04-19T17:39:21+00:00","text":"We're investigating reports of an issue with Gmail. We will provide more information shortly. Gmail customers may experience \"Contact Sharing\" disabled. Users will not see an address list in Gmail, but will still be able to send email.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-04-30T20:48:16+00:00","modified":"2024-04-30T20:48:16+00:00","when":"2024-04-30T20:48:16+00:00","text":"An incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/D6dP4yHWc2m93ZhpLYKw","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/XKnCyj2ED7njHXsC7a6E"},{"id":"Nr8avWyykjurdUqXYSDW","number":"8728561235399191547","begin":"2024-03-06T22:32:00+00:00","created":"2024-03-07T00:05:56+00:00","end":"2024-03-06T23:29:00+00:00","modified":"2024-03-13T21:27:00+00:00","external_desc":"**Summary:**\nAdmin Console is experiencing elevated errors with the Chrome setting pages.\n**Description:**\nMitigation work is currently underway by our engineering team.\nThe mitigation is expected to complete by Wednesday, 2024-03-06 16:15 US/Pacific.\nWe will provide more information by Wednesday, 2024-03-06 16:45 US/Pacific.\n**Customer Symptoms:**\nThe impacted customers would encounter a HTTP 500 error while accessing user or device setting pages within Chrome Admin Console. Additionally, customers would not be able to view or change any policies.\n**Workaround:**\nNone at this time.","updates":[{"created":"2024-03-13T21:26:43+00:00","modified":"2024-03-13T21:26:43+00:00","when":"2024-03-13T21:26:43+00:00","text":"# Incident Report\n## Summary\nOn Wednesday, 06 March 2024, Google Workspace users experienced elevated errors while accessing user or device setting pages within the Chrome Admin Console for a duration of 1 hour, 27 minutes.\nTo our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nGoogle Workspace Admin Console uses a backend authentication service to grant privileged access to internal accounts. As part of ongoing efforts to migrate the account authorization workflow to a new backend authentication service, Google engineers identified and cleaned up invalid account role assignments in the configuration.\nDuring this clean up effort, engineers identified a privilege which was undefined in the application manifest and removed it from the configuration. Upon further investigation, engineering confirmed the privilege was mapped to an access control setting used by the Admin Console for Chrome. This change resulted in users being unable to access Chrome user and device settings page.\n## Remediation and Prevention\nGoogle engineers were alerted to the outage via an internal monitoring alert on Wednesday, 06 March 2024 at 15:04 US/Pacific and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers initiated a rollback of the configuration change at 15:36 and the incident was fully resolved at 15:59 once the rollback completed.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n- Improve procedural documentation on how to safely cleanup access control settings from authorization configuration.\n- Complete thorough investigation of privilege usage for the Admin Console to prevent further issues of this type.\n- Identify and register a new owner for this privilege in the application manifest and create permission mapping for faster verification in the future.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\n## Detailed Description of Impact\nOn Wednesday, 06 March 2024 from 14:32 to 15:59 US/Pacific, Google Workspace users attempting to work within the Admin Console encountered an HTTP 500 error when accessing user or device setting pages within the console.\n---","status":"AVAILABLE"},{"created":"2024-03-07T21:06:27+00:00","modified":"2024-03-07T21:06:27+00:00","when":"2024-03-07T21:06:27+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 6 March 2023 14:32\n**Incident End:** 6 March 2023 16:29\n**Duration:** 1 hours, 57 minutes\n**Affected Services and Features:**\n* Admin Console Pages for Chrome user settings\n* Chrome device settings\n* Chrome devices\n**Regions/Zones:** Global\n**Description:**\nGoogle Workspace users attempting to work within the Admin Console encountered an HTTP 500 error when accessing user or device setting pages within the console. From preliminary analysis, the root cause of the issue is a change to the access control setting in a backend authentication service used by the Admin Console. This change resulted in users being unable to access the page.\nOur Engineering team was alerted by our internal monitoring systems and swiftly mitigated the issue by rolling back the change to the access control settings.\nThis impacted users ability to view the current user and device policies, change or edit the existing policies and their ability to create new policies.\nGoogle will complete a full Incident Report (IR) in the following days that will provide a detailed root cause.\n**Customer Impact:**\n* The impacted customers encountered a HTTP 500 error while accessing user or device setting pages within Chrome Admin Console.\n* Additionally, customers were unable to view or change any Chrome policies.\n---","status":"AVAILABLE"},{"created":"2024-03-07T00:39:52+00:00","modified":"2024-03-07T00:39:52+00:00","when":"2024-03-07T00:39:52+00:00","text":"**Summary:**\nAdmin Console experienced elevated errors with the setting pages.\n**Description:**\nThe issue with Admin Console has been resolved for all affected users as of Wednesday, 2024-03-06 16:29 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.\n**Symptoms:**\nThe impacted customers would have encountered a HTTP 500 error while accessing user or device setting pages within Chrome Admin Console. Additionally, customers were unable to view or change any policies.\nWorkaround:\nNone at this time...","status":"AVAILABLE"},{"created":"2024-03-07T00:05:57+00:00","modified":"2024-03-07T00:05:57+00:00","when":"2024-03-07T00:05:57+00:00","text":"**Summary:**\nAdmin Console is experiencing elevated errors with the Chrome setting pages.\n**Description:**\nMitigation work is currently underway by our engineering team.\nThe mitigation is expected to complete by Wednesday, 2024-03-06 16:15 US/Pacific.\nWe will provide more information by Wednesday, 2024-03-06 16:45 US/Pacific.\n**Customer Symptoms:**\nThe impacted customers would encounter a HTTP 500 error while accessing user or device setting pages within Chrome Admin Console. Additionally, customers would not be able to view or change any policies.\n**Workaround:**\nNone at this time.","status":"SERVICE_OUTAGE"}],"most_recent_update":{"created":"2024-03-13T21:26:43+00:00","modified":"2024-03-13T21:26:43+00:00","when":"2024-03-13T21:26:43+00:00","text":"# Incident Report\n## Summary\nOn Wednesday, 06 March 2024, Google Workspace users experienced elevated errors while accessing user or device setting pages within the Chrome Admin Console for a duration of 1 hour, 27 minutes.\nTo our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nGoogle Workspace Admin Console uses a backend authentication service to grant privileged access to internal accounts. As part of ongoing efforts to migrate the account authorization workflow to a new backend authentication service, Google engineers identified and cleaned up invalid account role assignments in the configuration.\nDuring this clean up effort, engineers identified a privilege which was undefined in the application manifest and removed it from the configuration. Upon further investigation, engineering confirmed the privilege was mapped to an access control setting used by the Admin Console for Chrome. This change resulted in users being unable to access Chrome user and device settings page.\n## Remediation and Prevention\nGoogle engineers were alerted to the outage via an internal monitoring alert on Wednesday, 06 March 2024 at 15:04 US/Pacific and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers initiated a rollback of the configuration change at 15:36 and the incident was fully resolved at 15:59 once the rollback completed.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n- Improve procedural documentation on how to safely cleanup access control settings from authorization configuration.\n- Complete thorough investigation of privilege usage for the Admin Console to prevent further issues of this type.\n- Identify and register a new owner for this privilege in the application manifest and create permission mapping for faster verification in the future.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.\n## Detailed Description of Impact\nOn Wednesday, 06 March 2024 from 14:32 to 15:59 US/Pacific, Google Workspace users attempting to work within the Admin Console encountered an HTTP 500 error when accessing user or device setting pages within the console.\n---","status":"AVAILABLE"},"status_impact":"SERVICE_OUTAGE","severity":"high","service_key":"xrkLpNMHAvPQczcU2uSb","service_name":"Admin Console","affected_products":[{"title":"Admin Console","id":"xrkLpNMHAvPQczcU2uSb"}],"uri":"incidents/Nr8avWyykjurdUqXYSDW"},{"id":"shD5VvSGTxETw1YLbsCt","number":"17367108176910994243","begin":"2024-03-05T15:25:00+00:00","created":"2024-03-05T16:22:52+00:00","end":"2024-03-05T17:22:00+00:00","modified":"2024-03-12T19:52:43+00:00","external_desc":"We're investigating reports of an issue with Gmail. We will provide more information shortly.","updates":[{"created":"2024-03-12T19:52:43+00:00","modified":"2024-03-12T19:52:43+00:00","when":"2024-03-12T19:52:43+00:00","text":"# Incident Report\n## Summary\nOn Tuesday, 5 March 2024, starting at 07:25 US/Pacific, 0.23% of Gmail UI requests were failing for a duration of 61 minutes and 1.1% of email messages experienced up to five minute delays for a duration of 1 hour, 57 minutes.\n## Root Cause\nThis report includes a revised version of root cause for the service issue Gmail customers experienced on 5 March 2024.\nAfter further investigation, we identified that a sudden and unexpected surge of email traffic initiated the issue, rather than an increase in authentication traffic as shared in the previous report.\nGmail frontend services HTTP, POP, or IMAP use an internal lookup service to route the requests to the specific backend instances that contain the user data. When this internal lookup service fails, users experience issues with Gmail UI interactions.\nThe same internal lookup service is also consulted during email delivery. When the lookup requests fail during email delivery, the requests are retried with an exponential backoff and this would materialize as delayed emails from an end-user perspective.\nOn 5 March 2024 at 07:25 US/Pacific, Gmail began experiencing a sudden and unexpected increase in real user traffic that coincided with a third-party service outage outside of Google’s infrastructure. At the time of the traffic increase, the internal lookup service in some geo locations was overloaded, which resulted in errors for a small fraction of Gmail UI requests and delayed email message delivery.\n## Remediation and Prevention\nGoogle engineers were alerted to the increase in traffic via internal systems on 5 March 2024 at 07:32 US/Pacific and immediately started an investigation. Once the scope and nature of the issue was identified, at 08:11 US/Pacific, Google engineers redirected traffic away from the affected locations. This immediately stopped the Gmail UI errors, and prevented delivery delays for new messages.\nBy 09:22 US/Pacific the backlog of emails in the delivery pipeline had cleared completely.\nIf your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you. Google is committed to preventing a repeat of this issue in the future and is completing the following actions:\n* Enhance Gmail’s lookup server traffic surge protection layer to be able to better handle these kinds of traffic surges.\n## Detailed Description of Impact\nOn 5 March 2024, between 07:25 and 09:22 US/Pacific, Gmail experienced increased error rates and delivery delays. During this time: * Approximately 0.23% of requests in the Gmail UI resulted in an error message. Affected customers encountered these errors for various actions on Gmail, including login. * Approximately 1.1% of email messages experienced delivery delays ranging between one and five minutes.\n---------","status":"AVAILABLE"},{"created":"2024-03-06T00:19:48+00:00","modified":"2024-03-06T00:19:48+00:00","when":"2024-03-06T00:19:48+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 05 March, 2024 07:25\n**Incident End:** 05 March, 2024 09:22\n**Duration:** 1 hour, 57 minutes\n**Affected Services and Features:**\nGmail\n**Regions/Zones:** Global\n**Description:**\nGmail experienced 0.23% UI error rates for 61 minutes, and 1 to 5 minute email delays for just under two hours.\nFrom preliminary analysis, the trigger of the issue was a sudden and unexpected increase in authentication traffic which was caused by user re-login attempts following device restarts. The device restarts were a result of users attempting to mitigate a third party application issue outside of Google’s infrastructure. This additional load led to elevated errors in our backend authentication service, which in turn impacted Gmail.\nGoogle engineers responded promptly to the increased error rates and mitigated the issue by adding compute resources to handle the additional load. By 08:26 AM US/Pacific, Gmail UI error rates were largely resolved, and the error rate returned to baseline by 08:51. Delivery delays continued until 09:22 at which point the impact to Gmail was completely mitigated.\n**Customer Impact:**\n* Affected customers would have noticed an “oops” error message in the Gmail UI. Customers encountered these errors for various actions on Gmail, including login.\n* Some customers would have experienced delays with email delivery ranging between one and five minutes.","status":"AVAILABLE"},{"created":"2024-03-05T17:34:26+00:00","modified":"2024-03-05T17:34:26+00:00","when":"2024-03-05T17:34:26+00:00","text":"We saw a surge in traffic starting around 7:25 AM Pacific Time on March 5, and we scaled up our systems to serve the additional load.","status":"AVAILABLE"},{"created":"2024-03-05T16:35:18+00:00","modified":"2024-03-05T16:35:18+00:00","when":"2024-03-05T16:35:18+00:00","text":"Our team is continuing to investigate this issue. We will provide an update by Mar 5, 2024, 5:15 PM UTC with more information about this problem. Thank you for your patience. **Symptoms:**\nSome Gmail users are experiencing elevated error rates while performing various actions and may also see delays in email delivery","status":"SERVICE_DISRUPTION"},{"created":"2024-03-05T16:22:52+00:00","modified":"2024-03-05T16:25:51+00:00","when":"2024-03-05T16:22:52+00:00","text":"We're investigating reports of an issue with Gmail. We will provide more information shortly.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-03-12T19:52:43+00:00","modified":"2024-03-12T19:52:43+00:00","when":"2024-03-12T19:52:43+00:00","text":"# Incident Report\n## Summary\nOn Tuesday, 5 March 2024, starting at 07:25 US/Pacific, 0.23% of Gmail UI requests were failing for a duration of 61 minutes and 1.1% of email messages experienced up to five minute delays for a duration of 1 hour, 57 minutes.\n## Root Cause\nThis report includes a revised version of root cause for the service issue Gmail customers experienced on 5 March 2024.\nAfter further investigation, we identified that a sudden and unexpected surge of email traffic initiated the issue, rather than an increase in authentication traffic as shared in the previous report.\nGmail frontend services HTTP, POP, or IMAP use an internal lookup service to route the requests to the specific backend instances that contain the user data. When this internal lookup service fails, users experience issues with Gmail UI interactions.\nThe same internal lookup service is also consulted during email delivery. When the lookup requests fail during email delivery, the requests are retried with an exponential backoff and this would materialize as delayed emails from an end-user perspective.\nOn 5 March 2024 at 07:25 US/Pacific, Gmail began experiencing a sudden and unexpected increase in real user traffic that coincided with a third-party service outage outside of Google’s infrastructure. At the time of the traffic increase, the internal lookup service in some geo locations was overloaded, which resulted in errors for a small fraction of Gmail UI requests and delayed email message delivery.\n## Remediation and Prevention\nGoogle engineers were alerted to the increase in traffic via internal systems on 5 March 2024 at 07:32 US/Pacific and immediately started an investigation. Once the scope and nature of the issue was identified, at 08:11 US/Pacific, Google engineers redirected traffic away from the affected locations. This immediately stopped the Gmail UI errors, and prevented delivery delays for new messages.\nBy 09:22 US/Pacific the backlog of emails in the delivery pipeline had cleared completely.\nIf your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you. Google is committed to preventing a repeat of this issue in the future and is completing the following actions:\n* Enhance Gmail’s lookup server traffic surge protection layer to be able to better handle these kinds of traffic surges.\n## Detailed Description of Impact\nOn 5 March 2024, between 07:25 and 09:22 US/Pacific, Gmail experienced increased error rates and delivery delays. During this time: * Approximately 0.23% of requests in the Gmail UI resulted in an error message. Affected customers encountered these errors for various actions on Gmail, including login. * Approximately 1.1% of email messages experienced delivery delays ranging between one and five minutes.\n---------","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/shD5VvSGTxETw1YLbsCt"},{"id":"5tY9wqrGbUpMEttzGbeq","number":"13911551035416450664","begin":"2024-02-28T12:23:00+00:00","created":"2024-02-28T13:08:35+00:00","end":"2024-02-28T13:41:00+00:00","modified":"2024-02-28T17:23:05+00:00","external_desc":"We're investigating reports of an issue with Google Voice. We will provide more information shortly.","updates":[{"created":"2024-02-28T14:50:06+00:00","modified":"2024-02-28T17:23:05+00:00","when":"2024-02-28T14:50:06+00:00","text":"The problem with Google Voice has been resolved. After further investigation, we believe the issue existed outside of Google's infrastructure. No further updates will be added to this dashboard. The affected users are able to access Google Voice, but are seeing error messages, high latency, and/or other unexpected behavior.","status":"AVAILABLE"},{"created":"2024-02-28T14:44:20+00:00","modified":"2024-02-28T14:44:20+00:00","when":"2024-02-28T14:44:20+00:00","text":"We are continuing to investigate this issue. We will provide an update by Feb 28, 2024, 4:00 PM UTC detailing when we expect to resolve the problem. The affected users are able to access Google Voice, but are seeing error messages, high latency, and/or other unexpected behavior. - Placing and receiving the calls that are external to the caller may fail in France. - The calls within the customer company network will work.","status":"SERVICE_OUTAGE"},{"created":"2024-02-28T13:56:38+00:00","modified":"2024-02-28T13:56:38+00:00","when":"2024-02-28T13:56:38+00:00","text":"Our team is continuing to investigate this issue. We will provide an update by Feb 28, 2024, 3:00 PM UTC with more information about this problem. Thank you for your patience. The affected users are able to access Google Voice, but are seeing error messages, high latency, and/or other unexpected behavior. Placing and receiving calls may fail in France.","status":"SERVICE_OUTAGE"},{"created":"2024-02-28T13:09:47+00:00","modified":"2024-02-28T13:09:47+00:00","when":"2024-02-28T13:09:47+00:00","text":"We're aware of a problem with Google Voice affecting a significant subset of users. We will provide an update by Feb 28, 2024, 2:00 PM UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change. Placing and receiving calls may fail in France.","status":"SERVICE_OUTAGE"},{"created":"2024-02-28T13:08:36+00:00","modified":"2024-02-28T13:08:36+00:00","when":"2024-02-28T13:08:36+00:00","text":"We're investigating reports of an issue with Google Voice. We will provide more information shortly. Placing and receiving calls may fail in France.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-28T14:50:06+00:00","modified":"2024-02-28T17:23:05+00:00","when":"2024-02-28T14:50:06+00:00","text":"The problem with Google Voice has been resolved. After further investigation, we believe the issue existed outside of Google's infrastructure. No further updates will be added to this dashboard. The affected users are able to access Google Voice, but are seeing error messages, high latency, and/or other unexpected behavior.","status":"AVAILABLE"},"status_impact":"SERVICE_OUTAGE","severity":"high","service_key":"gh79SrQk6wqonWoxPdCm","service_name":"Google Voice","affected_products":[{"title":"Google Voice","id":"gh79SrQk6wqonWoxPdCm"}],"uri":"incidents/5tY9wqrGbUpMEttzGbeq"},{"id":"bgABqRBhgnSP8FNM9Hwt","number":"16417451330693840686","begin":"2024-02-26T09:42:00+00:00","created":"2024-02-26T17:33:43+00:00","end":"2024-02-26T21:02:00+00:00","modified":"2024-02-29T23:45:55+00:00","external_desc":"We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific.\nFor further details on product impact, please see: https://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6","updates":[{"created":"2024-02-29T23:45:55+00:00","modified":"2024-02-29T23:45:55+00:00","when":"2024-02-29T23:45:55+00:00","text":"# Incident Report\n## Summary\nOn Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.\nThis is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nA third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.\n## Remediation and Prevention\nGoogle engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n- Adding additional system limits to prevent a single Apps Script project from creating overload scenarios\n- Improving the process for blocking problematic Apps Script projects\n## Detailed Description of Impact\nOn Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:\n### Apps Script\nApps Script encountered errors during the creation or loading of scripts.\n### Google Calendar\nA subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.\n### Google Forms, Google sheets and Google docs\nCustom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.","status":"AVAILABLE"},{"created":"2024-02-26T23:32:24+00:00","modified":"2024-02-26T23:32:24+00:00","when":"2024-02-26T23:32:24+00:00","text":"A mini incident report has been posted on the Google\nWorkspace Status Dashboard\nhttps://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6","status":"AVAILABLE"},{"created":"2024-02-26T21:26:26+00:00","modified":"2024-02-26T21:26:26+00:00","when":"2024-02-26T21:26:26+00:00","text":"The issue with Google Calendar, Google Docs, Google Forms, Google Sheets is believed to be resolved for the majority of users.\nWe received an earlier report of issues with triggers/API related errors however, engineering confirmed there was a queued backlog which caused delays. We expect the backlog should be cleared in a few hours.\nIf you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-26T17:33:43+00:00","modified":"2024-02-26T17:33:43+00:00","when":"2024-02-26T17:33:43+00:00","text":"We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific.\nFor further details on product impact, please see: https://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-29T23:45:55+00:00","modified":"2024-02-29T23:45:55+00:00","when":"2024-02-29T23:45:55+00:00","text":"# Incident Report\n## Summary\nOn Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.\nThis is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nA third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.\n## Remediation and Prevention\nGoogle engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n- Adding additional system limits to prevent a single Apps Script project from creating overload scenarios\n- Improving the process for blocking problematic Apps Script projects\n## Detailed Description of Impact\nOn Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:\n### Apps Script\nApps Script encountered errors during the creation or loading of scripts.\n### Google Calendar\nA subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.\n### Google Forms, Google sheets and Google docs\nCustom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"GsNjeYHsogeRgTiC5coU","service_name":"Google Sheets","affected_products":[{"title":"Google Sheets","id":"GsNjeYHsogeRgTiC5coU"}],"uri":"incidents/bgABqRBhgnSP8FNM9Hwt"},{"id":"7ewuZHQvXBszfpR4mEcq","number":"12497026859061866190","begin":"2024-02-26T09:42:00+00:00","created":"2024-02-26T17:34:44+00:00","end":"2024-02-26T21:02:00+00:00","modified":"2024-02-29T23:44:29+00:00","external_desc":"We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific.\nFor further details on product impact, please see: https://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6","updates":[{"created":"2024-02-29T23:44:29+00:00","modified":"2024-02-29T23:44:29+00:00","when":"2024-02-29T23:44:29+00:00","text":"# Incident Report\n## Summary\nOn Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.\nThis is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nA third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.\n## Remediation and Prevention\nGoogle engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n- Adding additional system limits to prevent a single Apps Script project from creating overload scenarios\n- Improving the process for blocking problematic Apps Script projects\n## Detailed Description of Impact\nOn Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:\n### Apps Script\nApps Script encountered errors during the creation or loading of scripts.\n### Google Calendar\nA subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.\n### Google Forms, Google sheets and Google docs\nCustom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.","status":"AVAILABLE"},{"created":"2024-02-26T23:31:24+00:00","modified":"2024-02-26T23:31:24+00:00","when":"2024-02-26T23:31:24+00:00","text":"A mini incident report has been posted on the Google\nWorkspace Status Dashboard\nhttps://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6","status":"AVAILABLE"},{"created":"2024-02-26T21:25:53+00:00","modified":"2024-02-26T21:25:53+00:00","when":"2024-02-26T21:25:53+00:00","text":"The issue with Google Calendar, Google Docs, Google Forms, Google Sheets is believed to be resolved for the majority of users.\nWe received an earlier report of issues with triggers/API related errors however, engineering confirmed there was a queued backlog which caused delays. We expect the backlog should be cleared in a few hours.\nIf you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-26T17:34:44+00:00","modified":"2024-02-26T17:34:44+00:00","when":"2024-02-26T17:34:44+00:00","text":"We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific.\nFor further details on product impact, please see: https://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-29T23:44:29+00:00","modified":"2024-02-29T23:44:29+00:00","when":"2024-02-29T23:44:29+00:00","text":"# Incident Report\n## Summary\nOn Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.\nThis is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nA third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.\n## Remediation and Prevention\nGoogle engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n- Adding additional system limits to prevent a single Apps Script project from creating overload scenarios\n- Improving the process for blocking problematic Apps Script projects\n## Detailed Description of Impact\nOn Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:\n### Apps Script\nApps Script encountered errors during the creation or loading of scripts.\n### Google Calendar\nA subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.\n### Google Forms, Google sheets and Google docs\nCustom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"wNHuVFtZEWU5Mmj2eRCK","service_name":"Google Docs","affected_products":[{"title":"Google Docs","id":"wNHuVFtZEWU5Mmj2eRCK"}],"uri":"incidents/7ewuZHQvXBszfpR4mEcq"},{"id":"ipDHw6kUSngP2zux2KoP","number":"8901458806698388399","begin":"2024-02-26T09:42:00+00:00","created":"2024-02-26T17:36:27+00:00","end":"2024-02-26T21:02:00+00:00","modified":"2024-02-29T23:46:33+00:00","external_desc":"We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific.\nFor further details on product impact, please see: https://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6","updates":[{"created":"2024-02-29T23:46:33+00:00","modified":"2024-02-29T23:46:33+00:00","when":"2024-02-29T23:46:33+00:00","text":"# Incident Report\n## Summary\nOn Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.\nThis is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nA third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.\n## Remediation and Prevention\nGoogle engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n- Adding additional system limits to prevent a single Apps Script project from creating overload scenarios\n- Improving the process for blocking problematic Apps Script projects\n## Detailed Description of Impact\nOn Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:\n### Apps Script\nApps Script encountered errors during the creation or loading of scripts.\n### Google Calendar\nA subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.\n### Google Forms, Google sheets and Google docs\nCustom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.","status":"AVAILABLE"},{"created":"2024-02-26T23:33:25+00:00","modified":"2024-02-26T23:33:25+00:00","when":"2024-02-26T23:33:25+00:00","text":"A mini incident report has been posted on the Google\nWorkspace Status Dashboard.\nhttps://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6","status":"AVAILABLE"},{"created":"2024-02-26T21:27:02+00:00","modified":"2024-02-26T21:27:02+00:00","when":"2024-02-26T21:27:02+00:00","text":"The issue with Google Calendar, Google Docs, Google Forms, Google Sheets is believed to be resolved for the majority of users.\nWe received an earlier report of issues with triggers/API related errors however, engineering confirmed there was a queued backlog which caused delays. We expect the backlog should be cleared in a few hours.\nIf you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-26T17:36:27+00:00","modified":"2024-02-26T17:36:27+00:00","when":"2024-02-26T17:36:27+00:00","text":"We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific.\nFor further details on product impact, please see: https://www.google.com/appsstatus/dashboard/incidents/DR1JYH2tW69g9BobX8e6","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-29T23:46:33+00:00","modified":"2024-02-29T23:46:33+00:00","when":"2024-02-29T23:46:33+00:00","text":"# Incident Report\n## Summary\nOn Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.\nThis is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nA third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.\n## Remediation and Prevention\nGoogle engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n- Adding additional system limits to prevent a single Apps Script project from creating overload scenarios\n- Improving the process for blocking problematic Apps Script projects\n## Detailed Description of Impact\nOn Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:\n### Apps Script\nApps Script encountered errors during the creation or loading of scripts.\n### Google Calendar\nA subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.\n### Google Forms, Google sheets and Google docs\nCustom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"n5hVexuq1PM9onY9qrmo","service_name":"Google Calendar","affected_products":[{"title":"Google Calendar","id":"n5hVexuq1PM9onY9qrmo"}],"uri":"incidents/ipDHw6kUSngP2zux2KoP"},{"id":"DR1JYH2tW69g9BobX8e6","number":"16390237619113481977","begin":"2024-02-26T09:42:00+00:00","created":"2024-02-26T17:26:24+00:00","end":"2024-02-26T21:02:00+00:00","modified":"2024-02-29T23:45:05+00:00","external_desc":"We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific.\nWe believe the issue is mostly resolved for impacted services and our engineers are continuing to work on preventing similar issues in the future.\nWe will provide an update by Monday, 2024-02-26 10:00 US/Pacific with current details.","updates":[{"created":"2024-02-29T23:45:05+00:00","modified":"2024-02-29T23:45:05+00:00","when":"2024-02-29T23:45:05+00:00","text":"# Incident Report\n## Summary\nOn Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.\nThis is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nA third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.\n## Remediation and Prevention\nGoogle engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n- Adding additional system limits to prevent a single Apps Script project from creating overload scenarios\n- Improving the process for blocking problematic Apps Script projects\n## Detailed Description of Impact\nOn Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:\n### Apps Script\nApps Script encountered errors during the creation or loading of scripts.\n### Google Calendar\nA subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.\n### Google Forms, Google sheets and Google docs\nCustom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.","status":"AVAILABLE"},{"created":"2024-02-26T23:27:46+00:00","modified":"2024-02-26T23:27:46+00:00","when":"2024-02-26T23:27:46+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 26 Feb, 2024 1:42\n**Incident End:** 26 Feb, 2024 13:02\n**Duration:** 11 hours, 20 minutes\n**Affected Services and Features:**\nWorkspace product integrations with Apps Script\n**Regions/Zones:** Global\n**Description:**\nMultiple Google Workspace products experienced intermittent failures while attempting various operational actions for the incident’s duration of 11 hours, 20 minutes. From preliminary analysis, the root cause of the issue appears to be due to a queued backlog which led to the performance delays. Google will complete a full Incident Report in the following days that will provide a detailed root cause.\n**Customer Impact:**\nDuring the incident time frame customers may have experienced the following product impact:\n* Apps Script: Apps Script is returning errors on creation or loading of scripts\n* Google Calendar: Calendar 3rd party video conferencing integrations are impacted\n* Google Forms: Custom Functions using Apps Script could be impacted\n* Google Sheets, Google Docs: Add-ons which are built on Apps Script in Google Sheets and Google Docs can also be impacted\nOther Workspace product integrations with Apps Script may have also encountered elevated error rate. The impact includes any third party integrations to Apps Script.","status":"AVAILABLE"},{"created":"2024-02-26T21:21:10+00:00","modified":"2024-02-26T21:21:10+00:00","when":"2024-02-26T21:21:10+00:00","text":"The issue with Google Calendar, Google Docs, Google Forms, Google Sheets is believed to be resolved for the majority of users.\nWe received an earlier report of issues with triggers/API related errors however, engineering confirmed there was a queued backlog which caused delays. We expect the backlog should be cleared in a few hours.\nIf you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-26T20:21:51+00:00","modified":"2024-02-26T20:21:51+00:00","when":"2024-02-26T20:21:51+00:00","text":"We believe the issue is mostly resolved for impacted services. Our engineers have implemented a new mitigation to prevent recurrence of this issue and are showing signs of recovery.\nWe are investigating a few reports of triggers/API related errors which may be impacting a small number of users.\nWe will provide an update by Monday, 2024-02-26 13:20 US/Pacific with current details.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-26T19:09:48+00:00","modified":"2024-02-26T19:09:48+00:00","when":"2024-02-26T19:09:48+00:00","text":"We believe the issue is mostly resolved for impacted services. Our engineers have implemented a new mitigation to prevent recurrence of this issue in the future which is showing signs of improvement.\nWe will provide an update by Monday, 2024-02-26 12:10 US/Pacific with current details.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-26T18:08:55+00:00","modified":"2024-02-26T18:08:55+00:00","when":"2024-02-26T18:08:55+00:00","text":"We believe the issue is mostly resolved for impacted services. Mitigation work is ongoing and our engineers are continuing to work on preventing similar issues in the future.\nWe will provide an update by Monday, 2024-02-26 10:30 US/Pacific with current details.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-26T17:26:25+00:00","modified":"2024-02-26T17:26:25+00:00","when":"2024-02-26T17:26:25+00:00","text":"We are experiencing an Apps Script issue impacting Google Calendar 3rd party video conferencing integrations, Google Forms, Google Docs, and Google Sheets beginning on Monday, 2024-02-26 00:11 US/Pacific.\nWe believe the issue is mostly resolved for impacted services and our engineers are continuing to work on preventing similar issues in the future.\nWe will provide an update by Monday, 2024-02-26 10:00 US/Pacific with current details.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-29T23:45:05+00:00","modified":"2024-02-29T23:45:05+00:00","when":"2024-02-29T23:45:05+00:00","text":"# Incident Report\n## Summary\nOn Monday, 26 February 2024 at 1:42 US/Pacific, Google Apps Script experienced a higher rate of failures while attempting various operational actions for a duration of 11 hours, 20 minutes. Apps Script integrations in Google Workspace products, such as those that integrate with Google Calendar and Google Docs, were partially unavailable. This includes 3rd party Google Workspace add-ons that were built using Apps Script. During the incident, the issue was intermittent for most users with overall error rates of up to 25% for the listed services, and customers retrying actions were eventually successful.\nThis is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nA third-party Google Workspace Add-on with a large pre-existing install base added a dependency on Google Apps Script, which triggered a significant increase in load for the Apps Script service. Due to this high QPS, latencies increased, followed by an increase in error rates. All server threads supporting Apps Script integrations were busy processing requests, causing our servers to start rejecting requests and resulting in errors. Ultimately this resulted in temporary service interruption for users who rely on Apps Script functionality within their Google Workspace environments.\n## Remediation and Prevention\nGoogle engineering was alerted to the outage via internal monitoring on Monday 26 February 2024 at 00:16 US/Pacific and immediately started an investigation. Engineering identified the problematic deployment, rolled back the change, and temporarily blocked access to the add-on so the service could fulfill requests as expected. This resulted in a significant error rate decrease, services became available, and the issue was fully recovered on Monday 26 February, 2024 13:02 US/Pacific.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n- Adding additional system limits to prevent a single Apps Script project from creating overload scenarios\n- Improving the process for blocking problematic Apps Script projects\n## Detailed Description of Impact\nOn Monday, 26 February 2024 from 01:42 to 13:02 US/Pacific customers utilizing products with Apps Script integrations may have experienced the following:\n### Apps Script\nApps Script encountered errors during the creation or loading of scripts.\n### Google Calendar\nA subset of customers encountered failures while listing the third-party conferencing solutions. Additionally, failures were observed while attaching a conference to an event with the Google Calendar.\n### Google Forms, Google sheets and Google docs\nCustom functions and Add-ons created with Apps Script would have experienced disruptions when used in Google Docs, Sheets, Slides, and Forms.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"3mtqLXm6QxeM6bAVkVez","service_name":"Google Forms","affected_products":[{"title":"Google Forms","id":"3mtqLXm6QxeM6bAVkVez"}],"uri":"incidents/DR1JYH2tW69g9BobX8e6"},{"id":"aMjTw17JG2FAysbmhTdQ","number":"14286879852283004251","begin":"2024-02-13T15:30:00+00:00","created":"2024-02-15T23:30:35+00:00","end":"2024-02-16T19:00:00+00:00","modified":"2024-02-28T17:36:19+00:00","external_desc":"We are investigating issues with Google Tasks starting at 07:23 PT on 13-February-2024. There is no workaround at this time.","updates":[{"created":"2024-02-28T17:33:47+00:00","modified":"2024-02-28T17:33:47+00:00","when":"2024-02-28T17:33:47+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-20T21:11:40+00:00","modified":"2024-02-20T21:11:40+00:00","when":"2024-02-20T21:11:40+00:00","text":"A mini incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-16T23:14:52+00:00","modified":"2024-02-16T23:14:52+00:00","when":"2024-02-16T23:14:52+00:00","text":"We experienced an issue that impacted multiple products including Gmail, Google Calendar, Google Chat, Google Drive, Google Meet, Google Cloud Search, Google Voice, Google Tasks, Google Docs and Chrome OS.\nOur engineering team completed the mitigation actions and the backlog of the changes has cleared up for most of the customers.\nA small subset of customers may still experience delays with previously implemented changes to their workspace products and we expect full resolution within the next 24 hours.\nDuring the issue customers experienced longer than expected delays (sometimes over 24 hours) in propagating changes made to various Workspace products. These delays in some cases may have resulted in errors. The affected changes are not limited to enterprise setting changes.\nIf changes are taking longer than 24 hours going forward or you continue to see errors, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-16T19:33:19+00:00","modified":"2024-02-16T19:33:19+00:00","when":"2024-02-16T19:33:19+00:00","text":"Please refer to https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV for updates on this issue going forward.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T17:24:59+00:00","modified":"2024-02-16T17:24:59+00:00","when":"2024-02-16T17:24:59+00:00","text":"**Summary**\nWe are experiencing an issue with Google Tasks.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 11:30 US/Pacific.\n**Customer symptom:**\n- New changes to enterprise settings done by customers should be working as expected.\n- Previously implemented changes may still take longer than expected to take effect as we process the backlog.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T13:50:52+00:00","modified":"2024-02-16T13:50:52+00:00","when":"2024-02-16T13:50:52+00:00","text":"**Summary**\nWe are experiencing an issue with Google Tasks.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 09:15 US/Pacific.\n**Customer symptom:**\nNew changes to enterprise settings done by customers should be working as expected.\nPreviously implemented changes may still take longer than expected to take effect.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T08:42:57+00:00","modified":"2024-02-16T08:42:57+00:00","when":"2024-02-16T08:42:57+00:00","text":"**Summary**\nWe are experiencing an issue with Google Tasks.\nMitigation work is currently underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 06:00 US/Pacific.\n**Customer symptom:**\nChanges to enterprise settings done by customers may take longer than expected to take effect\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T02:08:06+00:00","modified":"2024-02-16T07:35:15+00:00","when":"2024-02-16T02:08:06+00:00","text":"**Summary**\nWe are experiencing an issue with Google Tasks.\nA rollback has been performed, but the issue persists. Our engineering team is continuing their investigation and working towards a solution.\nWe will provide more information by Friday, 2024-02-16 00:30 US/Pacific.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T01:04:14+00:00","modified":"2024-02-16T01:04:14+00:00","when":"2024-02-16T01:04:14+00:00","text":"**Summary**\nWe are experiencing an issue with Google Tasks.\nThe engineers have identified a mitigation strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 18:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T00:19:54+00:00","modified":"2024-02-16T00:19:54+00:00","when":"2024-02-16T00:19:54+00:00","text":"**Summary**\nThe engineers have identified a mitigating strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 17:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-15T23:30:36+00:00","modified":"2024-02-15T23:30:36+00:00","when":"2024-02-15T23:30:36+00:00","text":"We are investigating issues with Google Tasks starting at 07:23 PT on 13-February-2024. There is no workaround at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-28T17:33:47+00:00","modified":"2024-02-28T17:33:47+00:00","when":"2024-02-28T17:33:47+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"Vups7kniH52V5x4UX1gM","service_name":"Google Tasks","affected_products":[{"title":"Google Tasks","id":"Vups7kniH52V5x4UX1gM"}],"uri":"incidents/aMjTw17JG2FAysbmhTdQ"},{"id":"jVNGKzkagzeaN7kWCu37","number":"2419471840878979611","begin":"2024-02-13T15:30:00+00:00","created":"2024-02-15T23:27:35+00:00","end":"2024-02-16T19:00:00+00:00","modified":"2024-02-28T17:31:47+00:00","external_desc":"We are investigating issues with Google Cloud Search starting at 07:23 PT on 13-February-2024. There is no workaround at this time.","updates":[{"created":"2024-02-28T17:31:30+00:00","modified":"2024-02-28T17:31:30+00:00","when":"2024-02-28T17:31:30+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-20T21:12:32+00:00","modified":"2024-02-20T21:12:32+00:00","when":"2024-02-20T21:12:32+00:00","text":"A mini incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-16T23:16:09+00:00","modified":"2024-02-16T23:16:09+00:00","when":"2024-02-16T23:16:09+00:00","text":"We experienced an issue that impacted multiple products including Gmail, Google Calendar, Google Chat, Google Drive, Google Meet, Google Cloud Search, Google Voice, Google Tasks, Google Docs and Chrome OS.\nOur engineering team completed the mitigation actions and the backlog of the changes has cleared up for most of the customers.\nA small subset of customers may still experience delays with previously implemented changes to their workspace products and we expect full resolution within the next 24 hours.\nDuring the issue customers experienced longer than expected delays (sometimes over 24 hours) in propagating changes made to various Workspace products. These delays in some cases may have resulted in errors. The affected changes are not limited to enterprise setting changes.\nIf changes are taking longer than 24 hours going forward or you continue to see errors, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-16T19:33:51+00:00","modified":"2024-02-16T19:33:51+00:00","when":"2024-02-16T19:33:51+00:00","text":"Please refer to https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV for updates on this issue going forward.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T17:25:45+00:00","modified":"2024-02-16T17:25:45+00:00","when":"2024-02-16T17:25:45+00:00","text":"**Summary**\nWe are experiencing an issue with Google Cloud Search.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 11:30 US/Pacific.\n**Customer symptom:**\n- New changes to enterprise settings done by customers should be working as expected.\n- Previously implemented changes may still take longer than expected to take effect as we process the backlog.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T13:48:14+00:00","modified":"2024-02-16T13:48:14+00:00","when":"2024-02-16T13:48:14+00:00","text":"**Summary**\nWe are experiencing an issue with Google Cloud Search.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 09:15 US/Pacific.\n**Customer symptom:**\nNew changes to enterprise settings done by customers should be working as expected.\nPreviously implemented changes may still take longer than expected to take effect.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T08:45:23+00:00","modified":"2024-02-16T08:45:23+00:00","when":"2024-02-16T08:45:23+00:00","text":"**Summary**\nWe are experiencing an issue with Google Cloud Search.\nMitigation work is currently underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 06:00 US/Pacific.\n**Customer symptom:**\nChanges to enterprise settings done by customers may take longer than expected to take effect\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T02:06:50+00:00","modified":"2024-02-16T07:36:53+00:00","when":"2024-02-16T02:06:50+00:00","text":"**Summary**\nWe are experiencing an issue with Google Cloud Search.\nA rollback has been performed, but the issue persists. Our engineering team is continuing their investigation and working towards a solution.\nWe will provide more information by Friday, 2024-02-16 00:30 US/Pacific.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T01:06:31+00:00","modified":"2024-02-16T01:06:31+00:00","when":"2024-02-16T01:06:31+00:00","text":"**Summary**\nWe are experiencing an issue with Google Cloud Search.\nThe engineers have identified a mitigation strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 18:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T00:20:33+00:00","modified":"2024-02-16T00:20:33+00:00","when":"2024-02-16T00:20:33+00:00","text":"**Summary**\nThe engineers have identified a mitigating strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 17:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-15T23:27:36+00:00","modified":"2024-02-15T23:27:36+00:00","when":"2024-02-15T23:27:36+00:00","text":"We are investigating issues with Google Cloud Search starting at 07:23 PT on 13-February-2024. There is no workaround at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-28T17:31:30+00:00","modified":"2024-02-28T17:31:30+00:00","when":"2024-02-28T17:31:30+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"iJPnujhwzow3TaPE1dM8","service_name":"Google Cloud Search","affected_products":[{"title":"Google Cloud Search","id":"iJPnujhwzow3TaPE1dM8"}],"uri":"incidents/jVNGKzkagzeaN7kWCu37"},{"id":"mvR44baNg5v3UMTjsYr4","number":"14879639063615360444","begin":"2024-02-13T15:30:00+00:00","created":"2024-02-15T23:31:58+00:00","end":"2024-02-16T19:00:00+00:00","modified":"2024-02-28T17:30:46+00:00","external_desc":"We are investigating issues with Google Voice starting at 07:23 PT on 13-February-2024. There is no workaround at this time.","updates":[{"created":"2024-02-28T17:30:33+00:00","modified":"2024-02-28T17:30:33+00:00","when":"2024-02-28T17:30:33+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-20T21:12:53+00:00","modified":"2024-02-20T21:12:53+00:00","when":"2024-02-20T21:12:53+00:00","text":"A mini incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-16T23:16:35+00:00","modified":"2024-02-16T23:16:35+00:00","when":"2024-02-16T23:16:35+00:00","text":"We experienced an issue that impacted multiple products including Gmail, Google Calendar, Google Chat, Google Drive, Google Meet, Google Cloud Search, Google Voice, Google Tasks, Google Docs and Chrome OS.\nOur engineering team completed the mitigation actions and the backlog of the changes has cleared up for most of the customers.\nA small subset of customers may still experience delays with previously implemented changes to their workspace products and we expect full resolution within the next 24 hours.\nDuring the issue customers experienced longer than expected delays (sometimes over 24 hours) in propagating changes made to various Workspace products. These delays in some cases may have resulted in errors. The affected changes are not limited to enterprise setting changes.\nIf changes are taking longer than 24 hours going forward or you continue to see errors, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-16T19:34:06+00:00","modified":"2024-02-16T19:34:06+00:00","when":"2024-02-16T19:34:06+00:00","text":"Please refer to https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV for updates on this issue going forward.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T17:26:05+00:00","modified":"2024-02-16T17:26:05+00:00","when":"2024-02-16T17:26:05+00:00","text":"**Summary**\nWe are experiencing an issue with Google Voice.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 11:30 US/Pacific.\n**Customer symptom:**\n- New changes to enterprise settings done by customers should be working as expected.\n- Previously implemented changes may still take longer than expected to take effect as we process the backlog.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T13:47:16+00:00","modified":"2024-02-16T13:47:16+00:00","when":"2024-02-16T13:47:16+00:00","text":"**Summary**\nWe are experiencing an issue with Google Voice.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 09:15 US/Pacific.\n**Customer symptom:**\nNew changes to enterprise settings done by customers should be working as expected.\nPreviously implemented changes may still take longer than expected to take effect.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T08:49:56+00:00","modified":"2024-02-16T08:49:56+00:00","when":"2024-02-16T08:49:56+00:00","text":"**Summary**\nWe are experiencing an issue with Google Voice.\nMitigation work is currently underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 06:00 US/Pacific.\n**Customer symptom:**\nChanges to enterprise settings done by customers may take longer than expected to take effect\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T02:05:37+00:00","modified":"2024-02-16T07:37:34+00:00","when":"2024-02-16T02:05:37+00:00","text":"**Summary**\nWe are experiencing an issue with Google Voice.\nA rollback has been performed, but the issue persists. Our engineering team is continuing their investigation and working towards a solution.\nWe will provide more information by Friday, 2024-02-16 00:30 US/Pacific.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T01:07:32+00:00","modified":"2024-02-16T01:07:32+00:00","when":"2024-02-16T01:07:32+00:00","text":"**Summary**\nWe are experiencing an issue with Google Voice.\nThe engineers have identified a mitigation strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 18:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T00:21:10+00:00","modified":"2024-02-16T00:21:10+00:00","when":"2024-02-16T00:21:10+00:00","text":"**Summary**\nThe engineers have identified a mitigating strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 17:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-15T23:31:59+00:00","modified":"2024-02-15T23:31:59+00:00","when":"2024-02-15T23:31:59+00:00","text":"We are investigating issues with Google Voice starting at 07:23 PT on 13-February-2024. There is no workaround at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-28T17:30:33+00:00","modified":"2024-02-28T17:30:33+00:00","when":"2024-02-28T17:30:33+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"gh79SrQk6wqonWoxPdCm","service_name":"Google Voice","affected_products":[{"title":"Google Voice","id":"gh79SrQk6wqonWoxPdCm"}],"uri":"incidents/mvR44baNg5v3UMTjsYr4"},{"id":"cjb5FEny3AJDi6xu9nXu","number":"16228721308366841646","begin":"2024-02-13T15:30:00+00:00","created":"2024-02-15T23:26:10+00:00","end":"2024-02-16T19:00:00+00:00","modified":"2024-02-28T17:32:31+00:00","external_desc":"We are investigating issues with Google Docs starting at 07:23 PT on 13-February-2024.\nThere is no workaround at this time","updates":[{"created":"2024-02-28T17:32:31+00:00","modified":"2024-02-28T17:32:31+00:00","when":"2024-02-28T17:32:31+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-20T21:12:10+00:00","modified":"2024-02-20T21:12:10+00:00","when":"2024-02-20T21:12:10+00:00","text":"A mini incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-16T23:15:49+00:00","modified":"2024-02-16T23:15:49+00:00","when":"2024-02-16T23:15:49+00:00","text":"We experienced an issue that impacted multiple products including Gmail, Google Calendar, Google Chat, Google Drive, Google Meet, Google Cloud Search, Google Voice, Google Tasks, Google Docs and Chrome OS.\nOur engineering team completed the mitigation actions and the backlog of the changes has cleared up for most of the customers.\nA small subset of customers may still experience delays with previously implemented changes to their workspace products and we expect full resolution within the next 24 hours.\nDuring the issue customers experienced longer than expected delays (sometimes over 24 hours) in propagating changes made to various Workspace products. These delays in some cases may have resulted in errors. The affected changes are not limited to enterprise setting changes.\nIf changes are taking longer than 24 hours going forward or you continue to see errors, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-16T19:33:37+00:00","modified":"2024-02-16T19:33:37+00:00","when":"2024-02-16T19:33:37+00:00","text":"Please refer to https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV for updates on this issue going forward.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T17:25:24+00:00","modified":"2024-02-16T17:27:16+00:00","when":"2024-02-16T17:25:24+00:00","text":"**Summary**\nWe are experiencing an issue with Google Docs.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 11:30 US/Pacific.\n**Customer symptom:**\n- New changes to enterprise settings done by customers should be working as expected.\n- Previously implemented changes may still take longer than expected to take effect as we process the backlog.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T13:50:00+00:00","modified":"2024-02-16T13:50:00+00:00","when":"2024-02-16T13:50:00+00:00","text":"**Summary**\nWe are experiencing an issue with Google Docs.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 09:15 US/Pacific.\n**Customer symptom:**\nNew changes to enterprise settings done by customers should be working as expected.\nPreviously implemented changes may still take longer than expected to take effect.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T08:43:46+00:00","modified":"2024-02-16T08:43:46+00:00","when":"2024-02-16T08:43:46+00:00","text":"**Summary**\nWe are experiencing an issue with Google Docs.\nMitigation work is currently underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 06:00 US/Pacific.\n**Customer symptom:**\nChanges to enterprise settings done by customers may take longer than expected to take effect\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T02:08:51+00:00","modified":"2024-02-16T07:35:53+00:00","when":"2024-02-16T02:08:51+00:00","text":"**Summary**\nWe are experiencing an issue with Google Docs.\nA rollback has been performed, but the issue persists. Our engineering team is continuing their investigation and working towards a solution.\nWe will provide more information by Friday, 2024-02-16 00:30 US/Pacific.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T01:05:19+00:00","modified":"2024-02-16T01:05:19+00:00","when":"2024-02-16T01:05:19+00:00","text":"**Summary**\nWe are experiencing an issue with Google Docs.\nThe engineers have identified a mitigation strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 18:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T00:21:18+00:00","modified":"2024-02-16T00:21:18+00:00","when":"2024-02-16T00:21:18+00:00","text":"**Summary**\nThe engineers have identified a mitigating strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 17:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-15T23:26:10+00:00","modified":"2024-02-15T23:26:10+00:00","when":"2024-02-15T23:26:10+00:00","text":"We are investigating issues with Google Docs starting at 07:23 PT on 13-February-2024.\nThere is no workaround at this time","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-28T17:32:31+00:00","modified":"2024-02-28T17:32:31+00:00","when":"2024-02-28T17:32:31+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"wNHuVFtZEWU5Mmj2eRCK","service_name":"Google Docs","affected_products":[{"title":"Google Docs","id":"wNHuVFtZEWU5Mmj2eRCK"}],"uri":"incidents/cjb5FEny3AJDi6xu9nXu"},{"id":"By2PE4BYiuCuJ1wZQ4XV","number":"15987434828945763696","begin":"2024-02-13T15:30:00+00:00","created":"2024-02-15T23:25:28+00:00","end":"2024-02-16T19:00:00+00:00","modified":"2024-02-28T17:28:05+00:00","external_desc":"We are investigating issues with Gmail starting at 07:23 PT on 13-February-2024.\nThere is no workaround at this time","updates":[{"created":"2024-02-28T17:27:36+00:00","modified":"2024-02-28T17:27:36+00:00","when":"2024-02-28T17:27:36+00:00","text":"# Incident Report\n## Summary\nBetween 13 February 2024 at 07:30 PT and 16 February 2024 at 11:00 PT, multiple Google Workspace products experienced propagation delays for enterprise customer policy changes for a duration of 3 days, 3 hours, and 30 minutes.\nTo our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nGoogle Workspace products expose settings through the Admin Console that Customer Admins can group together to create policies that govern how their users interact with Workspace products and services.\nWhen a Customer Admin makes changes to policy settings in Admin Console, those changes are processed by a compilation server, which computes effective settings for the applicable entities. This process includes optimizations to avoid the repeated processing of multiple changes that affect the same set of entities.\nOn 13 February 2024 at approximately 07:30 PT, an unusual pattern of policy changes triggered a latent issue where compilation optimizations did not work as intended, resulting in increased resource consumption, eventually leading to resource exhaustion. The diversion of resources away from useful work caused an increasing number of compilations to accumulate in the queue and consume resources, further exacerbating the latency.\n## Remediation and Prevention\nGoogle engineers were alerted to the issue via internal alerting on 13 February 2024 at 14:27 PT and immediately started an investigation. Engineers rolled back recent binary changes for the compilation process as a potential mitigation on 14 February 2024 at 15:35 PT, however this did not improve latency.\nEngineers continued to attempt further mitigation steps throughout the day on 15 February 2024, including pausing non-essential processing. Once the cause of the issue was identified at 20:23 PT, engineers put in place temporary measures to increase available memory for compilation while keeping non-essential processing paused, and tuned parameters that drive compilation optimizations. These temporary mitigations allowed compilation processing to begin to recover.\nCompilation processing returned to pre-outage levels by 16 February 2024, at 11:00 PT.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n* We have fixed the underlying issue that caused compilation optimizations to not work as intended as of 21 February 2024.\n* Introduce additional optimizations to improve compilation queue processing to better handle scenarios where a large number of changes affect the same set of entities.\n## Detailed Description of Impact\nBetween 13 February 2024 at 07:30 PT and 16 February 2024 at 11:00 PT, Google Workspace policy updates initiated by customer admins via Admin Console would have experienced delays in propagation.","status":"AVAILABLE"},{"created":"2024-02-20T21:08:20+00:00","modified":"2024-02-20T21:08:20+00:00","when":"2024-02-20T21:08:20+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 13 February 2024 08:00\n**Incident End:** 16 February 2024 14:37\n**Duration:** 3 days, 6 hours, 37 minutes\n**Affected Services and Features:**\nMultiple Google Workspace products including\n* Gmail\n* Google Calendar\n* Google Chat\n* Google Drive\n* Google Meet\n* Google Cloud Search\n* Google Voice\n* Google Tasks\n* Google Docs\n* Chrome OS\n**Regions/Zones:** Global\n**Description:**\nMultiple Workspace products experienced delays in propagating changes made to the products by the Workspace administrators for a duration of 3 days, 6 hours, 37 minutes. Some of the affected changes are changes to enterprise settings, policy related changes, changes to user accounts etc.\nFrom preliminary analysis, the root cause of the issue is elevated latency in job processing by backend systems that support the storage and compilation of Workspace policies for enterprise customers. This elevated latency in turn led to queuing of more jobs exhausting the memory resources required to complete the jobs.\nThe issue was mitigated by manually reconfiguring the service to allow the job processing to get back to previous levels.\n**Customer Impact:**\nDuring the issue customers experienced longer than expected delays (sometimes over 24 hours) in propagating changes made to various Workspace products. These delays in some cases may have resulted in errors.\n--------","status":"AVAILABLE"},{"created":"2024-02-16T23:10:51+00:00","modified":"2024-02-16T23:10:51+00:00","when":"2024-02-16T23:10:51+00:00","text":"We experienced an issue that impacted multiple products including Gmail, Google Calendar, Google Chat, Google Drive, Google Meet, Google Cloud Search, Google Voice, Google Tasks, Google Docs and Chrome OS.\nOur engineering team completed the mitigation actions and the backlog of the changes has cleared up for most of the customers.\nA small subset of customers may still experience delays with previously implemented changes to their workspace products and we expect full resolution within the next 24 hours.\nDuring the issue customers experienced longer than expected delays (sometimes over 24 hours) in propagating changes made to various Workspace products. These delays in some cases may have resulted in errors. The affected changes are not limited to enterprise setting changes.\nIf changes are taking longer than 24 hours going forward or you continue to see errors, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-16T19:30:08+00:00","modified":"2024-02-16T19:30:08+00:00","when":"2024-02-16T19:30:08+00:00","text":"**Summary:**\nWe are experiencing service issues with multiple Google Workspace products.\n**Description:**\nWe are experiencing an issue that is impacting multiple products including Gmail, Google Calendar, Google Chat, Google Drive, Google Meet, Google Cloud Search, Google Voice, Google Tasks, Google Docs and Chrome OS.\nOur engineering team completed the mitigation actions and the backlog of the enterprise setting changes has cleared up for most of the customers.\nA small subset of customers would still experience delays with previously implemented changes to their enterprise settings and we expect full resolution within the next 24 hours.\nWe will provide another update by Saturday, 2024-02-17 11:30 US/Pacific.\n**Customer symptom:**\n- Customers are experiencing longer than expected delays (sometimes over 24 hours) in propagating changes made to various Workspace products. These delays in some cases may result in errors. The affected changes are not limited to enterprise setting changes.\n- New changes to enterprise settings done by customers should be working as expected.\n- Previously implemented changes may still take longer than expected to take effect for a small subset of customers while we process the backlog .\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T17:22:10+00:00","modified":"2024-02-16T17:22:49+00:00","when":"2024-02-16T17:22:10+00:00","text":"**Summary**\nWe are experiencing an issue with Gmail.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 11:30 US/Pacific.\n**Customer symptom:**\n- New changes to enterprise settings done by customers should be working as expected.\n- Previously implemented changes may still take longer than expected to take effect as we process the backlog.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T13:54:08+00:00","modified":"2024-02-16T13:54:08+00:00","when":"2024-02-16T13:54:08+00:00","text":"**Summary**\nWe are experiencing an issue with Gmail.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 09:15 US/Pacific.\n**Customer symptom:**\nNew changes to enterprise settings done by customers should be working as expected.\nPreviously implemented changes may still take longer than expected to take effect.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T08:38:21+00:00","modified":"2024-02-16T08:38:21+00:00","when":"2024-02-16T08:38:21+00:00","text":"**Summary**\nWe are experiencing an issue with Gmail.\nMitigation work is currently underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 06:00 US/Pacific.\n**Customer symptom:**\nChanges to enterprise settings done by customers may take longer than expected to take effect\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T02:05:14+00:00","modified":"2024-02-16T07:27:46+00:00","when":"2024-02-16T02:05:14+00:00","text":"**Summary**\nWe are experiencing an issue with Gmail\nA rollback has been performed, but the issue persists. Our engineering team is continuing their investigation and working towards a solution.\nWe will provide more information by Friday, 2024-02-16 00:30 US/Pacific.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T00:58:10+00:00","modified":"2024-02-16T01:04:53+00:00","when":"2024-02-16T00:58:10+00:00","text":"**Summary**\nWe are experiencing an issue with Gmail.\nThe engineers have identified a mitigation strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 18:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T00:18:34+00:00","modified":"2024-02-16T00:18:34+00:00","when":"2024-02-16T00:18:34+00:00","text":"**Summary**\nThe engineers have identified a mitigating strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 17:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-15T23:25:30+00:00","modified":"2024-02-15T23:25:30+00:00","when":"2024-02-15T23:25:30+00:00","text":"We are investigating issues with Gmail starting at 07:23 PT on 13-February-2024.\nThere is no workaround at this time","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-28T17:27:36+00:00","modified":"2024-02-28T17:27:36+00:00","when":"2024-02-28T17:27:36+00:00","text":"# Incident Report\n## Summary\nBetween 13 February 2024 at 07:30 PT and 16 February 2024 at 11:00 PT, multiple Google Workspace products experienced propagation delays for enterprise customer policy changes for a duration of 3 days, 3 hours, and 30 minutes.\nTo our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nGoogle Workspace products expose settings through the Admin Console that Customer Admins can group together to create policies that govern how their users interact with Workspace products and services.\nWhen a Customer Admin makes changes to policy settings in Admin Console, those changes are processed by a compilation server, which computes effective settings for the applicable entities. This process includes optimizations to avoid the repeated processing of multiple changes that affect the same set of entities.\nOn 13 February 2024 at approximately 07:30 PT, an unusual pattern of policy changes triggered a latent issue where compilation optimizations did not work as intended, resulting in increased resource consumption, eventually leading to resource exhaustion. The diversion of resources away from useful work caused an increasing number of compilations to accumulate in the queue and consume resources, further exacerbating the latency.\n## Remediation and Prevention\nGoogle engineers were alerted to the issue via internal alerting on 13 February 2024 at 14:27 PT and immediately started an investigation. Engineers rolled back recent binary changes for the compilation process as a potential mitigation on 14 February 2024 at 15:35 PT, however this did not improve latency.\nEngineers continued to attempt further mitigation steps throughout the day on 15 February 2024, including pausing non-essential processing. Once the cause of the issue was identified at 20:23 PT, engineers put in place temporary measures to increase available memory for compilation while keeping non-essential processing paused, and tuned parameters that drive compilation optimizations. These temporary mitigations allowed compilation processing to begin to recover.\nCompilation processing returned to pre-outage levels by 16 February 2024, at 11:00 PT.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n* We have fixed the underlying issue that caused compilation optimizations to not work as intended as of 21 February 2024.\n* Introduce additional optimizations to improve compilation queue processing to better handle scenarios where a large number of changes affect the same set of entities.\n## Detailed Description of Impact\nBetween 13 February 2024 at 07:30 PT and 16 February 2024 at 11:00 PT, Google Workspace policy updates initiated by customer admins via Admin Console would have experienced delays in propagation.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/By2PE4BYiuCuJ1wZQ4XV"},{"id":"E6Qpq3HEwLS8R2CTynNi","number":"9874820750312052949","begin":"2024-02-13T15:30:00+00:00","created":"2024-02-15T23:30:47+00:00","end":"2024-02-16T19:00:00+00:00","modified":"2024-02-28T17:35:32+00:00","external_desc":"We are investigating issues with ChromeOS starting at 07:23 PT on 13-February-2024","updates":[{"created":"2024-02-28T17:34:53+00:00","modified":"2024-02-28T17:34:53+00:00","when":"2024-02-28T17:34:53+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-20T21:10:21+00:00","modified":"2024-02-20T21:10:21+00:00","when":"2024-02-20T21:10:21+00:00","text":"A mini incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-16T23:11:45+00:00","modified":"2024-02-16T23:11:45+00:00","when":"2024-02-16T23:11:45+00:00","text":"We experienced an issue that impacted multiple products including Gmail, Google Calendar, Google Chat, Google Drive, Google Meet, Google Cloud Search, Google Voice, Google Tasks, Google Docs and Chrome OS.\nOur engineering team completed the mitigation actions and the backlog of the changes has cleared up for most of the customers.\nA small subset of customers may still experience delays with previously implemented changes to their workspace products and we expect full resolution within the next 24 hours.\nDuring the issue customers experienced longer than expected delays (sometimes over 24 hours) in propagating changes made to various Workspace products. These delays in some cases may have resulted in errors. The affected changes are not limited to enterprise setting changes.\nIf changes are taking longer than 24 hours going forward or you continue to see errors, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-16T19:31:46+00:00","modified":"2024-02-16T19:31:46+00:00","when":"2024-02-16T19:31:46+00:00","text":"Please refer to https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV for updates on this issue going forward.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T17:23:24+00:00","modified":"2024-02-16T17:23:24+00:00","when":"2024-02-16T17:23:24+00:00","text":"**Summary**\nWe are experiencing an issue with ChromeOS.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 11:30 US/Pacific.\n**Customer symptom:**\n- New changes to enterprise settings done by customers should be working as expected.\n- Previously implemented changes may still take longer than expected to take effect as we process the backlog.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T13:53:24+00:00","modified":"2024-02-16T13:53:24+00:00","when":"2024-02-16T13:53:24+00:00","text":"**Summary**\nWe are experiencing an issue with ChromeOS.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 09:15 US/Pacific.\n**Customer symptom:**\nNew changes to enterprise settings done by customers should be working as expected.\nPreviously implemented changes may still take longer than expected to take effect.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T08:39:05+00:00","modified":"2024-02-16T08:39:05+00:00","when":"2024-02-16T08:39:05+00:00","text":"**Summary**\nWe are experiencing an issue with ChromeOS.\nMitigation work is currently underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 06:00 US/Pacific.\n**Customer symptom:**\nChanges to enterprise settings done by customers may take longer than expected to take effect\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T02:10:00+00:00","modified":"2024-02-16T07:30:03+00:00","when":"2024-02-16T02:10:00+00:00","text":"**Summary**\nWe are experiencing an issue with ChromeOS\nA rollback has been performed, but the issue persists. Our engineering team is continuing their investigation and working towards a solution.\nWe will provide more information by Friday, 2024-02-16 00:30 US/Pacific.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T00:18:08+00:00","modified":"2024-02-16T00:18:08+00:00","when":"2024-02-16T00:18:08+00:00","text":"**Summary**\nThe engineers have identified a mitigating strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 17:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-15T23:30:48+00:00","modified":"2024-02-15T23:30:48+00:00","when":"2024-02-15T23:30:48+00:00","text":"We are investigating issues with ChromeOS starting at 07:23 PT on 13-February-2024","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-28T17:34:53+00:00","modified":"2024-02-28T17:34:53+00:00","when":"2024-02-28T17:34:53+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"RjKbsvuGeE7M1pvBHmvX","service_name":"ChromeOS","affected_products":[{"title":"ChromeOS","id":"RjKbsvuGeE7M1pvBHmvX"}],"uri":"incidents/E6Qpq3HEwLS8R2CTynNi"},{"id":"A4uJ9SvoKy16DvCVgAg1","number":"13157613487705130506","begin":"2024-02-13T15:30:00+00:00","created":"2024-02-15T23:25:43+00:00","end":"2024-02-16T19:00:00+00:00","modified":"2024-02-28T17:35:21+00:00","external_desc":"We are investigating issues with Google Chat starting at 07:23 PT on 13-February-2024.\nThere is no workaround at this time.","updates":[{"created":"2024-02-28T17:35:08+00:00","modified":"2024-02-28T17:35:08+00:00","when":"2024-02-28T17:35:08+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-20T21:09:41+00:00","modified":"2024-02-20T21:09:41+00:00","when":"2024-02-20T21:09:41+00:00","text":"A mini incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-16T23:11:23+00:00","modified":"2024-02-16T23:11:23+00:00","when":"2024-02-16T23:11:23+00:00","text":"We experienced an issue that impacted multiple products including Gmail, Google Calendar, Google Chat, Google Drive, Google Meet, Google Cloud Search, Google Voice, Google Tasks, Google Docs and Chrome OS.\nOur engineering team completed the mitigation actions and the backlog of the changes has cleared up for most of the customers.\nA small subset of customers may still experience delays with previously implemented changes to their workspace products and we expect full resolution within the next 24 hours.\nDuring the issue customers experienced longer than expected delays (sometimes over 24 hours) in propagating changes made to various Workspace products. These delays in some cases may have resulted in errors. The affected changes are not limited to enterprise setting changes.\nIf changes are taking longer than 24 hours going forward or you continue to see errors, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-16T19:31:31+00:00","modified":"2024-02-16T19:31:31+00:00","when":"2024-02-16T19:31:31+00:00","text":"Please refer to https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV for updates on this issue going forward.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T17:21:32+00:00","modified":"2024-02-16T17:21:32+00:00","when":"2024-02-16T17:21:32+00:00","text":"**Summary**\nWe are experiencing an issue with Google Chat.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 11:30 US/Pacific.\n**Customer symptom:**\n- New changes to enterprise settings done by customers should be working as expected.\n- Previously implemented changes may still take longer than expected to take effect as we process the backlog.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T13:54:37+00:00","modified":"2024-02-16T13:54:37+00:00","when":"2024-02-16T13:54:37+00:00","text":"**Summary**\nWe are experiencing an issue with Google Chat.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 09:15 US/Pacific.\n**Customer symptom:**\nNew changes to enterprise settings done by customers should be working as expected.\nPreviously implemented changes may still take longer than expected to take effect.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T08:36:15+00:00","modified":"2024-02-16T08:36:15+00:00","when":"2024-02-16T08:36:15+00:00","text":"**Summary**\nWe are experiencing an issue with Google chat.\nMitigation work is currently underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 06:00 US/Pacific.\n**Customer symptom:**\nChanges to enterprise settings done by customers may take longer than expected to take effect\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T02:07:13+00:00","modified":"2024-02-16T07:22:26+00:00","when":"2024-02-16T02:07:13+00:00","text":"**Summary**\nWe are experiencing an issue with Google chat.\nA rollback has been performed, but the issue persists. Our engineering team is continuing their investigation and working towards a solution.\nWe will provide more information by Friday, 2024-02-16 00:30 US/Pacific.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T01:01:43+00:00","modified":"2024-02-16T01:01:43+00:00","when":"2024-02-16T01:01:43+00:00","text":"**Summary**\nWe are experiencing an issue with Google chat.\nThe engineers have identified a mitigation strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 18:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T00:13:58+00:00","modified":"2024-02-16T00:13:58+00:00","when":"2024-02-16T00:13:58+00:00","text":"**Summary**\nThe engineers have identified a mitigating strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 17:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-15T23:25:44+00:00","modified":"2024-02-15T23:25:44+00:00","when":"2024-02-15T23:25:44+00:00","text":"We are investigating issues with Google Chat starting at 07:23 PT on 13-February-2024.\nThere is no workaround at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-28T17:35:08+00:00","modified":"2024-02-28T17:35:08+00:00","when":"2024-02-28T17:35:08+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"Ht9Vx5PFzDPHYvrnrvSk","service_name":"Google Chat","affected_products":[{"title":"Google Chat","id":"Ht9Vx5PFzDPHYvrnrvSk"}],"uri":"incidents/A4uJ9SvoKy16DvCVgAg1"},{"id":"Y34LjSscuViZ673o5HT9","number":"16032236717615185227","begin":"2024-02-13T15:30:00+00:00","created":"2024-02-15T23:25:57+00:00","end":"2024-02-16T19:00:00+00:00","modified":"2024-02-28T17:36:08+00:00","external_desc":"We are investigating issues with Google Meet starting at 07:23 PT on 13-February-2024.\nThere is no workaround at this time","updates":[{"created":"2024-02-28T17:34:04+00:00","modified":"2024-02-28T17:34:04+00:00","when":"2024-02-28T17:34:04+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-20T21:11:25+00:00","modified":"2024-02-20T21:11:25+00:00","when":"2024-02-20T21:11:25+00:00","text":"A mini incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-16T23:13:54+00:00","modified":"2024-02-16T23:13:54+00:00","when":"2024-02-16T23:13:54+00:00","text":"We experienced an issue that impacted multiple products including Gmail, Google Calendar, Google Chat, Google Drive, Google Meet, Google Cloud Search, Google Voice, Google Tasks, Google Docs and Chrome OS.\nOur engineering team completed the mitigation actions and the backlog of the changes has cleared up for most of the customers.\nA small subset of customers may still experience delays with previously implemented changes to their workspace products and we expect full resolution within the next 24 hours.\nDuring the issue customers experienced longer than expected delays (sometimes over 24 hours) in propagating changes made to various Workspace products. These delays in some cases may have resulted in errors. The affected changes are not limited to enterprise setting changes.\nIf changes are taking longer than 24 hours going forward or you continue to see errors, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-16T19:33:02+00:00","modified":"2024-02-16T19:33:02+00:00","when":"2024-02-16T19:33:02+00:00","text":"Please refer to https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV for updates on this issue going forward.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T17:24:39+00:00","modified":"2024-02-16T17:24:39+00:00","when":"2024-02-16T17:24:39+00:00","text":"**Summary**\nWe are experiencing an issue with Google Meet.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 11:30 US/Pacific.\n**Customer symptom:**\n- New changes to enterprise settings done by customers should be working as expected.\n- Previously implemented changes may still take longer than expected to take effect as we process the backlog.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T13:51:32+00:00","modified":"2024-02-16T13:51:32+00:00","when":"2024-02-16T13:51:32+00:00","text":"**Summary**\nWe are experiencing an issue with Google Meet.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 09:15 US/Pacific.\n**Customer symptom:**\nNew changes to enterprise settings done by customers should be working as expected.\nPreviously implemented changes may still take longer than expected to take effect.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T08:41:17+00:00","modified":"2024-02-16T08:41:17+00:00","when":"2024-02-16T08:41:17+00:00","text":"**Summary**\nWe are experiencing an issue with Google Meet.\nMitigation work is currently underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 06:00 US/Pacific.\n**Customer symptom:**\nChanges to enterprise settings done by customers may take longer than expected to take effect\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T02:08:15+00:00","modified":"2024-02-16T07:33:46+00:00","when":"2024-02-16T02:08:15+00:00","text":"**Summary**\nWe are experiencing an issue with Google Meet.\nA rollback has been performed, but the issue persists. Our engineering team is continuing their investigation and working towards a solution.\nWe will provide more information by Friday, 2024-02-16 00:30 US/Pacific.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T01:02:50+00:00","modified":"2024-02-16T01:02:50+00:00","when":"2024-02-16T01:02:50+00:00","text":"**Summary**\nWe are experiencing an issue with Google meet.\nThe engineers have identified a mitigation strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 18:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T00:20:51+00:00","modified":"2024-02-16T00:20:51+00:00","when":"2024-02-16T00:20:51+00:00","text":"**Summary**\nThe engineers have identified a mitigating strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 17:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-15T23:25:57+00:00","modified":"2024-02-15T23:25:57+00:00","when":"2024-02-15T23:25:57+00:00","text":"We are investigating issues with Google Meet starting at 07:23 PT on 13-February-2024.\nThere is no workaround at this time","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-28T17:34:04+00:00","modified":"2024-02-28T17:34:04+00:00","when":"2024-02-28T17:34:04+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"sUH4BQXzYXma7NiS94Hi","service_name":"Google Meet","affected_products":[{"title":"Google Meet","id":"sUH4BQXzYXma7NiS94Hi"}],"uri":"incidents/Y34LjSscuViZ673o5HT9"},{"id":"LAGEnZ2zTgpMfcoUTBg4","number":"3569527912800347868","begin":"2024-02-13T15:30:00+00:00","created":"2024-02-15T23:25:38+00:00","end":"2024-02-16T19:00:00+00:00","modified":"2024-02-28T17:35:45+00:00","external_desc":"We are investigating issues with Google Calendar starting at 07:23 PT on 13-February-2024.\nThere is no workaround at this time","updates":[{"created":"2024-02-28T17:34:34+00:00","modified":"2024-02-28T17:34:34+00:00","when":"2024-02-28T17:34:34+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-20T21:10:35+00:00","modified":"2024-02-20T21:10:35+00:00","when":"2024-02-20T21:10:35+00:00","text":"A mini incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-16T23:12:09+00:00","modified":"2024-02-16T23:12:09+00:00","when":"2024-02-16T23:12:09+00:00","text":"We experienced an issue that impacted multiple products including Gmail, Google Calendar, Google Chat, Google Drive, Google Meet, Google Cloud Search, Google Voice, Google Tasks, Google Docs and Chrome OS.\nOur engineering team completed the mitigation actions and the backlog of the changes has cleared up for most of the customers.\nA small subset of customers may still experience delays with previously implemented changes to their workspace products and we expect full resolution within the next 24 hours.\nDuring the issue customers experienced longer than expected delays (sometimes over 24 hours) in propagating changes made to various Workspace products. These delays in some cases may have resulted in errors. The affected changes are not limited to enterprise setting changes.\nIf changes are taking longer than 24 hours going forward or you continue to see errors, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-16T19:32:01+00:00","modified":"2024-02-16T19:32:01+00:00","when":"2024-02-16T19:32:01+00:00","text":"Please refer to https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV for updates on this issue going forward.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T17:23:49+00:00","modified":"2024-02-16T17:23:49+00:00","when":"2024-02-16T17:23:49+00:00","text":"**Summary**\nWe are experiencing an issue with Google Calendar.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 11:30 US/Pacific.\n**Customer symptom:**\n- New changes to enterprise settings done by customers should be working as expected.\n- Previously implemented changes may still take longer than expected to take effect as we process the backlog.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T13:52:37+00:00","modified":"2024-02-16T13:52:37+00:00","when":"2024-02-16T13:52:37+00:00","text":"**Summary**\nWe are experiencing an issue with Google Calendar.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 09:15 US/Pacific.\n**Customer symptom:**\nNew changes to enterprise settings done by customers should be working as expected.\nPreviously implemented changes may still take longer than expected to take effect.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T08:39:53+00:00","modified":"2024-02-16T08:39:53+00:00","when":"2024-02-16T08:39:53+00:00","text":"**Summary**\nWe are experiencing an issue with Google Calendar.\nMitigation work is currently underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 06:00 US/Pacific.\n**Customer symptom:**\nChanges to enterprise settings done by customers may take longer than expected to take effect\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T02:06:06+00:00","modified":"2024-02-16T03:53:27+00:00","when":"2024-02-16T02:06:06+00:00","text":"**Summary**\nWe are experiencing an issue with Google Calendar.\nA rollback has been performed, but the issue persists. Our engineering team is continuing their investigation and working towards a solution.\nWe will provide more information by Friday, 2024-02-16 00:30 US/Pacific.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T00:59:27+00:00","modified":"2024-02-16T01:05:13+00:00","when":"2024-02-16T00:59:27+00:00","text":"**Summary**\nWe are experiencing an issue with Google Calendar.\nThe engineers have identified a mitigation strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 18:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T00:20:20+00:00","modified":"2024-02-16T00:20:20+00:00","when":"2024-02-16T00:20:20+00:00","text":"**Summary**\nThe engineers have identified a mitigating strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 17:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-15T23:25:39+00:00","modified":"2024-02-15T23:25:39+00:00","when":"2024-02-15T23:25:39+00:00","text":"We are investigating issues with Google Calendar starting at 07:23 PT on 13-February-2024.\nThere is no workaround at this time","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-28T17:34:34+00:00","modified":"2024-02-28T17:34:34+00:00","when":"2024-02-28T17:34:34+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"n5hVexuq1PM9onY9qrmo","service_name":"Google Calendar","affected_products":[{"title":"Google Calendar","id":"n5hVexuq1PM9onY9qrmo"}],"uri":"incidents/LAGEnZ2zTgpMfcoUTBg4"},{"id":"VXeA6EZJzVKE53BAaoae","number":"3540362291114407947","begin":"2024-02-13T15:30:00+00:00","created":"2024-02-15T23:25:40+00:00","end":"2024-02-16T19:00:00+00:00","modified":"2024-02-28T17:35:57+00:00","external_desc":"We are investigating issues with Google Drive starting at 07:23 PT on 13-February-2024. There is no workaround at this time.","updates":[{"created":"2024-02-28T17:34:17+00:00","modified":"2024-02-28T17:34:17+00:00","when":"2024-02-28T17:34:17+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-20T21:11:00+00:00","modified":"2024-02-20T21:11:00+00:00","when":"2024-02-20T21:11:00+00:00","text":"A mini incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},{"created":"2024-02-16T23:12:35+00:00","modified":"2024-02-16T23:12:35+00:00","when":"2024-02-16T23:12:35+00:00","text":"We experienced an issue that impacted multiple products including Gmail, Google Calendar, Google Chat, Google Drive, Google Meet, Google Cloud Search, Google Voice, Google Tasks, Google Docs and Chrome OS.\nOur engineering team completed the mitigation actions and the backlog of the changes has cleared up for most of the customers.\nA small subset of customers may still experience delays with previously implemented changes to their workspace products and we expect full resolution within the next 24 hours.\nDuring the issue customers experienced longer than expected delays (sometimes over 24 hours) in propagating changes made to various Workspace products. These delays in some cases may have resulted in errors. The affected changes are not limited to enterprise setting changes.\nIf changes are taking longer than 24 hours going forward or you continue to see errors, please open a case with the Support Team and we will work with you until this issue is resolved.","status":"AVAILABLE"},{"created":"2024-02-16T19:32:33+00:00","modified":"2024-02-16T19:32:33+00:00","when":"2024-02-16T19:32:33+00:00","text":"Please refer to https://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV for updates on this issue going forward.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T17:24:19+00:00","modified":"2024-02-16T17:24:19+00:00","when":"2024-02-16T17:24:19+00:00","text":"**Summary**\nWe are experiencing an issue with Google Drive.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 11:30 US/Pacific.\n**Customer symptom:**\n- New changes to enterprise settings done by customers should be working as expected.\n- Previously implemented changes may still take longer than expected to take effect as we process the backlog.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T13:52:01+00:00","modified":"2024-02-16T13:52:01+00:00","when":"2024-02-16T13:52:01+00:00","text":"**Summary**\nWe are experiencing an issue with Google Drive.\nMitigation work is still underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 09:15 US/Pacific.\n**Customer symptom:**\nNew changes to enterprise settings done by customers should be working as expected.\nPreviously implemented changes may still take longer than expected to take effect.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T08:40:44+00:00","modified":"2024-02-16T08:40:44+00:00","when":"2024-02-16T08:40:44+00:00","text":"**Summary**\nWe are experiencing an issue with Google Drive.\nMitigation work is currently underway by our engineering team and we believe that the issue has been mitigated for most customers.\nWe do not have an ETA for a complete mitigation at this point.\nWe will provide another update by Friday, 2024-02-16 06:00 US/Pacific.\n**Customer symptom:**\nChanges to enterprise settings done by customers may take longer than expected to take effect\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T02:08:43+00:00","modified":"2024-02-16T07:32:42+00:00","when":"2024-02-16T02:08:43+00:00","text":"**Summary**\nWe are experiencing an issue with Google Drive.\nA rollback has been performed, but the issue persists. Our engineering team is continuing their investigation and working towards a solution.\nWe will provide more information by Friday, 2024-02-16 00:30 US/Pacific.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-16T00:19:09+00:00","modified":"2024-02-16T00:19:09+00:00","when":"2024-02-16T00:19:09+00:00","text":"**Summary**\nThe engineers have identified a mitigating strategy and rolled it out. We do not have ETA for the mitigation to be completed.\nWe will provide more information by Thursday, 2024-02-15 17:15 US/Pacific.\nWe regret the inconvenience caused.\n**Customer symptom:**\nEnterprise settings being changed / edited by customers would take more time than normal to take effect and work as expected.\n**Workaround:**\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2024-02-15T23:25:40+00:00","modified":"2024-02-15T23:25:40+00:00","when":"2024-02-15T23:25:40+00:00","text":"We are investigating issues with Google Drive starting at 07:23 PT on 13-February-2024. There is no workaround at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2024-02-28T17:34:17+00:00","modified":"2024-02-28T17:34:17+00:00","when":"2024-02-28T17:34:17+00:00","text":"A detailed incident report has been posted at\nhttps://www.google.com/appsstatus/dashboard/incidents/By2PE4BYiuCuJ1wZQ4XV","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"VHNA7p3Z5p3iakj5sA8V","service_name":"Google Drive","affected_products":[{"title":"Google Drive","id":"VHNA7p3Z5p3iakj5sA8V"}],"uri":"incidents/VXeA6EZJzVKE53BAaoae"},{"id":"jXDsQKfxuz5DoSiMZC25","number":"11885224178787377939","begin":"2023-12-11T14:00:00+00:00","created":"2023-12-11T19:56:16+00:00","end":"2023-12-12T06:05:00+00:00","modified":"2023-12-12T06:09:58+00:00","external_desc":"We're aware of a problem with Google Chat affecting a significant subset of users. We will provide an update by Dec 11, 2023, 9:00 PM UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change. The affected users are unable to access Google Chat.","updates":[{"created":"2023-12-12T06:09:48+00:00","modified":"2023-12-12T06:09:48+00:00","when":"2023-12-12T06:09:48+00:00","text":"The issue with Google Chat has been resolved for all affected users as of Monday, 2023-12-11 22:05 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2023-12-12T01:29:59+00:00","modified":"2023-12-12T01:29:59+00:00","when":"2023-12-12T01:29:59+00:00","text":"**Summary:**\nSome Google Chat spaces are not loading when attempting to access the Chat space.\n**Description:**\nOur engineering team continues to work on mitigating the impact.\nThe mitigation is expected to complete by Monday, 2023-12-11 21:30 US/Pacific.\nWe will provide more information by Monday, 2023-12-11 23:30 US/Pacific.\nWe sincerely appreciate your patience and understanding as we work to resolve it as quickly as possible.\n**Diagnosis:**\nThe impacted customers will encounter \"Failed to Load Chat\" error message while accessing Google Chat space.\n**Workaround:**\nNone at this time.","status":"SERVICE_INFORMATION"},{"created":"2023-12-11T21:36:16+00:00","modified":"2023-12-11T21:36:16+00:00","when":"2023-12-11T21:36:16+00:00","text":"Mitigation work is currently underway by our engineering team.\nThe mitigation is expected to complete by Monday, 2023-12-11 18:00 US/Pacific.\nWe will provide more information by Monday, 2023-12-11 18:15 US/Pacific.\nCustomer Symptoms : Customers may observe issues with \"Failed to Load Chat\" toasts and UI loading issues with Google Chat.\nWorkaround : None at this time.","status":"SERVICE_INFORMATION"},{"created":"2023-12-11T20:59:26+00:00","modified":"2023-12-11T20:59:34+00:00","when":"2023-12-11T20:59:26+00:00","text":"We are experiencing an issue with Google Chat beginning at Friday, 2023-12-08.\nOur engineering team continues to investigate the issue and working on multiple mitigation strategies at the highest priority. We apologize for the service interruption.\nCustomers may observe issues with \"Failed to Load Chat\" toasts and UI loading issues with Google Chat.\nWe will provide an update by Monday, 2023-12-11 14:00 US/Pacific with current details.\nWorkaround : None at this time.","status":"SERVICE_INFORMATION"},{"created":"2023-12-11T19:56:17+00:00","modified":"2023-12-11T19:56:17+00:00","when":"2023-12-11T19:56:17+00:00","text":"We're aware of a problem with Google Chat affecting a significant subset of users. We will provide an update by Dec 11, 2023, 9:00 PM UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change. The affected users are unable to access Google Chat. Customers may observe issues with \"Failed to Load Chat\" toasts and UI loading issues with Google Chat. None at this time.","status":"SERVICE_INFORMATION"}],"most_recent_update":{"created":"2023-12-12T06:09:48+00:00","modified":"2023-12-12T06:09:48+00:00","when":"2023-12-12T06:09:48+00:00","text":"The issue with Google Chat has been resolved for all affected users as of Monday, 2023-12-11 22:05 US/Pacific.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},"status_impact":"SERVICE_INFORMATION","severity":"low","service_key":"Ht9Vx5PFzDPHYvrnrvSk","service_name":"Google Chat","affected_products":[{"title":"Google Chat","id":"Ht9Vx5PFzDPHYvrnrvSk"}],"uri":"incidents/jXDsQKfxuz5DoSiMZC25"},{"id":"CdDuxDjCM6PzMbwe7Jck","number":"8618882370325683493","begin":"2023-12-07T19:00:00+00:00","created":"2023-12-07T20:16:48+00:00","end":"2023-12-07T20:32:00+00:00","modified":"2023-12-14T00:37:09+00:00","external_desc":"We're investigating reports of an issue with Gmail. We will provide more information shortly.","updates":[{"created":"2023-12-14T00:37:09+00:00","modified":"2023-12-14T00:37:09+00:00","when":"2023-12-14T00:37:09+00:00","text":"A full incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},{"created":"2023-12-08T11:23:04+00:00","modified":"2023-12-08T15:31:06+00:00","when":"2023-12-08T11:23:04+00:00","text":"A mini incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},{"created":"2023-12-07T20:52:42+00:00","modified":"2023-12-07T20:52:42+00:00","when":"2023-12-07T20:52:42+00:00","text":"The problem with Gmail has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.","status":"AVAILABLE"},{"created":"2023-12-07T20:16:48+00:00","modified":"2023-12-07T20:37:25+00:00","when":"2023-12-07T20:16:48+00:00","text":"We're investigating reports of an issue with Gmail. We will provide more information shortly. Gmail users are unable to login. Users may see errors in the Google login page. Users can retry their login attempts if they see errors.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2023-12-14T00:37:09+00:00","modified":"2023-12-14T00:37:09+00:00","when":"2023-12-14T00:37:09+00:00","text":"A full incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/CdDuxDjCM6PzMbwe7Jck"},{"id":"fm2yv2H4BFHfmANwQro7","number":"8624690055516946968","begin":"2023-12-07T19:00:00+00:00","created":"2023-12-07T20:41:00+00:00","end":"2023-12-07T20:32:00+00:00","modified":"2023-12-14T00:37:54+00:00","external_desc":"We're investigating reports of an issue with Google Calendar. We will provide more information shortly.","updates":[{"created":"2023-12-14T00:37:54+00:00","modified":"2023-12-14T00:37:54+00:00","when":"2023-12-14T00:37:54+00:00","text":"A full incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},{"created":"2023-12-08T15:27:53+00:00","modified":"2023-12-08T15:27:53+00:00","when":"2023-12-08T15:27:53+00:00","text":"A mini incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},{"created":"2023-12-07T20:59:52+00:00","modified":"2023-12-07T20:59:52+00:00","when":"2023-12-07T20:59:52+00:00","text":"The problem with Google Calendar has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. During the issue, some users were unable to access Google Calendar services.","status":"AVAILABLE"},{"created":"2023-12-07T20:41:00+00:00","modified":"2023-12-07T20:41:00+00:00","when":"2023-12-07T20:41:00+00:00","text":"We're investigating reports of an issue with Google Calendar. We will provide more information shortly.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2023-12-14T00:37:54+00:00","modified":"2023-12-14T00:37:54+00:00","when":"2023-12-14T00:37:54+00:00","text":"A full incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"n5hVexuq1PM9onY9qrmo","service_name":"Google Calendar","affected_products":[{"title":"Google Calendar","id":"n5hVexuq1PM9onY9qrmo"}],"uri":"incidents/fm2yv2H4BFHfmANwQro7"},{"id":"Xd1eMTBRFYqp8vVA3zbf","number":"353312009958345674","begin":"2023-12-07T19:00:00+00:00","created":"2023-12-07T20:40:37+00:00","end":"2023-12-07T20:32:00+00:00","modified":"2023-12-14T00:39:35+00:00","external_desc":"We're investigating reports of an issue with Google Docs. We will provide more information shortly.","updates":[{"created":"2023-12-14T00:39:35+00:00","modified":"2023-12-14T00:39:35+00:00","when":"2023-12-14T00:39:35+00:00","text":"A full incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},{"created":"2023-12-08T15:25:42+00:00","modified":"2023-12-08T15:25:42+00:00","when":"2023-12-08T15:25:42+00:00","text":"A mini incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},{"created":"2023-12-07T20:57:59+00:00","modified":"2023-12-07T20:57:59+00:00","when":"2023-12-07T20:57:59+00:00","text":"The problem with Google Docs has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. During the issue, some users were unable to access Google Docs services.","status":"AVAILABLE"},{"created":"2023-12-07T20:40:37+00:00","modified":"2023-12-07T20:40:37+00:00","when":"2023-12-07T20:40:37+00:00","text":"We're investigating reports of an issue with Google Docs. We will provide more information shortly.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2023-12-14T00:39:35+00:00","modified":"2023-12-14T00:39:35+00:00","when":"2023-12-14T00:39:35+00:00","text":"A full incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"wNHuVFtZEWU5Mmj2eRCK","service_name":"Google Docs","affected_products":[{"title":"Google Docs","id":"wNHuVFtZEWU5Mmj2eRCK"}],"uri":"incidents/Xd1eMTBRFYqp8vVA3zbf"},{"id":"W1TidtMivxSwJRzVL8gu","number":"9967204862700307363","begin":"2023-12-07T19:00:00+00:00","created":"2023-12-07T20:40:22+00:00","end":"2023-12-07T20:32:00+00:00","modified":"2023-12-14T00:41:41+00:00","external_desc":"We're investigating reports of an issue with Google Forms. We will provide more information shortly.","updates":[{"created":"2023-12-14T00:41:41+00:00","modified":"2023-12-14T00:41:41+00:00","when":"2023-12-14T00:41:41+00:00","text":"A full incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},{"created":"2023-12-08T15:19:15+00:00","modified":"2023-12-08T15:26:59+00:00","when":"2023-12-08T15:19:15+00:00","text":"A mini incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},{"created":"2023-12-07T20:56:30+00:00","modified":"2023-12-07T20:56:30+00:00","when":"2023-12-07T20:56:30+00:00","text":"The problem with Google Forms has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. During the issue, some users were unable to access Google Forms services.","status":"AVAILABLE"},{"created":"2023-12-07T20:40:23+00:00","modified":"2023-12-07T20:40:23+00:00","when":"2023-12-07T20:40:23+00:00","text":"We're investigating reports of an issue with Google Forms. We will provide more information shortly.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2023-12-14T00:41:41+00:00","modified":"2023-12-14T00:41:41+00:00","when":"2023-12-14T00:41:41+00:00","text":"A full incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"3mtqLXm6QxeM6bAVkVez","service_name":"Google Forms","affected_products":[{"title":"Google Forms","id":"3mtqLXm6QxeM6bAVkVez"}],"uri":"incidents/W1TidtMivxSwJRzVL8gu"},{"id":"UPy9KLTygPr93GWrMiUX","number":"13562783280520663581","begin":"2023-12-07T19:00:00+00:00","created":"2023-12-07T20:41:56+00:00","end":"2023-12-07T20:32:00+00:00","modified":"2023-12-14T00:38:52+00:00","external_desc":"We're investigating reports of an issue with Google Chat. We will provide more information shortly.","updates":[{"created":"2023-12-14T00:38:52+00:00","modified":"2023-12-14T00:38:52+00:00","when":"2023-12-14T00:38:52+00:00","text":"A full incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},{"created":"2023-12-08T11:23:43+00:00","modified":"2023-12-08T15:29:47+00:00","when":"2023-12-08T11:23:43+00:00","text":"A mini incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},{"created":"2023-12-07T20:53:57+00:00","modified":"2023-12-07T20:53:57+00:00","when":"2023-12-07T20:53:57+00:00","text":"The problem with Google Chat has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. During the issue, some users were unable to access Google Chat services.","status":"AVAILABLE"},{"created":"2023-12-07T20:41:56+00:00","modified":"2023-12-07T20:41:56+00:00","when":"2023-12-07T20:41:56+00:00","text":"We're investigating reports of an issue with Google Chat. We will provide more information shortly.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2023-12-14T00:38:52+00:00","modified":"2023-12-14T00:38:52+00:00","when":"2023-12-14T00:38:52+00:00","text":"A full incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"Ht9Vx5PFzDPHYvrnrvSk","service_name":"Google Chat","affected_products":[{"title":"Google Chat","id":"Ht9Vx5PFzDPHYvrnrvSk"}],"uri":"incidents/UPy9KLTygPr93GWrMiUX"},{"id":"yBzNPqLpw1qh3YnJ69XL","number":"18431689773159722118","begin":"2023-12-07T19:00:00+00:00","created":"2023-12-07T20:38:19+00:00","end":"2023-12-07T20:32:00+00:00","modified":"2023-12-14T00:40:21+00:00","external_desc":"We're investigating reports of an issue with Google Drive. We will provide more information shortly.","updates":[{"created":"2023-12-14T00:40:21+00:00","modified":"2023-12-14T00:40:21+00:00","when":"2023-12-14T00:40:21+00:00","text":"A full incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},{"created":"2023-12-08T15:29:05+00:00","modified":"2023-12-08T15:29:05+00:00","when":"2023-12-08T15:29:05+00:00","text":"A mini incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},{"created":"2023-12-07T20:55:47+00:00","modified":"2023-12-07T20:55:47+00:00","when":"2023-12-07T20:55:47+00:00","text":"The problem with Google Drive has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. During the issue, some users were unable to access Google Drive services.","status":"AVAILABLE"},{"created":"2023-12-07T20:38:19+00:00","modified":"2023-12-08T15:32:35+00:00","when":"2023-12-07T20:38:19+00:00","text":"We're investigating reports of an issue with Google Drive. We will provide more information shortly.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2023-12-14T00:40:21+00:00","modified":"2023-12-14T00:40:21+00:00","when":"2023-12-14T00:40:21+00:00","text":"A full incident report has been posted on the Google Workspace Status Dashboard [1].\n[1]. https://www.google.com/appsstatus/dashboard/incidents/jPWZ4p7bTSq65P1b9Gfn","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"VHNA7p3Z5p3iakj5sA8V","service_name":"Google Drive","affected_products":[{"title":"Google Drive","id":"VHNA7p3Z5p3iakj5sA8V"}],"uri":"incidents/yBzNPqLpw1qh3YnJ69XL"},{"id":"jPWZ4p7bTSq65P1b9Gfn","number":"6220459159394509537","begin":"2023-12-07T18:00:00+00:00","created":"2023-12-07T20:39:05+00:00","end":"2023-12-07T20:32:00+00:00","modified":"2023-12-14T00:45:03+00:00","external_desc":"We're investigating reports of an issue with Classroom. We will provide more information shortly.","updates":[{"created":"2023-12-14T00:35:25+00:00","modified":"2023-12-14T00:45:03+00:00","when":"2023-12-14T00:35:25+00:00","text":"# Incident Report\n## Summary\nOn 07 December 2023, various Google Workspace and Google Cloud customers experienced intermittent authentication issues when routed to one of our data centers in the US East region. As a result, a limited number of users could not sign-up, login, or re-authenticate for a duration of 1 hour and 32 minutes.\nTo our Google Workspace and Google Cloud customers who were impacted during this issue, we sincerely apologize – this is not the level of quality and reliability we strive to offer you.\n## Root Cause\nGoogle’s authentication stack is broad and deep, running across every region and zone to provide functionality globally across all Google products. This particular issue was scoped to a single microservice's serving jobs within a single data center; this microservice handles serving some of the user-facing pages involved in sign-up, login, and re-authentication flows.\nTo improve resource usage efficiency, engineers had been implementing more advanced autoscaling algorithms across these serving jobs globally. These changes were rolled out gradually over a period of several weeks without any issues.\nOn 7 December, the rollout kicked off rolling restarts across the individual tasks within the serving jobs in a single data center in the US East region. Due to internal network routing at the time, this data center was serving traffic from the US East area as well as South America. These factors placed additional demand on each task's thread pool, both overloading some tasks and affecting those tasks' ability to report metrics back to load-balancers. This caused requests to be intermittently dropped between the load-balancer and the individual serving tasks; these dropped requests led to user-facing 500 error pages.\n## Remediation and Prevention\nGoogle engineers were alerted by our internal monitoring system on 07 December at 11:16 US/Pacific and immediately started investigating the issue. Once the nature and scope of the issue became clear, Google engineers worked to confirm that traffic could be safely re-routed away from the impacted serving jobs as a mitigation strategy. Traffic was rerouted to other data centers, and the customer impact was mitigated by 12:32 US/Pacific. Our engineers then reviewed and tuned the stability and performance of both this and related serving jobs globally before declaring the incident fully resolved.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business. We are committed to preventing a repeat of this issue in the future and are completing the following actions:\n* Enhance metrics and dashboards to make these complex load-balancing dynamics more legible to engineers, reducing the time to mitigate similar future issues.\n* Improve performance-testing of resource-allocation profiles for each individual serving job\n## Detailed Description of Impact\n**Google Workspace:**\n* During the issue, a limited number Google Workspace users served from this data center had issues with log-in and sign-up for part of unauthenticated traffic, including some users going through \"Sign in with Google '' interactive flows using their Google Workspace account\n* Multiple Google Workspace product log-in attempts, reauthentication attempts, and some new account creation requests from users routed to this data center failed with error 502s.\n* Users that were already logged in were not affected by the issue.\n**Google OAuth:**\n* Some users served from this data center intermittently experienced error 502s when attempting to \"Sign in with Google\" and encountering interactive sign-in or re-authentication flows.\n**Google Cloud Console:**\n* A limited number of users served from this data center experienced error 502s intermittently when attempting to log-in to the Cloud Console.\n---------","status":"AVAILABLE"},{"created":"2023-12-08T11:24:04+00:00","modified":"2023-12-08T11:24:04+00:00","when":"2023-12-08T11:24:04+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues.\nIf you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 7 December 2023 11:00\n**Incident End:** 7 December 2023 12:32\n**Duration:** 1 hours, 32 minutes\n**Affected Services and Features:**\nSign-up and log-in for parts of unauthenticated traffic on Google Workspace and Google Cloud products.\n**Regions/Zones:** us-east (partial impact)\n**Description:**\nStarting on 7 December 2023 at 11:00, users trying to sign up or login to Google Workspace products or Google Cloud Console services in the us-east region experienced intermittent issues for a duration of 1 hour and 32 minutes.\nFrom our preliminary investigation, this was likely caused by performance issues on the sign up and login frontend servers in one of the datacenters in the affected region. To mitigate the impact, our engineering team rerouted traffic and optimized server distribution. Service was restored on 7 December 2023 at 12:32.\nGoogle will complete a detailed Incident Report in the following days that will provide a full root cause.\n**Customer Impact:**\nSome unauthenticated users in parts of the us-east region intermittently experienced difficulties accessing the impacted products. These users saw 502 errors when trying to sign up, login, or re-authenticate services.","status":"AVAILABLE"},{"created":"2023-12-07T20:53:26+00:00","modified":"2023-12-07T20:53:26+00:00","when":"2023-12-07T20:53:26+00:00","text":"The problem with Classroom has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. During the issue, some users were unable to access Classroom services.","status":"AVAILABLE"},{"created":"2023-12-07T20:39:05+00:00","modified":"2023-12-07T20:46:35+00:00","when":"2023-12-07T20:39:05+00:00","text":"We're investigating reports of an issue with Classroom. We will provide more information shortly.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2023-12-14T00:35:25+00:00","modified":"2023-12-14T00:45:03+00:00","when":"2023-12-14T00:35:25+00:00","text":"# Incident Report\n## Summary\nOn 07 December 2023, various Google Workspace and Google Cloud customers experienced intermittent authentication issues when routed to one of our data centers in the US East region. As a result, a limited number of users could not sign-up, login, or re-authenticate for a duration of 1 hour and 32 minutes.\nTo our Google Workspace and Google Cloud customers who were impacted during this issue, we sincerely apologize – this is not the level of quality and reliability we strive to offer you.\n## Root Cause\nGoogle’s authentication stack is broad and deep, running across every region and zone to provide functionality globally across all Google products. This particular issue was scoped to a single microservice's serving jobs within a single data center; this microservice handles serving some of the user-facing pages involved in sign-up, login, and re-authentication flows.\nTo improve resource usage efficiency, engineers had been implementing more advanced autoscaling algorithms across these serving jobs globally. These changes were rolled out gradually over a period of several weeks without any issues.\nOn 7 December, the rollout kicked off rolling restarts across the individual tasks within the serving jobs in a single data center in the US East region. Due to internal network routing at the time, this data center was serving traffic from the US East area as well as South America. These factors placed additional demand on each task's thread pool, both overloading some tasks and affecting those tasks' ability to report metrics back to load-balancers. This caused requests to be intermittently dropped between the load-balancer and the individual serving tasks; these dropped requests led to user-facing 500 error pages.\n## Remediation and Prevention\nGoogle engineers were alerted by our internal monitoring system on 07 December at 11:16 US/Pacific and immediately started investigating the issue. Once the nature and scope of the issue became clear, Google engineers worked to confirm that traffic could be safely re-routed away from the impacted serving jobs as a mitigation strategy. Traffic was rerouted to other data centers, and the customer impact was mitigated by 12:32 US/Pacific. Our engineers then reviewed and tuned the stability and performance of both this and related serving jobs globally before declaring the incident fully resolved.\nGoogle is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business. We are committed to preventing a repeat of this issue in the future and are completing the following actions:\n* Enhance metrics and dashboards to make these complex load-balancing dynamics more legible to engineers, reducing the time to mitigate similar future issues.\n* Improve performance-testing of resource-allocation profiles for each individual serving job\n## Detailed Description of Impact\n**Google Workspace:**\n* During the issue, a limited number Google Workspace users served from this data center had issues with log-in and sign-up for part of unauthenticated traffic, including some users going through \"Sign in with Google '' interactive flows using their Google Workspace account\n* Multiple Google Workspace product log-in attempts, reauthentication attempts, and some new account creation requests from users routed to this data center failed with error 502s.\n* Users that were already logged in were not affected by the issue.\n**Google OAuth:**\n* Some users served from this data center intermittently experienced error 502s when attempting to \"Sign in with Google\" and encountering interactive sign-in or re-authentication flows.\n**Google Cloud Console:**\n* A limited number of users served from this data center experienced error 502s intermittently when attempting to log-in to the Cloud Console.\n---------","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"3GoNVrVz48jeNZSyLJMV","service_name":"Classroom","affected_products":[{"title":"Classroom","id":"3GoNVrVz48jeNZSyLJMV"}],"uri":"incidents/jPWZ4p7bTSq65P1b9Gfn"},{"id":"LWGeK5YBtJL8kE4nfmNn","number":"2040168905245794817","begin":"2023-12-01T18:52:00+00:00","created":"2023-12-01T19:24:56+00:00","end":"2023-12-01T20:20:00+00:00","modified":"2023-12-07T00:19:26+00:00","external_desc":"We are experiencing an issue with Gmail beginning at Friday, 2023-12-01 10:52 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Friday, 2023-12-01 12:15 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\nCustomer Symptoms : Customers may experience delays in their email delivery\nWorkaround : None at this time.","updates":[{"created":"2023-12-07T00:19:26+00:00","modified":"2023-12-07T00:19:26+00:00","when":"2023-12-07T00:19:26+00:00","text":"The Incident Report is now available on the Google Workspace Status Dashboard : https://www.google.com/appsstatus/dashboard/incidents/Hjq8CpkhWroicw7R1wSh","status":"AVAILABLE"},{"created":"2023-12-01T20:36:58+00:00","modified":"2023-12-01T20:36:58+00:00","when":"2023-12-01T20:36:58+00:00","text":"The issue with Gmail has been resolved for all affected users as of Friday, 2023-12-01 12:20 US/Pacific.\nWe will publish an analysis of this incident once we have completed our internal investigation.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2023-12-01T20:16:26+00:00","modified":"2023-12-01T20:16:26+00:00","when":"2023-12-01T20:16:26+00:00","text":"Mitigation work is currently underway by our engineering team. Our engineers are seeing positive signs of recovery.\nHowever we are actively monitoring to ensure the systems are fully recovered.\nWe will provide more information by Friday, 2023-12-01 13:00 US/Pacific.\nCustomer Symptoms : Customers may experience delays in their email delivery. Customers may not be required to re-send the affected messages unless they bounce back. Customers with messages traversing in our US-based servers may be impacted.\nWorkaround : No action required from Customers unless the email that has bounced back.","status":"SERVICE_DISRUPTION"},{"created":"2023-12-01T20:06:15+00:00","modified":"2023-12-01T20:06:15+00:00","when":"2023-12-01T20:06:15+00:00","text":"We are experiencing an issue with Gmail beginning at Friday, 2023-12-01 10:52 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Friday, 2023-12-01 12:45 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\nCustomer Symptoms : Customers may experience delays in their email delivery. Customers may not be required to re-send the affected messages unless they bounce back. Customers with messages traversing in our US-based servers may be impacted.\nWorkaround : No action required from Customers unless the email that has bounced back.","status":"SERVICE_DISRUPTION"},{"created":"2023-12-01T19:24:57+00:00","modified":"2023-12-01T19:24:57+00:00","when":"2023-12-01T19:24:57+00:00","text":"We are experiencing an issue with Gmail beginning at Friday, 2023-12-01 10:52 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Friday, 2023-12-01 12:15 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\nCustomer Symptoms : Customers may experience delays in their email delivery\nWorkaround : None at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2023-12-07T00:19:26+00:00","modified":"2023-12-07T00:19:26+00:00","when":"2023-12-07T00:19:26+00:00","text":"The Incident Report is now available on the Google Workspace Status Dashboard : https://www.google.com/appsstatus/dashboard/incidents/Hjq8CpkhWroicw7R1wSh","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/LWGeK5YBtJL8kE4nfmNn"},{"id":"Hjq8CpkhWroicw7R1wSh","number":"16943205783932140564","begin":"2023-11-30T16:30:00+00:00","created":"2023-11-30T17:31:53+00:00","end":"2023-11-30T21:15:00+00:00","modified":"2023-12-07T00:15:54+00:00","external_desc":"We are experiencing an issue with Gmail beginning on Thursday, 2023-11-30 08:30 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Thursday, 2023-11-30 10:30 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\nCustomer Symptoms :\nCustomers impacted by this issue may see delay in sending the emails.\nWorkaround:\nNone at this time.","updates":[{"created":"2023-12-07T00:15:54+00:00","modified":"2023-12-07T00:15:54+00:00","when":"2023-12-07T00:15:54+00:00","text":"# Incident Report\n## Summary\nGmail’s core message processing system experienced delays for email delivery on Thursday, 30 November 2023 and Friday, 01 December 2023, for a cumulative duration of 6 hours and 23 minutes. To our Gmail customers whose email delivery was impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nGoogle runs a core message processing system that is responsible for delivering, monitoring, queuing, and receiving email messages. The root cause of both incidents was an atypical pattern of email traffic, resulting in high resource utilization in this system and delays in email delivery.\n## Remediation and Prevention\nOn both occasions, Google engineers were alerted to the delays via monitoring alerts at 08:39 US/Pacific on 30 November and again at 10:57 US/Pacific on 01 December.\nWe immediately began mitigation efforts, including increasing capacity for the impacted regions. We have also implemented reasonable limits to prevent such an atypical traffic pattern from recurring.\nWe apologize for the recurrence and severity of this incident and are taking the following step to improve reliability in the future:\n- Implement rate limit mechanisms in our systems.\n## Detailed Description of Impact\nOn Thursday, 30 November between 08:30 and 13:15 US/Pacific, and on Friday, 01 December between 10:52 and 12:30 US/Pacific, Gmail users experienced email delivery delays for at least 5 minutes. During this period, users may also have observed an increase in rejected emails.","status":"AVAILABLE"},{"created":"2023-12-01T01:43:16+00:00","modified":"2023-12-01T01:43:16+00:00","when":"2023-12-01T01:43:16+00:00","text":"# Mini Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n(All Times US/Pacific)\n**Incident Start:** 30 November 2023 08:30\n**Incident End:** 30 November 2023 13:15\n**Duration:** 4 hours, 45 minutes\n**Affected Services and Features:** Gmail\n**Regions/Zones:** Global\n**Description:**\nSome Gmail users experienced a delay in sending emails globally for a duration of 4 hours and 45 minutes. Any emails that were delayed after entering our system were auto retried and should have been delivered within a couple of hours after mitigation. From preliminary analysis, the root cause of the issue appears to be an overload in Google Message Router (GMR). Google will complete a full incident report in the following days that will provide a detailed root cause.\n**Customer Impact:**\nCustomers experienced delays while sending the emails.","status":"AVAILABLE"},{"created":"2023-11-30T22:03:26+00:00","modified":"2023-11-30T22:03:26+00:00","when":"2023-11-30T22:03:26+00:00","text":"The issue with email delays was mitigated for the majority of affected users at 11:45 US/Pacific.\nA few users may have observed delays until the full mitigation was done at 13:15 US/Pacific.\nWe will publish an analysis of this incident once we have completed our internal investigation.\nWe thank you for your patience while we worked on resolving the issue.","status":"AVAILABLE"},{"created":"2023-11-30T20:46:53+00:00","modified":"2023-11-30T20:46:53+00:00","when":"2023-11-30T20:46:53+00:00","text":"Google engineers have taken required mitigation measures.\nNew emails should generally not be seeing any issues. However older emails will be auto retried and should be delivered within the next couple of hours.\nOur engineers continue to monitor the issue until full recovery.\nWe will provide more information by Thursday, 2023-11-30 13:45 US/Pacific.\nCustomer Symptoms:\nCustomers impacted by this issue may see delay while sending the emails.\nWorkaround:\nCustomers can retry sending the email.","status":"SERVICE_DISRUPTION"},{"created":"2023-11-30T20:04:58+00:00","modified":"2023-11-30T20:04:58+00:00","when":"2023-11-30T20:04:58+00:00","text":"Google engineers have taken required mitigation measures.\nNew emails should generally not be seeing any issues. However older emails will be auto retried and should be delivered within the next couple of hours.\nOur engineers continue to monitor the issue until full recovery.\nWe will provide more information by Thursday, 2023-11-30 12:45 US/Pacific.\nCustomer Symptoms:\nCustomers impacted by this issue may see delay for a few mins while sending the emails.\nWorkaround:\nCustomers can retry sending the email.","status":"SERVICE_DISRUPTION"},{"created":"2023-11-30T19:13:05+00:00","modified":"2023-11-30T19:13:05+00:00","when":"2023-11-30T19:13:05+00:00","text":"Mitigation work is currently underway by our engineering team.\nOur engineers have taken a few mitigation steps at the highest priority and are actively validating the results.\nWe will provide more information by Thursday, 2023-11-30 12:00 US/Pacific.\nCustomer Symptoms:\nCustomers impacted by this issue may see delay for a few mins while sending the emails.\nWorkaround:\nCustomers can retry sending the email.","status":"SERVICE_DISRUPTION"},{"created":"2023-11-30T18:22:55+00:00","modified":"2023-11-30T18:22:55+00:00","when":"2023-11-30T18:22:55+00:00","text":"Mitigation work is currently underway by our engineering team.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Thursday, 2023-11-30 11:15 US/Pacific.\nCustomer Symptoms:\nCustomers impacted by this issue may see delay for a few mins while sending the emails.\nWorkaround:\nCustomers can retry sending the email.","status":"SERVICE_DISRUPTION"},{"created":"2023-11-30T17:41:17+00:00","modified":"2023-11-30T17:41:17+00:00","when":"2023-11-30T17:41:17+00:00","text":"Mitigation work is currently underway by our engineering team.\nWe do not have an ETA for mitigation at this point.\nWe will provide more information by Thursday, 2023-11-30 10:30 US/Pacific.\nCustomer Symptoms:\nCustomers impacted by this issue may see delay in sending the emails.\nWorkaround:\nNone at this time.","status":"SERVICE_DISRUPTION"},{"created":"2023-11-30T17:31:53+00:00","modified":"2023-11-30T17:31:53+00:00","when":"2023-11-30T17:31:53+00:00","text":"We are experiencing an issue with Gmail beginning on Thursday, 2023-11-30 08:30 US/Pacific.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Thursday, 2023-11-30 10:30 US/Pacific with current details.\nWe apologize to all who are affected by the disruption.\nCustomer Symptoms :\nCustomers impacted by this issue may see delay in sending the emails.\nWorkaround:\nNone at this time.","status":"SERVICE_DISRUPTION"}],"most_recent_update":{"created":"2023-12-07T00:15:54+00:00","modified":"2023-12-07T00:15:54+00:00","when":"2023-12-07T00:15:54+00:00","text":"# Incident Report\n## Summary\nGmail’s core message processing system experienced delays for email delivery on Thursday, 30 November 2023 and Friday, 01 December 2023, for a cumulative duration of 6 hours and 23 minutes. To our Gmail customers whose email delivery was impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nGoogle runs a core message processing system that is responsible for delivering, monitoring, queuing, and receiving email messages. The root cause of both incidents was an atypical pattern of email traffic, resulting in high resource utilization in this system and delays in email delivery.\n## Remediation and Prevention\nOn both occasions, Google engineers were alerted to the delays via monitoring alerts at 08:39 US/Pacific on 30 November and again at 10:57 US/Pacific on 01 December.\nWe immediately began mitigation efforts, including increasing capacity for the impacted regions. We have also implemented reasonable limits to prevent such an atypical traffic pattern from recurring.\nWe apologize for the recurrence and severity of this incident and are taking the following step to improve reliability in the future:\n- Implement rate limit mechanisms in our systems.\n## Detailed Description of Impact\nOn Thursday, 30 November between 08:30 and 13:15 US/Pacific, and on Friday, 01 December between 10:52 and 12:30 US/Pacific, Gmail users experienced email delivery delays for at least 5 minutes. During this period, users may also have observed an increase in rejected emails.","status":"AVAILABLE"},"status_impact":"SERVICE_DISRUPTION","severity":"medium","service_key":"prPt2Yra2CbGsbEm9cpC","service_name":"Gmail","affected_products":[{"title":"Gmail","id":"prPt2Yra2CbGsbEm9cpC"}],"uri":"incidents/Hjq8CpkhWroicw7R1wSh"}]