<?xml version="1.0" encoding="UTF-8"?> <feed xmlns="http://www.w3.org/2005/Atom"> <title>Google Workspace Status Dashboard updates</title> <updated>2024-11-22T17:10:18+00:00</updated> <link href="https://www.google.com/appsstatus/dashboard/" rel="alternate" type="text/html"></link> <link href="https://www.google.com/appsstatus/dashboard/en-SG/feed.atom" rel="self"></link> <author> <name>Google Workspace</name> </author> <id>https://www.google.com/appsstatus/dashboard/</id> <entry> <title>RESOLVED: Incident at 19 Nov 2024, 6:44鈥痯m UTC</title> <link href="https://www.google.com/appsstatus/dashboard/incidents/vU5ZcqrdVSm1zxxY1E5Q" rel="alternate" type="text/html"></link> <id>tag:www.google.com,2024:feed:vU5ZcqrdVSm1zxxY1E5Q.b328NgzoPHWSA3VCCgGf</id> <updated>2024-11-22T17:10:18+00:00</updated> <summary type="html">&lt;p&gt; Incident began at &lt;strong&gt;2024-11-19 18:44&lt;/strong&gt; and ended at &lt;strong&gt;2024-11-19 20:24&lt;/strong&gt; &lt;span&gt;(Times are in &lt;strong&gt;Coordinated Universal Time (UTC)&lt;/strong&gt;.)&lt;/span&gt;&lt;/p&gt;&lt;div class="cBIRi14aVDP__status-update-text"&gt;&lt;h1&gt;Incident Report&lt;/h1&gt; &lt;h2&gt;Summary&lt;/h2&gt; &lt;p&gt;On 19 November 2024 at 10:44 US/Pacific, Google Calendar experienced service disruptions globally for web users due to a misconfigured denial-of-service rule. This resulted in some users being unable to access their calendars for a duration of 1 hour and 40 minutes. Google Calendar鈥檚 mobile application users were not impacted.&lt;/p&gt; &lt;p&gt;To our Google Calendar users impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform鈥檚 performance and availability.&lt;/p&gt; &lt;h2&gt;Root Cause&lt;/h2&gt; &lt;p&gt;The incident was triggered by the implementation of a denial-of-service (DoS) rule, intended to mitigate malicious traffic. However, a misconfiguration led to the rule being applied to all traffic instead of traffic to a specific endpoint. Consequently, some users who triggered the rule may have had all of their web traffic to Google Calendar blocked, resulting in a denial of service for those users.&lt;/p&gt; &lt;p&gt;The root cause was an incorrect scope for a rule within the DoS configuration. The rule was applied too broadly with the purpose of blocking malicious traffic, and led to unintended blocking of legitimate user traffic.&lt;/p&gt; &lt;p&gt;Internal monitoring shows a traffic anomaly during the incident, but no alerts were triggered.&lt;/p&gt; &lt;h2&gt;Remediation and Prevention&lt;/h2&gt; &lt;p&gt;The issue was escalated by the support team to the engineering team at 12:12 US/Pacific. The on-call Engineer identified the misconfiguration and rolled back the faulty change to mitigate the issue at 12:24 US/Pacific.&lt;/p&gt; &lt;p&gt;Google is committed preventing a repeat of this issue in the future and is completing the following actions:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Ensuring oncall engineers are alerted when unusual amounts of traffic are blocked.&lt;/li&gt; &lt;li&gt;Updating processes and documentation around denial-of-service configurations to prevent recurrence.&lt;/li&gt; &lt;li&gt;Creating internal trainings on best practices for denial-of-service configurations to prevent recurrence.&lt;/li&gt; &lt;li&gt;Investigating validation mechanisms to catch bad configurations.&lt;/li&gt; &lt;/ul&gt; &lt;h2&gt;Detailed Description of Impact&lt;/h2&gt; &lt;p&gt;Google Calendar:&lt;/p&gt; &lt;p&gt;On 19 November 2024 at 10:44 US/Pacific,&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Google Calendar web users globally were impacted.&lt;/li&gt; &lt;li&gt;The primary impact was the inability to access calendar functionalities after transferring an event.&lt;/li&gt; &lt;li&gt;The issue was mitigated at 12:24 US/Pacific.&lt;/li&gt; &lt;/ul&gt; &lt;/div&gt;&lt;hr&gt;&lt;p&gt;Affected products: Google Calendar&lt;/p&gt;</summary> </entry> </feed>