GitHub Status - Incident History

<?xml version="1.0" encoding="UTF-8"?> <rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/"> <channel> <title>GitHub Status - Incident History</title> <link>https://www.githubstatus.com</link> <description>Statuspage</description> <pubDate>Mon, 31 Mar 2025 16:27:33 +0000</pubDate> <item> <title>Disruption with some GitHub services</title> <description> Mar <var data-var='date'>31</var>, <var data-var='time'>16:27</var> UTC Investigating - We are currently investigating this issue. </description> <pubDate>Mon, 31 Mar 2025 16:27:33 +0000</pubDate> <link>https://www.githubstatus.com/incidents/thpkk85p1sjy</link> <guid>https://www.githubstatus.com/incidents/thpkk85p1sjy</guid> </item> <item> <title>[Retroactive] Disruption with Pull Request Ref Updates</title> <description> Mar <var data-var='date'>28</var>, <var data-var='time'>22:50</var> UTC Resolved - Beginning at 21:24 UTC on March 28 and lasting until 21:50 UTC, some customers of github.com had issues with PR tracking refs not being updated due to processing delays and increased failure rates. We did not status before we completed the rollback, and the incident is currently resolved. We are sorry for the delayed post on githubstatus.com. </description> <pubDate>Fri, 28 Mar 2025 22:50:53 +0000</pubDate> <link>https://www.githubstatus.com/incidents/7s7z64qbl194</link> <guid>https://www.githubstatus.com/incidents/7s7z64qbl194</guid> </item> <item> <title>Disruption with some GitHub services</title> <description> Mar <var data-var='date'>28</var>, <var data-var='time'>18:14</var> UTC Resolved - This incident was opened by mistake. Public services are currently functional.Mar <var data-var='date'>28</var>, <var data-var='time'>17:53</var> UTC Investigating - We are currently investigating this issue. </description> <pubDate>Fri, 28 Mar 2025 18:14:56 +0000</pubDate> <link>https://www.githubstatus.com/incidents/vhzbr55p9jtx</link> <guid>https://www.githubstatus.com/incidents/vhzbr55p9jtx</guid> </item> <item> <title>Disruption with Pull Request Ref Updates</title> <description> Mar <var data-var='date'>28</var>, <var data-var='time'>01:40</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>28</var>, <var data-var='time'>01:40</var> UTC Update - This issue has been mitigated and we are operating normally.Mar <var data-var='date'>28</var>, <var data-var='time'>00:54</var> UTC Update - We are continuing to monitor for recovery.Mar <var data-var='date'>28</var>, <var data-var='time'>00:20</var> UTC Update - We believe we have identified the source of the issue and are monitoring for recovery.Mar <var data-var='date'>27</var>, <var data-var='time'>23:52</var> UTC Update - Pull Requests is experiencing degraded performance. We are continuing to investigate.Mar <var data-var='date'>27</var>, <var data-var='time'>23:49</var> UTC Investigating - We are currently investigating this issue. </description> <pubDate>Fri, 28 Mar 2025 01:40:38 +0000</pubDate> <link>https://www.githubstatus.com/incidents/lnvht7g03czs</link> <guid>https://www.githubstatus.com/incidents/lnvht7g03czs</guid> </item> <item> <title>[Retroactive] Incident with Migrations Submitted Via GitHub UI</title> <description> Mar <var data-var='date'>23</var>, <var data-var='time'>18:00</var> UTC Resolved - Between 2024-03-23 18:10 UTC and 2024-03-24 16:10 UTC, migration jobs submitted through the GitHub UI experienced processing delays and increased failure rates. This issue only affected migrations initiated via the web interface. Migrations started through the API or the command line tool continued to function normally. We are sorry for the delayed post on githubstatus.com. </description> <pubDate>Sun, 23 Mar 2025 18:00:00 +0000</pubDate> <link>https://www.githubstatus.com/incidents/cyf5mhwh42tt</link> <guid>https://www.githubstatus.com/incidents/cyf5mhwh42tt</guid> </item> <item> <title>Disruption with some GitHub services</title> <description> Mar <var data-var='date'>21</var>, <var data-var='time'>13:44</var> UTC Resolved - On March 21st, 2025, between 11:45 UTC and 13:20 UTC, users were unable to interact with GitHub Copilot Chat in GitHub. The issue was caused by a recently deployed Ruby change that unintentionally overwrote a global value. This led to GitHub Copilot Chat in GitHub being misconfigured with an invalid URL, preventing it from connecting to our chat server. Other Copilot clients were not affected. We mitigated the incident by identifying the source of the problematic query and rolling back the deployment. We are reviewing our deployment tooling to reduce the time to mitigate similar incidents in the future. In parallel, we are also improving our test coverage for this category of error to prevent them from being deployed to production.Mar <var data-var='date'>21</var>, <var data-var='time'>13:44</var> UTC Update - Copilot is operating normally.Mar <var data-var='date'>21</var>, <var data-var='time'>13:43</var> UTC Update - Mitigation is complete and we are seeing full recovery for GitHub Copilot Chat in GitHub.Mar <var data-var='date'>21</var>, <var data-var='time'>13:16</var> UTC Update - We have identified the problem and have a mitigation in progress.Mar <var data-var='date'>21</var>, <var data-var='time'>13:00</var> UTC Update - Copilot is experiencing degraded performance. We are continuing to investigate.Mar <var data-var='date'>21</var>, <var data-var='time'>12:42</var> UTC Update - We are investigating issues with GitHub Copilot Chat in GitHub. We will continue to keep users updated on progress toward mitigation.Mar <var data-var='date'>21</var>, <var data-var='time'>12:40</var> UTC Investigating - We are currently investigating this issue. </description> <pubDate>Fri, 21 Mar 2025 13:44:16 +0000</pubDate> <link>https://www.githubstatus.com/incidents/kgjdbjz65clj</link> <guid>https://www.githubstatus.com/incidents/kgjdbjz65clj</guid> </item> <item> <title>Intermittent GitHub Actions workflow failures</title> <description> Mar <var data-var='date'>21</var>, <var data-var='time'>09:34</var> UTC Resolved - On March 21st, 2025, between 05:43 UTC and 08:49 UTC, the Actions service experienced degradation, leading to workflow run failures. During the incident, approximately 2.45% of workflow runs failed due to an infrastructure failure. This incident was caused by intermittent failures in communicating with an underlying service provider. We are working to improve our resilience to downtime in this service provider and to reduce the time to mitigate in any future recurrences.Mar <var data-var='date'>21</var>, <var data-var='time'>09:34</var> UTC Update - Actions is operating normally.Mar <var data-var='date'>21</var>, <var data-var='time'>09:05</var> UTC Update - We have made progress understanding the source of these errors and are working on a mitigation.Mar <var data-var='date'>21</var>, <var data-var='time'>08:20</var> UTC Update - We're continuing to investigate elevated errors during GitHub Actions workflow runs. At this stage our monitoring indicates that these errors are impacting no more than 3% of all runs.Mar <var data-var='date'>21</var>, <var data-var='time'>07:27</var> UTC Update - We're continuing to investigate intermittent failures with GitHub Actions workflow runs.Mar <var data-var='date'>21</var>, <var data-var='time'>06:55</var> UTC Update - We're seeing errors reported with a subset of GitHub Actions workflow runs, and are continuing to investigate.Mar <var data-var='date'>21</var>, <var data-var='time'>06:21</var> UTC Investigating - We are investigating reports of degraded performance for Actions </description> <pubDate>Fri, 21 Mar 2025 09:34:23 +0000</pubDate> <link>https://www.githubstatus.com/incidents/slgf0l5smt3j</link> <guid>https://www.githubstatus.com/incidents/slgf0l5smt3j</guid> </item> <item> <title>Incident with Codespaces</title> <description> Mar <var data-var='date'>21</var>, <var data-var='time'>03:08</var> UTC Resolved - On March 21, 2025 between 01:00 UTC and 02:45 UTC, the Codespaces service was degraded and users in various regions experienced intermittent connection failures. The peak error error was 30% of connection attempts across 38% of Codespaces. This was due to a service deployment. The incident was mitigated by completing the deployment to the impacted regions. We are working with the service team to identify the cause of the connection losses and perform necessary repairs to avoid future occurrences.Mar <var data-var='date'>21</var>, <var data-var='time'>03:08</var> UTC Update - Codespaces is operating normally.Mar <var data-var='date'>21</var>, <var data-var='time'>03:08</var> UTC Update - We have seen full recovery in the last 15 minutes for Codespaces connections. GitHub Codespaces are healthy. For users who are still seeing connection problems, restarting the Codespace may help resolve the issue.Mar <var data-var='date'>21</var>, <var data-var='time'>02:53</var> UTC Update - We are continuing to investigate issues with failed connections to Codespaces. We are seeing recovery over the last 10 minutes.Mar <var data-var='date'>21</var>, <var data-var='time'>02:19</var> UTC Update - Customers may be experiencing issues connecting to Codespaces on GitHub.com. We are currently investigating the underlying issue.Mar <var data-var='date'>21</var>, <var data-var='time'>02:12</var> UTC Investigating - We are investigating reports of degraded performance for Codespaces </description> <pubDate>Fri, 21 Mar 2025 03:08:45 +0000</pubDate> <link>https://www.githubstatus.com/incidents/t1x87c1ntdcr</link> <guid>https://www.githubstatus.com/incidents/t1x87c1ntdcr</guid> </item> <item> <title>Incident with Pages</title> <description> Mar <var data-var='date'>20</var>, <var data-var='time'>20:54</var> UTC Resolved - On March 20, 2025, between 19:24 UTC and 20:42 UTC the GitHub Pages experience was degraded and returned 503s for some customers. We saw an error rate of roughly 2% for Pages views, and new page builds were unable to complete successfully before timing out. This was due to replication failure at the database layer between a write destination and read destination. We mitigated the incident by redirecting reads to the same destination as writes. The error with replication occurred while in this transitory phase, as we are in the process of migrating the underlying data for Pages to new database infrastructure. Additionally our monitors failed to detect the error. We are addressing the underlying cause of the failed replication and telemetry. Mar <var data-var='date'>20</var>, <var data-var='time'>20:53</var> UTC Update - We have resolved the issue for Pages. If you're still experiencing issues with your GitHub Pages site, please rebuild.Mar <var data-var='date'>20</var>, <var data-var='time'>20:38</var> UTC Update - Customers may not be able to create or make changes to their GitHub Pages sites. Customers who rely on webhook events from Pages builds might also experience a downgraded experience.Mar <var data-var='date'>20</var>, <var data-var='time'>20:33</var> UTC Update - Webhooks is experiencing degraded performance. We are continuing to investigate.Mar <var data-var='date'>20</var>, <var data-var='time'>20:04</var> UTC Investigating - We are investigating reports of degraded performance for Pages </description> <pubDate>Thu, 20 Mar 2025 20:54:07 +0000</pubDate> <link>https://www.githubstatus.com/incidents/kpkc9bwsd42f</link> <guid>https://www.githubstatus.com/incidents/kpkc9bwsd42f</guid> </item> <item> <title>Scheduled Migrations Maintenance</title> <description> Mar <var data-var='date'>19</var>, <var data-var='time'>05:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>18</var>, <var data-var='time'>21:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>18</var>, <var data-var='time'>19:28</var> UTC Scheduled - Migrations will be undergoing maintenance starting at 21:00 UTC on Tuesday, March 18 2025 with an expected duration of up to eight hours. During this maintenance period, users will experience delays importing repositories into GitHub. Once the maintenance period is complete, all pending imports will automatically proceed. </description> <pubDate>Wed, 19 Mar 2025 05:00:21 +0000</pubDate> <maintenanceEndDate>Wed, 19 Mar 2025 05:00:00 +0000</maintenanceEndDate> <link>https://www.githubstatus.com/incidents/tldgc85p3q2d</link> <guid>https://www.githubstatus.com/incidents/tldgc85p3q2d</guid> </item> <item> <title>Incident with Actions: Queue Run Failures</title> <description> Mar <var data-var='date'>19</var>, <var data-var='time'>00:55</var> UTC Resolved - On March 18th, 2025, between 23:20 UTC and March 19th, 2025 00:15 UTC, the Actions service experienced degradation, leading to run start delays. During the incident, about 0.3% of all workflow runs queued during the time failed to start, about 0.67% of all workflow runs were delayed by an average of 10 minutes, and about 0.16% of all workflow runs ultimately ended with an infrastructure failure. This was due to a networking issue with an underlying service provider. At 00:15 UTC the service provider mitigated their issue, and service was restored immediately for Actions. We are working to improve our resilience to downtime in this service provider to reduce the time to mitigate in any future recurrences.Mar <var data-var='date'>19</var>, <var data-var='time'>00:55</var> UTC Update - Actions is operating normally.Mar <var data-var='date'>19</var>, <var data-var='time'>00:55</var> UTC Update - The provider has reported full mitigation of the underlying issue, and Actions has been healthy since approximately 00:15 UTC.Mar <var data-var='date'>19</var>, <var data-var='time'>00:22</var> UTC Update - We are continuing to investigate issues with delayed or failed workflow runs with Actions. We are engaged with a third-party provider who is also investigating issues and has confirmed we are impacted.Mar <var data-var='date'>18</var>, <var data-var='time'>23:45</var> UTC Update - Some customers may be experiencing delays or failures when queueing workflow runsMar <var data-var='date'>18</var>, <var data-var='time'>23:45</var> UTC Investigating - We are investigating reports of degraded performance for Actions </description> <pubDate>Wed, 19 Mar 2025 00:55:47 +0000</pubDate> <link>https://www.githubstatus.com/incidents/lg4s05t6ttxb</link> <guid>https://www.githubstatus.com/incidents/lg4s05t6ttxb</guid> </item> <item> <title>Disruption with some GitHub services</title> <description> Mar <var data-var='date'>18</var>, <var data-var='time'>18:45</var> UTC Resolved - On March 18th, 2025, between 13:35 UTC and 17:45 UTC, some users of GitHub Copilot Chat in GitHub experienced intermittent failures when reading or writing messages in a thread, resulting in a degraded experience. The error rate peaked at 3% of requests to the service. This was due to an availability incident with a database provider. Around 16:15 UTC the upstream service provider mitigated their availability incident, and service was restored in the following hour. We are working to improve our failover strategy for this database to reduce the time to mitigate similar incidents in the future.Mar <var data-var='date'>18</var>, <var data-var='time'>18:28</var> UTC Update - We are seeing recovery and no new errors for the last 15mins.Mar <var data-var='date'>18</var>, <var data-var='time'>17:42</var> UTC Update - We are still investigating infrastructure issues and our provider has acknowledged the issues and is working on a mitigation. Customers might still see errors when creating messages, or new threads in Copilot Chat. Retries might be successful.Mar <var data-var='date'>18</var>, <var data-var='time'>16:42</var> UTC Update - We are still investigating infrastructure issues and collaborating with providers. Customers might see some errors when creating messages, or new threads in Copilot Chat. Retries might be successful.Mar <var data-var='date'>18</var>, <var data-var='time'>16:00</var> UTC Update - We are experiencing issues with our underlying data store which is causing a degraded experience for a small percentage of users using Copilot Chat in github.comMar <var data-var='date'>18</var>, <var data-var='time'>15:58</var> UTC Investigating - We are currently investigating this issue. </description> <pubDate>Tue, 18 Mar 2025 18:45:32 +0000</pubDate> <link>https://www.githubstatus.com/incidents/fpkvkrdmh033</link> <guid>https://www.githubstatus.com/incidents/fpkvkrdmh033</guid> </item> <item> <title>macos-15-arm64 hosted runner queue delays</title> <description> Mar <var data-var='date'>18</var>, <var data-var='time'>17:15</var> UTC Resolved - On March 18, between 13:04 and 16:55 UTC, Actions workflows relying on hosted runners using the beta MacOS 15 image experienced increased queue time waiting for available runners. An image update was pushed the previous day that included a performance reduction. The slower performance caused longer average runtimes, exhausting our available Mac capacity for this image. This was mitigated by rolling back the image update. We have updated our capacity allocation to the beta and other Mac images and are improving monitoring in our canary environments to catch this potential issue before it impacts customers.Mar <var data-var='date'>18</var>, <var data-var='time'>16:56</var> UTC Update - We are seeing improvements in telemetry and are monitoring for full recovery.Mar <var data-var='date'>18</var>, <var data-var='time'>16:36</var> UTC Update - We've applied a mitigation to fix the issues with queuing Actions jobs on macos-15-arm64 Hosted runner. We are monitoring.Mar <var data-var='date'>18</var>, <var data-var='time'>15:43</var> UTC Update - The team continues to investigate issues with some Actions macos-15-arm64 Hosted jobs being queued for up to 15 minutes. We will continue providing updates on the progress towards mitigation.Mar <var data-var='date'>18</var>, <var data-var='time'>15:05</var> UTC Investigating - We are currently investigating this issue. </description> <pubDate>Tue, 18 Mar 2025 17:15:01 +0000</pubDate> <link>https://www.githubstatus.com/incidents/yz50lpc8d4xq</link> <guid>https://www.githubstatus.com/incidents/yz50lpc8d4xq</guid> </item> <item> <title>Incident with Issues</title> <description> Mar <var data-var='date'>17</var>, <var data-var='time'>23:02</var> UTC Resolved - Between March 17, 2025, 18:05 UTC and March 18, 2025, 09:50 UTC, GitHub.com experienced intermittent failures in web and API requests. These issues affected a small percentage of users (mostly related to pull requests and issues), with a peak error rate of 0.165% across all requests. We identified a framework upgrade that caused kernel panics in our Kubernetes infrastructure as the root cause. We mitigated the incident by downgrading until we were able to disable a problematic feature. In response, we have investigated why the upgrade caused the unexpected issue, have taken steps to temporarily prevent it, and are working on longer term patch plans while improving our observability to ensure we can quickly react to similar classes of problems in the future.Mar <var data-var='date'>17</var>, <var data-var='time'>23:01</var> UTC Update - We saw a spike in error rate with issues related pages and API requests due to some problems with restarts in our kubernetes infrastructure that, at peak, caused 0.165% of requests to see timeouts or errors related to these API surfaces over a 15 minute period. At this time we see minimal impact and are continuing to investigate the cause of the issue.Mar <var data-var='date'>17</var>, <var data-var='time'>21:25</var> UTC Update - We are investigating reports of issues with service(s): Issues We're continuing to investigate. Users may see intermittent HTTP 500 responses when using Issues. Retrying the request may succeed.Mar <var data-var='date'>17</var>, <var data-var='time'>20:51</var> UTC Update - We are investigating reports of issues with service(s): Issues We're continuing to investigate. We will continue to keep users updated on progress towards mitigation.Mar <var data-var='date'>17</var>, <var data-var='time'>19:19</var> UTC Update - We are investigating reports of issues with service(s): Issues. We will continue to keep users updated on progress towards mitigation.Mar <var data-var='date'>17</var>, <var data-var='time'>18:39</var> UTC Investigating - We are investigating reports of degraded performance for Issues </description> <pubDate>Mon, 17 Mar 2025 23:02:33 +0000</pubDate> <link>https://www.githubstatus.com/incidents/y8x4xw6j3q84</link> <guid>https://www.githubstatus.com/incidents/y8x4xw6j3q84</guid> </item> <item> <title>Some Actions users are seeing their workflow jobs failing to start</title> <description> Mar <var data-var='date'>12</var>, <var data-var='time'>14:07</var> UTC Resolved - On March 12, 2025, between 13:28 UTC and 14:07 UTC, the Actions service experienced degradation leading to run start delays. During the incident, about 0.6% of workflow runs failed to start, 0.8% of workflow runs were delayed by an average of one hour, and 0.1% of runs ultimately ended with an infrastructure failure. The issue stemmed from connectivity problems between the Actions services and certain nodes within one of our Redis clusters. The service began recovering once connectivity to the Redis cluster was restored at 13:41 UTC. These connectivity issues are typically not a concern because we can fail over to healthier replicas. However, due to an unrelated issue, there was a replication delay at the time of the incident, and failing over would have caused a greater impact on our customers. We are working on improving our resiliency and automation processes for this infrastructure to improve the speed of diagnosing and resolving similar issues in the future.Mar <var data-var='date'>12</var>, <var data-var='time'>13:55</var> UTC Update - We have applied a mitigation for the affected Redis node, and are starting to see recovery with Action workflow executions.Mar <var data-var='date'>12</var>, <var data-var='time'>13:28</var> UTC Investigating - We are investigating reports of degraded performance for Actions </description> <pubDate>Wed, 12 Mar 2025 14:07:12 +0000</pubDate> <link>https://www.githubstatus.com/incidents/nhcpszxtqxtm</link> <guid>https://www.githubstatus.com/incidents/nhcpszxtqxtm</guid> </item> <item> <title>Incident with Actions and Pages</title> <description> Mar <var data-var='date'> 8</var>, <var data-var='time'>18:11</var> UTC Resolved - On March 8, 2025, between 17:16 UTC and 18:02 UTC, GitHub Actions and Pages services experienced degraded performance leading to delays in workflow runs and Pages deployments. During this time, 34% of Actions workflow runs experienced delays, and a small percentage of runs using GitHub-hosted runners failed to start. Additionally, Pages deployments for sites without a custom Actions workflow (93% of them) did not run, preventing new changes from being deployed. An unexpected data shape led to crashes in some of our pods. We mitigated the incident by excluding the affected pods and correcting the data that led to the crashes. We鈥檝e fixed the source of the unexpected data shape and have improved the overall resilience of our service against such occurrences.Mar <var data-var='date'> 8</var>, <var data-var='time'>18:11</var> UTC Update - Actions is operating normally.Mar <var data-var='date'> 8</var>, <var data-var='time'>18:10</var> UTC Update - Actions run start delays are mitigated. Actions runs that failed will need to be re-run. Impacted Pages updates will need to re-run their deployments.Mar <var data-var='date'> 8</var>, <var data-var='time'>18:00</var> UTC Update - Pages is operating normally.Mar <var data-var='date'> 8</var>, <var data-var='time'>17:50</var> UTC Update - We are investigating impact to Actions run start delays, about 40% of runs are not starting within five minutes and Pages deployments are impacted for GitHub hosted runners.Mar <var data-var='date'> 8</var>, <var data-var='time'>17:45</var> UTC Investigating - We are investigating reports of degraded performance for Actions and Pages </description> <pubDate>Sat, 08 Mar 2025 18:11:56 +0000</pubDate> <link>https://www.githubstatus.com/incidents/m7vl0x8k3j9c</link> <guid>https://www.githubstatus.com/incidents/m7vl0x8k3j9c</guid> </item> <item> <title>Disruption with some GitHub services</title> <description> Mar <var data-var='date'> 7</var>, <var data-var='time'>11:24</var> UTC Resolved - On March 7, 2025, from 09:30 UTC to 11:07 UTC, we experienced a networking event that disrupted connectivity to our search infrastructure, impacting about 25% of search queries and indexing attempts. Searches for PRs, Issues, Actions workflow runs, Packages, Releases, and other products were impacted, resulting in failed requests or stale data. The connectivity issue self-resolved after 90 minutes. The backlog of indexing jobs was fully processed and saw recovery soon after, and queries to all indexes also saw an immediate return to normal throughput. We are working with our cloud provider to identify the root cause and are researching additional layers of redundancy to reduce customer impact in the future for issues like this one. We are also exploring mitigation strategies for faster resolution.Mar <var data-var='date'> 7</var>, <var data-var='time'>10:54</var> UTC Update - We continue investigating degraded experience with searching for issues, pull, requests and actions workflow runs.Mar <var data-var='date'> 7</var>, <var data-var='time'>10:27</var> UTC Update - Actions is experiencing degraded performance. We are continuing to investigate.Mar <var data-var='date'> 7</var>, <var data-var='time'>10:12</var> UTC Update - Searches for issues and pull-requests may be slower than normal and timeout for some usersMar <var data-var='date'> 7</var>, <var data-var='time'>10:06</var> UTC Update - Pull Requests is experiencing degraded performance. We are continuing to investigate.Mar <var data-var='date'> 7</var>, <var data-var='time'>10:05</var> UTC Update - Issues is experiencing degraded performance. We are continuing to investigate.Mar <var data-var='date'> 7</var>, <var data-var='time'>10:03</var> UTC Investigating - We are currently investigating this issue. </description> <pubDate>Fri, 07 Mar 2025 11:24:06 +0000</pubDate> <link>https://www.githubstatus.com/incidents/lb0d8kp99f2v</link> <guid>https://www.githubstatus.com/incidents/lb0d8kp99f2v</guid> </item> <item> <title>Incident with Issues, Git Operations and API Requests</title> <description> Mar <var data-var='date'> 3</var>, <var data-var='time'>05:31</var> UTC Resolved - On March 3rd 2025 between 04:07 UTC and 09:36 UTC various GitHub services were degraded with an average error rate of 0.03% and peak error rate of 9%. This issue impacted web requests, API requests, and git operations. This incident was triggered because a network node in one of GitHub's datacenter sites partially failed, resulting in silent packet drops for traffic served by that site. At 09:22 UTC, we identified the failing network node, and at 09:36 UTC we addressed the issue by removing the faulty network node from production. In response to this incident, we are improving our monitoring capabilities to identify and respond to similar silent errors more effectively in the future.Mar <var data-var='date'> 3</var>, <var data-var='time'>05:30</var> UTC Update - We have seen recovery across our services and impact is mitigated.Mar <var data-var='date'> 3</var>, <var data-var='time'>05:20</var> UTC Update - Git Operations is operating normally.Mar <var data-var='date'> 3</var>, <var data-var='time'>05:20</var> UTC Update - Webhooks is operating normally.Mar <var data-var='date'> 3</var>, <var data-var='time'>04:54</var> UTC Update - We are investigating intermittent connectivity issues between our backend and databases and will provide further updates as we have them. The current impact is you may see elevated latency while using our services.Mar <var data-var='date'> 3</var>, <var data-var='time'>04:23</var> UTC Update - We are seeing intermittent timeouts across our various services. We are currently investigating and will provide updates.Mar <var data-var='date'> 3</var>, <var data-var='time'>04:21</var> UTC Update - Webhooks is experiencing degraded performance. We are continuing to investigate.Mar <var data-var='date'> 3</var>, <var data-var='time'>04:20</var> UTC Investigating - We are investigating reports of degraded performance for API Requests, Git Operations and Issues </description> <pubDate>Mon, 03 Mar 2025 05:31:19 +0000</pubDate> <link>https://www.githubstatus.com/incidents/291w7fn43fy1</link> <guid>https://www.githubstatus.com/incidents/291w7fn43fy1</guid> </item> <item> <title>Scheduled Codespaces Maintenance</title> <description> Mar <var data-var='date'> 1</var>, <var data-var='time'>02:00</var> UTC Completed - The scheduled maintenance has been completed.Feb <var data-var='date'>28</var>, <var data-var='time'>17:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Feb <var data-var='date'>27</var>, <var data-var='time'>21:09</var> UTC Scheduled - Codespaces will be undergoing maintenance in Europe and Southeast Asia from 17:00 UTC Friday February 28 to 02:00 UTC Saturday March 1. Maintenance will begin in North Europe at 17:00 UTC Friday February 28, followed by Southeast Asia, concluding in UK South. Each region will take 2-3 hours to complete. During this time period, users may experience connectivity issues with new and existing Codespaces. Please ensure that any uncommitted changes that you may need during the maintenance window are committed and pushed. Codespaces with any uncommitted changes will be accessible as usual once maintenance is complete. Thank you for your patience as we work to improve our systems. </description> <pubDate>Sat, 01 Mar 2025 02:00:21 +0000</pubDate> <maintenanceEndDate>Sat, 01 Mar 2025 02:00:00 +0000</maintenanceEndDate> <link>https://www.githubstatus.com/incidents/5ylj8dpvg096</link> <guid>https://www.githubstatus.com/incidents/5ylj8dpvg096</guid> </item> <item> <title>Elevated Request Latency for Write operations on github.com and api.github.com</title> <description> Feb <var data-var='date'>28</var>, <var data-var='time'>06:55</var> UTC Resolved - On February 28th, 2025, between 05:49 UTC and 06:55 UTC, a newly deployed background job caused increased load on GitHub鈥檚 primary database hosts, resulting in connection pool exhaustion. This led to degraded performance, manifesting as increased latency for write operations and elevated request timeout rates across multiple services. The incident was mitigated by halting execution of the problematic background job and disabling the feature flag controlling the job execution. To prevent similar incidents in the future, we are collaborating on a plan to improve our production signals to better detect and respond to query performance issues.Feb <var data-var='date'>28</var>, <var data-var='time'>06:29</var> UTC Update - Issues and Pull Requests are experiencing degraded performance. We are continuing to investigate.Feb <var data-var='date'>28</var>, <var data-var='time'>06:12</var> UTC Investigating - We are currently investigating this issue. </description> <pubDate>Fri, 28 Feb 2025 06:55:57 +0000</pubDate> <link>https://www.githubstatus.com/incidents/36ftd36c921f</link> <guid>https://www.githubstatus.com/incidents/36ftd36c921f</guid> </item> <item> <title>Disruption with some GitHub services</title> <description> Feb <var data-var='date'>27</var>, <var data-var='time'>12:22</var> UTC Resolved - On February 27, 2025, between 11:30 UTC and 12:22 UTC, Actions experienced degraded performance, leading to delays in workflow runs. On average, 5% of Actions workflow runs were delayed by 31 minutes. The delays were caused by updates in a dependent service that led to failures in Redis connectivity in one region. We mitigated the incident by failing over the impacted service and re-routing the service鈥檚 traffic out of that region. We are working to improve monitoring and processes of failover to reduce our time to detection and mitigation of issues like this one in the future.Feb <var data-var='date'>27</var>, <var data-var='time'>12:22</var> UTC Update - The team is confident that recovery is complete. Thank you for your patience as this issue was investigated.Feb <var data-var='date'>27</var>, <var data-var='time'>12:16</var> UTC Update - Our mitigations have rolled out successfully and have seen recovery for all Actions run starts back within expected range. Users should see Actions runs working normally. We will keep this incident open for a short time while we continue to validate these results.Feb <var data-var='date'>27</var>, <var data-var='time'>12:01</var> UTC Update - We have identified the cause of the delays to starting Action runs. Our team is working to roll out mitigations and we hope to see recovery as these take effect in our systems over the next 10-20 minutes. Further updates as we have more information.Feb <var data-var='date'>27</var>, <var data-var='time'>11:39</var> UTC Update - We are seeing an increase in run start delays since 1104 UTC. This is impacting ~3% of Action runs at this time. The team is working to understand the causes of this and to mitigate impact. We will continue to update as we have more information.Feb <var data-var='date'>27</var>, <var data-var='time'>11:31</var> UTC Update - Actions is experiencing degraded performance. We are continuing to investigate.Feb <var data-var='date'>27</var>, <var data-var='time'>11:28</var> UTC Investigating - We are currently investigating this issue. </description> <pubDate>Thu, 27 Feb 2025 12:22:34 +0000</pubDate> <link>https://www.githubstatus.com/incidents/3zss4vv50thx</link> <guid>https://www.githubstatus.com/incidents/3zss4vv50thx</guid> </item> <item> <title>Incident with Actions and Packages</title> <description> Feb <var data-var='date'>26</var>, <var data-var='time'>17:19</var> UTC Resolved - On February 26, 2025, between 14:51 UTC and 17:19 UTC, GitHub Packages experienced a service degradation, leading to billing-related failures when uploading and downloading Packages. During this period, the billing usage and budget pages were also inaccessible. Initially, we reported that GitHub Actions was affected, but we later determined that the impact was limited to jobs interacting with Packages services, while jobs that did not upload or download Packages remained unaffected. The incident occurred due to an error in newly introduced code, which caused containers to get into a bad state, ultimately leading to billing API calls failing with 503 errors. We mitigated the issue by rolling back the contributing change. In response to this incident, we are enhancing error handling, improving the resiliency of our billing API calls to minimize customer impact, and improving change rollout practices to catch these potential issues prior to deployment.Feb <var data-var='date'>26</var>, <var data-var='time'>17:19</var> UTC Update - Actions and Packages are operating normally.Feb <var data-var='date'>26</var>, <var data-var='time'>16:41</var> UTC Update - We're continuing our investigation into Billing interfaces and retrieval of packages causing Actions workflow run failures.Feb <var data-var='date'>26</var>, <var data-var='time'>16:17</var> UTC Update - We鈥檙e investigating issues related to billing and the retrieval of packages that are causing Actions workflow run failures.Feb <var data-var='date'>26</var>, <var data-var='time'>15:56</var> UTC Update - We're investigating issues related to the Billing interfaces and Packages downloads failing for enterprise customers.Feb <var data-var='date'>26</var>, <var data-var='time'>15:51</var> UTC Investigating - We are investigating reports of degraded performance for Actions and Packages </description> <pubDate>Wed, 26 Feb 2025 17:19:09 +0000</pubDate> <link>https://www.githubstatus.com/incidents/2lxm4wb8wy3r</link> <guid>https://www.githubstatus.com/incidents/2lxm4wb8wy3r</guid> </item> <item> <title>Disruption with some GitHub services</title> <description> Feb <var data-var='date'>25</var>, <var data-var='time'>16:50</var> UTC Resolved - On February 25th, 2025, between 14:25 UTC and 16:44 UTC email and web notifications experienced delivery delays. At the peak of the incident the delay resulted in ~10% of all notifications taking over 10 minutes to be delivered, with the remaining ~90% being delivered within 5-10 minutes. This was due to insufficient capacity in worker pools as a result of increased load during peak hours. We also encountered delivery delays for a small number of webhooks, with delays of up-to 2.5 minutes to be delivered. We mitigated the incident by scaling out the service to meet the demand. The increase in capacity gives us extra headroom, and we are working to improve our capacity planning to prevent issues like this occurring in the future.Feb <var data-var='date'>25</var>, <var data-var='time'>16:49</var> UTC Update - Web and email notifications are caught up, resolving the incident.Feb <var data-var='date'>25</var>, <var data-var='time'>16:16</var> UTC Update - We're continuing to investigate delayed web and email notifications.Feb <var data-var='date'>25</var>, <var data-var='time'>15:43</var> UTC Update - We're continuing to investigate delayed web and email notifications.Feb <var data-var='date'>25</var>, <var data-var='time'>15:13</var> UTC Update - We're investigating delays in web and email notifications impacting all customers.Feb <var data-var='date'>25</var>, <var data-var='time'>15:12</var> UTC Investigating - We are currently investigating this issue. </description> <pubDate>Tue, 25 Feb 2025 16:50:14 +0000</pubDate> <link>https://www.githubstatus.com/incidents/flt2rxl1dg1t</link> <guid>https://www.githubstatus.com/incidents/flt2rxl1dg1t</guid> </item> <item> <title>Claude 3.7 Sonnet Partially Unavailable</title> <description> Feb <var data-var='date'>25</var>, <var data-var='time'>15:45</var> UTC Resolved - On February 25, 2025 between 13:40 UTC and 15:45 UTC the Claude 3.7 Sonnet model for GitHub Copilot Chat experienced degraded performance. During the impact, occasional requests to Claude would result in an immediate error to the user. This was due to upstream errors with one of our infrastructure providers, which have since been mitigated. We are working with our infrastructure providers to reduce time to detection and implement additional failover options, to mitigate issues like this one in the future.Feb <var data-var='date'>25</var>, <var data-var='time'>15:25</var> UTC Update - We have disabled Claude 3.7 Sonnet models in Copilot Chat and across IDE integrations (VSCode, Visual Studio, JetBrains) due to an issue with our provider. Users may still see these models as available for a brief period but we recommend switching to a different model. Other models were not impacted and are available. Once our provider has resolved the issues impacting Claude 3.7 Sonnet models, we will re-enable them.Feb <var data-var='date'>25</var>, <var data-var='time'>14:44</var> UTC Update - Copilot is experiencing degraded performance. We are continuing to investigate.Feb <var data-var='date'>25</var>, <var data-var='time'>14:43</var> UTC Update - We are currently experiencing partial availability for the Claude 3.7 Sonnet and Claude 3.7 Thinking models in Copilot Chat, VSCode and other Copilot products. This is due to problems with an upstream provider. We are working to resolve these issues and will update with more information as it is made available. Other Copilot models are available and working as expected.Feb <var data-var='date'>25</var>, <var data-var='time'>14:40</var> UTC Investigating - We are currently investigating this issue. </description> <pubDate>Tue, 25 Feb 2025 15:45:44 +0000</pubDate> <link>https://www.githubstatus.com/incidents/tskzz9n0bjpt</link> <guid>https://www.githubstatus.com/incidents/tskzz9n0bjpt</guid> </item> <item> <title>Incident with Packages</title> <description> Feb <var data-var='date'>25</var>, <var data-var='time'>01:08</var> UTC Resolved - On February 25, 2025, between 00:17 UTC and 01:08 UTC, GitHub Packages experienced a service degradation, leading to failures uploading and downloading packages, along with increased latency for all requests to GitHub Packages registry. At peak impact, about 14% of uploads and downloads failed, and all Packages requests were delayed by an average of 7 seconds. The incident was caused by the rollout of a database configuration change that resulted in a degradation in database performance. We mitigated the incident by rolling back the contributing change and failing over the database. In response to this incident, we are tuning database configurations and resolving a source of deadlocks. We are also redistributing certain workloads to read replicas to reduce latency and enhance overall database performance.Feb <var data-var='date'>25</var>, <var data-var='time'>01:08</var> UTC Update - We have confirmed recovery for the majority of our systems. Some systems may still experience higher than normal latency as they catch up.Feb <var data-var='date'>25</var>, <var data-var='time'>00:41</var> UTC Update - We have identified the issue impacting packages and have rolled out a fix. We are seeing signs of recovery and continue to monitor the situation.Feb <var data-var='date'>25</var>, <var data-var='time'>00:17</var> UTC Investigating - We are investigating reports of degraded performance for Packages </description> <pubDate>Tue, 25 Feb 2025 01:08:43 +0000</pubDate> <link>https://www.githubstatus.com/incidents/pnl2xrj64d7p</link> <guid>https://www.githubstatus.com/incidents/pnl2xrj64d7p</guid> </item> </channel> </rss>

CINXE.COM

GitHub Status - Incident History