Resolved -
Currently service is restored and stable. At this time, the incident is resolved.
May 19, 22:40 EDT
Monitoring -
We have successfully reverted service impacting changes. The service is currently operational and stable. We will continue to monitor. Next update in 30 minutes or as new information becomes available.
May 19, 22:11 EDT
Update -
We are still investigating the root cause and continue to work with our infrastructure provider on restoring service. Next update in 30 minutes or as new information becomes available.
May 19, 21:22 EDT
Update -
We are still investigating the root cause and continue to work with our infrastructure provider on restoring service. Next update in 30 minutes or as new information becomes available.
May 19, 21:19 EDT
Update -
We are still investigating the root cause and continue to work with our infrastructure provider on restoring service. Next update in 30 minutes or as new information becomes available.
May 19, 20:48 EDT
Update -
We are still investigating the root cause and will continue to update the status page. We are continuing to work with our infrastructure provider to restore service. Next update in 30 minutes or whenever we have new information.
May 19, 20:13 EDT
Update -
We are still investigating the root cause and will continue to update the status page. We are continuing to work with our infrastructure provider to restore service. Next update in 30 minutes or whenever we have new information.
May 19, 19:32 EDT
Update -
We are still investigating the root cause and will continue to update the status page. We are continuing to work with our infrastructure provider to restore service. Next update in 30 minutes or whenever we have new information.
May 19, 19:03 EDT
Update -
We are still investigating the root cause and will continue to update the status page. We have engaged our infrastructure provider to assist with service restoration. Next update in 30 minutes or whenever we have new information.
May 19, 18:20 EDT
Update -
We are still investigating the root cause and will continue to update the status page. Next update in 30 minutes or whenever we have new information.
May 19, 18:13 EDT
Update -
We are still investigating the root cause and will continue to update the status page. Next update in 30 minutes or whenever we have new information.
May 19, 17:37 EDT
Update -
We are still investigating the root cause and will continue to update the status page. Next update in 30 minutes or whenever we have new information.
May 19, 16:58 EDT
Investigating -
We are still investigating the root cause and will continue to update the status page. Next update in 30 minutes or whenever we have new information.
May 19, 16:29 EDT
Update -
Scaling up the database has caused the service to go back in full outage. We are still investigating root cause and further ways to workaround the issue.
May 19, 15:47 EDT
Update -
We are scaling up the database instance which we hope will fix the occasional errors we are seeing. We will post a new update within the next 30 minutes as we confirm this has resolved the remaining issues.
May 19, 15:26 EDT
Identified -
We are occasionally seeing 504 errors when pulling images even after turning off connection pooling. Our team is investigating further.
May 19, 14:52 EDT
Update -
We are continuing to monitor for any further issues.
May 19, 14:17 EDT
Monitoring -
Quay.io push operations have been re-enabled. The service is stable and we are monitoring the situation while we work on a definitive fix and a complete root cause analysis
May 19, 14:14 EDT
Update -
We are working on further stabilizing the service and on re-enabling write operations. The workaround is still in place and root cause is still unknown.
May 19, 13:43 EDT
Identified -
The temporary fix is proving to be effective and the service is stabilizing. Quay images pulling works albeit slower than normal. We are working on re-enabling write operations and a definitive fix.
May 19, 13:11 EDT
Update -
Turning database connection pooling off seems to have mitigated the issue for now. We're monitoring closely and working on root cause analysis and a permanent fix
May 19, 12:43 EDT
Update -
We are still investigating the root cause of the DB connections spike. Attempts at failing over to a standby cluster have been unsuccessful.
May 19, 12:10 EDT
Update -
We are still investigating the root cause of the DB connections spike. Attempts at failing over to a standby cluster have been unsuccessful.
May 19, 11:38 EDT
Update -
The root cause of the outage is still unknown. Our attempts at recovering the service have not been successful so far. We are working across the board to bring the service back up.
May 19, 11:02 EDT
Update -
We are still working on fixing the issues that were identified previously.
May 19, 10:05 EDT
Update -
We've identified multiple causes for this outage and continue to work on fixing these. We acknowledge that our consumers depend on Quay.io for their workloads and are attempting to resolve this with the highest urgency.
May 19, 09:16 EDT
Update -
We are still investigating the root cause and will continue to update the status page. Next update in 30 minutes or whenever we have new information.
May 19, 08:45 EDT
Update -
We are still investigating the root cause and will continue to update the status page. Next update in 30 minutes or whenever we have new information.
May 19, 08:12 EDT
Update -
Quay.io is currently in read-only mode. We are still investigating the root cause and will continue to update the status page.
May 19, 07:39 EDT
Update -
We are still working on bringing quay.io back online in read-only mode. We are currently able to bring it back online for a brief period of time, but the service continues to degrade due to the underlying database connections load.
May 19, 07:21 EDT
Update -
We are working on bringing quay.io back online in read-only mode. Once we accomplish that we will continue working on bringing it up with full read/write support.
May 19, 06:38 EDT
Update -
The number of open connections to the Quay.io database increased significantly causing an outage of the service, we are still investigating the root cause of this spike in DB connections. Right now we don't have an ETA for full service restoration.
May 19, 06:14 EDT
Investigating -
We are currently investigating this issue.
May 19, 03:30 EDT
