Currently service is restored and stable. At this time, the incident is resolved.
Posted May 19, 2020 - 22:40 EDT
Monitoring
We have successfully reverted service impacting changes. The service is currently operational and stable. We will continue to monitor. Next update in 30 minutes or as new information becomes available.
Posted May 19, 2020 - 22:11 EDT
Update
We are still investigating the root cause and continue to work with our infrastructure provider on restoring service. Next update in 30 minutes or as new information becomes available.
Posted May 19, 2020 - 21:22 EDT
Update
We are still investigating the root cause and continue to work with our infrastructure provider on restoring service. Next update in 30 minutes or as new information becomes available.
Posted May 19, 2020 - 21:19 EDT
Update
We are still investigating the root cause and continue to work with our infrastructure provider on restoring service. Next update in 30 minutes or as new information becomes available.
Posted May 19, 2020 - 20:48 EDT
Update
We are still investigating the root cause and will continue to update the status page. We are continuing to work with our infrastructure provider to restore service. Next update in 30 minutes or whenever we have new information.
Posted May 19, 2020 - 20:13 EDT
Update
We are still investigating the root cause and will continue to update the status page. We are continuing to work with our infrastructure provider to restore service. Next update in 30 minutes or whenever we have new information.
Posted May 19, 2020 - 19:32 EDT
Update
We are still investigating the root cause and will continue to update the status page. We are continuing to work with our infrastructure provider to restore service. Next update in 30 minutes or whenever we have new information.
Posted May 19, 2020 - 19:03 EDT
Update
We are still investigating the root cause and will continue to update the status page. We have engaged our infrastructure provider to assist with service restoration. Next update in 30 minutes or whenever we have new information.
Posted May 19, 2020 - 18:20 EDT
Update
We are still investigating the root cause and will continue to update the status page. Next update in 30 minutes or whenever we have new information.
Posted May 19, 2020 - 18:13 EDT
Update
We are still investigating the root cause and will continue to update the status page. Next update in 30 minutes or whenever we have new information.
Posted May 19, 2020 - 17:37 EDT
Update
We are still investigating the root cause and will continue to update the status page. Next update in 30 minutes or whenever we have new information.
Posted May 19, 2020 - 16:58 EDT
Investigating
We are still investigating the root cause and will continue to update the status page. Next update in 30 minutes or whenever we have new information.
Posted May 19, 2020 - 16:29 EDT
Update
Scaling up the database has caused the service to go back in full outage. We are still investigating root cause and further ways to workaround the issue.
Posted May 19, 2020 - 15:47 EDT
Update
We are scaling up the database instance which we hope will fix the occasional errors we are seeing. We will post a new update within the next 30 minutes as we confirm this has resolved the remaining issues.
Posted May 19, 2020 - 15:26 EDT
Identified
We are occasionally seeing 504 errors when pulling images even after turning off connection pooling. Our team is investigating further.
Posted May 19, 2020 - 14:52 EDT
Update
We are continuing to monitor for any further issues.
Posted May 19, 2020 - 14:17 EDT
Monitoring
Quay.io push operations have been re-enabled. The service is stable and we are monitoring the situation while we work on a definitive fix and a complete root cause analysis
Posted May 19, 2020 - 14:14 EDT
Update
We are working on further stabilizing the service and on re-enabling write operations. The workaround is still in place and root cause is still unknown.
Posted May 19, 2020 - 13:43 EDT
Identified
The temporary fix is proving to be effective and the service is stabilizing. Quay images pulling works albeit slower than normal. We are working on re-enabling write operations and a definitive fix.
Posted May 19, 2020 - 13:11 EDT
Update
Turning database connection pooling off seems to have mitigated the issue for now. We're monitoring closely and working on root cause analysis and a permanent fix
Posted May 19, 2020 - 12:43 EDT
Update
We are still investigating the root cause of the DB connections spike. Attempts at failing over to a standby cluster have been unsuccessful.
Posted May 19, 2020 - 12:10 EDT
Update
We are still investigating the root cause of the DB connections spike. Attempts at failing over to a standby cluster have been unsuccessful.
Posted May 19, 2020 - 11:38 EDT
Update
The root cause of the outage is still unknown. Our attempts at recovering the service have not been successful so far. We are working across the board to bring the service back up.
Posted May 19, 2020 - 11:02 EDT
Update
We are still working on fixing the issues that were identified previously.
Posted May 19, 2020 - 10:05 EDT
Update
We've identified multiple causes for this outage and continue to work on fixing these. We acknowledge that our consumers depend on Quay.io for their workloads and are attempting to resolve this with the highest urgency.
Posted May 19, 2020 - 09:16 EDT
Update
We are still investigating the root cause and will continue to update the status page. Next update in 30 minutes or whenever we have new information.
Posted May 19, 2020 - 08:45 EDT
Update
We are still investigating the root cause and will continue to update the status page. Next update in 30 minutes or whenever we have new information.
Posted May 19, 2020 - 08:12 EDT
Update
Quay.io is currently in read-only mode. We are still investigating the root cause and will continue to update the status page.
Posted May 19, 2020 - 07:39 EDT
Update
We are still working on bringing quay.io back online in read-only mode. We are currently able to bring it back online for a brief period of time, but the service continues to degrade due to the underlying database connections load.
Posted May 19, 2020 - 07:21 EDT
Update
We are working on bringing quay.io back online in read-only mode. Once we accomplish that we will continue working on bringing it up with full read/write support.
Posted May 19, 2020 - 06:38 EDT
Update
The number of open connections to the Quay.io database increased significantly causing an outage of the service, we are still investigating the root cause of this spike in DB connections. Right now we don't have an ETA for full service restoration.
Posted May 19, 2020 - 06:14 EDT
Investigating
We are currently investigating this issue.
Posted May 19, 2020 - 03:30 EDT
This incident affected: Registry, API, Build System, and Frontend.