Organization administrators can now receive personalized incident information in Atlassian Administration. Subscribe to System Health to receive notifications if a reliability incident affects your organization.
We sent an library upgrade to backend services at 14:44 UTC, during blue-green deploy phase everything seemed OK. When we shut down old instances transaction durations started to increase. We understood that the problem was in the caching layer. Revert decision made at 15:45 UTC, but a bug in deployment automation prevented us to solve the problem immediately. We started to revert instances manually. Between 16:28 UTC and 16:42 UTC backend services receiving API v1 requests, fully went down. Also we experienced delays at outgoing integrations up to 20 minutes. API v2 and alerting / notification flows were functional during the incident. A detailed post-mortem will be published later on.
Posted Feb 22, 2018 - 12:35 UTC
Monitoring
Update is rolled back. Everything is functional.
Posted Feb 21, 2018 - 17:25 UTC
Identified
Latest update to the production caused timeouts in the cache layer
Posted Feb 21, 2018 - 17:10 UTC
Investigating
We are experiencing long transactions in the backend. Investigating the root cause.
Posted Feb 21, 2018 - 15:35 UTC
This incident affected: US (Outgoing Integration Flow, Configuration REST APIs).
