We observe some false deduplications of alerts
Incident
Report for Opsgenie
Resolved
Alert deduplication problem is resolved now. No new alerts are affected due to this problem. Approximately affected alert request ratio is 0.8% during this incident.
For alerts created between 9 Jan 2020, 09:01 AM UTC and 10 Jan 2020, 01:15 PM UTC, we have set proper alert aliases to them.
This could have caused only one more alert to be created with the same alert alias although there is a "not closed" alert with same alias. Additionally, closing the previous alerts by alias created within the above time range could have been affected for minority of alerts.
For alerts created between 9 Jan 2020, 09:01 AM UTC and 10 Jan 2020, 01:15 PM UTC, we have set proper alert aliases to them.
This could have caused only one more alert to be created with the same alert alias although there is a "not closed" alert with same alias. Additionally, closing the previous alerts by alias created within the above time range could have been affected for minority of alerts.
Update
We are still running the fix script with monitoring its progress. Another update will be given after completion of the script.
Update
We have started to fix the corrupted data. It is expected to take couple of hours.
Identified
We have identified the root cause of the problem. Our engineering teams have implemented the fix that will prevent false deduplications of the previously impacted alerts.
Update
This problem did not affect EU region.
Investigating
We are investigating a problem with false alert deduplications. Some alerts are not created due to deduplication over a closed alert with same alias. Or it causes two alerts with same alias to be created. Our engineering team is currently working on a fix now.
This incident affected: US (Alert Flow) and EU (Alert Flow).