Alert deduplication problem is resolved now. No new alerts are affected due to this problem. Approximately affected alert request ratio is 0.8% during this incident.
For alerts created between 9 Jan 2020, 09:01 AM UTC and 10 Jan 2020, 01:15 PM UTC, we have set proper alert aliases to them. This could have caused only one more alert to be created with the same alert alias although there is a "not closed" alert with same alias. Additionally, closing the previous alerts by alias created within the above time range could have been affected for minority of alerts.
Posted Jan 11, 2020 - 10:31 UTC
We are still running the fix script with monitoring its progress. Another update will be given after completion of the script.
Posted Jan 10, 2020 - 21:35 UTC
We have started to fix the corrupted data. It is expected to take couple of hours.
Posted Jan 10, 2020 - 16:41 UTC
We have identified the root cause of the problem. Our engineering teams have implemented the fix that will prevent false deduplications of the previously impacted alerts.
Posted Jan 10, 2020 - 15:33 UTC
This problem did not affect EU region.
Posted Jan 10, 2020 - 15:07 UTC
We are investigating a problem with false alert deduplications. Some alerts are not created due to deduplication over a closed alert with same alias. Or it causes two alerts with same alias to be created. Our engineering team is currently working on a fix now.
Posted Jan 10, 2020 - 15:04 UTC
This incident affected: US (Alert Flow) and EU (Alert Flow).