On May 15, 2018, OpsGenie experienced an incident that resulted in some Solarwinds integrations not to receive alerts. We want to share the details of the impact, root-cause, resolution, and corrective actions we’re taking.

Incident Summary & Impact On April 2018, OpsGenie integration infrastructure has changed. In new infrastructure, some services are redirected to new integration applications. Some requests that are sent by customers’ systems are modified by the internal redirection. On April 27, 2018 We have received HTTP 414 “request too large” errors from some of the integrations (Papertrail, Zendesk, NewRelic, Solarwinds, Logentries). We determined that the redirection was the cause of these errors, and to solve the problem the form data was sent to the redirected application in both request uri and request body. On May 15, We made some changes about redirection by sending data only in the request body. This change only impacted some of Solarwinds clients that used 'solarwinds/triggerActionBody.txt'. Some OpsGenie customers use custom scripts that send alert data to OpsGenie. The problem occurred only on integrations where the custom script was used and the payload included “§#§” and “µ#µ” characters. Testing and fixes to this issue went over 2 days and and fully resolved with a fix update on May 16 05:00 PM EST.

What Happened Some OpsGenie integrations support customizing the data payload sent by the integration plugins to OpsGenie servers. This flexibility in functionality introduces challenges when testing the integrations since the payload varies and integration tests may not cover all the potential different payload possibilities. After the integration infrastructure was changed, we’ve run into problems with redirection of requests to new services. We’ve found out that the framework that we have used for redirection makes changes on http requests and the encoding of the data depending on the data format. We noticed that the request body was encoded for some requests but not for all of them. This problem was fixed by decoding the requests for all formats. Once the problem was fixed, the fix was tested based on the standard integration scripts that are included in OpsGenie integrations, however the customized scripts were not tested. We found out that decoding all the data caused some unexpected problems. When the payload included “§#§” characters, the request was not properly processed by OpsGenie servers.

What We’re Doing About It We have reviewed our internal process to improve the resolution time for issues and fix deployment. We are implementing capabilities for faster detection of such problems by enabling engineering teams to easily observe how incoming requests processed whenever any integration related code changes are made.

Final Words First, all of us at OpsGenie would like apologize for the impact of this incident. We know that you rely on OpsGenie to notify the right people as soon as possible to prevent or minimize impact of incidents on your business. Even if all customers were not affected, this was unacceptable incident for us. We also know that words go only so far and we are committed to doing everything we can to prevent such problems from impacting your business in the future.