Alerts being sent for some detector rules previously disabled via API

Incident Report for Splunk Observability Cloud US2

Resolved

This incident has been resolved.

Posted Feb 09, 2024 - 17:27 PST

Identified

We have identified a bug that is causing alerts to be sent for detector rules that have been previously disabled via the API. This issue does not impact detectors that have rules disabled via the UI.

Starting January 18, some detectors that had rules disabled via the API (such as through Terraform) continued to send alerts. We are currently rolling out a fix to production systems and expect to resolve the issue within the next 3 hours.

Posted Feb 09, 2024 - 12:16 PST