This incident has been resolved. At the beginning of the incident, 0.3% of Splunk APM trace spans were dropped between 7:26am and 7:28am PST.
Posted Jan 08, 2022 - 08:26 PST
Update
The processing of Monitoring MetricSets has recovered and is back to real-time processing. We are continuing to work on stabilizing the rest of the services.
Posted Jan 08, 2022 - 08:01 PST
Update
We are continuing to work on a fix for this issue.
Posted Jan 08, 2022 - 07:52 PST
Identified
A degradation in the performance of the Splunk APM trace processing pipeline is causing Troubleshooting MetricSets to be delayed by more than fifteen minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data from approximately 5% of the traffic.
The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.
Posted Jan 08, 2022 - 07:51 PST
This incident affected: Splunk APM (Splunk APM Monitoring MetricSets, Splunk APM Troubleshooting MetricSets, Splunk APM Trace Data, Splunk APM Tag Spotlight, Splunk APM Business Workflows).