The Harder Problem in AI-Assisted Root Cause Analysis: Data Pipeline, Not Model Reasoning, Emerges as the New Frontier

A significant shift is occurring within the observability engineering community, with a growing consensus that the reasoning capabilities of large language models (LLMs) are no longer the primary obstacle in AI-assisted root cause analysis (RCA). Instead, the more complex challenge now lies in meticulously crafting the data pipelines that determine what information reaches these powerful AI models. For engineering teams integrating LLMs into their incident response workflows, this evolving understanding suggests that substantial effort invested in preparing high-quality, contextualized data may yield greater returns than simply opting for larger, more powerful LLMs.

The current landscape of AI-driven RCA solutions largely bifurcates into two distinct architectural philosophies. The first, often termed "agent-based" designs, empowers LLMs with a suite of tools and the autonomy to investigate incidents. These agents dynamically fetch telemetry data as they reason through potential causes. The second approach, characterized as "deterministic" designs, prioritizes upfront signal correlation. In this model, relevant data is pre-processed and synthesized into a single, focused context before being presented to the LLM. This latter camp is gaining considerable traction, as evidenced by the work of observability vendor Coroot and similar industry leaders. Dynatrace’s proprietary Davis AI, for instance, relies heavily on deterministic, topology-aware causal analysis. It navigates a real-time dependency map to precisely identify root causes, a stark contrast to the open-ended, exploratory nature of agent-based LLM loops. The critical distinction between these two approaches lies in their failure modes. When an AI diagnosis falters, it becomes challenging to definitively attribute the error to either the LLM’s reasoning deficiencies or to an inadequate data pipeline that supplied the wrong evidence.

To address this ambiguity and isolate the performance of LLMs from the effectiveness of their data inputs, recent research from Coroot offers a novel perspective. Engineer Nikolay Sivko, in a seminal piece of research published on the Coroot blog, conceptually divided the task of LLM-assisted RCA into two discrete functions: the intrinsic reasoning ability of the AI over available data, and the "harness"—the system responsible for selecting, shaping, and delivering that data to the model. Sivko argues that the perennial question, "Can AI perform RCA?", is fundamentally misframed because these two critical functions require separate evaluation.

Coroot’s innovative pipeline architecture is designed to streamline this process. It correlates disparate signals into actionable findings and then presents these findings to the LLM as a singular, focused context, deliberately omitting any agent-based data retrieval loops. This design choice is crucial because it ensures that any incorrect diagnosis can be confidently attributed to the LLM itself, rather than being obscured by missing or irrelevant data. To rigorously test this hypothesis, Sivko engineered a specific experimental scenario. He utilized Chaos Mesh, a popular Kubernetes-native chaos engineering platform, to inject a deliberate network delay between a catalogue service and its associated PostgreSQL database. This simulated latency was designed to slow down database queries, ultimately triggering 502 errors on the front-end. Crucially, the data context provided to the LLM for analysis intentionally included misleading signals, such as inflated query timings that were skewed by the simulated network round-trip delays. Sivko then subjected eleven different LLMs to the same prompt, comprising approximately 9,800 tokens, requesting each model to identify the root cause of the issue, delineate the cause-and-effect chain, and propose an immediate corrective action.

The results of this controlled experiment provided compelling insights into the current state of LLM reasoning. Leading closed-frontier models, including Anthropic’s Claude Opus 4.8, OpenAI’s GPT-5.5, and Google’s Gemini 3.1 Pro, successfully navigated the complex scenario, correctly identifying the Chaos Mesh NetworkChaos experiment as the root cause and prescribing the necessary steps to delete both the experiment and its associated schedule. Larger, open-weight models generally performed comparably. However, at the lower end of the performance spectrum, Gemma 4 31B emerged as the sole self-hostable model capable of accurately pinpointing the root cause. More surprisingly, even larger open-weight models such as Qwen3.6 35B and Qwen3 Coder Next failed to correctly diagnose the problem.

While these findings do not definitively settle the ongoing debate surrounding the optimal approach to AI-powered RCA, they strongly underscore the significance of the data preparation phase. Agent-based approaches retain a distinct advantage in scenarios involving novel or unforeseen incidents. In such cases, an LLM empowered to fetch its own data can potentially uncover critical signals that a pre-defined, fixed pipeline might never anticipate. This flexibility, however, comes at a considerable cost in terms of operational complexity and debugging. Anecdotal evidence from platforms like ZenML and Incident.io highlights the profound challenges associated with debugging multi-agent LLM investigations in production environments. Unlike traditional software, a failed agentic run often leaves behind no clear stack trace, only a tangled web of unpredictable prompt interactions and emergent, uncoordinated behavior among the agents. This inherent fragility is a significant deterrent for many practitioners, prompting a palpable shift towards more deterministic workflows. Discussions on developer forums, such as a notable thread on Reddit, reveal engineers actively abandoning fully agentic designs in favor of predominantly deterministic workflows that incorporate a narrowly defined LLM step. The primary drivers for this pivot are enhanced reliability, more predictable outcomes, and significantly lower token consumption costs. Deterministic pipelines, while potentially sacrificing some degree of ad-hoc flexibility, offer superior repeatability and facilitate cleaner, more rigorous evaluation methodologies. This trade-off is a key factor contributing to the industry’s growing recognition of the data harness, rather than the LLM itself, as the more formidable challenge in achieving effective AI-assisted RCA.

The economic implications of this architectural shift are also noteworthy. Sivko observed that even a single, concise LLM call, even on state-of-the-art frontier models, incurs a cost of mere cents. This affordability stems directly from the fact that the computationally intensive signal correlation and data preparation work is completed before the LLM is invoked. He posits that the core reasoning aspect of AI RCA is "basically solved," and that the critical engineering focus must now shift to "preparing the right, compact context for the model before you call it." This perspective clearly directs the next wave of engineering innovation towards refining the data harness, rather than solely pursuing incremental improvements in LLM capabilities. This strategic framing resonates strongly with a broader industry-wide emphasis on "context engineering." Guidance from leading AI research organizations like Anthropic and development frameworks such as LangChain, alongside insights from observability vendors like Mezmo, all converge on a singular, crucial principle: the meticulous curation of the most concise set of high-signal data represents a fundamental discipline for achieving reliable and effective LLM-based reasoning and observability. This burgeoning field of context engineering is rapidly becoming a cornerstone of modern AI-driven operational intelligence.

The Evolution of AI in Incident Response

The journey towards AI-powered incident response has been marked by rapid advancements and evolving priorities. Initially, the focus was squarely on enhancing the cognitive abilities of machines to understand and interpret complex system behaviors. Early research and development efforts were heavily invested in improving the natural language understanding and reasoning capabilities of AI models, with the hope that they could autonomously diagnose and resolve system failures. This era saw the emergence of sophisticated algorithms designed to process vast amounts of log data, trace distributed system interactions, and identify patterns indicative of anomalies.

However, as LLMs have become more sophisticated and readily available, a new set of challenges has come to the forefront. The sheer volume and velocity of data generated by modern, distributed systems present a significant hurdle. Without effective mechanisms to filter, aggregate, and contextualize this data, even the most powerful LLMs can be overwhelmed or misled. This has led to the development of specialized techniques for data preprocessing, including log parsing, metric aggregation, and event correlation, aimed at distilling the essential information for AI analysis.

Agentic vs. Deterministic Architectures: A Comparative Analysis

The dichotomy between agentic and deterministic AI architectures for RCA reflects a fundamental trade-off between flexibility and controllability.

Agentic Architectures: The Promise of Autonomous Investigation

Agentic AI systems empower LLMs to act as independent investigators. They are equipped with a toolkit of APIs and functions that allow them to query various data sources, execute diagnostic commands, and even make changes to the system under investigation. The core idea is that the LLM, guided by its reasoning engine, can dynamically adapt its investigation strategy based on real-time findings.

• Strengths:

  • Adaptability: Can handle novel and unexpected incidents by dynamically seeking out relevant information.
  • Discovery: Potential to uncover root causes that might be missed by pre-defined correlation rules.
  • Proactive Response: Can potentially initiate automated remediation steps based on its findings.

• Weaknesses:

  • Debugging Complexity: Highly challenging to trace and debug failures due to the emergent behavior of multiple interacting agents.
  • Unpredictability: Outcomes can be inconsistent, making it difficult to guarantee reliable performance in production.
  • Resource Intensive: Can consume significant computational resources and tokens due to iterative data fetching and reasoning.
  • Security Concerns: Granting agents broad access to systems can introduce security risks if not meticulously managed.

Deterministic Architectures: The Power of Curated Context

Deterministic AI approaches, conversely, rely on pre-processing and structuring data before it is fed to the LLM. This involves identifying key signals, correlating them based on predefined rules and topological information, and then presenting a synthesized context to the model. The LLM’s role is primarily to reason over this carefully prepared information.

• Strengths:

  • Reliability and Repeatability: Produces consistent results, making it easier to validate and trust.
  • Debuggability: Failures are more easily traceable to specific data inputs or correlation logic.
  • Cost Efficiency: Generally more cost-effective due to reduced token consumption and focused LLM calls.
  • Clarity of Attribution: Isolates LLM reasoning performance from data quality issues.

• Weaknesses:

  • Limited Novelty Handling: May struggle with entirely new types of incidents that fall outside the scope of predefined correlations.
  • Requires Upfront Engineering: Significant effort is needed to build and maintain robust data correlation pipelines.
  • Potential for Information Overload: While curated, the context must still be sufficiently comprehensive to avoid missing critical clues.

The Coroot Experiment: Isolating Variables

Coroot’s research, spearheaded by engineer Nikolay Sivko, represents a significant step towards demystifying the performance of LLMs in RCA. By architecting a system that strictly separates data acquisition and preparation from LLM reasoning, Sivko’s work provides a clearer lens through which to evaluate the LLMs themselves. The experiment’s design, employing Chaos Mesh to simulate a realistic network latency scenario, offered a controlled environment to test how different LLMs would interpret a deliberately ambiguous data set. The inclusion of misleading signals, such as inflated query timings, served as a critical test of the models’ ability to discern true causality from superficial correlations.

The subsequent evaluation across eleven diverse LLMs, ranging from leading proprietary models to open-weight alternatives, yielded valuable comparative data. The success of top-tier models like Claude Opus 4.8 and GPT-5.5 in accurately identifying the root cause, even with misleading data, highlights their advanced reasoning capabilities. The performance variance among open-weight models, particularly the struggle of larger models like Qwen3.6 35B while Gemma 4 31B succeeded, underscores that model size alone is not a deterministic predictor of RCA efficacy. This suggests that architectural nuances, training data composition, and fine-tuning strategies play a crucial role.

Industry Implications and Future Directions

The prevailing sentiment in the observability community—that the data pipeline is the new bottleneck—carries profound implications for how engineering teams will approach AI integration. The focus is shifting from simply acquiring more powerful LLMs to investing in the sophisticated tooling and expertise required to prepare high-quality, context-rich data.

This necessitates a deeper understanding of:

• Context Engineering: The art and science of distilling vast amounts of observability data into the most relevant, concise, and actionable information for LLMs. This involves sophisticated signal correlation, anomaly detection, and pattern recognition techniques.
• Data Pipeline Optimization: Building robust, scalable, and efficient data pipelines that can ingest, process, and transform telemetry data in real-time. This includes leveraging techniques like stream processing and distributed data stores.
• Model Evaluation Frameworks: Developing standardized benchmarks and methodologies for evaluating LLM performance specifically within the context of RCA, separating the model’s reasoning from the quality of its input.

The increasing adoption of deterministic workflows, driven by the pursuit of reliability and debuggability, suggests a future where AI-assisted RCA is not an "all-or-nothing" proposition. Instead, it is likely to involve a hybrid approach, where deterministic pipelines handle the bulk of data correlation and filtering, presenting a refined context to a narrowly focused LLM for final reasoning and diagnosis. This pragmatic approach balances the power of advanced AI with the operational realities of complex, distributed systems. As organizations continue to navigate the complexities of AI in their operations, the emphasis on intelligent data curation will undoubtedly define the next era of effective incident response.

The transition towards prioritizing the data pipeline in AI-assisted RCA is not merely an academic discussion; it represents a tangible shift in engineering strategy. Teams that can master the art of context engineering and build resilient data pipelines will be best positioned to unlock the full potential of LLMs for faster, more accurate, and more reliable incident resolution. The era of simply "throwing more compute" at the problem is giving way to a more nuanced approach, where intelligent data management becomes the key differentiator in the quest for operational excellence.