From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry
The rapid evolution of Artificial Intelligence, particularly in the realm of large language models (LLMs), presents both immense opportunities and complex challenges for software development. At a recent presentation hosted by InfoQ, Ben O’Mahony, Principal AI Engineer at Thoughtworks, detailed a compelling approach to harnessing the power of these advanced models by distilling their behavior into smaller, more efficient models, leveraging production telemetry captured through OpenTelemetry (OTEL). This methodology promises to democratize advanced AI capabilities, making them more accessible, cost-effective, and controllable for individual developers and organizations alike.
The core of O’Mahony’s presentation revolved around a practical demonstration of how developers can transform raw telemetry data into actionable insights for AI model fine-tuning. This process, he argued, empowers developers to actively shape the AI tools they use, rather than passively being shaped by them. The foundational elements of this approach include the Language Server Protocol (LSP), a ubiquitous technology for enabling intelligent code editing features across various IDEs, and OpenTelemetry, the industry standard for observability.
The Language Server Protocol: A Foundation for Intelligent Development
O’Mahony began by establishing a common understanding of the Language Server Protocol. "The Language Server Protocol is a backbone of modern editor language-specific smarts," he explained. "It’s why you can use the same Python language features in VS Code, Neovim, Emacs, any other editor." The LSP functions by separating the language understanding logic from the editor itself. The editor acts as a client, sending requests to a language server—a separate process that possesses deep knowledge of a specific programming language. This server then responds with relevant information, such as code diagnostics, definitions, references, and suggestions.
While traditional LSPs, like those powering features in tools such as pyright or rust-analyzer, are rule-based and excel at precise, reliable code analysis through abstract syntax trees (ASTs), they often struggle with understanding semantic intent. O’Mahony highlighted the limitations: "They can’t really understand semantic intent. They can’t tell you the name of this function no longer matches what this function actually does. That’s too complex a thing to be rules-based."
This is where the concept of an AI-powered LSP emerges. O’Mahony’s personal project involved building an AI LSP that could provide semantic code analysis, inline hints, and intelligent suggestions directly within his development workflow. The goal was not just to have AI suggestions available, but for them to be proactive and "agentic"—actively assisting the developer without requiring explicit prompts for every action. A significant challenge he encountered was the high cost associated with constantly sending data to large language models (LLMs) for analysis, a common experience for developers who "burn tokens" throughout the day.
OpenTelemetry: Capturing the Gold Standard of User Feedback
The breakthrough came with the realization that his existing OpenTelemetry instrumentation was already capturing the precise data needed for AI model training. "I was using OpenTelemetry. It was capturing everything, every interaction, every suggestion I accepted, every suggestion I dismissed, every suggestion I asked to regenerate," O’Mahony stated. This rich dataset, he realized, could be used to train a smaller, faster, and more cost-effective model.
OpenTelemetry, an industry standard for observability, provides developers with distributed tracing, metrics, and logs, allowing them to track requests across entire systems. For generative AI systems, OTel’s capabilities are even more potent. O’Mahony emphasized that OTel offers specific benefits for AI, including the ability to capture events such as system messages, user prompts, assistant responses, and tool calls. Crucially, it provides metrics like token usage, operation duration, and time to first token, which are invaluable for debugging and platform development.
"For generative AI systems, these traces are even more valuable," he asserted. "Each step’s a span, with timing information, metadata, and context. For generative AI systems, these traces are even more valuable." He elaborated on how OTel can represent model spans and agent spans, composed of operations like agent creation, workflow execution, chat completion, and tool execution.
The key insight for O’Mahony was that every developer interaction with an AI tool could be logged as a signal. "Every agent should produce traces, in my opinion. Instrument them from day one. Don’t wait until you’re thinking about distillation, just get the telemetry in place. Capture the context, the prompts, all those things, the user action." He highlighted that auto-instrumentation, readily available in many frameworks, can capture a significant portion of this data with minimal effort.
The Data Flywheel: From Telemetry to Fine-Tuned Models
The captured telemetry data, specifically user actions within the AI LSP, became the foundation for a "data flywheel." O’Mahony identified three key types of user feedback captured through his instrumentation:
- Accepted: When a diagnostic appears and the developer clicks "apply fix," it signifies a strong positive signal that the AI not only found an issue but also provided a correct and desirable solution. This is explicit confirmation of helpfulness.
- Regenerated: Clicking "regenerate AI suggestion" indicates a weak positive signal. The developer likely agrees with the diagnostic but not the proposed solution, signaling a desire for alternatives. This suggests the AI is on the right track but needs refinement.
- Dismissed: This represents strong negative feedback. When a suggestion is dismissed, it indicates the AI’s output was unhelpful or incorrect, providing crucial negative examples that are often difficult to obtain.
This granular feedback, O’Mahony argued, is far more powerful than simple "thumbs up/thumbs down" mechanisms. It reflects actual product usage rather than stated opinions. By collecting these signals over time, a developer can build a dataset that labels itself.
"This allows us to use RLHF, any other fine-tuning methodology that you like that incorporates human feedback, super useful," he explained. The data flywheel concept posits that as more data is collected through user interactions, models can be improved, leading to better performance, which in turn encourages more usage and thus more data.
Distillation: Making Frontier Models Accessible
The process of turning this rich telemetry into a usable AI model is called "distillation." O’Mahony’s approach involves exporting OTEL traces to a backend (such as Logfire, Honeycomb, or Datadog), processing them to extract relevant training examples, and then using these examples to fine-tune a smaller language model (SLM).
"The mechanics of actually turning that OpenTelemetry traces into training data, this is where the rubber hits the road," O’Mahony noted. He described an extraction pipeline that queries traces for user action spans, filters out irrelevant interactions, and formats the data into a simple JSONL file, where each line is a training example.
He then uses frontier models, like Google’s Gemini, as a source for distillation. The goal is not to train a general-purpose coding model from scratch but to fine-tune an existing open-source model (in the 7-billion to 13-billion parameter range) with the specific behaviors observed in his production telemetry. This personalized tuning makes the model more efficient, cheaper to run, and better aligned with his individual coding style and codebase.
"Now I’ve got something that’s local, cheaper, faster. I can use it more easily on client projects that maybe don’t allow me to use AI that goes externally," he said. While acknowledging that the distilled model might not match the performance of frontier models on highly complex or novel tasks (estimating around 80-85% quality), the trade-offs in cost, speed, and privacy are significant advantages.
Platform Capabilities and Generalization
O’Mahony emphasized that this approach is not a one-off solution but a set of transferable platform capabilities. The core pattern involves:
- Instrumenting agents with OpenTelemetry: Capturing user actions as training signals.
- Distilling model behavior into an SLM: Creating a specialized, efficient model.
- Deploying with safety rails: Ensuring responsible AI usage.
- Continuously iterating and improving: Leveraging the data flywheel.
This architecture can be generalized across multiple AI agents within an organization. A shared instrumentation layer, extraction pipeline, and fine-tuning infrastructure can support various agents, from customer support bots to code review assistants. This allows for a unified approach to AI development and improvement, fostering a "learning organization."
He also touched upon the economic advantages. Traditional machine learning often requires expensive data labeling processes. By leveraging product usage as a natural byproduct of user interaction, the "data flywheel" approach significantly reduces costs and scales more effectively. "Your product usage is the labeling process. Your users, or in my case, me, the labeler, they’re labeling for free as a natural byproduct of using the tool," he highlighted.
The Future of Developer Tools: Personalization and Control
The presentation concluded with a discussion on the broader implications of this methodology for the future of developer tools. O’Mahony drew a parallel to how IDEs and version control systems like Git have fundamentally altered how developers work. He believes that AI tools, too, will shape our workflows and thinking. The critical question for developers is whether these tools make them "better engineers" or merely faster but less capable in the long run.
His approach aims to provide developers with greater control over their AI tools. "For me, that control means like I want to be able to configure it, be able to shape it. I want to be able to give it feedback myself personally, not just how it works generally." The ability to distill AI behavior into personalized models allows developers to remain in the "driving seat," actively shaping the tools that shape them.
This vision aligns with a future of AI tooling that is personalized, local, and continuously improving, moving away from a singular focus on Artificial General Intelligence (AGI) towards a multitude of specialized intelligences. This approach offers portability, cost-effectiveness, and the ability to run AI solutions locally, which is particularly beneficial for projects with strict data privacy requirements or those where external AI services are not permitted.
Key Takeaways for Immediate Application
O’Mahony offered three concrete takeaways for developers and organizations looking to implement this strategy:
- Instrument all agents with OpenTelemetry: Implement comprehensive telemetry collection from day one, focusing on capturing user actions. Auto-instrumentation can significantly simplify this process.
- Design user experiences for observable decisions: Create interfaces where user interactions provide clear, observable signals for AI improvement. This goes beyond simple feedback mechanisms to embedding feedback loops directly into the user workflow.
- Embrace distillation when data is sufficient: Recognize that distillation is becoming increasingly accessible and cost-effective. Tooling is accelerating, enabling more individuals and teams to fine-tune models locally, even on personal laptops, without requiring massive compute budgets or large ML engineering teams.
He encouraged attendees to start small, pick a single agent or workflow, instrument it, collect data, and then experiment with distillation. The results, he suggested, can be surprisingly effective in significantly improving personal workflows and driving AI adoption. The future of AI tooling, he concluded, lies in empowering users to shape their tools, fostering a continuous cycle of improvement and control.
