Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

AI Engineering Podcast

AI Engineering Podcast artwork

From Blind Spots to Observability: Operationalizing LLM Apps with OpenLit

Feb 15, 202651 minAudio
ShareXLinkedInFacebookWhatsAppEmail

Summary In this episode of the AI Engineering Podcast, Aman Agarwal, creator of OpenLit, discusses the operational foundations required to run LLM-powered applications in production. He highlights common early blind spots teams face, including opaque model behavior, runaway token costs, and brittle prompt management, emphasizing that strong observability and cost tracking must be established before an MVP ships. Aman explains how OpenLit leverages OpenTelemetry for vendor-neutral tracing across models, tools, and data stores, and introduces features such as prompt and secret management with versioning, evaluation workflows (including LLM-as-a-judge), and fleet management for OpenTelemetry collectors. The conversation covers experimentation patterns, strategies to avoid vendor lock-in, and how detailed stepwise traces reshape system design and debugging. Aman also shares recent advancements like a Kubernetes operator for zero-code instrumentation, multi-database configurations for environment isolation, and integrations with platforms such as Grafana and Dash0. They conclude by discussing lessons learned from building in the open, prioritizing reliability, developer experience, and data security, and preview future work on context management and closing the loop from experimentation to prompt/dataset improvements. Announcements Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems Unlock the full potential of your AI workloads with a seamless and composable data infrastructure. Bruin is an open source framework that streamlines integration from the command line, allowing you to focus on what matters most - building intelligent systems. Write Python code for your business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. With native support for ML/AI workloads, Bruin empowers data teams to deliver faster, more reliable, and scalable AI solutions. Harness Bruin's connectors for hundreds of platforms, including popular machine learning frameworks like TensorFlow and PyTorch. Build end-to-end AI workflows that integrate seamlessly with your existing tech stack. Join the ranks of forward-thinking organizations that are revolutionizing their data engineering with Bruin. Get started today at aiengineeringpodcast.com/bruin, and for dbt Cloud customers, enjoy a $1,000 credit to migrate to Bruin Cloud. Your host is Tobias Macey and today I'm interviewing Aman Agarwal about the operational investments that are necessary to ensure you get the most out of your AI models Interview Introduction How did you get involved in the area of AI/data management? Can you start by giving your assessment of the main blind spots that are common in the existing AI application patterns? As teams adopt agentic architectures, how common is it to fall prey to those same blind spots? There are numerous tools/services available now focused on various elements of "LLMOps". What are the major components necessary for a minimum viable operational platform for LLMs? There are several areas of overlap, as well as disjoint features, in the ecosystem of tools (both open source and commercial). How do you advise teams to navigate the selection process? (point solutions vs. integrated tools, and handling frameworks with only partial overlap) Can you describe what OpenLit is and the story behind it? How would you characterize the feature set and focus of OpenLit compared to what you view as the "major players"? Once you have invested in a platform like OpenLit, how does that change the overall development workflow for the lifecycle of AI/agentic applications? What are the most complex/challenging elements of change management for LLM-powered systems? (e.g. prompt tuning, model changes, data changes, etc.) How can the information collected in OpenLit be used to develop a self-improvement flywheel for agentic systems? Can you describe the architecture and implementation of OpenLit? How have the scope and goals of the project changed since you started working on it? Given the foundational aspects of the project that you have built, what are some of the adjacent capabilities that OpenLit is situated to expand into? What are the sharp edges and blind spots that are still challenging even when you have OpenLit or similar integrated? What are the most interesting, innovative, or unexpected ways that you have seen OpenLit used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on OpenLit? When is OpenLit the wrong choice? What do you have planned for the future of OpenLit? Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data/AI management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@aiengineeringpodcast.com with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers. Links OpenLit Fleet Hub OpenTelemetry LangFuse LangSmith TensorZero AI Engineering Podcast Episode Traceloop Helicone Clickhouse The intro and outro music is from Hitman's Lovesong feat. Paola Graziano by The Freak Fandango Orchestra/CC BY-SA 3.0

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Hosts and Guests

Hosts

  • Tobias Macey Host Feed-stated

Guests

  • Aman Agarwal Guest Feed-stated

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. What Do Humans Do Now? Rethinking Software Engineering in the Age of AI Agents

    Oct 15, 20261 hr 8 minAudio

    Summary In this episode, I spoke with Jonathan Ellis about how AI agents are changing the economics of software development and forcing teams to rethink where human time is best spent. We explored his argument that, now...

    Credit the people on this episode

  2. How to Evaluate RAG Systems When Ground Truth Keeps Changing

    Oct 7, 20261 hr 3 minAudio

    Summary In this episode Ofer Mendelevitch shares what it really takes to evaluate RAG systems in production when your source data is incomplete, constantly changing, or difficult to validate against a clean ground...

    Credit the people on this episode

  3. Harness Engineering for Reliable, Governed AI Agents

    Sep 19, 20261 hr 4 minAudio

    Summary In this episode Nikunj Bajaj, co-founder and CEO of TrueFoundry, talks about the challenge of building reliable agents on top of inherently variable foundation models. He explores the idea of the agent harness...

    Credit the people on this episode

  4. Kubernetes, Compliance, and Control: The Operational Backbone of AI Sovereignty

    Feb 25, 20261 hr 1 minAudio

    Summary In this episode of the AI Engineering Podcast, Steven Watt, leader of the Office of the CTO at Red Hat, discusses practical paths to achieving AI sovereignty for organizations. He shares his two-decade...

    Credit the people on this episode

  5. Taming Voice Complexity with Dynamic Ensembles at Modulate

    Feb 8, 202659 minAudio

    Summary In this episode of the AI Engineering Podcast, Carter Huffman, co-founder and CTO of Modulate, discusses the engineering behind low-latency, high-accuracy Voice AI. He explains why voice is a uniquely...

    Credit the people on this episode