Internal Developer Platforms were built to simplify software development. But as engineering teams begin building AI-powered applications, are traditional developer platforms ready for what comes next? From integrating large language models and managing inference costs to supporting AI evaluations, retrieval systems, and model gateways, platform engineering is entering a new phase. In this episode of DevOps Academy Deep Dive, we explore how Internal Developer Platforms (IDPs) are evolving to support AI development and what this transformation means for DevOps engineers, platform teams, and engineering organizations. We discuss the rise of AI-enabled developer platforms, how AI golden paths can simplify development, and why shared model access, evaluation pipelines, and cost visibility are becoming important platform capabilities. We also examine whether every organization needs its own AI infrastructure, how platform engineering and MLOps can work together, and why the next generation of developer platforms should focus on making AI development easier without introducing unnecessary complexity. 🔑 Key Takeaways Why traditional Internal Developer Platforms are evolving to support AI workloads How AI applications introduce new infrastructure and developer experience requirements The difference between providing AI model access and building a complete AI development platform Why approved model catalogs can balance developer flexibility with governance The role of model evaluations and AI testing in CI/CD pipelines Whether every AI platform needs GPU infrastructure Why AI-powered applications require closer collaboration with data engineering The relationship between platform engineering and MLOps The skills platform engineers need as AI becomes part of software development How to measure the success of an AI-enabled developer platform Where organizations should begin when expanding their existing IDPs 🎧 Tune in to discover how platform engineering is evolving beyond traditional software delivery - and what it takes to build developer platforms that support the next generation of AI-powered applications. 💬 Is your Internal Developer Platform ready for AI, or are your developers still building AI infrastructure from scratch?
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Imagine spending like three agonizing weeks building this perfectly secure, highly automated software deployment pipeline. Oh yeah, the dream scenario. Right. But then you realize the application you just launched can be tricked into giving away sensitive company secrets simply because a user, you know, asked it politely. It's terrifying. It really is. I mean, welcome to the new era of software development where the old reliable rules of platform engineering are just being completely rewritten overnight. Yeah, the ground is basically shifting under our feet. Exactly.
So welcome everyone to today's deep dive. We've got a fascinating stack of architectural notes and industry insights today and we're looking at the collision of internal developer platforms or IDPs and the really explosive inherently messy rise of AI applications. It's a huge shift. It is. And our mission today is to uncover whether developer platforms are actually transforming into AI platforms. And exactly what that means for you, whether you're a developer in the trenches, an engineering leader wondering where your budget went, or just someone trying to understand how
modern software is built. Because it impacts all of us, really. Definitely. OK, let's unpack this. You know, traditional software engineering has always felt a bit like following a really good, well -tested recipe. You have your list of ingredients. standard measurements. And as long as you follow the steps, like configuring the database, setting up the server. You know exactly what kind of cake is coming out of the oven. Exactly. But lately, building AI applications feels entirely different. It's no longer a recipe. It feels like being tossed into this frantic
experimental test kitchen where half the ingredients are invisible. The oven temperature fluctuates based on the time of day. And occasionally, the cake talks back to you. Yes. The cake talks back. It's wild. It really is a remarkably accurate description of the landscape right now. I mean, we are watching this massive industry migration from a world of deterministic, predictable systems where code does exactly what you tell it to. Right. Into a world that is inherently probabilistic and frankly a little chaotic. Which is exactly
why we need to establish a baseline before we get into the AI chaos. What is a developer platform actually meant to solve in the first place? We absolutely have to start there. So platform engineering originally emerged to solve a very specific crisis of complexity. Right, like the move to the cloud. Exactly. As the industry moved toward cloud -native technologies over the last decade, we gave developers this massive amount of power. Which sounds great on paper. Right. But the hidden cost of that power was that A standard developer was suddenly expected to be an expert in an overwhelming number
of specialized tools. Oh, like Kubernetes and all that. Yeah, managing orchestration with Kubernetes, setting up continuous integration and deployment or CI -CD pipelines, configuring complex cloud networking, handling secrets management, security policies. I mean, the list of non -coding tasks was endless. the cognitive load was just completely unsustainable. You're asking a brilliant engineer to like build a highly responsive user interface, but first they have to spend three weeks banging their head against the wall configuring a pipeline
just to get their code onto a server. Precisely. And so internal developer platforms were created to reduce that unnecessary friction. To kind of smooth things out. Exactly. The fundamental goal is to provide what the industry calls golden paths. Golden paths. It's a great concept. A golden path is essentially a supported, well designed route for a common engineering task. So instead of starting from scratch and wiring together 15 different cloud services. Which is a nightmare. Total nightmare. Instead, a developer
uses an internal portal to just click a button and say, I need a new web service. And the platform just takes over. Exactly. It mechanically takes over. It automatically provisions the code repository, sets up the deployment pipeline, configures the infrastructure, and applies all the corporate security defaults. So it gives them a secure starting point in minutes instead of weeks. Exactly. It's like a paved road through the wilderness, but here is my question. I want to push back on this a little bit. Sure, go for it. If you're an engineering manager listening to this, you
might be thinking, isn't the whole promise of modern cloud development supposed to be about developer autonomy. Right. Freedom to choose. Yeah. If we force everyone in the company onto these pre -built golden paths, aren't we just taking away their toys and dictating exactly how they have to work? Doesn't that stifle innovation? Well, if we connect this to the bigger picture, a golden path doesn't actually eliminate alternatives or stifle innovation. OK. It just provides a reliable, frustration -free way to do things correctly. If a product team has a highly specialized
use case, Like they need something totally bespoke? Yeah, if they absolutely need to go off -roading, they usually still can. But the trade -off is they take on the operational burden of maintaining that dirt road themselves. Ah, I see. You break it, you buy it. Exactly. The platform's real product isn't the infrastructure itself. It's a superior developer experience. By automating the boilerplate, developers can focus all their cognitive energy on the actual product, not the plumbing. That makes a lot of sense. The paved
road is there if you want it, and most people want it because they just want to reach their destination without, you know, rebuilding the car. Right. But this is exactly where the current wave of AI comes in and completely shatters this neat, tidy model. Oh, absolutely shatters it. Because developers aren't just building traditional web apps anymore. They're pulling in large language models. They're building retrieval augmented generation, or RAG, systems to... chat with internal docs. Yeah, they're experimenting with autonomous agents. Right, and based on the architectural trends, this introduces an entirely new layer
of incredibly messy dependencies. I mean, we are suddenly talking about integrating vector search systems, managing the mathematical embedding processes. Which is incredibly complex. Right, turning text into numbers, configuring complex prompts, handling model -specific access controls. And the harsh reality is that those new dependencies simply do not fit into the traditional CI -CD pipelines we've spent the last 10 years perfecting. They just break the mold entirely. They do. Think about the mechanics of a simple internal AI assistant.
A developer wants to build a tool that answers employee questions based on company HR documents. Sounds simple enough. Right. But to make that work, they need access to an improved language model. They need a data pipeline to retrieve relevant documents. They need a service to turn that text into embeddings, a specialized vector database to query those embeddings at scale. And a robust way to check if the AI's answers are actually legally accurate. Exactly. And if they don't have a centralized platform helping them wire all of that together. Without a shared
platform, you get absolute chaos. One team builds a custom authentication layer to connect to OpenAI. Another team down the hall builds a completely different incompatible integration for Anthropic. Oh, wow. A third team sets up their own disconnected vector database. You end up with a sprawling web of shadow IT. So it's just duplicated effort everywhere. Duplicated effort, totally separate cost tracking, and massive security vulnerabilities. Every team is building custom integrations, and half of them are accidentally routing sensitive
HR data through unencrypted public endpoints. Yikes. So what does this all mean? If integrating AI is this complex, the seemingly obvious solution a lot of companies jump to is just giving developers a unified API key. Oh, I see this all the time. Right. You might be thinking, can't I just hand my team an endpoint, say, here is your AI access and let them build? But industry experience is proving really quickly that a raw API connection is not a platform. It's a very common misconception, but a really dangerous one. Providing an AI model
API. solves exactly one problem, connectivity. Right. It completely ignores everything else required to run software in production safely. An API key does not solve how you evaluate the model's quality over time. Or data integration. Exactly. Or how you deploy consistently. Or how you actually monitor and attribute costs. Think of it like this. If a platform team just handed you the login credentials to a raw, unconfigured Kubernetes cluster. That would be useless. Right. You wouldn't call that a complete developer experience.
You still need deployment templates, monitoring, guardrails. The exact same logic applies to AI. Which brings up a critical architectural shift. If handing out a raw API key is a recipe for disaster, how do you actually give developers access to these models without losing control? You have to put a centralized choke point between the application and the model. The AI gateway. Exactly. This is what the industry is calling an AI gateway. It acts as a centralized control plane. So how does that work mechanically? Let's
trace it. When an app wants to ask a language model a question, it doesn't talk to the external provider directly. It sends a generic request to the internal AI gateway. Okay. The gateway looks at the request, checks if that specific app has the budget and permission to make the call, and then retrieves the highly secure provider credentials from a vault. Meaning the developer never even touches the real API key. Never. The gateway injects that credential, applies rate limits, so one buggy loop doesn't drain your entire monthly budget in an hour. Which definitely
happens. Yeah, it happens. And then it routes the request to the approved model. Okay, but hold on. If the gateway provides this single unified generic interface, doesn't that create a dangerous illusion? How do you mean? Well, does it make developers think these models are perfectly interchangeable commodities? Like, can I just go into the gateway config and blindly swap an open AI model for an anthropic one with zero consequences? That is a phenomenal question, and the answer is absolutely not. Really? Yeah.
Switching providers still requires rigorous, dedicated testing. Different models have totally different context window limits, meaning how much text they can remember at once. Ah, okay. They respond differently to specific input formats. Their tool calling behaviors and quirky ways of formatting JSON data vary wildly. Right, so even if the API call looks identical on the surface, The cognitive behavior and reasoning underneath are fundamentally different. The gateway supports control change, but these models are rarely one -to -one drop -in replacements. You cannot swap
them blindly and expect the same results. Right. You might swap the model in the gateway and suddenly your professional customer support bot starts speaking in pirate slang because it interpreted the system prompt differently. Exactly. Which highlights a major tension for engineering teams. Freedom versus standardization. Because, let's be honest, developers love playing with the newest, shiniest model they just read about. Of course they do. If these models aren't perfectly interchangeable, how much freedom should developers actually have to choose them? It's a really delicate balance. You definitely don't want to force every single
application onto one massive, expensive flagship model if they don't actually need it. Right, like a simple document classification service might run perfectly well on a cheap open source model. Exactly. Whereas a complex coding assistant needs a cutting -edge proprietary model. But unrestricted freedom is an operational nightmare. Yeah, I can imagine. If developers can pick literally any model off the internet, they might choose ones that route sensitive company data through unapproved third -party servers. Yikes. Or they pick obscure models that the operations team
has no tooling to monitor. So the solution we are seeing emerge to handle this is the concept of an approved model catalog. Yes. The platform team curates a centralized catalog. For each approved model, the catalog details what it is best used for, the specific access requirements, exactly how much it costs per token. And the data privacy restrictions, right? Most importantly, the data privacy restrictions. This way, developers get the flexibility to choose the right tool for the job, but they are doing it within safe,
legally compliant boundaries. And this brings us all the way back to that golden path concept we discussed earlier, but now specifically tailored for AI. What does an AI golden path actually look like in practice? Well, the most common example we are seeing is a ready to use template for a RAG application. Retrieval augmented generation. Exactly. So if I'm a developer, instead of spending two weeks figuring out how to securely glue a language model to a vector database, I just go into my developer portal and select this RAG template. And behind the scenes... The platform
spins up an approved model integration, a supported document retrieval component, secure credential handling, standardized telemetry so I can actually see my application logs, and a basic evaluation framework. Spot on. You get all that incredibly complex plumbing for free. Right. And because of that, you can spend your time designing the business logic, like figuring out what documents the bot should read, instead of wrestling with infrastructure. Exactly. The platform team focuses on standardizing the common repeated use cases across the company. But wait, let me play devil's
advocate here. If you're an enterprise architect listening to this, you might be thinking, to ensure maximum consistency across a massive company, shouldn't the platform team just build a massive proprietary internal AI framework? Oh, boy. Like a giant internal wrapper that abstracts all the AI complexities away and mandate that every single app must use this bespoke tool. Wouldn't that? permanently solve the chaos? I would strongly push back against that approach. And the history of software development backs me up here. How so? The AI space is evolving far too quickly
for any company to maintain a heavy proprietary abstraction layer. If you spend six months building a massive internal framework today, the underlying technology will shift, and your framework will be obsolete the day you launch it. Wow. Yeah. That makes sense. Tied coupling to an internal wrapper creates a massive maintenance bottleneck. It also deeply frustrates your developers who want to use new external capabilities but can't because the internal framework doesn't support them yet. So what's the golden rule then? Standardize the foundation. The authentication, the credential
management, the usage tracking, the model access. Do not attempt to standardize every single implementation detail or orchestration logic. Standardize the foundation. not the application logic. I really like that framing. It's a lot of pain. Okay, so now we have our golden path. We've built our AI app and we're ready to deploy it to users. Here's where it gets really interesting. Oh yes, evals. Exactly. How do you actually know that AI isn't completely hallucinating? This brings us to the wild west of AI testing or evaluations.
Because traditional software testing checks for known static outputs. If I put in 2 plus 2, I write an automated test to ensure the system returns exactly four. If it returns five, the test fails, deployment stops. It's binary. But AI models are non -deterministic by nature. You can ask the exact same language model the exact same question twice and get two slightly different paragraphs back. Right. So traditional testing methods like pinging an endpoint and checking for a 200 OK status code are dangerously insufficient.
Because the server is up, but the answer is garbage. Exactly. An AI app can be technically available while confidently outputting absolute hallucinatory garbage to your end users. So how do you test that? And where does platform engineering fit into this wildly subjective process? Platform engineering's role is to provide the reusable evaluation infrastructure. Instead of every product team writing their own bespoke Python scripts to test their prompts, the platform provides a centralized system. Okay. This system stores
versioned golden data sets, collections of standard questions and ideal answers. When a developer makes a change, the platform mechanically executes the tests against that data set, logs the results, and provides visual tools to compare the new model version against the old one. And the goal is to integrate these evaluations directly into the automated CI -CD pipeline. Exactly. So mechanically speaking, if a developer tweaks a system prompt to make the AI sound more polite, and their traditional code tests all pass, but that new polite prompt
accidentally causes the AI to fail at summarizing financial data, the pipeline can actually catch that performance drop. Yes, it flags it before it goes live. But wait, let me stop you there. Sure. If evaluation results are inherently noisy and often rely on things like an LLM as a judge scoring an output on a one to five scale for helpfulness. Right. How can you possibly use a subjective vibe check as an automated gate to block a production deployment? This raises an important question about the separation of
concerns. You have to separate the mechanism from the business logic. OK, unpack that for me. The platform provides the mechanism to run the test and halt the deployment. But the platform does not decide the rules. Ah. The platform team might enforce strict automated binary gates for security, like If a data loss prevention tool detects a social security number, the deployment is blocked, full stop. Makes sense. But for subjective quality, the application team defines that rubric. The platform simply provides the infrastructure
to run the LLM as a judge, score the output based on the app team's specific criteria, and route borderline cases to a human for manual review. So the platform doesn't decide what is correct or polite, it just... provides the scalable tools for the app team to make that call predictably. Exactly. So the platform team builds the testing arena and sets up the scoreboard, but the application team sets the rules of the game and decides what a winning score is. That is a perfect way to
put it. That makes perfect sense. Now, let's pivot to the elephant in the room whenever AI is mentioned, hardware and budget. Yes, the GPUs. Does building an AI platform mean an organization has to go out and buy a massive warehouse full of GPUs? What's fascinating here is how often people conflate AI software development with AI hardware management. They are very different things. Very different. The short answer is definitely not. An internal developer platform for AI does not inherently require building a custom GPU cluster. Right. If your developers are primarily
using managed external APIs, like calling out to OpenAI or AWS Bedrock, The platform's focus is entirely on secure access, routing, and cost control, not hardware provisioning. Right, because deciding to self -host your own open source models on physical GPUs introduces an entirely new universe of operational pain. Immense complexity. You're suddenly dealing with highly specialized hardware capacity planning, fighting with specific NVIDIA CUDA driver compatibilities, managing GPU memory
constraints. Sounds awful. You should only take on that massive operational burden if you have clear overriding business requirements, like incredibly low latency for high -frequency trading, or heavy continuous fine -tuning, or strict data privacy laws that legally mandate your data can never leave your building. Otherwise you're just taking on a massive headache for absolutely no reason. Exactly. But even if you avoid GPUs and stick to managed APIs, those API calls still cost money. Oh, they do. Every token processed is a fraction of a cent, and in a busy application,
they add up incredibly fast. Very fast. Cost management has to be a crucial capability for these platforms, because without it, a junior developer might write a recursive loop that accidentally calls a language model 10 ,000 times in a minute. And suddenly, you have a cloud build that requires an emergency meeting with the CFO. Yes. This is exactly where that AI gateway we talked about proves its worth mechanically. Right, because it's intercepting everything. Exactly. It physically intercepts every request and records the token usage tagged by the specific application. It
can expose exactly how many tokens an app is consuming and translate that into estimated dollar charges in real time. So they can see their daily spend right next to their code deployment metrics. Precisely. The financial data is surfaced directly to developers in their internal portal. OK, but if you have that visibility and costs start spiraling, shouldn't the platform just implement a hard automated spending limit? Like, the second an app hits its $500 daily budget, the gateway just forcefully blocks all further API requests. You
have to be incredibly careful with blunt instruments like that. Really, why? Well, in a sandbox or a development environment, yes, a hard limit makes total sense to stop a runaway script. Sure. But doing that in a production environment, think about the mechanics. You would be intentionally causing a customer facing outage over an internal budget threshold. Oh Right. If a customer is trying to check out of your e -commerce store and the AI product recommender hits a budget limit and takes the entire checkout page down
You've lost four more revenue than the API call would have cost. That would be disastrous. It would The platform needs to provide alerting and visibility, but the application teams and business owners need to define the graceful degradation strategies when thresholds are met. That is a huge distinction. Visibility versus blunt force. Okay, moving on to our final segment. Boundaries, data, and what the industry calls the help desk trap. This is a big one. We've covered prompt testing, gateways, hardware, and costs. But where
does platform engineering actually end? Because AI applications are incredibly thirsty for data. They aren't. If you're building a rag application, you need massive data pipelines to ingest company wikis, create those mathematical embeddings, index them. Does the platform team now have to own all the company's data engineering pipelines, too? No, and they absolutely shouldn't try. Thank goodness. Right. The platform shouldn't own the actual data sets or the complex data transformations. What they should do is provide supported integration patterns. What does that look like? They provide
the blessed storage services, like an approved vector database, and they provide standardized, secure ways for an application to connect to that database. We also need to draw a clear line between platform engineering and MLOps, machine learning operations. For anyone confused, how do those two differ? They are deeply related, but fundamentally distinct disciplines. MLOps is focused on the data science side of the house, mathematically training the models, managing the training data sets and operationalizing the
machine learning itself. Okay. And platform engineering? Platform engineering is about providing reusable capabilities to deliver software. If your company is large enough to train custom models, you might have an MLOps team handling the training and a platform team handling how app developers actually integrate and deploy those models into web apps. They overlap at the edges, but they shouldn't automatically be merged into one giant unfocused team. Exactly. Which brings us to the help desk trap. Because the AI ecosystem is so new and
exciting, every developer wants to try a different cutting -edge tool. Shiny Object Syndrome. Yes. And if the platform team just says yes to every single bespoke request, like, sure, we'll integrate that new vector database just for you, or sure, we'll build support for this obscure new agent framework just for your team, they stop being a strategic platform team. And they just become a reactive IT service desk. Right. And falling into that trap defeats the entire purpose of scalable platform engineering. A successful platform
roadmap has to be driven by overarching architectural patterns, not single feature requests. Okay. If you notice that five different product teams are all trying to build document retrieval systems from scratch, then you invest the time to build a standardized, secure rag template that solves the problem for everyone. So if I'm understanding this correctly, the platform team... needs to act more like a city planner, looking at traffic patterns and building an efficient highway system that benefits everyone, rather than acting like a personal chauffeur driving every individual
developer to their exact unique destination. That is a perfect analogy. And crucially, we can't forget that traditional platform engineering doesn't just vanish. Right. The basics still matter. You still need robust cloud networking. You still need secure secrets management, incident response protocols. AI is just an added, highly complex capability layer built on top of those traditional software foundations. So bringing this all together. So how do organizations actually measure success when they undertake these massive platform shifts? Because the industry consensus
is very clear on one thing. Success is not measured by the sheer number of AI applications a company deploys. Oh, I take it a step further. Measuring success by the number of deployments is a vanity metric. Yeah. A frictionless platform could make it incredibly easy to deploy 50 totally useless AI experiments that generate zero business value. Right. So what is the actual metric? It's measured by developer outcomes. It's the measurable reduction in custom infrastructure work. It's the aggregate time saved waiting for servers to be provisioned.
It's the tangible reliability of the services running in production. Exactly. Let's look at a practical example to summarize everything we've talked about. Imagine a team building an internal support assistant. Without an AI platform, they spend weeks researching providers, begging for a budget, managing API credentials and sticky notes. Spending up unapproved vector databases, writing custom logging scripts, and just fighting with infrastructure. But with a mature developer platform, that same team logs into their internal
portal, they select a standard RAG template, and instantly they are handed an isolated environment. It's just ready to go. Exactly. They get an approved model, a secure retrieval service, and a CI -CD pipeline with evaluation hooks already built in. Because the infrastructure is handled, they spend their valuable time defining the rubric of what a good support answer actually looks like for their specific users instead of wrestling with API keys. Platform engineering 2 .0 isn't about abandoning the original principles of reducing cognitive load and the standardizing infrastructure.
It's just about applying those exact same time -tested principles to these wild new probabilistic AI workloads. The next generation of internal developer platforms will not be defined by how many flashy AI tools they manage to integrate into a dashboard. Right. They will be defined by how effectively they help developers turn the raw, chaotic potential of AI into reliable, operational, and secure software. And that leaves us with a question for you listening right now. Look at your own organization's architecture.
Are you building an AI platform that paves the road and handles the heavy lifting? Or are your teams out there in the dark, independently reinventing the wheel and accidentally exposing your data along the way? It's a critical question. It really is. And as we wrap up, I want to leave you with one final thought to mull over, building on what we discussed about subjective testing and evals. We established that the platform team shouldn't decide if an AI -generated answer is correct. They only provide the testing tools for the application team to make that call. But think about the long
-term implications of this. As developer platforms get better and better, as they successfully automate away all the infrastructure, all the hardware configurations, and all the deployment bottlenecks of building AI, will the ultimate bottleneck in software engineering simply become human judgment? When the friction of building software disappears entirely, Will the hardest part of a developer's job in the future just be defining what correct actually means? It's a profound shift in what
it means to be an engineer. I mean, once the friction of building is gone, the only friction left is deciding what is actually worth building and how we measure its truth. A fascinating problem to have. Thanks for joining us for this deep dive. We'll see you next time.
Your engineering team may be using more AI tools than you realize. But do you know which ones have access to your code, credentials, and production systems? From AI coding assistants and locally running models to...
AI agents are moving beyond generating answers. They're connecting to real infrastructure, accessing internal systems, and interacting with production environments. But who controls what they're allowed to do? The Model...
What happens when your AI application is running perfectly, but its answers are wrong? Traditional Site Reliability Engineering focuses on availability, latency, error rates, and infrastructure performance. But...
AI coding assistants are helping developers write code, generate tests, and build applications faster than ever. But is the rest of the software delivery process keeping up? In this episode of DevOps Academy Deep Dive,...
Modern engineering teams collect more telemetry than ever, yet getting meaningful visibility into production can still require extensive application instrumentation. In this episode of DevOps Academy Deep Dive, we...