A Power Automate flow can start as a simple productivity shortcut and quietly become business-critical production software. In this episode, Alex and Maya examine what changes when automation has to operate reliably at scale, including workload management, concurrency, downstream limits, retries, error handling, observability, ownership, application lifecycle management, and architectural boundaries. The discussion begins by reframing a production flow as a small distributed system rather than a simple sequence of actions. Alex and Maya explore the multiplication problem, why broad triggers and loops can generate far more work than the flow designer suggests, and why increasing concurrency can simply move a bottleneck downstream. They then examine throttling, HTTP 429 responses, service protection limits, dynamic backoff, safe retries, stable business keys, and idempotent operations. The episode then moves into production engineering practices: explicit failure paths using scopes and run-after conditions, actionable observability with Application Insights and KQL, service-principal ownership, managed solutions, environment variables, connection references, and modular flow design. The conversation also examines when Power Automate should hand work off to Azure Functions, desktop flows, or message queues instead of trying to perform every task itself. Finally, Alex and Maya distinguish deterministic workflows from agentic workflows and explore how AI agents can complement governed automation when reasoning is required without replacing the predictable execution boundaries of production systems. The central lesson is straightforward: a production flow should be engineered for failure, observability, scale, and change - not simply designed to work when everything goes right. This episode is based on the Support Engineering Blog article “When a Power Automate Flow Becomes Production Software”, including its production architecture model, runtime and performance guidance, resilience patterns, observability practices, ALM approach, modularity principles, and supporting diagrams. Chapters 00:00 - From Simple Flow to Production Software 02:00 - The Three Planes of Production 03:43 - The Multiplication Problem 05:35 - Concurrency, Throttling, and Safe Retries 09:08 - Failure Handling and Observability 12:09 - Production Ownership, ALM, and Modularity 16:00 - Choosing the Right Architecture 18:00 - Deterministic Workflows and AI 20:08 - Engineer for Failure, Not Luck Learn more: The full technical article, references and supporting resources are available through the Support Engineering Blog at https://blog.sadhan.ch/.
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
You are listening to Support Engineering Weekly. You know the feeling. It's, um, it's honestly incredibly satisfying. You just drag and drop a trigger onto the canvas, add a few actions, authenticate a connector. And within minutes, you have a working power, automate flow. Oh, totally. I mean, it really feels like magic, right? The barrier to entry is just beautifully low. Yeah, exactly. You can automate some tedious daily tasks, like, I don't know, moving email attachments to a folder before your morning coffee even gets cold.
Right. But then, you know, something happens. That simple little flow gets shared. Yeah, always. Another department starts using it. And suddenly your cute little five recorded a convenient script is processing thousands of transactions. It has quietly become, well, business critical software. Yep. And when it breaks, it is no longer just a minor annoyance. Oh no. It is a full blown production incident. I'm Alex, by the way. And thank you so much for joining us. And I'm Maya. We're really glad to have you here. Welcome to this deep dive for sport engineering weekly.
Today, we have a very specific mission based on some brilliant engineering analysis by Satan Chandra. We do. We are going to fundamentally transform how you think about this platform. Right. We really need to shift from treating power, automate as just a simple workflow designer. We need to start engineering it as a robust, resilient production automation platform. And to make that transition, I mean, you have to realize that a flow in a production environment is never just a sequence of steps running in isolation.
Right. It's much bigger than that. Exactly. Architecturally speaking, it is a small distributed system. So it carries all the risks, the latency issues and the failure requirements of any distributed architecture. Let's break down how we should actually be visualizing that architecture, because honestly viewing it as a vertical list of boxes on a screen, it kind of hides the actual complexity. It really does hide it. If you look at the architecture of true production automation, you need to visualize it across three distinct planes. Okay. Three planes. What's the first one?
First, you have the control plane. This sits above your flow. It encompasses your governance, your application, lifecycle management, your data loss prevention policies and, you know, your environment strategy. Right. So the high level rules. Yeah. Then second, you have the execution plane. This is the power automate runtime itself, where your triggers, your actions and your concurrency are actually evaluated. Okay. And the third, finally sitting underneath all of that, you have the visibility plane. So this is your observability, your telemetry alerts, audit logs, all of that.
That makes a lot of sense. So building a basic flow in the designers, really just building a single car. You put the engine in, attach the wheels and it drives down the driveway. Cause that's right. Exactly. You built a car. But building production automation is, well, it's like designing the entire highway system. You can't just put cars on the road. You have to account for traffic jams. Oh, absolutely. You need speed limits to prevent chaos, emergency lanes for when things break down and traffic cameras to see what's happening. The surrounding planes like governance and observability, that's the infrastructure
making the highway viable. I love that analogy because without that infrastructure, a high speed pileup is basically inevitable. Right. When you deploy to production, that execution plane has to account for harsh realities. Volume dictates how much data is moving through. Concurrency dictates how many things happen at once. Yeah. And you have those constantly shifting dependencies on external APIs. Yeah. You always have to deal with APIs, which means, you know, the biggest danger isn't necessarily a flaw in the road itself. The biggest danger is how much traffic you dump onto it.
Yes. And that brings us to performance engineering, specifically what is referred to as the multiplication problem. Well, this is a big one. It is because there seems to be a fundamental disconnect in how people visualize the work a flow is actually doing. Break that down for us. What do you mean? Well, the number of actions you see on your screen in the flow designer does not equal the workload generated by the flow. It almost always starts at the trigger. Right. The very first step. Yeah. The most massive anti-pattern we see is starting with a broad trigger, pulling
in every possible record from a system and then using a condition action inside the flow to filter out what you don't need. So wait, you're triggering the flow 10,000 times just to say, never mind, to 9,900 of them. Exactly. And you pay a massive compute and platform limit penalty for work you didn't even need to do. Wow. That is hugely inefficient. It really is. The correct approach is to filter at the source. The pattern should always be a relevant event triggers the flow. You retrieve only what is strictly needed using scoped O data queries or trigger
conditions, and then you process it. So if the API can do the filtering before the data ever reaches power automate, you let the API do the heavy lifting spot on, let the API do the work. And this multiplication problem gets exponentially wilder when we introduce loops, doesn't it? Oh, it gets terrifying. Let's say you have an apply to each loop inside that loop. You only have three simple actions. Right. So it looks tiny on the screen. It looks harmless. But if your trigger pulls in 2000 records and passes them into that loop, you didn't just
build a three action flow. You generated 6000 individual connector requests instantly. Six thousand just from three little boxes. Yep. And every single one of those requests consumes API calls, memory and processing time. OK, here is where I want to push back a little bit on how we handle that volume. Sure. Go for it. Because if I am staring at a massive queue of thousands of records, my instinct as a problem solver is efficiency, right? Naturally, if I leave the concurrency control off, it processes one at a time.
It might take hours or even hit a timeout limit. So why shouldn't I just go into the settings, crank up the concurrency control to like 50 and run all those loops in parallel? Let the flow blast through the queue as fast as possible. It sounds like a great idea, but faster execution is not always better execution and a distributed system. Why not? Because your flow does not exist in a vacuum. Every action inside that loop is calling an API. If you crank up the concurrency, yeah, you shorten the runtime of the flow itself, but you simultaneously overwhelm your downstream dependencies.
Oh, right, because they're taking all those hits at once. Exactly. You are firing thousands of requests per second at a SharePoint list and SQL database or some third party sauce app. So you aren't fixing the bottleneck? No, you're just moving it. And usually you're breaking the downstream system in the process. The downstream system isn't just going to quietly accept being d-dossed by our own internal automation, is it? No, it's going to defend itself. How does it do that? It will defend itself through throttling. When you overwhelm a dependency, you get slapped with an HTTP 429 error.
Too many requests. The dreaded 429. Yep. Or if you are hitting data verse, you hit service protection limits. The API essentially tells your flow to back off because it simply cannot process any more data. So a resilient production flow needs a built in strategy for those 429 errors. It can't just fail the whole run. Right. It requires dynamic back off intervals. The flow needs to read the retry after header from the rejected request, wait that specific amount of time and then try again.
That makes sense. But your retry strategy has to be a conscious part of your workload model. It can't be an afterthought. Think about buying concept tickets online. You know, the website freezes, you panic and you double click the checkout button. We've all been there. Right. If a flow sends a request to create a record, times out waiting for the response and then automatically retries, we risk creating a duplicate record. Exactly. We know we need a dampadency so we don't accidentally buy two tickets. But actually achieving a dampadency across different third party APIs and power
automated is notoriously difficult. Yeah. How do we build that in? So retries are actually safe. You have to utilize stable business keys instead of relying on auto generated database IDs. OK, give me an example of that. Let's say you are sinking an invoice. You don't just send a create record command. You first query the system using the unique invoice number as a correlation ID and wherever possible you use up cert style operations. Up cert, meaning up data insert. Exactly. An up cert checks if the record exists based on that stable key.
If it exists, it updates it. If it doesn't, it inserts it. Odd smart. Right. The operation is designed so that if the exact same business event is retried three times because of network latency, it yields the exact same intended outcome without unintended side effects. So if your flow crosses a system boundary, making retrial safe is absolutely non-negotiable, but what happens if a retry isn't safe or what if the back off maxes out after four attempts and the action just completely permanently fails, right?
In a production system, it can't just freeze and hope someone notices. No, the system needs to know exactly where to route the failure. How do we do that? Error handling must be an intentional explicit part of the architecture. In a production flow, you use what are called scopes combined with run after conditions to build those routing paths. That sounds exactly like a try catch block in traditional software engineering. It is entirely the power automated equivalent. You put your main business logic inside a primary scope. That's your try block. OK. And if everything succeeds, the flow moves down the happy path.
Exactly. But you create a second scope directly underneath it, configured to only run after the first scope fails or times out. That is your catch block. So the failure path takes over, but the failure path needs to be smart enough to know what kind of failure just happened. Yes, it has to differentiate because transient failures like throttling get those controlled retries we just discussed. But what if it's a business rule failure, like, I don't know, a missing approval manager? In that case, the flow needs to capture that and route it to a human queue for review.
Or what about a configuration failure? Say a service count password expires and a connector gets revoked. Retrying a bad password 50 times isn't going to fix it. Definitely not. In that scenario, the failure path must immediately terminate the retry loop, stop the flow and escalate a critical alert to an operations team. But if our flow is escalating these critical failures, the operator on the other end is flying blind unless they have the tool of telemetry to see what just crashed. Very true. My instinct here is to say we should just log absolutely every single step,
every variable and every JSON payload in the flow to a custom database. That way troubleshooting is incredibly easy. Why is that considered an anti-pattern? Because you will drown yourself in noise and you'll fit data storage limits incredibly fast. Oh, I see. The goal of observability is actionable evidence. If your operating model requires an engineer to manually open individual, graphical flow runs in the power automate UI to figure out what went wrong. Your model is not sustainable at scale. Yeah, clicking through a haystack of green check marks and red Xs for a flow
that runs 10,000 times a day, that sounds impossible. It is impossible. So how do we get that actionable evidence without logging the entire 10 megabyte payload of a file transfer? A true production operating model uses application insights and KQL Cousteau query language for fleet level diagnostics. OK, so moving away from the UI. Exactly. Instead of clicking through a UI, an engineer writes a KQL query to instantly aggregate failure rates across thousands of runs. They can pinpoint a sudden spike in four hundred and twenty nine throttling errors
across the whole environment in seconds. That is so much faster. Right. You only use custom logging for specific business context. You log the business key, the correlation ID and the specific error state. You leave the heavy payload in the source system. That makes perfect sense. Yeah. But if a system requires this level of custom telemetry, explicit failure routing and I dampen it in retries, it is highly engineered software. Yes, it is. You can't just leave that running under the user credentials of the person who originally clicked create flow, right?
Tying a distributed architecture to a single employees user account feels like a massive operational risk. It is. The human element is the most dangerous dependency. If a business critical automation depends on a single employee's credentials, the workflow stops entirely the moment they change roles or leave the company or their password policies force a reset during a weekend. Exactly. For production flows, you must advocate for service principal ownership. You have to separate the identity running the flow from any actual human user,
which ties directly into how we deploy the software. We use application lifecycle management or ALM. Yes, ALM is crucial. We know development happens in unmanaged solutions. But test and production environments should only receive managed solutions. Why do managed solutions specifically protect the production environment? Because a managed solution locks down the definition of the flow, it prevents users from making direct edits in the production environment. So no quick fixes on the fly. Right. You cannot just hit edit, change a variable and hit save.
All changes must be made in the unmanaged development environment committed to version control and pushed through a deployment pipeline. It guarantees that the code running in production is the exact code that passed testing moving a managed solution from development to production introduces a mechanical challenge, though. The database connections or email addresses are going to change between those environments. How do we handle that without editing the flow? You keep configuration entirely out of the flow definition. Oh, interesting. How you use environment variables for your settings,
like SharePoint site URLs or email addresses and connection references for your authentication. I see. So when the managed solution moves from dev to test to prod, the flow definition remains identical, but the environment variables dynamically swap in the correct production endpoints when the package arrives. That is incredibly clean. Oh, so we have it properly deployed. It's running under a service principle. It's fully managed. That's perfect so far. But over time, the business keeps asking for just one more feature. Oh, always scope creep.
Right. They want the flow to send a message to a team channel and then update a spreadsheet and then generate a PDF report. Suddenly we have created the dreaded mega flow. A mega flow. Yes. How do we know when a flow has officially become a mega flow that needs to be broken apart? Well, the most glaring warning sign is unrelated responsibilities. If your flow is reading an inbox, parsing data, updating a database and handling employee onboarding all at once, you have built a monolith. Wow. Yeah, that is way too much.
Another sign is deep impossible to read nesting of conditional statements where you are eight levels deep in if yes, branches. Do those mega flows fail differently than modular ones? They do. And that reveals the most critical red flag unrelated failure domains. Think about the execution boundaries. OK. If the step that sends a notification email fails because of a temporary network blip, should it cause the entire database update process to fail and roll back? No, obviously not. Right. So if the answer is no, those two tasks should not be tightly coupled in the same
execution path. Got it. So you split the flow to create clear execution and failure boundaries. One parent flow can receive the workload and it can hand off the heavy data processing to a child flow and hand off the notification to another child flow. Yes, they execute independently. Modularity isolates your failures. The parent flow acts as an orchestrator while the child flows act as specialized workers. Exactly. But splitting a mega flow apart often reveals something a bit uncomfortable. Some of these tasks probably shouldn't be handled by power automated at all.
Oh, for sure. I've seen cloud flows timeout when they try to process a massive data set or run complex mathematical loops. It feels like trying to run a marathon in a sprint lane. If power automate can't handle the heavy compute, where do we upload that? That is the ultimate architectural maturity moment. The goal of enterprise governance is not to maximize your usage of power automate. That's a good point. The goal is to build the most reliable automation system using the right tools. Cloud flows are orchestration engines, not compute engines.
They're subject to strict timeout limits and API constraints. So what's the alternative for the heavy lifting for heavy complex compute? You need to hand that workload off to a dedicated service like Azure functions, which can process the code efficiently and return the result to the flow. OK, but what if the task involves interacting with a legacy mainframe or say a web portal that doesn't have an API available? A cloud flow can't send an HTTP request if there's no endpoint to receive it. That is when you pivot to desktop flows, which is robotic process automation or RPA.
RPA, right? RPA simulates human interaction with the user interface. It literally clicks the buttons and reads the screen when modern API simply don't exist. And what if the workload is incredibly spiky? Let's say we get zero traffic all week and then 10,000 requests hit the system in five minutes on a Friday afternoon, a classic scenario. If we dumped 10,000 concurrent triggers directly into a flow, it's going to hit platform limits, start returning four twenty nines to the source and we will permanently drop data. Yes, which is why for spiky workloads, you never expose the flow directly to the source
event. What do you do instead? You put a message queue in front of the process like Azure service bus. The message QAC is a shock absorber. It durably accepts and holds all 10,000 requests instantly. Oh, I see. Then your flow is configured to pull messages off that queue at a safe controlled concurrency rate. The queue absorbs the spike and the flow processes the backlog steadily without ever hitting a limit. That makes perfect architectural sense. It is all incredibly structured deterministic logic. But you know, looking at the broader landscape of technology right now, I have
to ask about the impact of artificial intelligence. AI, I knew we'd get here. We have to with AI agents becoming capable of independent reasoning, writing code and navigating systems. Are these rigid, carefully engineered workflows becoming obsolete? Should we stop building complex try catch scopes and just hand the goal to an AI agent to figure out the routing on the fly? It is the most common architectural debate happening right now. But assuming AI replaces structured automation relies on a misunderstanding of what workflows actually do.
Really? How so? We have to clarify the distinction between a deterministic workflow and an agent workflow. Where is the line between them? Well, a deterministic workflow, what we've been talking about this whole time, is used for processes that require strict, predictable and audible rules. Think of financial compliance, updating a system of record or sinking inventory. Right. You absolutely do not want creativity in those processes. No, you want the system to do exactly what it is told, the exact same way, every single time producing a predictable JSON schema.
Yeah, I definitely don't want a non deterministic, large language model getting creative with my payroll deductions or guessing how to update a database schema. Exactly. An agentic workflow, on the other hand, is utilized when reasoning and adaptation are genuinely needed because the exact path to the solution isn't known in advance. Like what? For example, analyzing a furious customer email, extracting the sentiment, determining which internal department is best equipped to handle it and drafting a customer response, that requires reasoning for they aren't competitors fighting
for the same workloads. Not at all. They are completely complimentary. The deterministic workflow handles the rigid governed data movement when it reaches a step that requires human like reasoning, the workflow calls out to an AI agent. I see. It waits for the agent to make a decision and then takes that decision and executes it safely within the governed, auditable boundaries of the deterministic system. That is fascinating. Let's synthesize everything we've covered today because this really is a massive paradigm shift.
Truly is. What we've learned is that a production flow is fundamentally a distributed system. It requires intentional workload management, filtering at the source, controlling loops and respecting downstream limits. Yes. And it demands safe, a depotent retries and explicit error paths using scopes. Right. It requires actionable observability using tools like KQL rather than just logging noise and it relies on durable application lifecycle management, service principles and the right architectural tools to survive in the real world. Applying those principles means you stop hoping the flow succeeds and you start
actively engineering it to survive when things inevitably fail, which brings us to a final provocative thought for you to take away from this deep dive. I want you to think about the environment you manage right now. Look at the most successful flows in your organization, the ones that have been running quietly in the background for months, the ones nobody worries about. Yeah. Take a close look at those. Are they actually engineered for failure, observability and scale or have they just been lucky so far? Because in a distributed system, luck always runs out eventually.
You've been listening to Support Engineering Weekly from a Support Engineering Blog. For the full articles, references and technical resources from today's discussion, visit the URL blog.sadhan.ch. Until next week, keep troubleshooting, keep learning and keep engineering. Thank you for listening to Support Engineering Weekly. This podcast is available on Apple podcast, Spotify, iHeartRadio and other major podcast platforms. Follow the show on your preferred platform and never miss an episode.
What happens when a routine network maintenance operation triggers a failure deep inside a cloud provider's authentication infrastructure? In this week's episode, Alex and Maya investigate how a seemingly isolated...
How can a routine network capacity upgrade trigger a global cloud outage - and why can the very systems designed to protect availability make the failure worse? In this Support Engineering Weekly bonus episode, Alex and...
Microsoft 365 administrators deal with information from several places: the Microsoft 365 Roadmap, Message Center, and Service Health. But knowing where to find an update is only part of the challenge. The real question...
A server under pressure fails its health checks. Automated traffic management removes it from service, pushing even more load onto the remaining infrastructure. What was designed to protect the system becomes part of...
Microsoft Project Online is approaching retirement - or, in this case, its retirement date has already passed. But replacing a project management platform is not as simple as moving tasks from one product to another. In...