Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Support Engineering Weekly

Support Engineering Weekly artwork

Dataverse Isn't Just a Database

Oct 4, 202625 minAudio

Why is Microsoft Dataverse so much more than a database? In this episode, Alex and Maya examine Dataverse as a managed application data platform and explore how metadata, security, APIs, storage, capacity, integration, analytics, governance, and application lifecycle management work together to support enterprise workloads. The discussion follows what happens when a request reaches Dataverse and explains why the platform's layered architecture changes how engineers should think about schema changes, metadata, standard versus custom tables, storage models, capacity planning, and security. The episode also examines the difference between operational and analytical workloads, API service protection limits, HTTP 429 responses, backoff, bounded concurrency, idempotency, and the performance implications of synchronous plugins. The conversation then moves into governance and ALM, including solutions, managed versus unmanaged changes, deployment discipline, and why direct production customization can undermine the integrity of the deployment pipeline. Finally, Alex and Maya look at the growing role of AI agents and why clean metadata, well-defined security, lifecycle repeatability, predictable capacity, and resilient integrations become even more important as Dataverse becomes an operational data foundation for AI-enabled workloads. The central lesson is straightforward: Dataverse should be engineered as a governed application platform, not treated as an isolated database. This episode is based on the Support Engineering Blog article “Dataverse Isn't Just a Database”, including its layered architecture, metadata model, storage and capacity guidance, security model, integration patterns, ALM principles, and supporting diagrams. Chapters 00:00 - Dataverse Isn't Just a Database 02:00 - The Six-Layer Dataverse Architecture 04:28 - Metadata as the Platform Contract 07:00 - Storage Engines and Capacity 11:12 - The Seven-Layer Security Gauntlet 14:48 - APIs, Analytics, and Throttling 18:09 - Plugins, Governance, and ALM 20:50 - Dataverse Anti-Patterns 22:12 - AI and the Platform Contract Learn more: The full technical article, references and supporting resources are available through the Support Engineering Blog at https://blog.sadhan.ch/.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

You are listening to Support Engineering Weekly. You know, it is incredibly common for engineers, architects, I mean, even really seasoned developers to treat Microsoft data versus just like a simple backend database. Oh, absolutely. It happens all the time. Right. You spin up a power platform application. You realize you need a place to just kind of dump the user data and boom, you just create some table. Yeah, you just treat it like any other SQL bucket. Exactly. It feels very familiar, very standard. But assuming it is just a relational SQL bucket where rows and columns just, you know,

live in isolation, yeah, that assumption is an absolute enterprise trap. It really is. I mean, treating data versus like a basic database completely ignores the reality of what the system actually is mechanically. Yeah, it's so much more than that. It is a massive, multi-layered, managed application data platform. Like the moment you try to force it to behave like a standard isolated database. Using raw queries and stuff. Yeah, writing raw queries bypassing the API or just ignoring its life cycle rules entirely, things start breaking down catastrophically at scale.

I'm Alex, by the way. And I'm Maya. And we want to sincerely thank you for joining today's deep dives into our source material, because our mission today is to completely dissect the dataverse architecture. We really want to get into the weeds here. We do. We are going to look at how metadata, security, API's capacity, and application life cycle management all interact under the hood. Because once you understand how these specific gears turned together, you can actually engineer a predictable governed platform.

Right. Rather than just kind of crossing your fingers and hoping your apps don't crash when, you know, user traffic spikes. Yeah, hope is not a great engineering strategy to really understand why this platform is fundamentally different from a standalone SQL server. We have to trace what actually happens when a request hits the system. Okay, let's unpack this. Because the architecture here is often described in our sources as a six layer kick. Right. That's right. When an application saves a record, it is not a straight line to a hard drive. It doesn't just, you know, write to disk.

So think of dataverse, not as a simple storage unit, but as a, like a high security corporate headquarters, you don't just walk in off the street and drop a file on a desk in the basement vault. No, they would tackle you. Exactly. First you have to go through the front desk. Those are your API's. Then you have to get badge clearance, which is the security layer. Right. Checking if you're even allowed in the building. Yeah. After that, you have to follow company policy, meaning the platform services and business logic layers have to approve your action, which is a whole other

set of checks. Right. And only after you clear all those hurdles, does your file actually take the elevator down to the vault, which is the storage layer. That is a highly accurate way to visualize it. And covering all of that, you have the experience layer at the very top. Oh, like the user interfaces. Yeah. This includes your canvas apps, your model driven apps, and your co-pilot agents. You also have the integration and analytics layer, which maps the data outward to other enterprise systems. Plus, there is a governance and operations layer overlaying the entire stack. So when a JSON payload comes

in from say, a custom app or power automate flow, it has to physically pass through that API access layer. It has to run the gauntlet. Exactly. It navigates the platform services, the metadata checks, the security engine, all of that before a single byte is ever committed to storage, which means we completely have to change how we think about making system updates. Because if you tweak something in dataverse, you aren't just like altering a table definition in a vacuum. What's fascinating here is why this matters operationally. A schema change in this environment is not just a database tweet because of all

those layers. Yes, because of those interconnected layers, adding a column or changing a relationship alters the behavior of applications, API's and security almost instantly instantly. Yeah, you modify a table and suddenly the API latency might shift because of how the data is being routed. Or a synchronous plugin firing at the platform services layer might behave completely differently because the data structure it expects just changed. Wow. So if a single request has to pass through the front desk, the security gate, and the policy checks, how does the system actually know what rules to apply to that specific request in real time?

Well, the architecture uses metadata as the glue holding all these layers together. But wait, isn't metadata just data about data? Like if I right click a photo on my laptop, the metadata just the data file was created or the camera model used. Yeah. Why is it treated as such a big deal in this architecture? I mean, in a traditional file system, you absolutely write metadata is mostly just descriptive tags sitting passively on a file. But data versus fundamentally a metadata driven architecture. It's not passive. Not at all. Here, metadata is active and structural. It dictates engine

behavior. What does that mean? Practically, it means the table metadata defines not just what data type is stored in a column, but how records are created, what specific actions can be performed on them by the API. Oh, wow. Yeah. And how forms and views are dynamically generated for the user interface and even how the cascading security rules behave. It is the active rule book that the platform services layer reads continuously. So it's not just describing the table. The metadata is actively generating the API endpoints

and the application behavior for that table on the fly. Exactly that. The source material explicitly refers to the schema as a platform contract, not just a database structure. A contract. I like that. When you define a table in Dataverse, you are signing a strict contract with the rest of the stack about how that entity will behave across every single layer we just talked about, which totally explains why the architectural guidance is so strict about using standard out of the box tables whenever they fit the business concept. Oh, absolutely. Like if you need a table to track accounts or contacts, the recommendation is to use the built

in ones rather than building your own custom, I don't know, my company accounts table from scratch. But why? If I build it myself, I control all the columns. Well, you do control the column, sure, but you lose the platform contract. You really have to question your assumptions here about control versus technical debt. Okay, so if you build unnecessary custom tables, instead of using the standard ones, you are throwing away thousands of hours of Microsoft's engineering. Right, because they already did the work. Exactly. The standard tables already have

highly optimized relationships, complex security models, and specific API behaviors hardwired into the platform's engine. So when you reinvent the wheel, you lose all that out of the box optimization. And you take on the burden of maintaining a much more fragile custom contract that frankly might not perform as well under heavy load. Right. So assuming we've honored the metadata contract and we aren't reinventing the wheel, the data finally makes it down to the vault, the storage layer, the bottom of the cake. Right. But again, the trap we mentioned earlier is thinking

data versus just tosses everything into one massive relational SQL database. It absolutely doesn't. And assuming it does, we'll ruin your capacity planning. So what is it actually doing down there? Dataverse acts as a router. It sends data to highly specialized storage engines under the hood based on the actual workload. Like different types of vaults. Yeah. For your transactional, relational business, beta like accounts, invoices, complex relationships, it uses standard or custom tables, which are backed by Azure SQL. Okay, that makes sense for standard business logic.

But for high volume, bursty semi structured workloads like IoT sensor data or interaction logs, Dataverse offers elastic tables. And those are actually backed by Azure Cosmos DB. And this is a huge trap I see in enterprise deployments all the time. People treat elastic tables like they're just a toggle switch for faster standard tables. Oh, I've seen that too. They look at their data. Think, Oh, I have a lot of rows coming in quickly. I'll just use the elastic table for better performance. Why is applying an elastic table to a standard

business problem? Such a dangerous architectural move. Because Cosmos DB, which powers those elastic tables operates on completely different database principles than a traditional relational SQL database. Right. It's not SQL. No, when we talk about standard SQL tables, we rely on AC transactions, atomicity, consistency, isolation, durability. Right. The golden rules of SQL. Exactly. If you save an account in a contact at the exact same time as part of a single transaction,

ACD guarantees that either both save successfully or neither saves. It's perfectly consistent. But Cosmos DB doesn't play by those exact same rules. Right. Cosmos DB is designed for a horizontal scale and speed, often relying on eventual consistency. The eventual consistency. So not instant. Right. The data is partitioned differently. If you try to build a highly relational ACD dependent financial business process on top of an elastic table, just because you wanted the rights to be, you know, fast, you are going to encounter massive architectural friction.

It sounds messy. It is. The data might not be instantly available across all nodes. The millisecond is written. You have to match the data model mechanically to the specific workload. And the mechanical complexity that storage directly impacts capacity planning to is significantly because capacity and dataverse isn't just a single flat metric of how many gigabytes do I have left? The source material breaks it into three distinct buckets. Yes. Database capacity for your structured data and active metadata file capacity for heavy things like PDFs and image

attachments and log capacity for audit and trace data. If we connect this to the bigger picture, treating those three distinct buckets as one single storage problem leads to massive cost overruns. I can imagine. Let's look closely at auditing actually because that is where organizations really bleed money. It's a huge issue. Say a company migrates their legacy system to dataverse and the internal compliance team issues a mandate. We need to audit everything. Oh no. Right. Turn on auditing for every single table, every column and track every single read and write.

I can hear the billing alarms going off in my head already. You should because all of that tracking data flows directly into that third bucket log capacity. Right. Mechanically, every time a user simply opens a record to look at it, the system has to generate a separate log entry and write it to storage just for opening it. Yes. If you blindly turn on comprehensive auditing for the entire system, your log capacity explodes exponentially. It will dwarf your actual business data in a matter of weeks. That's insane. And that directly impacts your compliance operations

and your monthly storage costs. Exactly. Audit design is fundamentally a capacity and cost decision, not just a checkbox on a security document. You have to forecast the growth curve for database, file and log capacity completely separately. Okay. So we have data landing in these highly optimized storage buckets and we've got audit logs generating massive capacity to monitor activity. But monitoring after the fact isn't actual protection. No, it's just a record of what happened. Right. The real question is how the system guards the front door and the vault at the exact same time.

Security in this architecture is cumulative. It's not a single toggle switch where you are either an admin or a guest. It's not binary. It is a gauntlet of seven distinct layers that every single request has to survive sequentially to gain effective access to a piece of data. Let's run through them because of the sheer depth of this is wild. First, the request hits Microsoft Antra ID for baseline authentication. Right. Just to prove you are who you say you are. Right. Assuming the system knows who you are, it checks licensing to see if you are legally

allowed to use the platform. Then it checks the environment boundary. Are you allowed in this specific sandbox? And that's just to get into the outer perimeter. Exactly. Once you were in the environment, the system checks your specific security roles. But roles are enough on their own. Right. There's more. It then checks your business unit or team affiliation to see where you sit in the company hierarchy. Then it evaluates ownership and sharing. Do you actually own the record you were trying to read or has someone explicitly shared it with you? And finally,

even if you pass all of that, it checks column security. Yes. Like you might be able to see the employee record, but the system strips out their social security number column before sending it to your screen. It's exactly like traveling internationally. That's an excellent way to visualize the mechanics of it. Think about it. You need your passport just to get through the airport doors. That's Antra ID. You need a valid plane ticket. That's your dataverse license. You need clearance to enter a specific terminal. That's your environment. Right. Right. And then

you need your specific seat assignment on the plane. Those are your roles, business units, and ownership rules. If you are missing just one of those pieces of paper, you aren't flying. The access is entirely denied. And that cumulative nature is exactly why application access does not automatically equal data access. Well, that's a big one. This confuses developers new to the platform all the time. Just because a user has the link to open a power app and the app loads on their phone doesn't mean they can see the records inside it because they still have to pass the data

security column. Precisely. If their business unit restricts them or they lack the specific record ownership, the system will render the app interface perfectly, but the galleries will just be blank. The data is blocked at the platform level. The architecture warning from our source material is incredibly clear on this point. Security must be designed before a large-scale data migration. I've seen the aftermath of ignoring that warning. You import like five million records and just assign them all to a generic system admin account to get it done quickly. Oh, the

classic quick fix. But then you realize you need granular security. So you try to retrofit those seven layers of cumulative security afterward. It is an absolute nightmare to untangle because the system has to recalculate the cascading access rules for millions of rows is practically impossible to retrofit without causing severe operational downtime. The system just locks up. The platform will literally choke on the recapulation jobs. You have to mechanically map out those roles, teams and ownership models and migrate the data with the correct owner from day one.

Okay, so guarding the data from internal users clicking around a UI is one piece of the puzzle. The security gauntlet handles that. But what happens when an automated external system, like a legacy enterprise platform or a data warehousing tool, tries to pull millions of rows out of data verse programmatically. That brings us back up to the API and integrations layer. Okay. The primary interfaces developers use to talk to the system are the Dataverse Web API, which implements the OData version four standard, the SDK for .NET and the TDS endpoint. Right, TDS.

That stands for tabular data stream, which basically provides read only SQL style query access for reporting tool. Exactly. And the crucial rule here, the one that architects seem to get wrong time and time again, is keeping analytical workloads completely out of the operational transaction path. Absolutely vital. Dataverse is an operational platform. Its primary job is to run your live business transactions smoothly. Right. It's not a warehouse. No. If you have an external data science team that wants to run massive historical aggregation queries,

like summing up five years of sales data, and they point their BI tools directly at the Dataverse Web API, they're going to consume all the computing resources that your live users need to do their jobs, which brings the whole system to a crawl. Dataverse data should flow to Microsoft fabric or one lake for analytics. Yeah. You do your heavy data warehousing there safely away from the APIs that your live apps rely on. Because if you overwhelm those live APIs with heavy queries, the system is going to protect itself. Yeah. Dataverse has built in service protection limits.

Yes, it will push back. If an external system hammers the API too hard, Dataverse throws an HTTP 429 error, which explicitly includes a retry after signal. And here's where it gets really interesting. I've heard it's fascinating. I've seen integrations where developers treat a 429 error like a standard network blip. So you just program their system to aggressively retry the request immediately, which is the worst thing you can do. Wait, but from a developer's perspective, if a server drops my request, my first instinct is to try again immediately. So my user

doesn't get a failure message. Why does Dataverse punish that? Because a 429 isn't a network drop. It is a deliberate protective action. Dataverse is explicitly telling you my CPU is overwhelmed, stop talking to me for exactly this many seconds. Oh, so it's a command, not a glitch. Exactly. If you ignore that mathematical pause and hit the API again instantly, the system registers you as a threat to its stability. It penalizes you further, pushing you deeper into the penalty box, and your application crashes harder.

So you end up taking your own integration offline just by being impatient. Precisely why robust integrations require specific mechanical designs. You need back off logic, which means waiting exponentially longer between retries. You need bounded concurrency, which means strictly limiting how many active threads your external app opens against Dataverse at once. And most importantly, you need idempotency. Let's define idempotency for a second, because that's a turn that gets thrown around a lot mechanically. What does that actually look like? Idempotency means designing your request so that if a network drop does cause your external app to

retry sending a create account payload three times, Dataverse knows to only create that account once. Rather than generating a mess of duplicate records. Exactly. You usually achieve this by passing a unique transaction ID with the payload. So Dataverse recognizes the duplicate request and safely ignores it. You have to design external systems to cooperate with Dataverse's limits, not fight them. And that philosophy extends to how we build internal business logic too. The source material strongly warns against turning every single business requirement

into a synchronous plugin. Yes, a synchronous plugin operates on the main thread of the transaction. If a user clicks save and you have custom C sharp code running synchronously, the database transaction literally halts until your code finishes executing, which means if your code is reaching out to an external web service that takes five seconds to respond, the user is stuck staring at a loading spinner for five seconds. And doing that across thousands of users creates massive performance bottlenecks. Yeah, that scales terribly. It does. You have to ask yourself mechanically. Does this logic actually need to happen right

this exact second to maintain data integrity? Or can it be handled by an asynchronous web hook or a service bus event in the background? It's all about respecting the platform's mechanical boundaries. But if an organization ignores all of this, if they ignore the metadata contract, if they dump everything into one storage type, if they let analytical queries hammer the live API's, they end up creating an incredibly fragile system. The total mess. Yeah. So how do support engineers prevent this chaos over the lifespan of the platform? Well, that brings us to governance

and application lifecycle management or ALM, right? Data versus deeply integrated with power platform ALM, which uses the concept of solutions to package up your custom tables, security roles, flows and plugins into a single, deplorable file. Okay, solutions. The golden rule of ALM here is the strict flow of code from an unmanaged development environment to a managed testing environment. And finally, to a managed production environment. I've been on the other side of this though. You get a frantic call from a department head that a crucial form is broken in production.

Oh boy. The fastest way to fix it is just to log in as an admin and tweak the form right there in the live environment. Yeah. But the source material treats this like a cardinal sin. Mechanically, it actually breaks when I do that. When you make an unmanaged change directly in production, you are creating an active unmanaged solution layer that sits on top of your managed code. Okay. Think of it like stacking transparencies on an overhead projector. Because of how the platform resolves conflicts, unmanaged layers always win. They always sit on top. I see where this is

going. Yeah. So the next time your engineering team pushes a legitimate tested managed update from the development environment to fix that form permanently, your unmanaged quick fix is going to block it. It silently overrides the deployment pipeline. So the update says it deployed successfully, but the users don't see the changes because my late night hotfix is still sitting on top overriding the new code. Exactly. It is the single fastest way to destroy the integrity of your platform. That is terrifying. Production must remain strictly managed, meaning no direct

customizations are ever made outside the pipeline. So what does this all mean for the boots on the ground support engineer who gets paged at 2 a.m. Because a legacy dataverse setup is falling apart. Well, the source outlines some specific enterprise anti patterns to look out for. Things like running broad select queries over the TDS endpoint, which joke the memory, right, or broad blanket auditing that fills up log capacity. High frequency polling from external apps that trigger 429 errors, or using dataverse as a telemetry warehouse. This raises an important question. What do all of

those distinct anti patterns have in common? What's the thread? The root cause is always the exact same conceptual failure. It is an organization treating dataverse as an isolated database, instead of a governed application platform. Right back to where we started. Exactly. When you pull an API every two seconds, you are treating it like a local SQL server sitting under your desk. When you dump raw telemetry logs into it, you are completely ignoring the computational cost of the metadata and security layers that have to wrap every single row you insert. It's like trying to

use a luxury sports car to tow a freight train. The engine isn't the problem. You're just applying the wrong machine to the workload. And eventually the transmission is going to blow. That's a perfect analogy. You have to engineer for the complex platform you actually have, not the simple database, you assume it is. And that architectural discipline is only going to become more critical as we move forward, right? Oh, absolutely. If we look at the source materials notes on the 2026 AI roadmap, Microsoft is heavily positioning gate averse as an AI grounded operational data platform. Wow. We are talking about features like work IQ, complex agent program ability,

and deep native integration with Microsoft 365 co pilot, which means AI agents are essentially just the newest, most demanding consumers of the data versus API. They are an AI acts as a massive operational magnifier. How so? If your metadata is perfectly clean, your relational architecture is well defined. And your seven layer security gauntlet is properly constructed. An AI agent can do incredible things safely because it understands the platform contract. But if your metadata is messy, your security layers are flat and retrofitted and your

integrations are brittle. AI will only amplify those underlying problems at light speed. Exactly. It will confidently surface sensitive data to users who shouldn't see it, or it will hallucinate business processes based on broken table relationships. Good AI requires a perfectly engineered data platform underneath it. It truly all comes back to the foundation security correctness, lifecycle repeatability through a lm, capacity predictability across those three buckets, and integration resilience against 429 limits. It's all connected. Understanding those mechanisms is the difference between simply

using data verse and truly engineering a resilient data verse platform. Before we wrap up, I want to leave you with one final thought to mow over based on that roadmap. If an autonomous AI agent running in co pilot tries to execute a business process on behalf of a user, it has to mechanically pass through that exact same seven layer security gauntlet we discussed earlier. So ask yourself, if an AI agent rapidly acted on your current data verse security model today, would it fail safely? Or would it expose things it shouldn't? Because the agents are already here and they're reading your platform contract right now. You've been listening to support

engineering weekly from the Support Engineering Blog. For the full articles, references, and technical resources from today's discussion, visit the URL blog.sadhan.ch. Until next week, keep troubleshooting, keep learning, and keep engineering. Thank you for listening to support engineering weekly. This podcast is available on Apple podcast, Spotify, iHeartRadio, and other major podcast platforms. Follow the show on your preferred platform and never miss an episode.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Microsoft Weekly — November 16-20, 2026

    Nov 20, 202656 minAudio

    What happens when a routine network maintenance operation triggers a failure deep inside a cloud provider's authentication infrastructure? In this week's episode, Alex and Maya investigate how a seemingly isolated...

    Credit the people on this episode

  2. When Redundancy Fails: Inside Microsoft’s Network Stampede

    Nov 17, 202633 minAudio

    How can a routine network capacity upgrade trigger a global cloud outage - and why can the very systems designed to protect availability make the failure worse? In this Support Engineering Weekly bonus episode, Alex and...

    Credit the people on this episode

  3. From Microsoft Announcement to Completed Change

    Nov 15, 202623 minAudio

    Microsoft 365 administrators deal with information from several places: the Microsoft 365 Roadmap, Message Center, and Service Health. But knowing where to find an update is only part of the challenge. The real question...

    Credit the people on this episode

  4. Microsoft Weekly — November 09–13, 2026

    Nov 13, 202623 minAudio

    A server under pressure fails its health checks. Automated traffic management removes it from service, pushing even more load onto the remaining infrastructure. What was designed to protect the system becomes part of...

    Credit the people on this episode

  5. Planner Premium Isn't Project Online — So What Is the Gap?

    Nov 8, 202621 minAudio

    Microsoft Project Online is approaching retirement - or, in this case, its retirement date has already passed. But replacing a project management platform is not as simple as moving tasks from one product to another. In...

    Credit the people on this episode