Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Support Engineering Weekly

Support Engineering Weekly artwork

Microsoft Weekly — September 28–October 2, 2026

Oct 2, 202621 minAudio

This week on Support Engineering Weekly, Alex and Maya examine a series of Microsoft 365 incidents that reveal a common engineering pattern: as cloud services become more deeply connected, failures increasingly occur in the connective tissue between systems rather than in the systems themselves. The episode begins with Microsoft’s transition from SharePoint One-Time Passcodes to Microsoft Entra B2B for external sharing, and what that means for legacy sharing links, guest identities, Conditional Access, and help-desk preparation. From there, the discussion moves into Teams device-management consolidation, iOS enrollment failures, and a Windows Autopilot routing issue where an internal network change caused seemingly unrelated device-management operations to fail. The deeper dive focuses on Microsoft 365 Copilot, including licensing failures, agent deployment problems, traffic-related capacity issues, and a particularly significant semantic-indexing gap that left Exchange Online email available in storage but absent from Copilot's semantic understanding. The episode then turns to traditional Office clients, examining a Word PDF-save failure that redirected files into a hidden legacy cache location, an Exchange Online add-in synchronization issue, and a Teams workflow regression involving run-only permissions. The common thread is silent and indirect failure: systems can appear healthy while an underlying dependency, routing path, synchronization pipeline, or indexing process is no longer behaving as expected. For support engineers, that raises a difficult question for the AI era: how do you troubleshoot a system that is functioning normally while quietly operating on incomplete information? Chapters 00:00 - Cloud Dependencies Under Stress 01:47 - SharePoint External Access Changes 05:43 - Teams Device Management Consolidation 06:43 - Intune Routing and Autopilot Failure 10:03 - Copilot Licensing and AI Index Gaps 13:02 - Copilot Scale and Agent Failures 14:05 - Office Clients and Cloud Dependencies 16:44 - Exchange Add-ins and Teams Workflows 19:28 - The Enterprise AI Monitoring Challenge Learn more: Visit the Support Engineering Blog at https://blog.sadhan.ch/ for the full articles, technical references, and additional support-engineering resources behind the discussion.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

You are listening to support engineering weekly. Imagine you ask your company's million dollar enterprise AI to summarize a crucial internal project. Right, something you rely on every day. Exactly. And it competently gives you a detailed answer. But, and here is the kicker, it's completely wrong. Which is terrifying. Yeah. And not because the AI hallucinated, but because a silent network pipeline failure caused it to, you know, just stop reading your organization's emails like two months ago.

No red dashboard. Right. No error code at all. Just confident, invisible failure. Welcome to the deep dive for the week of September 28th through October 2nd, 2026. This is where we're seeing massive architectural convergence just slamming directly into the fragility of deeply buried cloud migrations. It is a fascinating collision. I got to say it. Tracking a really clear industry push toward these unified zero trust platforms, but simultaneously

witnessing how tethering every single desktop action and, you know, every AI prompt to a remote microservice creates these incredibly obscure failure domains. Which is really the core of our mission today for you, the listener. I'm Alex, by the way, and I'm Maya. Thank you so much for joining us for this deep dive. So we are going to cut through the noise of this week's Microsoft Message Center and service health updates, really trying to find the underlying engineering stories for you and your support teams. We're unpacking major identity shifts, those silent AI failures we mentioned, and some truly

wild legacy desktop client mysteries. Yeah, the desktop stuff is wild this week. It really is. But let's start at the identity layer because there is an operational cliff arriving in mid-October 2026. It's all about how external users access your SharePoint and OneDrive environments. Right. The retirement of SharePoint OneTime passcodes or, you know, SPO OTP. Yeah. This has basically been the standard for ad hoc external sharing for years. You share a link, the external vendor gets an email, they click it, get a temporary six

digit code email to them, and they authenticate. It's super easy for the user. It was completely frictionless. Yeah. But from an architectural standpoint, it's, well, it's a dead end. Because you can't build a modern security parameter around a temporary ghost, right? The SPO OTP method, it issues a session token on the fly, but it doesn't actually create a persistent object in your tenant directory. Exactly. So if you're trying to implement a zero trust architecture, you literally have nothing to target. You can't apply conditional access. You can't enforce device compliance.

And you definitely can't run any meaningful lifecycle governance on, you know, an email address that only exists as a string of text in a SharePoint sharing lake. And that is the exact driving force behind this deprecation. Microsoft is shifting all external sharing to Microsoft, enter a B2B. So when you share a file now, a process that began rolling out back in May, I think. Yeah, May 2026. Oh, right. The system provisions a persistent guest account stub directly inside your enter ID directory. So that guest identity actually becomes a first class citizen, which is huge.

It is. You can enforce multi factor authentication natively now. You can lock down their access based on sign in risk. With the operational cliff we mentioned, that's about the legacy links. So if an external user tries to click an older specific people link that was generated using the old SBO OTP method and they don't already have that new entropy to be guest accounts stub provisioned in your directory, the link is just going to hard fail. Oh, yeah. Hard failure. They would just get an access denied error, which means your help desk needs to prepare

for a huge influx of tickets from external vendors who, you know, suddenly lost access to contracts or project files they've been collaborating on for months. Yeah, that's going to be a fun Monday morning, but the remediation is straightforward. If a bit manual, an internal user just needs to reshare the file or folder with that vendor. Right. Just reshare it. Yeah. And that action fires off a call to the enter a B2B invitation manager provisions the guest account and maps the permissions. And then all previously shared content access is instantly restored because the permission

claims now map to a valid, intra object ID, right? Instead of just a transient email token. Exactly. Alternatively, your identity administrators can proactively run scripts against your external sharing reports, you know, identify the highly active vendors relying on OTP and just pre provision those entropy to be accounts before the mid October cutoff, which would save the help desk a lot of pain. Taking that proactive step prevents the surge entirely. I like to think of it as upgrading a corporate facilities security posture.

Okay, how so? Well, SPO OTP is like the front desk handing a vendor, a generic temporary paper badge, every single time they walk in. It gets them through the turnstile, sure, but security can't track their movements or enforce restricted zone rules. Oh, that makes sense. Yeah. Whereas, on tributab is like printing them a permanent smart chip corporate ID that integrates directly with building central access control system. I like that analogy and that permanent ID allows the system to enforce rules dynamically,

which really creates a perfect bridge to the hardware side of this convergence. Right. Because you enforce a conditional access policy, say requiring a compliant device before a guest can open a SharePoint file, your device management infrastructure has to be completely unified and robust. And this week we saw Microsoft actively consolidating that hardware management layer. Well, you know, simultaneously struggling with its underlying routing. So true. Let's look at the consolidation first. Teams Android device management, which covers teams rooms on Android teams, phones and panels.

That's all being deprecated in the legacy teams admin center or TAC starting this October, right? Yep. This October, the workload is moving entirely to the teams rooms pro management portal, the PMP, which makes sense because the telemetry streams coming from Android based conference room hardware, they are vastly different from standard windows endpoints. PMP was basically purpose built to standardize that ingestion layer. Yeah, totally different data models. Right. So by forcing all teams devices into PMP, admins get a unified data model for inventory, remote

actions, health monitoring, rather than constantly context switching between different admin centers with completely different API capabilities. It's a necessary evolution for the device management lifecycle for sure. But the friction we saw this week wasn't actually in the new portals. It was deep in the underlying into an infrastructure. Yeah, we saw two pretty severe hiccups. We did. First, an authentication configuration issue took down iOS automated device enrollment for five whole days.

Admins literally couldn't use the Apple setup assistant to enroll new iPhones or iPads. Five days for a core provisioning pathway is a massive disruption. I mean, it really highlights the vulnerability of cross vendor API dependencies. It really does. But the second issue is the one I really want to dissect because it affected Windows autopilot. And the root cause description is just baffling on its face. Oh, the routing one. Yeah, admins were attempting to import hardware hashes via CSV, right? And the tasks were just locking up in an infinite processing state.

Worse, if an admin tried to update an autopilot group tag on an existing device, they would hit save, the UI would process, and the tag would just vanish. You wouldn't persist at all. Right. And the incident report blamed a network traffic routing change. I just had to push back on that logic. Why on earth would a network traffic routing update prevent a group tag from saving? I mean, a group tag is just a text string. The text string fails to save to a profile. That sounds like a database commit failure or I don't know, a corrupted storage table,

not a networking issue. I get why you'd say that. Okay. It's entirely counterintuitive. If you view cloud admin centers as traditional monolithic applications, but the in tune portal, you access in your browser, it doesn't execute SQL queries against a database. No, no, it acts as an API gateway. So when you upload that CSV of hardware hashes, the front end accepts the payload and forwards it to an internal in tune microservice, which then communicates with the Windows autopilot deployment service backend.

It's the internal handoffs. Yes, exactly. A backend routing table update severed the internal virtual network peering between those specific microservices. So the front end gateway successfully accepted the group tag string. It sent the HTTP request down the pipeline. But because the internal route was dropped, the TCP connection to the storage blob just timed out mid flight. So the front end UI is just sitting there spinning, waiting for an HTTP 200.ok confirmation that the database actually committed the change, right? But the confirmation never arrives because the internal postman couldn't find the route

to the database. Exactly. The database itself was perfectly healthy. The storage tables weren't corrupted at all. The API tether connecting the microservices was essentially just cut. The data evaporated in transit. That is a brilliant look at how isolated microservices can create incredibly confusing front end behaviors. The admin sees a database failure, but the infrastructure telemetry sees a dropped packet. Yeah, it's a huge disconnect. Now, if a severed internal network route can completely break something as foundational

as a text string in device enrollment, it sets a pretty grim stage for what happens to highly complex multi note AI infrastructure. Oh, for sure. Co-pilot really exposed that scaffolding this week. The AI platform requires this massive federation of agents, token validators, and storage indexes all working in perfect concert just to generate a single response. When one of those internal pathways degrades, the failure states are fascinating. They really are. Let's start with the licensing bug from September 22 to the 28th.

A flaw in the request flow caused co-pilot and co-pilot chat to view valid, fully licensed users as unlicensed. But it wasn't a hard crash, right? No, not at all. It was a silent feature drop. Users suddenly just couldn't generate images. They couldn't prompt the AI to generate content and co-pilot notebooks became completely inaccessible. See, that behavior proves co-pilot is checking license and claims at the individual service level, not just at the initial application launch. The token validation endpoint for the image generation microservice was clearly returning

a false negative while the core chat interface remained, you know, somewhat functional. But the absolute standout issue of the week, the one that truly blew my mind, was a semantic index gap. Oh, this one was wild. Insane. From July 22, all the way to September 28, a misconfiguration prevented exchange online emails from being indexed by the semantic index model. Think about the tier one support tech fielding this ticket. Oh, I feel bad for them. Right. A user complains co-pilot can't find an email about a specific project.

The tech remotes in runs a standard exact wording keyword search and co-pilot and boom, the email appears. The tech assumes its user error. Naturally. But then when the user types is semantic prompt, something like, summarize the recent pushback on this project, the AI claims it has no idea what they are talking about. And that scenario right there is the defining support engineering challenge of the enterprise AI era. It really requires understanding the absolute distinction between the storage index and

the semantic processing layer, which are totally different. Completely different. The emails were never missing. They were safely resting in the exchange online storage substrate. And that is exactly why the tech's exact keyword search worked. A keyword query bypasses the AI processing entirely and queries the traditional inverted storage index directly. Like find subject line containing X. Exactly. But a semantic prompt requires context. The semantic index is a secondary pipeline. It continuously reads the exchange storage, converts the concepts, relationships and context

of those emails into dense mathematical vectors and stores them in a separate vector database. So it's constantly translating. Right. So if the ingest pipeline feeding data from exchange to that vector database stalls, which it did for over two months, the AI's memory simply stops updating. It's a failure of architectural parity. The data exists, but the AI's map of the data is frozen in time. The most dangerous part to me at least is the complete lack of an error code. Yeah, silent failure. The copilot didn't tell the user warning.

My index is two months out of date. It just confidently generated a summary based on incomplete information. It doesn't know what it doesn't know, which forces us to rethink how we monitor system health entirely. We are also seeing the raw compute limitations of this new architecture. Microsoft reported that PowerPoint agents failed this week simply due to traffic spikes. Just too many people using it. Yeah. The new infrastructure literally couldn't scale fast enough to handle the concurrent request load. We also saw deployment pipeline bugs for the copilot agents themselves, like for users

on shared computer activation devices in the frontier preview, clicking open to install the legal agent for word just failed. The launch sequence exposed to missing component dependency. Well. And meanwhile, if you tried to open the analyst or researcher agents from the M365 store, a URL redirection bug failed to pass the agent context properly. It just dumped the user into a blank generic new chat window instead of launching the specific agent interface. The platform is being built, scaled, and deeply integrated into legacy apps simultaneously.

The capabilities are impressive, sure, but the connective tissue is still highly vulnerable to basic traffic and routing logic errors. And speaking of that connective tissue to legacy apps, we cannot ignore the classic desktop clients this week. While we're out here debating the intricacies of dense vector databases, the standard office suite was out there generating some of the most bizarre old school support tickets imaginable. Ah, though. Where did my file go incidents? Yes. They provide a perfect counterpoint to the AI discussion, really showing how deeply cloud

tethers impact traditional Win32 applications. I have to admit, I laughed out loud reading the incident report for the Microsoft Word bug. In word version 2609, there was a validation path failure. If a user tried to use Save As to output a document as a PDF directly to a SharePoint cloud location, the process failed. But not gracefully. No, word didn't throw an error message. It didn't crash. It quietly dumped the generated PDF into a deeply hidden local cache folder, C colon slash users

slash username slash app data slash local Microsoft Windows, iNet cache content dot MSO. And to make it worse, it saved it with an incorrect machine generated file name. I mean, picture the sheer panic of an executive who just finalized a critical quarterly report. They save it as a PDF to their team site, watch it disappear into the void, and then a support engineer has to go hunting through hidden internet explorer legacy app data directories just to find a misnamed temporary file. It is the ultimate IT ghost story.

But why on earth does a modern application like word fall back to the iNet cache directory from the Windows 7 era? It really exposes the architectural layers still present in the office suite. Word is a rich local application, but it relies on asynchronous graph API calls to save directly to SharePoint. Right. When the user clicked Save, the application initiated the PDF creation in its local memory buffer. However, the network validation path to the cloud backed location encountered a synchronous block. It just timed out.

So the app has a fully rendered PDF sitting in volatile memory, but the cloud door is locked. Exactly. And it has to dump that memory buffer to disk immediately to avoid a fatal application crash. Because the standard save pathways were blocked by the cloud failure, it executed an ungraceful fallback routine. They just panicked. Pretty much. It utilized the oldest lowest level temporary cache module available in the win 32 framework, the iNet cache directory to silently stash the data, the application panicked and hid the evidence in the basement.

That is a phenomenal example of how tethering legacy desktop code to cloud APIs can result in completely unpredictable client behavior when the network drops. And we saw a similar pipeline state failure in exchange online, but instead of hiding files, it was resurrecting dead add ins. Oh, the enabled status sync pipeline bug? Yes. A portion of the exchange sync pipeline was returning incorrect values for the enabled status parameter of various outlook add ins. Suddenly, unwanted third party add ins that administrators had been available in the catalog,

but explicitly chose not to automatically install, were deploying themselves across outlet clients globally. This is a failure in the global catalog sync mechanism. The M365 admin center deployment pipeline pushes manifest states down to the exchange mailbox substrate. Okay. So if the default client launches, it reads that manifest to determine which add ins to load. The pipeline responsible for verifying the true administrative intent failed to validate the state. So it defaulted to a binary enabled integer.

And the fix was a highly disruptive PowerShell run. Admins had to execute a specific set dash app command using the parameters dash private catalog and dash default state for user disabled across our entire environment, just to force the manifest back into alignment. It really demonstrates the blast radius of a single defaulted integer in a cloud configuration pipeline. It doesn't just affect one server. It instantly alters the client interface for every user worldwide. Insane scale rounding out our desktop client mysteries.

We had a localized logic failure in Microsoft Teams regarding workflows. Users were attempting to execute run only workflows they have been granted access to. Right. So the primary workflows button in the main UI, the automation ran perfectly. But if they hovered over a specific chat message, selected the more actions context menu and clicked create workflows, the system through a missing trigger URI error. Unless crucially, the user happened to be granted owner or co owner permissions to the

underlying workflow logic, right? Which completely defeats the security purpose of a run only permission model. Exactly. So the next thing you are allowed to push the button on the assembly line, but only if you hold the deed to the entire factory. It is a classic code regression in how the team's client processes JSON payloads through different user interface pathways. The hover context menu trigger URI failed to pass the execution context properly to the back end. So the back end got confused. Yeah, because the execution context was missing, the back end validation logic defaulted to

its strict dis posture, checking if the user owned the object itself, rather than checking if they simply had run permissions. Different UI buttons relying on entirely different validation logic for the exact same underlying action. So synthesizing everything we've dissected today, the week of September 28 to October 2 presents a very clear narrative. The transition toward unified platforms, you know, shifting ad hoc sharing to enter a B2B for zero trust governance, or standardizing device telemetry in the pro management portal,

is absolutely necessary for long term stability and security. We have to make those moves. We do. But the journey there exposes the immense fragility of these deeply nested cloud tethered pipelines. I mean, a severed internal VNet routing rule breaks a text string in device enrollment. A validation timeout forces Word to dump a critical PDF into a hidden legacy cache folder. The connective tissue of enterprise IT is currently under extreme tension. Which brings us to the most provocative challenge for you to take back to your engineering

teams this week. As your organization relies more heavily on technologies like semantic indexing for enterprise AI, how are you going to monitor the completeness of your AI's knowledge? It is the frontier we have to figure out next. Traditional IT monitoring is binary. When an API fails, you get an error code. When a network route drops, your latency dashboard turns red. But when an AI ingest pipeline silently skips a month of your company's data, the system doesn't look broken. It looks fine. Exactly.

It continues to operate confidently providing incomplete answers. How do support engineering practices evolve to troubleshoot missing context? You've been listening to Support Engineering Weekly from the Support Engineering Blog. Until next episode, keep troubleshooting, keep learning, and keep engineering. Thank you for listening to Support Engineering Weekly. This podcast is available on Apple Podcast, Spotify, iHeartRadio, and other major podcast platforms.

Follow the show on your preferred platform and never miss an episode.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Microsoft Weekly — November 16-20, 2026

    Nov 20, 202656 minAudio

    What happens when a routine network maintenance operation triggers a failure deep inside a cloud provider's authentication infrastructure? In this week's episode, Alex and Maya investigate how a seemingly isolated...

    Credit the people on this episode

  2. When Redundancy Fails: Inside Microsoft’s Network Stampede

    Nov 17, 202633 minAudio

    How can a routine network capacity upgrade trigger a global cloud outage - and why can the very systems designed to protect availability make the failure worse? In this Support Engineering Weekly bonus episode, Alex and...

    Credit the people on this episode

  3. From Microsoft Announcement to Completed Change

    Nov 15, 202623 minAudio

    Microsoft 365 administrators deal with information from several places: the Microsoft 365 Roadmap, Message Center, and Service Health. But knowing where to find an update is only part of the challenge. The real question...

    Credit the people on this episode

  4. Microsoft Weekly — November 09–13, 2026

    Nov 13, 202623 minAudio

    A server under pressure fails its health checks. Automated traffic management removes it from service, pushing even more load onto the remaining infrastructure. What was designed to protect the system becomes part of...

    Credit the people on this episode

  5. Planner Premium Isn't Project Online — So What Is the Gap?

    Nov 8, 202621 minAudio

    Microsoft Project Online is approaching retirement - or, in this case, its retirement date has already passed. But replacing a project management platform is not as simple as moving tasks from one product to another. In...

    Credit the people on this episode