Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Support Engineering Weekly

Support Engineering Weekly artwork

Microsoft Weekly — October 5–9, 2026

Oct 9, 202625 minAudio
ShareXLinkedInFacebookWhatsAppEmail

A resilience update designed to make a service stronger can end up taking it offline. A routine cleanup can break a critical feature. And a small increase in response time can trigger a traffic surge that overwhelms an entire system. In this episode of Support Engineering Weekly, Alex and Maya examine Microsoft 365 service incidents from October 5-9, 2026. They explore Outlook calendar caching failures and unwanted add-ins, cascading failures affecting Teams meetings and Copilot voice, request queuing and retry storms, and silent failures involving Teams contacts and PowerPoint Live recordings. The discussion also examines security and compliance issues involving Defender Conditional Access App Control, emergency calling, and legitimate DocuSign links being quarantined. The episode concludes with a practical operational change: Microsoft’s migration of Teams Rooms on Android, Teams phones, and Teams panels management from the legacy Teams admin center to the Teams Rooms Pro Management Portal. Alex and Maya also consider a broader reliability challenge: how should engineers load-test generative AI workloads when prompt complexity and processing times are unpredictable? Chapters 02:04 - Outlook Caching and Unwanted Add-ins 06:35 - Teams and Copilot Resilience Failures 09:01 - Cascading Traffic and Request Queues 12:47 - Silent Failures in Contacts and Recordings 16:31 - Security and Compliance Failures 21:01 - Teams Rooms Android Management Migration 22:51 - AI Load Testing and Cloud Reliability Learn more: Visit the Support Engineering Blog at https://blog.sadhan.ch/ for the full articles, technical references, and additional support-engineering resources behind the discussion.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

You are listening to Support Engineering Weekly. You know, usually when we talk about a system upgrade or like a maintenance push, there's this ingrained expectation of progress. Oh, absolutely. Yeah, we rely on that comfort. Right. Like in physical engineering, the rules are pretty rigid. You install a reinforced steel beam and a building, and the building is, well, it's objectively stronger. Right. Or a doctor gives you a vaccine, and your immune system is objectively better prepared.

Exactly. It's binary. We find a lot of comfort in that cause and effect, but then you step into the world of modern cloud architecture. Oh, boy. Yeah. Specifically global scale distributed systems, like the ones we navigate every day. And suddenly that fundamental logic just, it shatters. It really does. You expect technical improvements to be visible, you know, categorized as these strict net positives that move the needle forward, never backward. But we are operating in a deployment landscape

with a very mechanisms designed to make a system invincible or exactly what bring it down to its knees. I'm Alex. And I'm Maya. And we really want to thank you for joining us for this deep dive. Today, we are on a mission. A very chaotic mission. Yeah, seriously. We're taking a highly curated, deeply technical look at the Microsoft updates and incidents that rolled out during a very specific window. October 5th through October 9th, 2026. We are treating you like the third engineer in the room with us today.

Pull up a chair to the console because we are unpacking a truly wild week. We're talking about resilience updates causing massive cascading outages. Routine service cleanups deleting core foundational requirements. And my favorite. AI model migrations triggering catastrophic traffic jams. It was basically a brutal masterclass in how complexity breeds fragility. Every layer we added to make the service smarter, it just ended up confusing the layers beneath it.

Yeah, completely. So diving into the first major issue, we saw some incredibly frustrating behaviors in outlook and exchange. Oh, the caching issues. Yeah. As an engineer, you already know the delicate handshake of caching and state sync in a distributed system at Microsoft scale. I mean, that handshake is everything. It really is. A client application like Outlook, it keeps a local cache so it doesn't have to query the exchange server for every single interaction. Right. And state sync is the pipeline ensuring that what the server knows definitively matches

what the client displays locally. And when that pipeline functions, it's invisible. But when it breaks, it's a nightmare. Yeah. The user experience degrades in ways that actively gaslight the user. Which brings us to incident EX1483540. This maddening Outlook calendar bug. Users were getting an error stating can't retrieve calendar events right now, refresh the browser and try again. And the natural user behavior is to pay refresh. Obviously. And the system would acknowledge the refresh.

It would update the folder envelope information. But the event loading process itself got physically stuck. Right. It continued referencing outdated calendar data in the local cache. Instead of just requesting the fresh payload from the server. It was trapped in an infinite loop, stirring it a stale memory pointer. From an IT perspective, those are the absolute hardest tickets to troubleshoot. Because it looks like it's working. Exactly. The UI tells the user it's refreshing. The network logs might even show a tiny bit of traffic for the folder update,

but the heavy payload, the actual meetings, is pulling from the past. Wow. It completely undermines the user's trust in the basic mechanics of the interface. You end up telling executives to like clear their cache multiple times a day. Which is definitely not a scalable enterprise solution. Not at all. But you know, that was a localized annoyance compared to the absolute saga of incident EX1441488. This was a months long ordeal that finally wrapped up on October 5th. The add-ins. The unwanted add-ins.

Users across all Outlook clients, desktop, web, mobile, were logging in and finding these unwanted add-ins forcefully installing themselves. Yeah. Six new third-party integrations would just suddenly clutter the ribbon. Unprompted. Unprompted. So what was the mechanism behind this? It was a catastrophic state sync failure on the server side. A specific portion of the Exchange Online add-in synchronization pipeline failed to process state information correctly. Okay. So when the Outlook client performed its routine boot sequence and asked the server for the status of add-ins for that specific user profile, the pipeline returned an invalid enabled status value.

Okay. Let's unpack this for a second because I was trying to visualize how this happens architecturally. Sure. It's like, imagine a bouncer at a highly exclusive club. He's standing at the door with a digital VIP list on a tablet. Okay. I like where this is going. Suddenly, his tablet loses his hits connection to the backend database. The list goes completely blank and he panics. Naturally. And instead of freezing the line and stopping people from entering, his default fail-safe is to just assume everyone in line is supposed to be there. Right.

He just starts waving everyone in. Exactly. Right. That bouncer is the sync pipeline. It lost the actual enabled status of these add-ins, so it just defaulted to available and aggressively pushed them to every client that asked. Yet the fail-safe was entirely backwards. And the operational consequence for you managing a tenant was severe. It was a mess. Because the patch Microsoft developed to fix that pipeline logic was so deeply complex to deploy across the global infrastructure, it took weeks to fully saturate. Weeks.

So what did IT teams do in the meantime? Well, they were drowning in complaints about unauthorized software appearing on desktops. Some of you listening actually had to deploy a specific heavy-handed PowerShell command. Oh, running the set app command, right? Exactly. Running set app identity with the app ID, appending default state for user disabled and targeting the private catalog. That is heavy-handed. You were forced to manually override the bad states to rip the add-ins out before the permanent infrastructure fix arrived. Running tenant-wide PowerShell scripts just to mitigate a sync bug is exactly the kind of manual high-risk intervention cloud services are supposed to eliminate.

Completely. But that mechanism, you know, a system trying to protect itself or optimize itself and failing spectacularly is the exact pattern we saw take down core communications later in the week. Ah, yes. The infamous resilience updates. Right. So incident TM 1437780 was a massive team's meeting failure. Massive. Users were completely unable to join or create any meetings or calls. PSTN, one-on-one calls group calls, all just dead.

And right around the same time, incident CP148980 hit. Which severed the copilot voice experience, right? Entirely. Users couldn't use voice dictation anywhere. So what happened there? What's fascinating here is the underlying causality. These weren't random hardware failures or severed fiber cables. For teams, the outage was triggered by a code deployment explicitly designed to improve service resilience. Wait, a resilience update broke it? Yes. Microsoft pushed an update to make the call routing system more robust against failure.

But the update itself overloaded their internal traffic management processes. Oh, wow. As dependent downstream services and client applications noticed the initial routing hiccup, they tried to recover. By sending retry requests. Exactly. They sent automatic retry requests, which caused request volumes to search instantly, creating a massive cascading failure across the entire service tree. It's like a city planner adding a brand new express lane to a highway to reduce traffic. But the construction crew leads a bulldozer sitting horizontally across the new lane.

Right. So the cars aggressively merged to avoid it, causing a massive pileup that blocks all the old lanes too. The improvement is what caused the 50 mile backup. I'd actually take that a step further. The cars in your analogy are the client applications. And they are programmed to be relentlessly persistent. Relentlessly persistent. I like that. Yeah. And the co-pilot voice failure operated on a similar vein of administrative irony. A routine service cleanup was executed on the back end.

Probably an automated job to prune deprecated code. Her optimized storage arrays. Yeah. But the cleanup script lacked the proper dependency mapping. It accidentally purged required feature configurations, instantly severing the connectivity paths for the voice experience. So we just deleted the roadmap. Basically foundational routing and hygiene mechanisms are incredibly dangerous, precisely because they sit beneath the application layer. When they break, they don't just cause a glitch. They orchestrated total blackout.

Exactly. The concept of cascading request surges from the teen's outage wasn't an isolated incident of infrastructure buckling under its own weight. We saw the exact same mechanism take down co-pilot and the admin center. But the triggers were deeply tied to the nuances of AI and modern web traffic. Yeah. The request queuing issues. We observed the exact same exhaustion pattern hit both co-pilot and the admin center, driven by the architecture's inability to handle sudden density. Right. The Microsoft 365 admin center under incidents mo 1488 931 and mo 1489051 suffered brutal

intermittent access failures. Pages were stalling out entirely. Throwing raw 500 internal server errors. And around the same time, co-pilot and outlook that's incident CP 1491697 started throwing generic something went wrong errors to use a prompts. The mechanical failure for both is request queuing and timeouts. For the admin center, the portal was essentially de-dossed by legitimate administrative traffic. Legitimate traffic took it down. Yeah. An unexpected massive influx of requests hit the front end.

The underlying infrastructure simply couldn't spin up new compute instances fast enough, causing active requests to sit in a queue. When a queue gets too long, the web servers, worker threads lock up. Exactly. The request time out, the frustrated human user hits refresh, adding another request to the pile, and the problem compounds exponentially. So what was the fix? Microsoft literally had to roll back the infrastructure to a previous build and disable a recent service change just to stabilize the traffic flow. Okay.

I understand how human impatience causes a queue to pile up for a web portal. But the co-pilot issue was triggered by a recent backend model migration. Right. Microsoft moved the AI to a new, presumably smarter model. And the incident report states that migration resulted in a massive influx of user traffic. It did. But why would a backend AI model swap cause a traffic influx? A number of human beings typing prompts into outlook didn't suddenly triple. The humans didn't multiply, but their clients did.

What do you mean? In a heavily distributed architecture, client software is aggressively programmed for fault tolerance. Let's say you swap an AI model on the backend. The new model generates a more nuanced, heavily contextualized response, which means it takes an extra 200 milliseconds to process the prompt. Wait, really? Just two tenths of a second? Right. Barely noticeable to a human. But the Outlook client has a hard-coded timeout threshold for API calls. It might view that 200 millisecond delay as a silent packet drop.

Oh, no. So what does the client do? It abandons the initial connection and automatically fires off a retry request to ensure the user gets an answer. So the server is still processing the first request, tying up compute resources. While the client fires a second, duplicate request into the load balancer. Exactly. Now multiply that automatic microscopic retry behavior by 10 million users simultaneously prompting co-pilot across the globe. Wow. You haven't added any new users, but the system volume has doubled, maybe tripled in milliseconds.

The TCP connections stay open longer, waiting for the slower model. And the application thread pools exhaust because every worker thread is busy either waiting for a response or processing a duplicate retry. You got it. The load balancers begin failing their health checks because the backend servers are unresponsive. And suddenly the entire AI feature goes offline. It is the dark side of aggressive retry logic. It's terrifying when you realize the infrastructure basically suffocated under the weight of its own clients trying to be able to get their own clients trying to be helpful.

The system essentially attacked itself. Yeah. But if loud cascading traffic jams are terrifying, I actually find the silent failures even worse. Oh, definitely. The moments where data just vanishes into the ether without triggering an alarm. You're referring to the graceful degradation failures. Yes, the phantom contacts. Incident MO 1489432. Users open teams and their contacts and speed dials were just blank or gray placeholders. You go to call a colleague and their contact card is a ghost.

Right. And similarly, incident TM 1483625, where a PowerPoint live recordings in teams were silently emitting specific slides. Just dropping them. Yeah. You finish an hour-long presentation, review the mp4 file, and slide 12 is just completely absent from the video. If we connect this to the bigger picture, both of these are classic examples of poor graceful degradation in cloud computing. Let's look at the database architecture for the contacts first. Okay. Microsoft uses something called profile-based contacts or PVCs.

A processing component essentially broke during an update, and it incorrectly separated the PVCs from their actual user profiles. If specific contact info wasn't explicitly stored in the database. And the incident report noted this heavily impacted, intended contacts created using the notify when available presence option. Why that specific feature? When you tag a colleague for notify when available, the system often creates a highly localized, lightweight stub of a contact in your cache.

Rather than pulling the whole profile. Right. Rather than maintaining a constant sync with their massive, heavy active directory profile. Because that stub lacked explicit, deep profile data, this buggy processing component didn't know how to validate it. So just dumped it. Instead of throwing a sync error, it just detached the pointer entirely. The UI tried to render a contact that no longer had a data source, resulting in the blank gray box. That explains the database pointer issue. But how does a cloud rendering engine just drop a slide from a video recording without failing the recording process entirely?

The root cause was network bandwidth constraints on a specific subset of the recording infrastructure. The rendering service couldn't handle the localized network load and fell below required performance thresholds. When a PowerPoint live recording is being assembled in the cloud, it's acting like a stream consumer. It's pulling the audio stream, the webcam video stream, and the high res slide metadata concurrently. So if there is network congestion, it's dropping packets. Significant packet retransmissions and massive latency. Yeah. The download requests for the specific high res slide assets just timed out before completion.

So just skip them. Here is the architectural choice. The rendering engine is programmed to prioritize availability over consistency. It assumes you would rather have a slightly broken video file than no video file at all. I mean, I guess that makes sense. When the packet timed out, the compiler's internal clock just moved to the next time stamp to ensure the file didn't corrupt. It skipped the slide and finalized the video. I understand the philosophy of wanting to deliver a usable file, but from an operational standpoint, failing silently is a nightmare. Total nightmare.

As an IT professional, you rely on explicit error codes. If a system fails loudly, you get an event log, you get a telemetry alert, you can actually troubleshoot. You have something to work with. Exactly. Blank contacts and silently missing slides don't generate localized error codes for the user to copy and paste to the help desk. It just looks like random magic failing. Which makes the admin look incompetent when they have to tell a user, I don't know where your slide went. Yeah. It creates a massive visibility gap. You can't fix what the system refuses to admit is broken.

Which is incredibly frustrating. But if silent failures are frustrating, what happens when automated security and compliance tools decide to act unpredictably? That's a whole different level of stress. Because when security fails, it doesn't just drop a slide. It either locks the business down or opens the floodgates. It was a very busy week for SecOps teams relying on automated policy enforcement. Let's look at the machine learning and compliance overreach. Incident DZ 1489394. Defender Conditional Access App Control, or CAAC policies, were being unexpectedly bypassed or aggressively hardened.

Yeah. Then Incident TM 1488 543. E911 Teams calls, which are highly regulated emergency channels, were failing to automatically mute users upon joining. Very dangerous. And Incident EX1483 1619. Totally legitimate emails containing DocuSign URLs were being quarantined en masse across thousands of tenants. The underlying mechanisms here reveal the immense fragility of trusting automated policy enforcement without manual oversight.

Right. For the Defender CAAC issue, the architecture dictates that the CAAC proxy sits directly in line, inspecting traffic. To enforce a policy, it has to constantly verify the user's session state with a backend authentication dependency. And if it can't? Well, when that authentication component experienced a localized outage, it couldn't connect. So the proxy is flying blind. Exactly. When an inline proxy can't verify state, it falls back to its failsafe configurations.

The problem is, the failsafe's acted unpredictably. Unpredictably how? Some policies defaulted open, bypassing security protocols and allowing unverified access. Oh, wow. While others defaulted closed, aggressively hardening access and blocking entirely legitimate administrative work. Here's where it gets really interesting, especially regarding the ML side of security. When a machine learning model or a code dependency fails, it has absolutely no nuance. Zero nuance.

The DocuSign quarantine is a perfect example of mathematical overreach. Microsoft's URL evaluation process relies heavily on a machine learning model to detect phishing and credential harvesting. We know phishing actors heavily spoof DocuSign because users are conditioned to click those links. Right. The model calculates a risk score based on heuristics, natural language processing of the surrounding text and URL structure analysis. Right. But the model got slightly overtuned or ingested a batch of bad training data where the attackers used URL parameter obfuscation techniques

that mathematically resembled how DocuSign structures its own legitimate complex tracking links. The math was too similar. Because the math looked similar, the model's confidence score for malicious intent spiked past the quarantine threshold. Suddenly, it identified millions of mathematically safe, critical business contracts as high-risk phishing. It locked the doors completely. And that mathematical overreach is mirrored by code level fragility in the E911 incident. Right. Emergency calls are strictly regulated by compliance laws.

When multiple people join an active E911 call, the system is hard-coded to force mute new entrants to keep the audio channel completely clear for the emergency dispatcher. So what in the trigger fire? A code mismatch. The specific software components utilized when marking a team's call object as a multi-party call had a versioning discrepancy. Oh, a versioning discrepancy. Yeah. The system saw multiple people joining, but because the call object wasn't correctly flagged as multi-party in the back-end database,

the automated compliance trigger to mute the microphones simply never fired. So you have dispatchers dealing with background noise during emergencies because database flag didn't update. Exactly. It is a stark reminder that security and compliance aren't just conceptual rule books. They are living fragile code bases with deep dependencies. If the ML model gets paranoid based on bad math, business grinds to a halt. If the off-dependency goes down, your firewall might just leave the front door wide open.

This raises an important question about visibility. Having direct control and insight over your endpoints and administrative portals is critical. You cannot just trust the automation blindly. You have to know where the over-rights switches are. And that is the perfect thematic bridge to our final topic today. Let's bring all this theoretical back-end chaos back to something you, the listener, have direct practical control over right now. We need to talk about the PMP migration. Message center update MC1227622.

This is a critical operational shift for anyone managing physical office hardware or conference rooms. Specifically Android device management. Microsoft is officially moving the management of Teams Rooms on Android, Teams phones and Teams panels out of the legacy Teams admin center, the TAC, and exclusively into the Teams Rooms Pro Management Portal or PMP. The PMP. We are emphasizing this because you need to care about this today. The deprecation of SIP device management in the legacy TAC starts this week.

This is not a future roadmap item you can ignore until next quarter. This is an active rollout. If you support these physical devices, you must update your internal help desk documentation immediately to reference PMP. The Pro Management Portal is going to be the new exclusive home for all your device inventories, your remote actions like pushing a firmware update or restarting a frozen conference room panel, and your health monitoring telemetry. And if they don't update. If your junior admins or tier one support desk are still logging into the legacy TAC next week to troubleshoot a dropped SIP phone, they are going to find a deprecated empty interface and no way to assist the user.

Consolidating these tools is ultimately a good thing for unified management, but the transition window is incredibly tight. Get your documentation updated. Run a quick training session with your support desk and make sure everyone knows exactly where the new levers and dials are located. Don't let a simple management portal deprecation turn into a self-inflicted incident ticket because your team didn't know where to look. Right. So, what does this all mean? And to synthesize this chaotic week, we've essentially watched a deep technical masterclass in modern cloud complexity.

As these distributed systems get smarter, they get more heavily layered with resilience mechanisms and automated compliance triggers, the failures become incredibly nuanced. We aren't dealing with simple hard drive crashes or localized power outages anymore. We are dealing with machine learning models getting mathematically overzealous and quarantining real estate contracts. We are dealing with resilience updates causing cascading traffic jams that take down entire global communication suites.

It's a completely new paradigm of troubleshooting, which leaves me with a final forward looking question for you to mull over as you head back to the console. We talked about how an AI model migration triggered a traffic surge that took down the co-pilot feature itself purely because of aggressive client retry logic. Right. If a simple back-end model swap can cause a self-inflicted DDoS attack, how on earth do we properly load test the unpredictable nature of AI prompts in production environments?

It's a terrifying thought for any QA or reliability engineer. Traditional load testing assumes predictable deterministic repetitive queries. You hit the database with the same synthetic transaction 10,000 times. But generative AI is completely asymmetrical. Every prompt is different. Every processing duration is unique. Are our traditional load testing tools and philosophies becoming completely obsolete in the age of generative AI? It's something every cloud architect needs to be thinking about right now.

You've been listening to Support Engineering Weekly from the Support Engineering blog. Until next episode, keep troubleshooting, keep learning, and keep engineering. Thank you for listening to Support Engineering Weekly. This podcast is available on Apple Podcast, Spotify, iHeartRadio, and other major podcast platforms. Follow the show on your preferred platform and never miss an episode.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Microsoft Weekly — November 16-20, 2026

    Nov 20, 202656 minAudio

    What happens when a routine network maintenance operation triggers a failure deep inside a cloud provider's authentication infrastructure? In this week's episode, Alex and Maya investigate how a seemingly isolated...

    Credit the people on this episode

  2. When Redundancy Fails: Inside Microsoft’s Network Stampede

    Nov 17, 202633 minAudio

    How can a routine network capacity upgrade trigger a global cloud outage - and why can the very systems designed to protect availability make the failure worse? In this Support Engineering Weekly bonus episode, Alex and...

    Credit the people on this episode

  3. From Microsoft Announcement to Completed Change

    Nov 15, 202623 minAudio

    Microsoft 365 administrators deal with information from several places: the Microsoft 365 Roadmap, Message Center, and Service Health. But knowing where to find an update is only part of the challenge. The real question...

    Credit the people on this episode

  4. Microsoft Weekly — November 09–13, 2026

    Nov 13, 202623 minAudio

    A server under pressure fails its health checks. Automated traffic management removes it from service, pushing even more load onto the remaining infrastructure. What was designed to protect the system becomes part of...

    Credit the people on this episode

  5. Planner Premium Isn't Project Online — So What Is the Gap?

    Nov 8, 202621 minAudio

    Microsoft Project Online is approaching retirement - or, in this case, its retirement date has already passed. But replacing a project management platform is not as simple as moving tasks from one product to another. In...

    Credit the people on this episode