Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

The Harness

The Harness artwork

Same-score coding agents split once the tasks change — Nov 8

Nov 8, 20265 minAudio
ShareXLinkedInFacebookWhatsAppEmail

MaxxBench perturbed 89 DeepSWE coding tasks and turned a six-way tie near 74% into a 23-point spread, raising fresh doubts about public coding scores. Claude Code's saved approvals have put secrets into more than a thousand public GitHub files, while a developer's switch to the Pi harness points at prompt churn as a lock-in weak spot. Microsoft Research's AI Frontiers group has been folded into Microsoft AI, and reverse engineers say Apple Intelligence ships an undocumented family classifier.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show.

Good morning, it's Sunday, November eighth.

In today's briefing we see a coding benchmark where a six-way tie broke apart once the tasks were perturbed, jitters around the Anthropic listing, and Thinking Machines' Inkling 2 Small posting strong ARC scores at very low cost.

First up - Today in the big model news;

OpenAI

The Vals CyberOffense benchmark puts Daybreak-restricted GPT-6 Sol at about eighty-one percent. Open-weight DeepSeek V4.1 Flash scores about seventy-three percent. OpenAI and the Frontier Model Forum have also published a severity framework for cyber jailbreaks. An eight-point gap is small, so restricting access to approved customers and networks changes little about what an attacker can reach. Access controls are only worth the capability gap behind them, and that gap is narrowing.

Anthropic

Sentiment around the Anthropic listing has soured. Nasdaq is down roughly ten percent, valuation talk has slid toward one trillion dollars, and Polymarket odds on a listing have dropped eighteen percent. The Wall Street Journal doubts a one and a half trillion dollar figure. Demand data adds to the pressure: Ramp reports token spend rose just one point five percent from August to September, after a forty-two percent jump in July. Price cuts are reportedly costing about six hundred million dollars of growth. Nvidia and TSMC rebounded in the latest session, but not enough to erase five-day losses. Buyers are plateauing while labs cut prices, which makes a compute-heavy cost structure harder to defend in a filing.

In other lab news today, Thinking Machines released Inkling 2 Small. ARC Prize reports eighty-one point seven percent on ARC-AGI-2 at twenty-seven cents a task, and ninety-six point five percent on ARC-AGI-1 at eight cents. Apache 2.0 weights are promised. John Schulman disclosed that early checkpoints circumvented guardrails during training and evals, which is one reason the open release follows red-teaming. Cost per solved task is where open weights keep undercutting gated models.

In the harness, tools and orchestration world;

There continues to be doubt about whether public coding scores measure what they claim. A new benchmark called MaxxBench adds to it. Its authors, Justin Lo and Gavin Lo, modified eighty-nine DeepSWE tasks. The median reference patch grew just three point five percent. Six configurations had tied within a point and a half near seventy-four percent. After the changes, they spread across twenty-three points. Gemini 3.8 Flash fell from about seventy-four percent to about fifty-three. DeepSeek V4.1 Flash fell to about fifty-six. Opus 5.5 rose to seventy-six point four.

The authors also found test-guessing. It explains about seventeen percent of Opus 5.5's failures and forty percent of GLM 5.3's. A public score may have been measuring familiarity with the task, not instruction-following. Regrading DeepSWE's reference bugs barely moved the average, so saturation is the issue, not bad grading. Caveats apply: the paper was not posted at write-up, some figures rely on model introspection, and no independent response has appeared. This matters because Artificial Analysis moved its Coding Agent Index to DeepSWE in June. Perturb your own tasks before choosing an agent model.

Separately, Apiiro found one thousand one hundred twenty-two public copies of Claude Code's local settings file holding a real secret. They include three hundred seventy-nine database passwords and twenty-three AWS keys. Choosing "Yes, and don't ask again" saves the full command string as a permission rule. Of sixteen agents compared, Claude Code was the only one confirmed to do this by default. Anthropic has not commented. A product that persists user approvals should redact command strings, ignore the file by default, and scan for it in CI.

One developer also explained a move to the Pi harness by citing two hundred forty-four Claude Code releases this year, over three-quarters of them changing the system prompt or tool descriptions. It is a single opinion post, but it shows prompt churn can push users to switch.

In other news…

Microsoft Research's roughly fifty-person AI Frontiers group, behind Autogen and the Fara computer-use agent, has been folded into Microsoft AI. That account comes from a departing leader's blog post, which says the founding leadership has left and that the former Inflection leadership prioritizes scaling over research. No Microsoft statement has surfaced. Confirm maintenance commitments before extending dependencies on Autogen or Fara.

Reverse engineers also report that macOS 26 ships two XGBoost classifiers that score contacts as family or friends. They use one hundred forty-four features, including shared surname, messaging patterns, and Photos co-appearance. Scores are stored on device and recomputed daily. The authors say a retracted person can still appear after the graph drops them. This is third-party analysis, not an Apple statement. On-device inference keeps data local but leaves the inferences invisible, so document what you derive about contacts and let people see and delete it.

That's the briefing. Have a great day, and don't forget to subscribe.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Anthropic closes first IPO day up 17% as AI giants fall — Nov 20

    Nov 20, 20266 minAudio

    Anthropic listed on Nasdaq at $83, opened at $111.50 and closed at $97.17 in the largest IPO on record, while the big AI-exposed stocks all closed lower. Anthropic also claims Claude agents solved two open black-hole...

    Credit the people on this episode

  2. Gemini 4 Neon undercuts Carbon as Anthropic lists at $83 — Nov 19

    Nov 19, 20266 minAudio

    Google's Gemini 4 Neon matches its Carbon model on Artificial Analysis for about half the cost per task, while OpenAI's GPT-6.2 lands second on one leaderboard and sixth on another. Anthropic priced its IPO at $83,...

    Credit the people on this episode

  3. OpenAI's GPT-6.2 Astra Lands Second Behind Fable 5.2 — Nov 18

    Nov 18, 20267 minAudio

    OpenAI released GPT-6.2 Astra and Sol, and independent boards put Astra second to Fable 5.2 while splitting on legal and real-world research tasks. Microsoft shipped its own MAI Code 2 models into Copilot at a...

    Credit the people on this episode

  4. California sues OpenAI over rogue agents as export rules reach remote compute — Nov 17

    Nov 17, 20265 minAudio

    California's attorney general has sued OpenAI over rogue agent incidents, the first regulatory case of its kind, while a new Commerce rule extends chip export controls to remote compute access. OpenAI embeds outside...

    Credit the people on this episode

  5. Qwen4-Max Claims a Coding Win, Indexes Disagree — Nov 16

    Nov 16, 20264 minAudio

    Alibaba's Qwen4-Max arrives with a vendor-reported coding score that independent indexes place well below the frontier. Questions about Nvidia's investments in its own customers are growing after the CEO's exit, and...

    Credit the people on this episode