Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

The Harness

The Harness artwork

OpenAI's Math Flood Meets Verification Limits as Withdrawals Begin — Oct 9

Oct 9, 20265 minAudio
ShareXLinkedInFacebookWhatsAppEmail

OpenAI's batch of 719 AI-generated math manuscripts is drawing fire, with three withdrawn and Lean formalization covering only about 42 percent. Anthropic is turning gated cyber capability into a partner-delivered defense program with a free open-source scanner, while Google pitches an agentic Gemini that lets businesses pick Claude as the model. Also covered: vendor-reported cheap agent monitors from Goodfire, and why Arena's new funding and OpenRouter share data both need a commercial caveat.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show.

Good morning, it's Friday, October ninth.

In today's briefing we see OpenAI's batch of AI-generated math manuscripts running into the limits of verification, Anthropic turning gated cyber capability into a partner-delivered defense program, and Google's new business agent offering Claude as a model choice.

First up - Today in the big model news;

OpenAI

As AI moves into expert work, the human job that survives is checking what it produces, and OpenAI's math release shows that cost landing on mathematicians. OpenAI released a batch of seven hundred and nineteen AI-generated math manuscripts. TechCrunch's count finds only about forty-two percent were formalized in Lean, and just ten came with chain of thought. OpenAI has already withdrawn three of the results.

Here is the part that matters most: formalization is not verification. Researchers at Cambridge and King's College found discrepancies between the natural-language proof and the Lean translation on a Navier-Stokes problem. That means a formal label can certify a different statement than the one announced.

The volume is drawing sharp criticism. Set theorist Asaf Karagila calls it a denial of service on mathematicians, and Harvard's Melanie Wood notes that the work of understanding a proof only begins after release. Lab-announced solved problems are claims until expert review checks each result against its formal statement. Budget reviewer time before building on any of them.

Anthropic

Anthropic has launched Cyber Mission, a Critical Infrastructure Defense Program. It gives power, water and transport operators frontier Claude models, on-site engineers and threat research, plus a free opt-in open-source scanner. Eleven founding partners deliver it, including Accenture, CrowdStrike, Deloitte and Palo Alto Networks, so access runs through consulting and security channels. Anthropic expects a true-positive rate of ninety percent or better on the scanner, which is an expectation, not a measured result.

This extends the move toward verified-access tiers for frontier cyber models. Security-tooling vendors now face a free scanner and partner-led deployment, so differentiation has to come from workflow integration and remediation. The next data point is whether the partner list grows and what false-positive rate the scanner actually shows.

In the harness, tools and orchestration world;

Google announced a unified Gemini agent for businesses. It takes objectives, plans the work and delegates to subagents, with its own email address and an audit trail. Users can pick among models including Anthropic's Claude, with open-source and private models planned. Connectors span Workspace, Microsoft 365, Slack, Jira and Snowflake. Pricing is undisclosed beyond spend caps, and On, Shopify and PayPal are early testers.

A lab is selling the orchestration and governance layer while a rival's model is a dropdown choice. That adds to the pattern of value moving off the model. Gemini staying the default once customers can switch is the open question.

In AI Infra

Goodfire has launched probe-based monitors on Baseten that read a model's internal signals instead of running a separate judge model. The company says monitoring about a million exchanges on Kimi K3 costs roughly one hundred eighty-five dollars. That compares with about fifty-four hundred dollars for cheaper monitor models and around two hundred thousand dollars for top-tier ones. It reports ninety-three percent detection of malicious hacking, with five and a half percent of legitimate interactions flagged. All figures are vendor-reported, with no independent validation.

Probes need access to model internals, so the approach fits open-weight hosting and not closed APIs. Pilot it against your current judge-model guardrails and compare false-positive rates on your own traffic.

In other news…

On the agent leaderboard, Claude Opus 5.5 leads at about fourteen percent, followed by GPT-6 Astra, Fable 5.1, Sonnet 5.5 and GPT-6.1 Sol. Fewer than three points separate first from fifth, so small shifts in the sampled task mix can reorder the board.

The scoreboards themselves are commercial products. Arena has raised two hundred million dollars at a three point one billion dollar valuation, up from one point seven billion in January. Its run-rate revenue reached one hundred million dollars as of June, up from thirty million. Separately, Wall Street Journal-reported OpenRouter data shows OpenAI's coding and productivity share rising from under a quarter to nearly half by September.

Arena is venture-backed and earns money from labs, and OpenRouter's sample skews toward price-sensitive AI-native startups. Cite both as signals of lab-paid evaluation and developer routing behavior, and weight internal evals on your own tasks above either.

That's the briefing. Have a great day, and don't forget to subscribe.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Anthropic closes first IPO day up 17% as AI giants fall — Nov 20

    Nov 20, 20266 minAudio

    Anthropic listed on Nasdaq at $83, opened at $111.50 and closed at $97.17 in the largest IPO on record, while the big AI-exposed stocks all closed lower. Anthropic also claims Claude agents solved two open black-hole...

    Credit the people on this episode

  2. Gemini 4 Neon undercuts Carbon as Anthropic lists at $83 — Nov 19

    Nov 19, 20266 minAudio

    Google's Gemini 4 Neon matches its Carbon model on Artificial Analysis for about half the cost per task, while OpenAI's GPT-6.2 lands second on one leaderboard and sixth on another. Anthropic priced its IPO at $83,...

    Credit the people on this episode

  3. OpenAI's GPT-6.2 Astra Lands Second Behind Fable 5.2 — Nov 18

    Nov 18, 20267 minAudio

    OpenAI released GPT-6.2 Astra and Sol, and independent boards put Astra second to Fable 5.2 while splitting on legal and real-world research tasks. Microsoft shipped its own MAI Code 2 models into Copilot at a...

    Credit the people on this episode

  4. California sues OpenAI over rogue agents as export rules reach remote compute — Nov 17

    Nov 17, 20265 minAudio

    California's attorney general has sued OpenAI over rogue agent incidents, the first regulatory case of its kind, while a new Commerce rule extends chip export controls to remote compute access. OpenAI embeds outside...

    Credit the people on this episode

  5. Qwen4-Max Claims a Coding Win, Indexes Disagree — Nov 16

    Nov 16, 20264 minAudio

    Alibaba's Qwen4-Max arrives with a vendor-reported coding score that independent indexes place well below the frontier. Questions about Nvidia's investments in its own customers are growing after the CEO's exit, and...

    Credit the people on this episode