Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

The Harness

The Harness artwork

Self-graded benchmarks meet their usual problem — Aug 20

Aug 20, 20265 minAudio

A self-improving open model claims it beats Claude Opus 4.8 on its own harness, but the one independent check that exists points the other way. OpenAI matches Anthropic's zero-data-retention promise and previews a way to catch cross-session risk without reading anyone's prompts. Pennsylvania becomes the third government in two weeks to add a veto point to the AI data-center buildout, while Wired reconstructs a police search tool that finds people from movement patterns alone.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show.

Good morning, it's Thursday, August twentieth.

In today's briefing we look at a new open-weight model claiming to beat Claude Opus on its own benchmark with little independent evidence behind it, OpenAI matching Anthropic's data privacy stance while previewing a new cross-session safety check, and Pennsylvania becoming the third government in two weeks to add a veto point to the AI data center boom.

First up - Today in the big model news;

OpenAI OpenAI reaffirmed Zero Data Retention for eligible frontier-model API customers and previewed a new system called Private Safety Processing, meant to catch risk patterns across multiple related interactions without giving OpenAI staff access to the underlying content. It's testing with Microsoft and Databricks, with a technical paper due in September. The move closes a gap with Anthropic's existing zero-retention stance, but it goes further: current zero-retention safety systems only evaluate one interaction at a time, and longer autonomous agent tasks create risk that only shows up across a sequence of them. Enterprise buyers comparing the two labs' data handling terms now have a real difference to weigh, not just a matching promise. Whether Private Safety Processing can catch cross-session risk without the access it claims not to need is what September's paper has to prove.

In local model developments;

There's a recurring problem with self-graded benchmarks: labs keep shipping self-reported numbers days or weeks before anyone independent can check them. Ornith's new release fits that pattern closely. Ornith has released Ornith one point five, an MIT-licensed open-weight family spanning a three hundred ninety-seven billion parameter flagship down to a nine billion parameter model that runs on a phone. Its real advance is architectural: an end-to-end loop where the model proposes its own training tasks, builds scaffolds for them, and generates the reinforcement-learning data it trains on next. The flagship claims to beat Claude Opus four point eight on Terminal-Bench two point one by about a single point. That figure comes from Ornith's own harness, averaged over five of its own runs, a margin thin enough to be ordinary noise, with no independent group having checked it yet. The one outside data point that exists points the other way: an informal test of the family's thirty-five billion parameter variant found it scoring roughly half of what Qwen three point eight, twenty-seven B, managed on a coding benchmark called DeepSWE. That's precisely the failure mode a model grading its own homework would produce.

In the harness, tools and orchestration world;

TrueFoundry shipped TrueForge, a vendor-neutral agent harness claiming a thirty percent cut in token usage and seventy-five percent lower cost at equivalent accuracy, and Agent Arena launched alongside it: a benchmarking platform that scores harness behavior specifically, built on one point seven million real agent sessions rather than model outputs. TrueFoundry doesn't own a model, so it competes by treating the harness itself as a commodity worth undercutting on; Agent Arena only has a reason to exist if harness quality varies enough from model quality to be worth ranking on its own terms. Both add pressure to a market DeepSeek's own open-sourced harness is already gaining adopters in, while Anthropic and Google keep their harnesses proprietary. If TrueForge's numbers and Agent Arena's rankings hold up under real use, the harness layer commoditizes further and proprietary harness lock-in gets harder to defend than it looked a week ago.

On the regulatory and oversight front, two stories show infrastructure moving faster than the rules meant to govern it.

Pennsylvania Governor Josh Shapiro signed an executive order requiring any data center over twenty-five megawatts of peak demand to secure binding grid commitments and local community approval before state permit review even begins, banning non-disclosure agreements on the projects and stripping them of fast-track permitting status. It's the third distinct veto point on AI infrastructure buildout in two weeks, after Texas's statewide grid connection pause and the Cherokee Nation's ban on tribal land, and this one is procedural and pre-emptive rather than reactive, requiring commitments up front instead of auditing after the fact. Each veto point compounds site-selection risk independently of the others, so a developer now has to clear grid, tribal-sovereignty, and community-approval hurdles in whichever jurisdiction it lands, not just the cheapest power contract. A fourth state adopting Pennsylvania's pre-emptive-commitment model specifically would turn one state's order into a template rather than a one-off.

Separately, Wired reconstructed a police search tool built on the same kind of surveillance infrastructure. Flock Safety's OS Investigate tool, pieced together from files exposed through the company's own login pages, gives officers sixty-nine prewritten natural-language prompts to search by movement pattern alone, fourteen of which need no license plate, name, or physical description at all, including one that finds witnesses based on which vehicles were seen most often in a neighborhood during a given timeframe. Flock says the tool is still in development with a small group of law-enforcement partners and that its capabilities may change before any wider release. The product turns a surveillance network built for plate-matching into a general pattern-search engine over people's movements, a capability jump the plate-reader-specific rules many jurisdictions already have on the books were never written to cover. Any city currently governed by a plate-reader-specific ordinance is, in practice, ungoverned on this exact use.

Quick hits from the consumer side;

Amazon dropped its nineteen ninety-nine dollar monthly Alexa+ fee for compatible Fire TV devices, with no Prime subscription required, though Echo speakers and Ring features remain paywalled.

That's the briefing. Have a great day, and don't forget to subscribe.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Anthropic closes first IPO day up 17% as AI giants fall — Nov 20

    Nov 20, 20266 minAudio

    Anthropic listed on Nasdaq at $83, opened at $111.50 and closed at $97.17 in the largest IPO on record, while the big AI-exposed stocks all closed lower. Anthropic also claims Claude agents solved two open black-hole...

    Credit the people on this episode

  2. Gemini 4 Neon undercuts Carbon as Anthropic lists at $83 — Nov 19

    Nov 19, 20266 minAudio

    Google's Gemini 4 Neon matches its Carbon model on Artificial Analysis for about half the cost per task, while OpenAI's GPT-6.2 lands second on one leaderboard and sixth on another. Anthropic priced its IPO at $83,...

    Credit the people on this episode

  3. OpenAI's GPT-6.2 Astra Lands Second Behind Fable 5.2 — Nov 18

    Nov 18, 20267 minAudio

    OpenAI released GPT-6.2 Astra and Sol, and independent boards put Astra second to Fable 5.2 while splitting on legal and real-world research tasks. Microsoft shipped its own MAI Code 2 models into Copilot at a...

    Credit the people on this episode

  4. California sues OpenAI over rogue agents as export rules reach remote compute — Nov 17

    Nov 17, 20265 minAudio

    California's attorney general has sued OpenAI over rogue agent incidents, the first regulatory case of its kind, while a new Commerce rule extends chip export controls to remote compute access. OpenAI embeds outside...

    Credit the people on this episode

  5. Qwen4-Max Claims a Coding Win, Indexes Disagree — Nov 16

    Nov 16, 20264 minAudio

    Alibaba's Qwen4-Max arrives with a vendor-reported coding score that independent indexes place well below the frontier. Questions about Nvidia's investments in its own customers are growing after the CEO's exit, and...

    Credit the people on this episode