Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Complex Systems with Patrick McKenzie (patio11)

Complex Systems with Patrick McKenzie (patio11) artwork

AI and the great developer speed-up, with Joel Becker of METR

Aug 21, 20251 hr 4 minAudio

This week on Complex Systems, Patrick McKenzie (patio11) is joined by Joel Becker from METR. They discuss groundbreaking research on AI coding assistants. Joel et al’s randomized controlled trial of 16 expert developers working on major open source projects revealed a counterintuitive finding: despite predictions of 24-40% speed improvements, developers actually took 19% longer to complete tasks when using AI tools, even though they retrospectively believed they were 20% faster. The conversation explores why even sophisticated professionals struggle to accurately assess their own productivity with AI tools, the industrial organization of software development, and the implications for AI's recursive self-improvement in research and development. It also touches on other perspectives from software developers using these tools professionally, and where we can expect them to improve rapidly. - Full transcript available here: www.complexsystemspodcast.com/the-great-developer-speed-up-with-joel-becker/ - Sponsor: This episode is brought to you by Mercury, the fintech trusted by 200K+ companies - from first milestones to running complex systems. Mercury offers banking that truly understands startups and scales with them. Start today at Mercury.com Mercury is a financial technology company, not a bank. Banking services provided by Choice Financial Group, Column N.A., and Evolve Bank & Trust; Members FDIC. - Recommended in this episode: METR: https://metr.org/ Joel Becker’s site: https://joel-becker.com/ - Timestamps: (00:00) Intro (00:34) Understanding AI evaluation methods (02:04) METR's unique approach to AI evaluation (03:10) The evolution of AI capabilities (06:44) AI as coding assistants (09:15) Research on AI's impact on developer productivity (13:55) Sponsor: Mercury (15:07) Challenges in measuring developer productivity (20:38) Insights from the research paper (31:26) The formalities of software development (32:07) Automated tools and human discussions (32:47) AI and style transfer in software (34:35) The role of comments in AI coding (36:51) The future of AI in software engineering (40:25) Economic implications of AI in software (46:53) Challenges and risks of AI in software (59:03) Security concerns with AI-generated code (01:04:59) Wrap

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Hosts and Guests

Hosts

  • Patrick McKenzie Host Feed-stated

Guests

  • Joel Becker Guest Feed-stated

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Liquidating a whale by hand, 22 times

    Nov 12, 202622 minAudio

    Patrick McKenzie (patio11) reads his 2022 Bits About Money essay demystifying financial leverage, starting from the balance sheet math that lets you work out in your head how big a drop in asset value wipes out a firm’s...

    Credit the people on this episode

  2. Reading the Anthropic S-1, with Zvi Mowshowitz

    Nov 6, 20261 hr 20 minAudio

    Patrick McKenzie (patio11) is joined by returning guest Zvi Mowshowitz, AI watcher extraordinaire, to read Anthropic’s S-1 and what it shows about the AI industry. They cover a founder-controlled structure built on a...

    Credit the people on this episode

  3. Why SoftBank paid me 17 yen to skip the air conditioning

    Oct 29, 202627 minAudio

    Patrick McKenzie (patio11) reads his 2022 Bits about Money essay “Markets in power,” prompted by a SoftBank push notification asking him to cut his air conditioner use between 5 and 8 PM, and updated for the era of AI...

    Credit the people on this episode

  4. The charity that is also a time machine

    Oct 22, 202627 minAudio

    Patrick McKenzie (patio11) reads his 2022 Bits about Money essay on the infrastructure behind American charitable giving, prompted by the wave of AI-generated wealth expected to flow into philanthropy. He explains...

    Credit the people on this episode

  5. ⁠⁠The $900 million wire that wasn't final

    Oct 8, 202633 minAudio

    Patrick McKenzie (patio11) reads his 2022 Bits about Money essay "Finality does not exist in payments." It argues that how final a payment is depends on the rails, the parties, and the governing law, not on the payment...

    Credit the people on this episode