Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

AI Security Ops

AI Security Ops artwork

AI and Bug Bounties | Episode 51

May 11, 202614 minAudio
ShareXLinkedInFacebookWhatsAppEmail

In this episode of BHIS Presents: AI Security Ops, the team breaks down a growing problem in cybersecurity: AI-generated bug bounty “slop” overwhelming the system. What started as a powerful way to crowdsource vulnerability discovery is now hitting a breaking point. Programs like cURL’s bug bounty and platforms like HackerOne are seeing a massive surge in submissions - but fewer and fewer of them are actually valid. The result? Security teams spending hours reviewing reports that go nowhere, while real vulnerabilities risk getting buried in the noise. We dig into: • Why cURL shut down its bug bounty program after years of success • How valid reports dropped from 1-in-6 to 1-in-20 • What “death by a thousand slops” actually looks like in practice • How AI is flooding programs with low-quality vulnerability reports • The difference between “theoretical” vs. exploitable vulnerabilities • Why reviewing findings is now harder than generating them • How HackerOne is responding to the surge in submissions • Whether AI can be used to filter AI-generated noise • The role of reproducibility and proof-of-impact in triage • Why human expertise still matters in vulnerability validation This episode explores a critical shift in security operations: when vulnerability discovery becomes cheap and automated, validation and triage become the real bottleneck. ⸻ 📚 Key Concepts & Topics Bug Bounty Programs & Triage • Submission quality vs. volume imbalance • Signal-to-noise challenges in vulnerability pipelines • The growing burden of manual validation AI in Vulnerability Discovery • Automated scanning vs. real exploitability • AI-generated findings and false positives • The “editor’s dilemma” - review vs. generation AI Security Risks • Lower barrier to entry for vulnerability discovery • Over-reliance on AI without domain expertise • Flooding systems with low-quality submissions Defensive Strategy • Requiring reproducible steps and proof-of-impact • Using AI to pre-filter vulnerability reports • Combining human expertise with AI tooling Industry Impact • cURL bug bounty shutdown • HackerOne submission pause • Shifting economics of vulnerability research #AISecurity #BugBounty #CyberSecurity #LLMSecurity #ArtificialIntelligence #InfoSec #BHIS #AIAgents #AppSec ---------------------------------------------------------------------------------------------- (00:00) - Intro: Bug Bounty Burnout & AI Noise (01:14) - cURL Kills Its Bug Bounty Program (02:05) - “Death by a Thousand Slops” Explained (03:42) - AI vs Vulnerability Scanners: Signal vs Noise (04:38) - HackerOne Pauses Submissions & Industry Impact (05:41) - Can AI Filter AI? Proposed Solutions (07:49) - Why Humans Still Matter in Validation (12:55) - Final Takeaway: AI as a Tool, Not a Replacement Click here to watch this episode on YouTube. Creators & Guests Ethan Robish - Host Bronwen Aker - Host Brian Fehrman - Host Derek Banks - Host Brought to you by: Black Hills Information Security https://www.blackhillsinfosec.com Antisyphon Training https://www.antisyphontraining.com/ Active Countermeasures https://www.activecountermeasures.com Wild West Hackin Fest https://wildwesthackinfest.com 🔗 Register for FREE Infosec Webcasts, Anti-casts & Summits https://poweredbybhis.com Click here to view the episode transcript.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

Brian FehrmanHey, everyone. Welcome to this week's episode of AI Security Ops. So imagine that you wake up and you've got a bunch of new security vulnerability reports that were filed overnight. You spend your entire morning reading through them, checking code paths, trying to reproduce the issues, and you come up empty handed. Find out every single one was just sloppily submitted to you without extra review, and that's exactly a problem that some of our bug bounty program partners are facing out in the industry today, and it's something we'll talk about in this episode. But before we get started, let's talk about Black Hills information security. If you or your organization are in need of any security services, whether that's external testing, internal testing, AD testing, web apps, physical pen testing, wireless, security operation center. Any security needs you might have, check us out at blackhills info sec dot com. Additionally, we have a training branch, anti siphon, where our practitioners who are doing these things day in and day out package up their skills and knowledge in a very digestible way to put out to the public at a very affordable freight price. Check check them out at antisiphontraining.com.

So let's dive into this a little bit. So just to kick it off, first story is back in January 2026, cURL, Daniel Stenberg, the creator of cURL, actually killed this bug bounty program that he had out. And the six years that he had it out, it paid out about a $100,000 for 87 vulnerabilities that have been submitted. So it's a very successful program for them. But then things started to go downhill recently where the valid submissions went from a one in six ratio to about a one in 20 ratio.

And his breaking point was that he had seven submissions in a sixteen hour window, and none of those were real vulnerabilities, which he called death by a thousand slops. So let's talk about what's contributing to something like this. And what do you guys think? Since we're on the AI Tech podcast, what do you think is a driving factor here?

Derek BanksWell, if you believe my x feed, the security has been solved by everybody and their brothers at GentekAI platforms. Platforms. So I don't know. Don't know. The whole cybersecurity industry is is now solved, I guess.

Brian FehrmanWe're all done. We can go home.

Derek BanksYeah. So I'll credit John Strand for this, but it it's kind of akin to, you know, you're running vulnerability scans and working on them, doing basically un auth vulnerability scans, and then all of a sudden you decide one day, hey. I'm gonna give this thing creds and, like, log in to stuff and find vulnerabilities, and your vulnerability report went from 200 pages to 200,000 pages overnight, and then you said, the hell am I gonna do with all this? So I think there's a difference between finding technically a vulnerability, technically a vulnerability, versus actually being able to take advantage of that vulnerability and exploit it and deliver some kind of impact. Right?

And so I think that still takes, obviously, a human, apparently. I mean, I'm not saying, like you said, AI seems to be finding some legit vulnerabilities. Apparently, the the latest hotness, the copy fail thing was found by some some kind of AI, excellent or something. Right? But, yeah. One in 20 is a pretty big ratio when you have to, like, spend your time manually looking through all that stuff. Right?

Brian FehrmanYeah. Yeah. I mean, that that takes it takes real time. Right? I mean, to to tell the AI to go out and to do something is about I I think the scanner analogy is perfect. It's about the same amount of effort that it takes to tell a scanner to go out and do something at this point. I mean, maybe, like, marginally more depending on how they're using the AI, but actually going through the output, that's that can take a lot of time. That that that's where a lot of real work can can happen. And I think that's where a lot of people are getting bogged down right now. It's easy to generate content.

It's a lot harder and more time consuming to actually review the content.

Bronwen AkerSo Editor's dilemma.

Brian FehrmanYeah. And so curl's not alone in this. I think they were actually part of the hacker one program, which is what we'll mention now. I think that's where they withdrew their their bug bounty program from, if I'm not mistaken. But hacker one as a whole, as of March 27 and as still of this recording, I I checked that their program is actually still paused for new submissions, because they're finding that they are spending just an exorbitant amount of time going through these vulnerabilities that vulnerability reports that are getting submitted by people, and it just been overloaded with the volume.

And so they're having to work through how are they going to filter out, basically, the signal from the noise. Right? And that's that's a very that's a difficult problem that I think a lot of people are currently trying to to figure out in the industry as a whole is how do we effectively how do we strike that balance between, keeping up with the pace of being able to quickly find and triage vulnerabilities, but yet still be able to separate out what's not important from that. And and it's and it's kind of a tough problem. Right?

What are your what are your guys' thoughts, opinions?

Bronwen AkerWell, the hacker one story came up in the the regular newscast for BHIS, and I know that we did a lot of discussion about the fact that the bug bounty program rewarded people for finding vulnerabilities, but there isn't a bounty program or a reward system for the people who are performing the actual fixes. And now that a little more time has passed, it seems kind of like a silly, obvious solution would be to have develop AI to reality check the submissions before then handing it off to a human for resolution. That would help take a legitimate load off of the humans. It would also help speed up the pipeline of getting legitimate issues into the hands of people who can actually fix them. But, you know, the the challenge there is gonna be training a new model to identify what's a a valid vulnerability versus what is a well, let's call it vulnerability slop.

Ethan RobishThe if AI is the problem, the solution is more AI.

Derek BanksThat's typically how a data scientist would look at it. Yes. Exactly. How do we protect AI? Well, we put more AI in front of it. Sounds like a plan.

Ethan RobishNo. But I think, Bronwen, you're you're onto something, not necessarily like better training being the whole picture, but giving like, if if you are fielding bug bounty requests for a specific application, giving the AI access to, you know, a staging environment, a testing environment with your application that can take the reports and try to reproduce it, like, this even able to be exploitable? And that's like an immediate validation. But I mean, that requires setting up more tooling. But I mean, realm of possibility, right?

Yeah.

Derek BanksI mean, I guess it's not just the model, it's also the harness. Right? And I think at the end of the day, I'm still in the camp of it really a human to determine whether or not this is truly, like, legitimate vulnerability. I think AI is a tool, and we shouldn't lose sight of that as technology workers. Right? It's just a really fancy pattern matcher. It's very good, but, you know, the more you learn about how AI works, the more you realize it's not a person, it's not an entity, it's a really fancy pattern matcher matcher. There's a lot of math going on, but, I that's really, at the end of the day, what's happening. And so I I still think that, you know and I've spent a lot of time recently, you know, you trying to develop custom agentic solutions to do, you know, penetration testing work, and I've been really successful. But also, sometimes you look at what the quote AI found.

Right? And, you think, man, a human would have never thought this was an issue. This is not an issue kind of thing. So, I mean, I I think, you know, yes, you know, models are going to get better, harnesses are gonna get better, but, yeah, I think at the end of the day, as long as we're using transformers, like, is this is gonna be an issue that, you know, there's just gonna be a whole bunch of people who are pointing AI at stuff and going, see it, found a vulnerability, and and not using what we would call, like, domain knowledge, like, how much do you know about this thing, you, and applying that to it and being you know, figuring out is this for real or not.

Bronwen AkerOkay. So I have a question because I've never participated in a bug bounty program. If I submitted to hacker one, do I have to tell them how to replicate the error? Because if so again, Derek, you you hit the nail on the head. AIs are tools. They're good at following sort of directions. And maybe if they have that roadmap to follow, then they can go, yes, the steps worked. No, it didn't. If they get a yes, that can bump it up whatever weighting scale can be used to get things through the queue faster. Just a thought.

Brian FehrmanYeah. So typically, I've only done bug bounties a couple times, but usually, they they ask for reproduce, like, steps to reproduce so that, to make it quicker for their their people to go through.

Derek BanksYeah. I think the the better your write up, the better the chances you're gonna get paid. Right? It's like a little mini

Brian Fehrmanpen test. Right? That's that's Yeah. What it was supposed supposed to be about. And, yeah, certainly, so I I think that's, you know, a good point is a is a first filter level is, like, you know, only only taking reports that actually have the steps, the commands that are in them that allow for the AI to go through and reproduce without, like, error to try to bump up, you know, bump up certain submissions higher than others.

Derek BanksYeah. If I was gonna do it, like, right now, I'll just say our AI platform. Right? You find something, it actually outputs everything for every single finding. Here are the commands that a human should use to go and verify this. So the human should go and use, you know, those commands to go verify it. So to me, that that's like the starting point, not the ending point for the vulnerability. Right? And so I feel like, yes. Okay. Now a human has verified it. Can you actually cause, like, more impact with this? Can you, even though the AI even though I try and have our platform tie stuff together, does this, like, have, like, more tie ins to things? And and so I think to me, like, you have, like, a human pen tester and an AI pen tester that, you know, depending on what you're looking for, you might think that one's better than the other, but both of them together are better than either one by themselves, in my opinion.

Ethan RobishThat's what I was going to say too.

Derek BanksOh, damn it. Sorry, Ethan. Doesn't have to be

Ethan Robishno. No. No. I'm I'm agreeing with you. It doesn't have to be adversarial. You mentioned earlier, like, AI is a tool. So I I mean, yeah, we should use it to to make us better at our jobs. And as humans are really good at taking new tools and try to use them in ways they're not meant to be or finding out the limitations, like you have a hammer, everything starts looking like a nail.

Derek BanksYou once told me about that I don't with Jupyter Notebooks, and I'll have you know that I

Ethan Robishstill do use Jupyter Notebooks. Nice. Yeah. And I I don't think we've found the limits yet, right, of, like, where AI can stop being helpful. Like we've we've found limits, but not like the absolute, you know, here's we can't progress any further or whatever. Like we're still it's still changing at super rapid pace of both the innovations in the fundamental models and, like, what people are able to do with them.

Brian FehrmanYeah. I I completely agree. So, yeah, I think I think that'd be probably a good place to, wrap up and close and and just say that, I mean, a AI is definitely finding, like, real bugs. I mean, it is it is an extreme extremely useful tool. Like we've said, it's great for augmenting, you know, many many different tasks, but it's important that we're still understanding that that humans are still needed and, professional judgment is still needed in this space, to be able to, so we can augment one another, not try to, you know, replace or, completely shift off, tasks and responsibilities.

So, yeah, with the with that, any other closing thoughts?

Derek BanksNo. I'll just say with that, keep on prompting.

Brian FehrmanKeep on prompting.

Open the transcript file

Hosts and Guests

Hosts

  • Brian Fehrman Host Feed-stated
  • Bronwen Aker Host Feed-stated
  • Derek Banks Host Feed-stated
  • Ethan Robish Host Feed-stated

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Black October Never Ended Why AI Just Broke the Patch Clock | Episode 77

    Nov 18, 202622 minAudio

    In this episode of BHIS Presents: AI Security Ops, Derek Banks and Ethan Robish revisit Black October and ask whether the wave of critical cybersecurity vulnerabilities is finally slowing down - or just getting started....

    Credit the people on this episode

  2. Black October | Episode 76

    Nov 6, 202622 minAudio

    In this episode of BHIS Presents: AI Security Ops, Bronwen Aker, Dr. Brian Fehrman, and guest Ethan Robish examine Black October, a wave of major cybersecurity vulnerability disclosures that sent the industry into a...

    Credit the people on this episode

  3. More Rogue Agents: Who Watches the Watchers? | Episode 75

    Oct 29, 202630 minAudio

    In this episode of BHIS Presents: AI Security Ops, the team tackles a growing problem in AI security: What happens when the AI agents you’re monitoring start disabling the very systems designed to keep them in check?...

    Credit the people on this episode

  4. Will AI Take Over? [Part 2] Recorded Live From Wild West Hackin' Fest | Episode 74

    Oct 27, 202637 minAudio

    In this special episode of BHIS Presents: AI Security Ops, Bronwen Aker, Derek Banks, Ethan Robish, and Brian Fehrman come together in person at Black Hills Information Security headquarters in South Dakota during the...

    Credit the people on this episode

  5. AI Behaving Badly | Episode 73

    Oct 16, 202634 minAudio

    In this episode of BHIS Presents: AI Security Ops, the team digs into some of the strangest recent examples of AI behaving badly. An AI agent finds a way around its sandbox using DNS. Malware asks four different AI...

    Credit the people on this episode