In this episode of the Distilled Security Podcast, we break down the Delve scandal - flawed SOC 2 reports, copy-pasted content, and oversight failures that expose deeper issues in compliance-as-a-service. Joined by Matthew J. Schiavone, we examine auditor accountability, quality review gaps, and key differences between SOC 2 and ISO 27001. We also cover what companies should demand from auditors, the role of automation, and whether this scandal will drive real change in the industry. Topics Covered The Delve scandal - leaked reports, copy-pasted audits & pervasive deficiencies The AICPA peer review process & AC Corp's adverse findings SOC 2 vs ISO 27001 - oversight models, witness audits & accreditation The incentive structure driving compliance to the bottom Compliance automation - what works, what doesn't & AI's real role What to ask your auditor before signing anything Trust centers - done right vs. compliance theater Is SOC 2 dead? What needs to change & who has to change it Hosts Justin Leapline - @justinleapline Joe Wynn - @wynnjoe Rick Yocum - @rickyocum Hosts Matthew J. Schiavone - (Sikich) Connect with Us Website: distilledsecuritypodcast.com X: @DisSecPod Email: hello@distilledsecuritypodcast.com
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Justin LeaplineToday on the episode, the Delve scandal blows the lid off fake compliance as a service, and we break down what it means for the certification industry, the future of compliance automation, and the third party risk management. Plus, a special guest joins us. This is Distilled Security Podcast. Somewhere
Speaker 2right now, someone is digging through folders trying to find the right version of an evidence file for the third time this week. Controls scattered across a dozen system, owners who left the company months ago, due dates that already passed, nobody noticed. It's how it's always been done, and somehow, we just accept it. There is a modern, better way. Episkey. Visit us at episkey.com/dsp for a special offer.
Speaker 3If you're leading a company today, you're not short on ideas. You're buried under them. More tools, more frameworks, more initiatives that were supposed to help and somehow made everything harder. Minus Partners exists for leaders who don't need another thing added. Minus Partners helps leadership teams remove friction that's been normalized over time. So execution gets easier, not harder. Less noise, clear decisions, faster
Justin LeaplineWelcome back to Distilled Security Podcast. Thank you for joining us. Before we actually dive into some of the topics that we have today, which is an excellent episode, I'm really looking forward to this, We have Matt Chavone. Matt, thank you for joining us, man.
Matthew J. SchiavoneThanks for having me.
Justin LeaplineSo to start off with, get the audience to know you and everything. Why don't you introduce yourself? Where are you working now? How you got into the industry?
Matthew J. SchiavoneOh, yeah. So my name is Matt Chavone. I work for Sikich. Sikich is a top 25 CPA firm. I lead the third party attestation practice. We focus primarily on SOC two, SOC one, ISO, CMMC, and PCI.
Justin LeaplineOh, okay. All the certifications?
Matthew J. SchiavoneMost of them.
Justin LeaplineYeah. Yeah. We we try. We try. Gotcha. So what got you into this? Have you always been doing SOC two, SOC one, ISO?
Matthew J. SchiavoneI started my career actually in the Department of Defense. Oh, okay. I'm a CPA, so got into internal controls mostly around financial reporting. Later in my career shortly into my career, got into more security related controls. And 2012 is when SOC two was released and got into SOC two a little bit shortly thereafter. Okay. It didn't wasn't popular right after
Justin Leapline'70 and all this stuff. They started splitting it out and everything. Got it. Yeah. Yeah.
Matthew J. SchiavoneIt snowballed from there. I got introduced to Joe. Joe got me on got involved in ISO. Okay. That was, like, 02/1716.
Rick YocumSomething like that.
Matthew J. SchiavoneYeah. Yeah. So and then just kept snowballing from there.
Justin LeaplineOkay. Great. And how big is your team that you do all this stuff with?
Matthew J. SchiavoneThe risk advisory practice as a whole is about 60 people. Okay. And then across PCI, CMMC, anywhere from 15 to 20
Justin LeaplineOkay. So the certification. So decent size. You do a lot of them every single year and everything. How many clients roughly, you know, do you do a year?
Matthew J. SchiavoneWe do
Justin LeaplineA lot. Yeah. Yeah. Yeah. Now do you promise SOC two in a week? Most of
Joe Wynnthe time.
Justin LeaplineYeah. Right? Yeah. Yeah.
Matthew J. SchiavoneOr sooner.
Rick YocumYeah. Yeah. That's only for startups that have no security whatsoever.
Matthew J. SchiavoneYeah.
Justin LeaplineRight. It's easy
Joe Wynnto fail a SOC two.
Justin LeaplineSo yeah. So, yeah, really appreciate you joining us and everything, and I think it's absolutely fitting for what we wanted to talk about mostly here. So to let everybody know, if you haven't heard about Delve, Delve is a GRC automation tool. They've been I don't know. It's only been about five years they've been on the market here.
Joe WynnI actually think it's much less. I actually looked at this right before we got here. I think it was they went through Y Combinator in, like, 2024.
Justin LeaplineI thought it was earlier than that. Okay. Yeah. Either which way.
Joe WynnYeah. Not long Very new
Justin Leaplineon the industry, all that stuff, and they got a reputation. And I don't know if you guys you you follow Troy Fine and everything. He he's great on, like, blowing up people that promised the world in what we know is unrealistic timelines ever working with any client whatsoever, you know, type of thing. But these were one of these companies that were known for sock in a box. You know? The quick compliance, if you just need a SOC two, we got you. Just pay us $510, whatever it was, you know, type of thing. Well, about two weeks ago, week and a half ago, I think it was just a week and a half ago, two Fridays ago with this, they got breached a little bit. I don't know the particulars of the breach, but essentially, a a Google Sheet got released out with all their clients and linked to documents with SOC two, ISO, and SOC ones into there. And somebody was it Delve die deep deep delver?
Delver or delvediver, something like that. Deep delver. Deep delver.
Rick YocumYeah. Substack article. Right.
Justin LeaplineNew new publisher. This is all they've actually published into this. They went through and did a pretty big hit piece onto them. It it was detailed. Their part one went through all the stuff. Yeah. It went through their reports. It went through how they were basically copying and pasting stuff, and all the reports are, like, available. I actually went in and downloaded all of them, and you could actually see all the reports. And they put it up into publicly downloadable format and all that stuff.
It's really interesting, you know, and you start looking at it. And a lot of it is they're SOC two reports. So, you know, and some of the ISO. But they pull out that it's basic copy and paste, you know, into that. They what were some of the other things they had? It was about 500 clients, and some of them were known customers into their, like, Lovable. I don't know if you guys know. Right. Yeah. They they do that. Their name was in there. Bland, Cooley, Eleven AI. Like, some of those I don't know, but a lot of them were just, you know, listed in here. And some of it was, like, garbage stuff. You can tell they're internal only into that.
They also not only they looked at the reports, they also looked at who's rubber stamping these Yep. And followed up on some of these companies, and it was it was interesting to say. They say even to, like, Google mapped, like, some of the companies are like, what's their location in Pakistan? Oh, yeah. It's some right above a marketplace, you know?
Joe WynnRight.
Justin LeaplineWas there a headquarters and all this? I mean, it was pretty big. I guess, you know, first off, before we dive too deep into it, you guys' opinion on this, like Well,
Rick Yocumfirst, full disclaimer. This is a person doing a write up. There's been no independent verification that this is that. Everybody that that keeps seeing responses sees Delve saying this isn't true. We're just talking about the article. Yes. Talking about the claims that there
Justin Leaplinewanna get sued. Yeah.
Joe WynnSo retroactively and henceforth in this podcast, just assume we say allegedly before every Just
Rick Yocumjust assume that because, you know, until until this is verified as proof, we don't know. Yeah. But that's a very well put together piece of research that somebody did. You can click into the stuff like Justin Yeah. Was mentioning, and you can kinda walk through the process and the person puts together quite a convincing story.
Joe WynnPretty compelling.
Justin LeaplineI mean, it's least it's enough to deserve a response. And I don't think Delv has like, there's a couple of LinkedIn posts that where their CEO, like like, we're coming out with a bigger response, but it's not really true. Like, some of the like, he he basically said, like, the reports weren't, like, finalized yet or something like
Joe WynnI think I saw a slightly more formal response. Did you? Okay. That more or less shifted the blame to the CPA for, oh, we just do software.
Justin LeaplineYeah. Yeah. That's right.
Joe WynnIt's a CPA. It's their fault. They didn't look enough. Yeah. So we we'll get into that.
Rick YocumYeah. And even looking past all that, one of the things I'm excited to talk about tonight is how does reliance on evidence really matter and make a difference, and how do you get something you can trust. So when we leave here tonight, I'm hoping that we have some good takeaways for people for what they should do when they're looking at this stuff and when they're evaluating software and auditors and even vendors that rely on these audit reports.
Justin LeaplineYeah. So I guess, first off, I mean, Matt, seeing this, did this surprise you?
Joe WynnAbsolutely not. Oh, no. That's a terrifying answer, actually.
Rick YocumWhy not?
Justin LeaplineBecause of Delve specifically or something else? Both. Okay.
Joe WynnDid you know of Delve before this?
Matthew J. SchiavoneYes. Okay. Well, only through I had a few client interactions where two noteworthy. One lost the SOC two client in December because they were moving I can't say whether or not it was Dell, but it was a very similar situation.
Justin LeaplineOkay.
Matthew J. SchiavoneThis company promised them, this is in December, their SOC two report by their deadline of May March 31. So already having not They already
Justin Leaplinesigned up with you?
Matthew J. SchiavoneWe were in the midst of renewing this year's engagement.
Justin LeaplineOh, okay.
Matthew J. SchiavoneThings were really slow, and I knew something was kind of a kind of a miss.
Justin LeaplineAnd everything. But they've already had a SOC two through you guys. They have. Okay. That They were looking at just cheapening it, I guess, getting a discount on doing that SOC two.
Matthew J. SchiavoneWell, I Yeah. For less than our SOC two, they got a pen test, ISO certification
Justin LeaplineOh.
Matthew J. SchiavoneAnd vCISO services. And I questioned, you don't even have an ISMS yet in December or a statement of applicability, and they're promising you an ISO certification.
Joe WynnThat deadline was also by March. The ISO March 31.
Justin LeaplineYeah. Okay. Okay.
Matthew J. SchiavoneDidn't know what a statement of of applicability was. So nor did they care. What they cared about was they can have Give us a stamp. Everything by March 31. Oh, that's
Justin Leaplinethat's today. Of a price.
Matthew J. SchiavoneThat's I should have checked in to see if hey, guys.
Justin LeaplineHow's that going? We remote Zoom in, you know.
Joe WynnAnother guest is joining us.
Matthew J. SchiavoneYeah. So how I mean, how how many ISO programs have you stood up in three months from initiation to certification?
Justin LeaplineNot to certification. You know? I mean, remediation, yes, if it's most of the way there. I mean, documentation is usually the biggest lift. You
Matthew J. Schiavoneknow? Stage two.
Justin LeaplineOh, it's stage two. Yeah. It still takes a while.
Rick YocumNo. Everybody I talked to that has a fairly sophisticated no. These aren't startups where you have one application, one very small ISMS Right. And really writing security policies. In that case, we would never just rubber stamp them and just pull out a template. But, you know, they don't have to be super complex because the organization is simple. But any organization that has couple 100 of peep couple 100 people are selling to health care and financial services, those kind of organizations, they're, you know, I'm like, look, let's just set a timeline of eighteen months to figure out where we're gonna be at, and then we'll start shortening it.
Joe WynnRight.
Rick YocumAnd we need to see what
Justin Leaplineyou have. Right. What scope you're setting?
Rick YocumYeah. What is you what's in scope? What do you wanna be compliant for? If you're putting an ISF MS in place from an ISO's perspective, you're gonna do more things than I think you even have to do just to hit all the SOC two stuff. And so maybe along the way, if that is the goal is to get both, you know, we can probably cross the SOC two finish line a little bit sooner, at least get to a type one situation where you can have your, you know, your your your design down and test of one in place.
And then after that, let's keep going and being able to show that you have have the stuff in place. But three months is that's typically a timeline and given these more mature, more complex companies just to do the first gap assessment Right. Lay out the remediation road map and get some traction on it.
Justin LeaplineOr as you pointed out, a very simple scope. Like, if they have one web app in Vercel or something like that, you could you could probably push through that, you know, type of thing. You still have to do a gap, but if you're not doing any infrastructure stuff and all you're doing is coding and policies and stuff, you could probably get them ready, you know, at that point.
Rick YocumOh, yeah.
Justin LeaplineYou know?
Rick YocumQuicker than that. Quicker.
Justin LeaplineYeah. Exactly.
Rick YocumYou'd probably be in a good situation in, like, two to three months, and then you get all that evidence burn in.
Justin LeaplineYeah. Exactly. Yeah. So but, yeah, the the the promises and I think we talked about this. We had a talk that we did for ISAC a few weeks ago.
Rick YocumOh, yeah. That was huge. Plus, it was free drinks if you missed it.
Justin LeaplineYes. Exactly. Yeah. We're sponsoring with that. But one of the things that we talked it was actually on point here. It was actually before this delve incident, you know, that came out. But we were talking about how the incentive structure is basically set up to the grade, you know. Right. You got these companies that they're trying to get bigger deals that the bigger companies say, well, not without a SOC two or ISO certification. So what do those companies look for? They look for the cheapest way to get a SOC two or ISO certification. And without a, basically, a an anti incentive, like, quality pushing it up, you're gonna get it to the grade to the worst possible point, you know, into that. And I think that's we've all known it for a while. You know? You know, you've seen it, you know, with these players are like, one week suck, you know, or your money back.
You know? I've seen that a lot with some of the players and everything. But, yeah, it's it's an issue. It's an issue, like
Joe WynnYeah. The incentive alignment issue has been around forever, particularly with, like, third party reviewers. Right? I mean, like, the whole Enron thing and socks came out of that. Yeah.
Justin LeaplineYeah. In fact, I was recently I have a buddy going through a cheap provider. Mhmm. And I I think I told you this. The funniest thing, they sent them, like, the evidence request packet. And one of the things that they had in the policy, they're like, if you don't have your access control policy, just copy and paste this into a chat GPT prompt.
Joe WynnYeah. Here's your prompt to enter
Justin Leaplineinto your prompt
Joe WynnYeah, Jenna.
Justin LeaplineTo get the policy out. It was like, I've never seen an evidence request. Yeah. Yeah. It was like, oh, wow. Yeah. Yeah. I mean, we we do that, you know, but you you don't say the deal.
Rick YocumI wanna go back to something we talked about just a couple seconds ago, which is, you know, how did this all come about? And I I can remember, you know, what is it? Like, fifteen, twenty years ago. And we're just starting to get into the parts where you really need to start you should always have been vetting your vendors. But it became a little bit more mainstream. I remember this in in that time frame. And the things we would do is we would create these
Joe WynnOh. Big
Rick Yocumquestionnaires. Who who's the big questionnaire maker?
Justin LeaplineSIG. That was SIG. The SIG Lite. Yeah.
Rick YocumSIG Lite. And so you get the SIG Lite. You'd, you know, you'd put that together or you make your own list of questionnaires or questions for the questionnaire, and then you'd you'd make the high, medium, and low risk vendor versions of it and get that out, and it would come back. And you'd be spending hours and hours, even days going through these, and then you'd have all these vendors you're doing this for. And it got to the point where you're like, how can we sustain this? This is a lot of work. We have to hire a lot of people in order to review all this stuff, and some companies even still would do that once they got a SOC two audit Mhmm. Attestation report
Justin LeaplineYeah.
Rick YocumOr saw an ISO report. But you'd get the reports and you would say, well, how many of my questions can I not ask now because I can rely on this report? Right. And so if you start thinking about, oh, that's that's so good. It's gonna make my life so much easier. Can I get the report? And then you get the report, and then you start giving it to somebody who would say, hey, do you got a SOC two report? Yes. Alright. Checkbox. Did you did you did you read it, analyst? Right. Oh, we have to read those.
Justin LeaplineYeah.
Rick YocumYeah. Yeah. You gotta read those. You need to know what's inside of them. You need to know what the controls are. Do the controls in the application that it's even referring to? Is that the one you're buying?
Joe WynnYeah. Is the scope even aligned?
Justin LeaplineYeah. Exactly. Yeah.
Rick YocumYeah. And then my favorite part is skimming through and reading through the control and the testing at the end to see were there any exceptions, and then going and following up on those. So all of a sudden, you had your own list of questions, and then you also got the SOC two report that you also had to read, and then you went through and you had a follow-up on the exceptions. Now, I just created more work for my analysts because we weren't sure that we could stop asking the questions in the questionnaire if we could get the vendor, if they were you know, we were important enough customer that they would fill out my questions and pay and get us the SOC two report. At the end of the day, who is the actual customer of these SOC two reports?
Just the person who's not paying for it. It's the person who's receiving it Mhmm. Because they wanna val vet that the Yeah. Service provider is actually strong enough to to support them. Right. And so it got all the way through that and now we have these reports and we're up to what we do now in the industry Right. Which is what the So you thousand dollars report.
Justin LeaplineOr worse than it was?
Rick YocumOh, think it's worse.
Joe WynnIt's way worse. It's way worse. Well, because it's It can be way worse. Yeah. And it's it's the illusion.
Justin LeaplineJust be careful. We're attacking Matt's all, you know, service line here.
Matthew J. SchiavoneI can defend it. I, you know, I hear it all the time. It's all over LinkedIn, SOC two criticism Yeah. Yeah. Poor quality, lacking framework and security.
Justin LeaplineBut I don't think it's necessarily the the framework that's bad. It's the enforcement of the quality.
Joe WynnRight. Because when it's bad, it's the illusion of security or the illusion of compliance. And it'll trick people that might not be savvy enough or you're an entrepreneur that has no idea about any of this stuff, and you're just trying to figure out who to rely on. Right. And you do you ask generative AI, what should I look for in a vendor? And it says, oh, make sure they have a SOC report, and you do that. Right. But you don't know, you know, how to dig into that necessarily. So I think the illusion of it, into your point, that comes from potential issues with quality, right, or the lack of enforcement Right. Of quality.
Rick YocumYeah. Let's maybe we talk a little bit about unless it's coming later, about how do SOC two auditors how are they reviewed, and what's that peer review process like?
Matthew J. SchiavoneYeah. So the peer review process, CPA firms have to undergo peer review once every three years. Okay. And SOC two, SOC one is considered an a high risk area. So peer reviewers love to drill in. And, basically, when peer review time comes, you give the peer reviewer a list of all your SOC two engagements over the past three years. How does that
Justin Leaplinework exactly? So AICPA pairs you up with a peer or you seek out another peer?
Matthew J. SchiavoneSeek out your own.
Joe WynnOkay. Oh, I didn't
Justin LeaplineI know that. Then you have to basically do an engagement to say, here's my total population, and they do a sampling, and it'll cost, I don't know, $20, whatever the the Depending on the sample, how many binders they have to review. Gotcha. So they you engage somebody to audit you, and it's tracked by AICPA. They do they submit it back to AICPA when they're done and you or you?
Matthew J. SchiavoneAnd now there is well, we'll probably get to this, but the AICPA has an advanced oversight committee for SOC two because of some some of these problems.
Justin LeaplineBut who submits that? Is it you that submits that? Or Peer review. The peer review submits it directly to AICPA.
Matthew J. SchiavoneAnd then on the AICPA website, you can go see letters of acceptance or Gotcha. Yeah.
Justin LeaplineYeah. Then they hand it back down to you and say, pass fail findings, whatever. Here's your remediation plan if necessary, all that stuff. And one thing I was reading with the Delve, you have the option to make that public as the person that got reviewed. Like, they have a website that shows reviews, but you don't necessarily have to
Joe WynnSo the reviewee can publish their the results of their peer review? Yes. But they don't have to from what I read.
Matthew J. SchiavoneDon't think they have to post the letter of acceptance. I think the results pass or fail have to be and I should I should know this a little bit better, but I think the results are automatically posted, but the letters are optional.
Justin LeaplineOh, okay.
Joe WynnAnd the letters have,
Justin Leaplinelike, some additional detail about the whys around passing or failing? The management like, the management response.
Matthew J. SchiavoneSo if a CPA firm underwent peer review, if they post one of their letters of acceptance, letters review, it'll, like, it'll say SOC two or SOC one was included in this peer review. Right.
Justin LeaplineGot it. Okay. And then they'll have three months to fix and all that stuff and everything. Right? Yeah. Candidly, I don't know. I've never So
Joe WynnI was this is actually related. Again, don't wanna jump too far ahead, but I know the the CPA involved in this had just failed an AI CPA period.
Justin LeaplineYes. In January.
Joe WynnIn January. Yep.
Justin LeaplineAnd so I You can actually go look at that on the portal.
Joe WynnYeah. So I was curious about and and maybe you know or maybe I we need to look it up. But I was curious, like, what happens when you fail? Because I was surprised that they failed in January, but then continued to issue reports.
Matthew J. SchiavoneI didn't think you were able.
Joe WynnThat would have been my assumption.
Matthew J. SchiavoneYeah.
Justin LeaplineYeah. So I got that details for Delve specifically. So they went through a peer review in January of finalize. Jason F. Claus PC was the one that did it. And quote unquote That was a peer reviewer? That was a peer reviewer. Yeah. Found, quote, unquote, pervasive deficiencies across all engagements reviewed. No isolated incidents. And some of the specific findings, there's no serviceable assurance that SOC engagements were planned and performed in accordance with professional standards. There's actually a quote out there. Controls were not suitably designed, appropriately tested, or supported by sufficient evidence. Engagement teams couldn't demonstrate understanding of the systems. They were supposedly examining independence violations.
Threats to independence were not identified, evaluated, or documented. Oh, yeah. Monitoring process failed to catch any of these before the reports were issued, and deficiencies were described as pervasive. Yeah. I've already said that. The corrective action by 05/31/2026. So essentially, what is that? Four months? You know? Four or five months? You know? Depending on where you count. All staff must complete sixteen hours of SOC CPE training. Preissurance review required for their next SOC one and SOC two engagements. Must remediate all nonconforming engagements.
Must submit evidence of remediation to an independent reviewer. That's what the
Rick YocumAnd just to clarify, that wasn't Delve that was being, audited here. That was a different auditor. Right?
Justin LeaplineNo. It it was against Delve, but the this is the auditor that audited Delve's engagement.
Joe WynnThe peer reviewer was reviewing Delve's program, and those were their findings.
Matthew J. SchiavoneLike Delves auditors.
Rick YocumIt was Delves auditor, not Delves themselves. Delves was working with
Justin LeaplineOh, yeah. Yeah. Yeah.
Rick YocumYeah. Just for clarification. Yeah. I see. Delves isn't a CPA firm, but the CPA firm that was issuing these were peer reviewed by the by the person you mentioned.
Justin LeaplineAnd have you heard of the so the order was a corp? C c a c corp. Yeah. A c That was o r p.
Matthew J. SchiavoneOne that had the adverse finding.
Rick YocumYes.
Joe WynnYeah. That was the the firm.
Justin LeaplineHave you heard of them before? I haven't heard. No. So, yeah, they're the one that I forget where they were located. I thought it was, like, Pakistan or something like that.
Joe WynnYou know? I dug into when I was looking at the numbers, I dug into the the the note about independence a little bit. Yeah. Yeah. And I looked up AC Corp, and so they they claim, you know, around 700 ish clients. Mhmm. And they were I used the vast majority of the, like, 485 leaked reports they were the stamp on. Okay. And so and it was an eighteen month period, so, like, not a ton of those are gonna be repeats necessarily. So if you have 700 plus clients, right, and you could practically speaking probably three to three fifty of those as a lowball Yeah.
Is all through Delve as one source of revenue.
Justin LeaplineRight.
Joe WynnYeah. You're probably not terribly independent. That's a ton of concentration risk.
Matthew J. SchiavoneYeah. I would be very curious to see how many partners signing these reports A Corp had because if you figure 700 reports in what? An eighteen month period?
Joe WynnYeah. I well, 700 clients, it was their total claim, but it was, like, 500 ish reports in an eighteen month period, and they said they have, like, 55 staff. Do the math. Oh, yeah.
Justin LeaplineDo the math. Do the hours.
Matthew J. SchiavoneI mean, 700 reports for one or two partners. I mean, it takes more than a few partners to actually review the engagements Mhmm. You know, sign the reports, with credibility. Yeah. So I I don't know how many partners signing the reports they had, but 55 staff. That's
Joe WynnYeah. That's a bit suspicious Yeah. To to rock through that number. Yeah.
Justin LeaplineYeah. So I mean and, you know, obviously, Dell dealt with this. It's more of a confirmation, I would say, in the industry, you know, that way you had that. We know there there've been the sock mills essentially, you know, just churning out sock for the stamp of it. But, like, what do you guys think, you know, like, there are a number of other companies that do it this way. Oh, yeah. You know? I can count, you know, on one, maybe two hands and to varying degrees and everything. Like, there are, like one of those famous that Troy Fine often beats up on is, like, Comp dot ai. You know?
They're the ones that, like, saw two in a week, you know, all the claims and everything. But then you get the more automation platforms like Vanta Andrata, and it was actually interesting. The original article called out that there are public declarations of, like, anti competitiveness and, like, fair practice that, like, Vanta and Drata actually have, which was good. Like, they're not gonna peer up with a an otter and do cuts, you know, with that. Like, you can use our tooling
Joe WynnOh, I see.
Justin LeaplineWhomever you want out of that. The the article actually called out some of those companies to say, like, hey. They they at least have a standard that they're not gonna get kickbacks on
Joe WynnRight.
Justin LeaplineSome of the stuff here.
Rick YocumYeah. That's very important, especially in what we do at CISO when we're looking at compliance automation tooling. Mhmm. We're primarily focused on finding tooling that can truly use AI, use integrations
Justin LeaplineYep.
Rick YocumTo gather evidence because, I mean, what's better than doing screenshots of every endpoint
Justin LeaplineRight.
Rick YocumAnd trying to submit that and then doing that, you know, once a quarter or whatever it is when you can truly automatically at the click of a button or just within seconds, have it going gather all this information and Yeah. It's always there. And so, like, that's the first step, getting the evidence. And then being able to rely on the process that gathers the evidence, which should also be tested as part of a an audit. Mhmm. You know, can you rely on the tool that's gathering the evidence? And then, we always like to have the human in a loop. That's like a big AI thing these days. You know, what's gonna be there to make sure that you're not getting hallucinations or or whatever. And then at that point, you know, we you know, our team, what we wanna do is make sure that somebody is validating.
This evidence makes sense, a true second line of defense role, and then but just rely on automation, rely on tooling to bring it to us, look at it, make sure it's good, tidy it all up, and then have the auditor come in and review the process for gathering it, review the tooling to make sure that it makes sense. And then at that point, make sure that the evidence is actually proving that the design is effective.
Joe WynnSo Yeah. There's a huge difference between, like, using a car to go fast and letting the car drive you. Right? It's just a completely different level of trust in terms of, like, you use tools to go faster on things, but Yeah. How much you trust it matters.
Justin LeaplineYeah. And the level of complexity and
Joe WynnYeah. Absolutely.
Justin LeaplineThat go into that into there. And, yeah, I mean, as a, you know, a creator of a GRC tool, I really love all that, you know, integration and AI. I mean, there is so much waste in the GRC aspect that can be lightened from a load perspective. But they're not they're not perfect because, like, one of the things I've seen commonly used in automation tool and then trying to go for an audit is, do you have the right scope? Like, okay, I included AWS, but I only did one instance, and there's three that should be included in this Yep.
Rick YocumI I've heard about that.
Justin LeaplineYep. Like, those things aren't gonna caught by a tool. It's gonna say, yeah. Everything looks green, you know, type of thing. It's like, well, you only pointed to one thing. So Right. Yeah, it looks green, you know, you know, into that. And that's where, like, you need the human in the loop, you know, into that. And there is validation on, like, some of the automation techniques. Like, yeah, I need to get trust into that's pulling the right evidence. It's looking at the right config. It's looking at the right parameter that it's set. But once you get that comfort level, then you can repeat that thousands of times and say, yeah, it's a valid test.
Joe WynnAnd there's also a huge difference between AI doing an evaluation and AI building code to do a robotic style automation. Right? Like, if AI's building traditional code to evaluate a screenshot Oh. Well, that evaluation is gonna be done pretty consistently, more consistently most likely.
Justin LeaplineSo, yes, in a context. But keep in mind, AI always has to translate into text. So we did some testing in, like, in Oh, yeah. Yeah. And everything. The it it's easy it's better to pull parameters and configs. Way better to pull that than the screenshot. Yeah. So the screen but
Joe Wynnthe screenshot's not the point.
Justin LeaplineLike, the point is basically, like, if you use AI
Joe Wynnto build code and, like, the traditional code is doing the evaluation of the control Mhmm. That's different than just
Justin LeaplineOh, yeah. Yeah.
Joe WynnHaving AI. I mean, you you told me this Yes. A while back, which I love because you were doing some, like, data manipulation. You were like, well, I asked AI to do it, you know, I don't know, a 100,000 rows
Justin Leaplineor whatever. And you're
Joe Wynnlike, it didn't do great. It did okay. And you're like, but then I asked AI to build me Python to do to fix the data. And you're like, and then that worked perfectly. Yeah. Exactly. And that's such a clever thing. And I I think people get stuck, like, in what AI is sometimes Yeah. Because if you if you kinda shift the AI left a little bit and you end up with traditional code that can be QA'd and it's just gonna perform like traditional code consistently, you can use that really effectively to evaluate compliance.
Justin LeaplineAnd and the well, I wouldn't say new, but what a lot of people are leaning towards so the model context protocol, the ACPs. Yeah. It eats up a lot of tokens, and it has to give kind of references when you're referencing AI. A lot of different people are actually moving more to the CLI. So the actual console, it's having AI run console command. So it's running AWS commands Right. To actually pull back the saves. Right. And then just evaluate those configs and everything. It's way more efficient. Yeah. It's actually it's way better. It eats up less tokens. You get a more accurate picture because Yeah. It's pulling it straight.
Joe WynnIt's not gonna hallucinate the inputs. Like yeah.
Justin LeaplineYeah. Yeah, it's actually yeah. There's a big move. I I saw a meme the other day on LinkedIn. It was like, you know, 2001, make a CLI. You know, web web web, you know, 2026, make a CLI. You know? You're right.
Joe WynnBut I think it's an important point that people in this sometimes, like, if you're evaluating vendors that do compliance software y things. Right? It's like, well Mhmm. If you say you use AI, help me understand how you're using it specifically. Mhmm. Like, is it the AI itself that's evaluating my stuff and making a call, or is most of that kinda traditional code, but it's AI enabled in various other ways?
Rick YocumYeah. And and to build on that, when you're getting a tool that's gonna do that stuff, one of the things that like that article that we're talking about on Substack, he he really pointed out some things to think about, which is as you're picking a vendor, as you're picking a tool, how is that tool doing the things that they're saying they're doing? So for instance, if it says it's using AI to generate help you generate or help you review stuff, delve into that. Shouldn't have said delve into. Jump jump into?
Dive into that. Dive into that. I'll I'll try to be better. Dive into that and understand what's happening. If they're saying they do integrations and the claim that was made is that the way, you know, Dell did integrations was actually not the way you would think. It wasn't go enter your creds for a read only account to pull the data. Yeah. It was it's giving you forms to go fill out and you still collect all the information is what he was claiming. I haven't ever used Delve, so I'm not sure.
Justin LeaplineBut He had screenshots in there to Right. Legit, you know. And so, you know I've never used Delve as well, but
Rick YocumAnd if you're gonna go test out a piece of software to see if it's gonna do this stuff, ask the vendor, how exactly do you pull Right. This? What are the tests? Are they automated tests? Show me how these tests work. Mhmm. Can I actually look at the how the tests are are built so I can actually rely on these tests to make sure they're pulling the right stuff? Right. And is that all happening and pulling it back? Yeah. Then that's where the human loop, you gotta have your GRC team check it first, make sure it's accurate before you get audited.
Joe WynnRight. Matt, are you are you at liberty to talk about whether or not you're using AI to I was actually just
Justin Leaplinegonna ask them to.
Matthew J. SchiavoneI mean, who isn't?
Joe WynnYeah. Right. Right.
Matthew J. SchiavoneYeah. So we don't rely on it. It just speeds things up.
Justin LeaplineYeah. Exactly.
Matthew J. SchiavoneWe have evidence collection. We have testing procedures, report generation, mostly for, you know, just proofing, grammar, formatting, things like that. Why waste hours reading? Right. I mean, we read them.
Justin LeaplineYeah. Yeah. No. No.
Joe WynnBut like copy editing is different than content generation.
Justin LeaplineRight. Yeah. Yeah. Yeah, writing up a finding like, here are the facts, here are the evidence, write something up, you know, real quick and then you proofread You know? Yeah. That's one of the things we've talked about AI obviously on the podcast before, and one of the big things is like, okay. Somebody's always responsible. Yes. It's not AI, you know, type of thing. You can't just say AI did it. It's you, you know, at the end of the day. And I think, you know, doing these type of reports, we just, did something the other day. It was a report we generated off and it was like, it was great, you know, type of thing. It needed some modification, but, you know, you take the ownership at the end of the day, whatever it spits out. Right.
Rick YocumWell, speaking of you oh, go ahead. No. No, please. I was gonna say, I was gonna change the topic a little bit to talk go back to how CPA firms and how ISO 27,001 audit firms kinda get their ability to be accredited, so to speak.
Justin LeaplineYeah.
Rick YocumBut were you gonna hit on something relevant to the last topic?
Matthew J. SchiavoneNo. No. I I think that's a great
Rick YocumSo so I so full disclosure
Justin Leaplineabout AI and its practice.
Rick YocumSo so full disclosure, I used AI in order to create a comparison table here. So I'll run through and tell you what it says about the SOC two and then and the ISO one, and let me know if there's any thing that's different. So if you look at these areas, so what does a CPA firm do? Their oversight model is peer review, and the frequency you said this earlier is every three years, and they're audited by other CPA firms. And one of the things that I was unclear about that you clarified was that you get to pick your own auditor, so to speak, or Yeah.
Essentially. And and, you know, professional standards are the way that they're they're being enforced. And then on the ISO side, it's more of an accreditation model, and that comes down from, you know, the what is it? The AICPA? No. That's on the CPA side. On the ISO side, they they're working through an accreditation. They they usually have an an ongoing or annual
Justin LeaplineYeah. Yeah. Yeah.
Rick YocumAnd it's done by the, like, the global organization, and then they take the, like, the ANZ or and the other Accreditation bodies. Yeah. Accreditation bodies. ANAP. And they go through when they are able to certify the local ones, and then those are, you know, get accredited. And then the other thing that they do
Justin LeaplineRewind back. I didn't hear how they did quality assurance though.
Rick YocumOh, I'll get to that. And so Okay. One of the things they do is they actually send auditors on an audit with an actual customer to sit there with the auditor.
Justin LeaplineSo they send another auditor. Yeah. The shadow. Another peer auditing firm. Yep. And who organizes that? Or yeah. Well, the that
Rick Yocumthat's organized by, like, ANAP. And we'll assign the local
Joe WynnSo it's assigned though. It's not you choose.
Rick YocumYeah. Yeah. That's a sense I'm getting of it. It's assigned and frequency. They will yearly, like multiple times throughout the year. Oh, yeah. Yeah. They're getting they're getting that done. It's happening a little bit, you know, throughout the year. They get a and I'll find the name of what they call that audit, but they're they're sitting there with them, and they're basically auditing the auditor real time. And so and then they also audit by getting the reporting that comes out Right. And sampling it Natural. And then checking the
Justin LeaplineSo they're they're with it throughout the entire engagement. So I've done a handful of ISO audits, you know, into the the helping companies. Mhmm. You've done more, Joe. You've done several as well through your practice. Have you ever had somebody shadow you?
Matthew J. SchiavoneI have not. We've just started certification, but
Justin LeaplineOh, okay.
Matthew J. SchiavoneGotcha. Most of you was on internal.
Rick YocumNo. Luckily, none none of my audit customers when we were sitting through audits Yeah.
Justin LeaplineActually had And I've gone through several. I've never had that. So I'm curious. You said annually, which seems very frequent, but I don't know. Yeah.
Rick YocumYeah. It's on that's what I meant by ongoing. It's annual that they get a review and they're recertified. And I think twenty seven zero zero six is part of what the standards are for for doing some of that. And then and then the oh, it'll come to me. I'm trying to remember what the what it's called when they have the auditor audit the auditor Okay. During the actual customer engagement.
Justin LeaplineIt's just interesting to me. I've I've never heard the peer review and never ran into it. I would think I would have run into it at least once
Rick Yocumor Yeah. Some of auditors
Justin Leaplinerun into it. Yeah.
Rick YocumYeah. Some of auditors tell us that and and they tell us the stories about how they're sitting there and, you know, they find something that might be marginal, like, should they call this out as a nonconformity? Right. But he's like, you know, every time that they're in being audited during a live audit,
Joe Wynnyou know, they they have to or they're gonna get totally written up. So I I can tell you, I've seen it once.
Justin LeaplineYou have seen it?
Joe WynnOkay. But it wasn't on the tech side, so I was only, like, loosely related. Oh, okay. So one of the companies I was running security for, was getting a they're a manufacturing company. They were doing a quality audit, like, the 9,000 side.
Justin LeaplineOh, okay.
Joe WynnGotcha. The assessment process generally similar, and I and I remember specifically getting the request. It's like, oh, there's gonna be, like, three more auditors on-site.
Justin LeaplineYeah.
Joe WynnLike, can they have accounts where it's like, oh, this is weird. Like, last year, was only this many.
Justin LeaplineYeah.
Joe WynnAnd Got it. Okay. So, anyway, I've seen it once, although I wasn't, like, really directly. Right.
Justin LeaplineRight. Was more the client's name. Thousand. Yeah. Yeah. Got it. Yeah. It was just okay. So they do perform those and everything.
Matthew J. SchiavoneI think I think those two steps in the process is what should differentiate ISO Right. SOC two. However, we've seen some implications on the ISO certification bodies in the Dell scandal. Yep. But I think having that stronger barrier to entry for certification bodies governed by the accreditation bodies, but then also that ongoing potential ongoing review. You could be audited. The auditor could be audited at any time Mhmm. Not just at the conclusion of the process, the annual or the triannual peer review. So you have stronger barrier to entry, risk of audit at any time, and then, you know, annual renewal or whatever the renewal is.
Joe WynnIt is an interesting timing thing for the Delve one. Right? Because if it's an annual review and Delve and was is
Joe WynnNo. You're right. Because it wasn't the same Yeah. Yeah. It wasn't the same CPA firm, was it? Those two That's
Matthew J. SchiavoneISO ones?
Justin LeaplineNo. It's two different companies. Yeah. So I forget who the ISO one was, but it was a different company.
Joe WynnOkay. Because I would I I was gonna say, I would have expected the annual one, theoretically, the annual QA process checking the ISO stuff to flag issues in in the eighteen months where the reports were from, you know, before the AICPA one would have.
Matthew J. SchiavoneWell, that brings up a good point, Joe. You can check you don't have to check me, but I think peer review has to start eighteen months after your first issued report.
Justin LeaplineThey said that in the thing. Okay.
Matthew J. SchiavoneSo a CPA firm could theoretically operate for eighteen months issuing low quality reports before peer review even is, you know
Rick YocumAnd you can get hundreds of reports out in that time.
Joe Wynn485 or
Justin Leaplineat least 500 and Just need 55 people.
Joe WynnRight. Interesting.
Justin LeaplineYeah. So, I mean, from a fixing perspective, is there something broken that needs to be reworked from AICPA and or ISO governance body?
Joe WynnI mean, if you have pervasive issues like, I'm not I'm not saying they were or were not allowed to keep issuing reports because I don't know. But if you have a peer review
Justin Leaplinethat They fix what they they've found, then yes.
Joe WynnBut even that time frame, fix this within, what was it, four months or something? Yeah. If you have pervasive issues, like, you should be forced to to hop immediately in my like, are you gonna keep issuing reports before your stuff's fixed? I mean and again, I'm talking about pervasive systemic I'm not talking about, like, minor deficiencies.
Matthew J. SchiavoneRight.
Joe WynnRight? There's probably some risk here, like, some risk rationale we can apply here. But if you're like, 82 of these reports that went out weren't actually about the client's system. Yeah. They were about Dell's system. Yes. Like, that's bonkers. And and frankly, that's the type of thing also where I'm like, did the clients not recognize it? Or did the did the client's customers Nobody's reading it. Nobody in the whole
Justin Leaplinechain knew this? Right. The reports go out, nobody's reading it. What did
Rick Yocumthey say? A dentist office was audited or something like that? Dental was audited, but or their their report description.
Justin LeaplineWhy is it dental office being a SOC two?
Rick YocumYeah. Or or is it dental dental software maybe?
Justin LeaplineOh, okay. That
Rick Yocummakes more sense. And it was audited, and their system description, you know, was actually Delvs.
Joe WynnYeah. 82 of them. 82 of, like, the 480 were that were that. So that's pretty egregious. Like It was glo glockcert? Yeah. It's l l
Justin Leaplineo c e r t was their ISO 27,000.
Joe WynnSo Okay. Got
Matthew J. Schiavoneit. That was a certification body?
Justin LeaplineThe yeah. That's the company they used to issue out the ISO. Yeah.
Joe WynnNow I actually don't It's
Justin Leaplinea Indian cert mill according to the article.
Joe WynnOkay. I was gonna say, I don't have the detail on this. So I know I know the SOC reports had some egregious issues. Did the ISO reports also have egregious issues? Do we know? Or was it just they were implicated in the the Google Drive?
Justin LeaplineYeah. So they're implicated. They they weren't as bad. I'm looking at the article right now.
Rick YocumThe the article didn't really delve into, dive into the the ISO reports itself. And yeah. And all of the strangest that they found that was repeated in the SOC two reports. But they went into looking at the criteria of and who was the auditor for the ISO certification and where are they based. And, I and I if I recall, they all kinda had this model where they had a US Shell company name.
They were actually owned and run by an operating out of maybe India. And those were they were very similar in nature. Yeah. And I think there might even even have been some tied together in people who were highly owners or high up individuals between both the CPA firm and the ISO shop.
Justin LeaplineSure. Sure.
Rick YocumSo it and as they started to pull these pieces together, they were making connections between them all Mhmm. Which now you have your little ecosystem.
Joe WynnYeah. Yeah. I mean, so one, if if there are pervasive issues of that nature, it feels like you should be forced to stop, like, as opposed to keep going. Mhmm. Two, I it's just interesting to me that you could go 18 I don't know if this is necessarily wrong, but it does feel broken, and I don't know what the fix is. Yeah. Like, going eighteen months with no real review on something like that feels kinda weird. And then also, I don't really know what prevents these parties from, yeah, you know what? You're right. We'll close-up shop, and we'll just start up elsewhere.
Justin LeaplineHonestly, I think I don't know how effective it would be, but there should be consumer reporting, you know, into that. Like, if I get a SOC two and let's say, you know, they charge me $5 for it. Right. And then I get it. Maybe I'll be happy with it, but maybe it's like they misspelled stuff. They did all that stuff. Like, I should be able to report them and be like, this report is kinda garbage. You know? Now Well, now your SOC reports $6, but they have a
Joe Wynnwhole bunch of bots
Justin LeaplineRight.
Joe WynnMessing with the numbers.
Justin LeaplineRight? You know, at least from that perspective, like, it it should you should be able to at least report and then maybe get a little bit more like, hey. We need to audit you now because we're seeing this, you know, a a valid, you know, not just any public comment, but
Joe WynnRight.
Justin LeaplineYou know, a customer that came and it's like, hey. They they basically described Dell's environment, not ours. You know? Yeah. Right. Type of thing. Like, those type of things is like, okay. There's a quality issue here. We need to address it sooner rather than later or axe it, you know, type of thing. Yeah. Because they're in charge of maintaining reputation. Absolutely. You know? So there's if AICPA doesn't really do anything, I I don't think they can continue to do what they're doing today. They have to get a little bit more strict and tighten up some of those stuff, because they're getting a bad rep.
And if you get a bad rep, they're either gonna go away or something's gonna replace them or both.
Joe WynnYeah. The trust erodes. Right?
Justin LeaplineYeah. Exactly. And I think you even said you started to see some clients come back to you because of this issue. Is that right? That is correct. Yeah.
Matthew J. SchiavoneYeah.
Justin LeaplineSo so obviously, like, clients don't like, the clients want the SOC two, but they don't want the bad rep from the bad companies also. You know? Like so they don't want that noise as well, you know, into it.
Matthew J. SchiavoneAnd it's as much as you can or you can try you just can't educate a client on why they shouldn't go with somebody who costs less. Mhmm. Well It just
Joe Wynnif you're selling the service, you're not in the right chair to educate Exactly. Like, I mean, you can try but
Matthew J. SchiavoneWe can we can Yeah. You know, talk about independence and then, you know, why we can't do this and they shouldn't be able to do that and, you know, quality and it just doesn't register until something like this happens. So I I think this is going to be beneficial to, you know, SOC and ISO and and the credibility, we'll get there. And we look to other frameworks, look at CMMC and HITRUST. There's additional steps along those processes to issue these reports, you know, undergo those engagements, barriers to entry. So I'm hopeful to see AIC payable figure it out.
Justin LeaplineYeah. Yeah. Oh, by the it's We
Rick Yocumshall see. It's called a witness audit. Oh, that that's what happens when the auditor comes and audits the auditor. And I think every you know, from what I'm reading is every every auditor so if you have an ISO shop and you have lots of auditors, each one of those auditors might get a witness audit every, like, two to four years. And and so if I'm thinking about this the right way, then one certification organization will have multiple witness audits because and that they'll be spaced out.
So you might have a couple a year, but not the same auditor. And so and so if you and then if you think about the way it works is you're getting an annual audit. You're one week maybe of of the fifty two weeks a year that that's happening for that auditor. Maybe they're not do maybe they're doing, like, 40 a year or something. And as they do those, you know, you you might just get lucky. So you won't be in the one that has witness audit.
Matthew J. SchiavoneRight. And that's enforced by the accreditation body.
Rick YocumIt is. You know, and
Justin Leaplineyou But I'm curious who pays for that audit. I think the your fees to the
Matthew J. Schiavoneaccreditation body probably fund those
Justin LeaplineYou think? Activities. Okay. I mean, because those auditors aren't going for free out there.
Matthew J. SchiavoneRight? Yeah. I I don't know for certain, but I I I would imagine.
Justin LeaplineAnd in your case well, in the AICPA case, you're funding that directly for the peer review We are. You know, that kind
Rick Yocumof thing. The certification body pays for the witness audits part of
Justin Leaplinethe I
Joe Wynnwas gonna say, makes sense because again, you wouldn't want the funding source to potentially muddy. Yeah. Exactly.
Justin LeaplineYeah. Or be gouged or something like that, like, if it's not your choice and everything. Has so you gotta pick your own peer review audit. Has or Wait.
Matthew J. SchiavoneI'm sorry. So just to confirm, the certification body pays for the witness audit?
Justin LeaplineHe just had?
Matthew J. SchiavoneYeah. But they can occur at any time based on the accreditation bodies.
Joe WynnYeah. Decision. I'm sure there's, like, any time within, you know, not to exceed this frequency.
Rick YocumYeah. I I think I kinda get I'm
Matthew J. Schiavonea little fuzzy on that process. It was difficult. Okay.
Justin LeaplineYeah. But, like, has there ever been a scandal before where, like, you know, two CPA companies kinda do the handshake? Like, I'll do your peer review, you do mine, and we'll just sweep everything under the rug. Has that ever, like, come up before?
Matthew J. SchiavoneThat I know of? No.
Justin LeaplineNo. Okay. I'm sure. Because you look at that, you're like, well, why not? You know, like, you could see it happening. Yeah. You know?
Joe WynnIt's all a spectrum too. Right? Like, to to what extent is it, well, I know you and I trust you and you know me and you trust me and we can cut each other like friends and family rates versus like, hey, you're just gonna say yes. Right? Because I'll just say yes to you if you just say yes to me. Like, it's a totally different
Justin LeaplineIt is. You know, type of thing. But
Matthew J. SchiavoneI think something reciprocal where you do mine, I do yours is probably, you know, there's independence considerations there that
Joe WynnI would hope that there were prevents lags.
Matthew J. SchiavoneBut I there are certainly, I'm sure sure handshake deals that, hey, you know. Because if you look at some of these peer reviewers who are reviewing SOC two firms and you go to the peer reviewers website, they don't even have SOC two as a service of theirs.
Justin LeaplineSo there Really?
Matthew J. SchiavoneSo there are I find that in but there's nothing that
Justin LeaplineTheir service is peer reviews?
Matthew J. SchiavoneI what I can't do is prove that they haven't hired an outside practitioner to help them with peer review of those applicable SOC two reports. So I can't make that claim. I'm just saying there are instances I found where CPA firms conducting peer review with SOC two don't have SOC two on their website.
Justin LeaplineThat's interesting. Yeah.
Rick YocumSo they may not be trained in what all the pieces are.
Matthew J. SchiavoneBut again, they could hire outside consultants.
Rick YocumSomebody they might have to go and get somebody with competency to competency to do that.
Joe WynnSubtracted out. Subcontracted out to the same firm you're reviewing. Hey. Can you review your own
Justin LeaplineYeah. Right.
Joe WynnProgram through me? Yeah. Same person that did the report. I'm not suggesting anyone that has done that.
Rick YocumYeah. Now is there a place in the audit where you needed to clear any potential independence issues?
Matthew J. SchiavoneVery early. I mean, we yeah. We have a new engagement acceptance procedure where we consider other services we have performs or with the clients or are engaged currently to perform. We have to document that all before signing the SOW, sending out the engagement letters, then throughout the process of the audit, we've Gotcha. Document.
Joe WynnIs any of that documented in the report itself, or is it just part of your, like, internal risk practices that you must do these things?
Matthew J. SchiavoneIn the internal risk practices, in the audit binders and work papers, we, as a firm, we include SOWs if we are providing Oh, I got it. Other services, for clarity for the peer review. And in the management rep letter, which no one sees, the client signs a rep letter and a rep that they took responsibility for all non attest services that they
Justin Leaplinemay have performed. Yeah. So is there anything that's, like, really preventive? Like, if you did a pen test for them and coming in and do the SOC two, even though that's part of your evidence, like, that wouldn't prevent you from doing that. Right?
Matthew J. SchiavoneWe have to be able to demonstrate just our firm's standards and policies that they have appropriate subject matter expertise to evaluate the findings. They have people on staff who essentially understand. No. No. No.
Justin LeaplineI'm talking about if your company did the pen test and now you're coming in to do a SOC two, you know, into that, that independence wouldn't be a conflict or would it be? It does
Matthew J. SchiavoneIt could be.
Justin LeaplineIt could be?
Matthew J. SchiavoneIt could be.
Justin LeaplineYeah. So
Joe WynnBut you'd have a you'd you'd have to disclose it and a partner would potentially make a call one way or the other as to whether or not you're gonna do that work.
Matthew J. SchiavoneAnd we have to document. Yeah. If we do it, we have to document why we're comfortable.
Joe WynnYeah. Why here's why it's okay.
Justin LeaplineRight. Yeah. Yeah. I'm just curious. Like, a lot of times, at least in the PCI realm, they just they have a spot for independence, but we usually just say it's another team. We're totally separate. We don't report, you know, nothing's going through them, you know, type of thing. Like, I have no say on their findings. They have no say on my findings. You know?
Matthew J. SchiavoneThat yeah. That that doesn't matter to us. Yeah. Because, I mean, I would never conduct a pen test. So, it's gonna be performed by a separate team.
Justin LeaplineYeah. That's what I'm saying, like Yeah. Into that, like, that that there is a kind of internal independence, you know, type of thing. Like, even though truth be told, you could have a partner maybe muscle, you know, like, give them a clean bill of health. I mean, it
Joe WynnIt totally depends on structure and everything. Yeah. Like, because, I mean, all the firms
Justin Leaplineare set up differently.
Joe WynnBut you you could very easily say, oh, yeah. This partner is responsible for this type of technical testing. This other partner Right. Is responsible for this other thing. They're fully independent for these reasons. Here's how we do peer reviews. Here's how we
Justin Leaplinedo other cross checks. Like
Matthew J. SchiavoneI I would also make the argument though of as an auditor, I wouldn't I wouldn't care if the pen test necessarily came back riddled with findings as an auditor. I mean, I guess I would. I should probably shouldn't have said that.
Justin LeaplineI I guess it depends on the remediation, you know, type
Rick Yocumof Yeah.
Matthew J. SchiavoneThat's what I'm there's additional steps in the process that I should care about as the auditor. But Right. I would
Rick Yocumexpect you'd be interested in, hey, was that pen test a quality pen test that somebody that it really exposed the risk that this company has. And if it did uncover flaws and vulnerabilities, are why are those there? And what is the process that gets those remediated? But also, what's the process that fixes the underlying cause?
Justin LeaplineAre they not doing change management well, patch management well? Like, it could show
Rick YocumBaseline standards.
Justin LeaplineYeah. Exactly. You know? Or did
Joe Wynnyou just or did
Justin Leaplineyou just buy the software from someone else and it's your first one and this is the process where Yeah. This is your first pen test, then there you're gonna have findings because you're not aware of maybe some of the Yeah. Configuration baselines you needed, you know, type of thing. Yeah. There could be a lot of reasons why. No. It's a great point. Yeah. Yeah. But, that should be separate from the SOC two audit versus the pen test. I mean, they're they're different skills. You know?
Matthew J. SchiavoneWhat I meant by that is Yeah. There isn't an there shouldn't even if it was perceived as an independence impairment, it I don't see how it could be our pen testers would be incentivized to give them a clean report due to my audit.
Justin LeaplineYeah. Oh, yeah. Yeah. Right.
Joe WynnRight. And I'm sure there's some threshold too where it's like, oh, we're doing VC so, and we're doing the pen test, test and and and and and. There's eventually one too many ands and you go, well, let's not do this next thing.
Matthew J. SchiavoneYeah. Like VC so we wouldn't even
Justin LeaplineRight. Yeah.
Rick YocumYeah. That's that's because that's more on advice. That's more on getting configured the right way Mhmm. Build. And so, you know, one of the things you because you
Justin Leaplinecould be control design Right. Even just Exactly. Control. Right. Yeah.
Rick YocumRight. Yeah. So, I mean, one of the things Matt and I had talked a lot about over the years is, you know, what's what's the separation? And there were times when it might have been easy from a contract standpoint for me to bring, like, Matt's company in to get it done. And as that kind of stuff unfolds, it's like, no, we really need to have our contract signed independently by the customer. That way, there's independence just because I know Matt and I know he does quality work Mhmm. That becomes the the the most beside besides me vouching for him, that becomes the most, potential independence issue that there is. It's like Right. I I'm fully independent for Matt because I I know him for many years now. So when I suggest that he's a good auditor, his reports are gonna come out, they're gonna be something you can rely on, and in fact, this is something we talked about when we were presenting to the we did the fireside chat with the, you know, Isaaca.
Isaaca. Yep. Is, you know, when when you're getting that audit done and you really wanna be able to look at that report, go to your customers and say, yeah, our controls are good. And you can believe this because look at the quality of the auditor Right. That I have here doing it. And when that all happens, you know, then then you're gonna have something you can rely on. And But what if your customers don't care? Well, I guess it depends on you're you're probably not selling up market enough yet. Yeah. Someday, when you start selling to more sophisticated customers, they that are processing that, you know, higher level regulated data Yeah.
They're they're probably gonna care.
Justin LeaplineSo I I think I told this story before. I might have mentioned it in a second. So when I first went in to head up the GRC practice at Diebold, a big fortune services company, Fortune 500 into that. I came into the role. We just did, at that point, our first SOC two, and we're finishing it up. Mhmm. And it was within my first week or two. We hop on the phone with the CPA company. They're like, you got some qualified opinions into it. And for those who don't know, qualified opinions are we didn't meet the control fully, and now you need to do a management response and all that stuff and everything.
And it was so we did the manager's response. We, you know, stamped the the report, you know, got it all done, and it was up to me. It was like, alright. Do we hurry up and do it like a three month testing time window, clean this up, and get a clean report, you know, which is the minimum time to get, you know, clean? Do we hold off until next year? Like, it was solely on my shoulders on what to do. And I decided after thinking a little bit, I was like, I'm just gonna send it out. I'm gonna send it out as is. In a year, we'll clean it up, and we'll see how it goes, you know, type of thing. So we'd send it out with the bridge letters, all that stuff.
It took almost ten months for a customer to come back and be like, you had some deficiencies into this. I was like, yeah. We cleaned it up, and they're like, okay. Yeah. I mean, it's like, I got zero stuff back, like Yeah.
Rick YocumHow many customers did it go to and you got one? Hundreds.
Justin LeaplineHundreds. Hundreds.
Rick YocumSo, yeah, so hundreds either looked at it and said, oh, well, these deficiencies don't matter to me, or more likely, nobody read the report.
Joe WynnRight. Well, there's something interesting here too, because I can tell you, if you're at an organization that has an issue, right, whether it's a breach or even it's Mhmm. Something where so, like, rewind a bunch of years. I'm at a company called Black Box, and sometimes we would we would have this, like, sort of DMZ network where clients would send us network equipment that wasn't working, and we fix it. Yeah. Sometimes it was riddled with malware. Right? So we'd get a piece of network kit on a network that we owned. Mhmm. Right?
And it would start posting, trying to fall back and everything. So then the the bit sites and the Upguards and all the the the tools would start to be like, hey. We see weird traffic here. Your network, your score is going down. Right. Right? But I just think it's interesting. So, like, a SOC report finding is bad ten months before someone says something. But I can tell you, as soon as you drop a piece of network kit into the hardware, the security nerds, right, not necessarily the GRC nerds, the security nerds are like, hey. Our threat groups are telling us that you have an issue on your network.
Do you wanna look into that? And that happens, like, fast. Right. That happens, like I'll tell you, like, six or seven years ago, it was happening within a week or two. Now it's happening in a day or two.
Justin LeaplineInteresting. But it's not perfect there. Like, I remember the Oh,
Joe Wynnit's not perfect at all.
Justin LeaplineAWS in there as a vendor, and it's like, oh, they're a d. I I will tell you, well, that's because they're a hosting company and all the bad stuff coming out. I was like, you know.
Joe WynnI am certainly not vouching for those systems.
Justin LeaplineYeah. Right.
Joe WynnBut but I will say it's interesting in terms of the the perspective, and I think it kinda also speaks to how overwhelmed to your point before, Joe, right, in terms of, well, now we have SOC reports to review and questionnaires to review, and I have way more vendors than I ever used to have. And I've been pressured to increase the ratings of what's high and critical based on these other standards and regulations.
Rick YocumYeah. I wanna come back to that after the break.
Justin LeaplineYeah. Yeah. Yeah. Yeah. And talk about solutions.
Joe WynnBut it's just interesting, like, how fast the GRC teams can react to this stuff and versus how fast the the technical security teams, which are a bit typically a bit closer to automation, are reacting to other signals that
Justin Leaplineare coming out. Yeah. Yeah. I guess, you know, to your point of, like, collecting and getting good stuff, I mean, how many times have we run out across customers? They collect the report. They throw it into the folder, and they checkbox the we've done our due diligence for this vendor.
Joe WynnThey have one. Yeah. We received it.
Justin LeaplineI remember, like, I did another story. I was doing a PCI client, and they got I I think it was actually a SaaS 70 back in the day. But they had a data center vendor got the report got the report back, and all the findings and the, like or the controls and the description and the the testing of that was deleted out of the report. It only had the description of the environment into it. And I'm like, guys, where's the rest of the report? They're like, well, this is what they gave us. You know? And the customer didn't know any better. Like, they didn't know there was a whole, like, listing of controls and whether it, like you know, what's the statement like no no not nothing noted or something like that. Yeah.
No exceptions noted or something like that. Yeah. There was none of that in the report. And then so we're like, you know, we reached back out to the data center. We're like, guys, what's up? They're like, that's confidential. It was like, you got a SaaS 70 to say it's confidential. It's like, come on, guys. You gotta find it. Like, we know you gotta find it, you know, into this.
Rick YocumThat's what the NDA is for so you can share the confidential stuff about the controls I'm relying on in my company that you're providing.
Joe WynnAnd that's when they invented a SOC three.
Justin LeaplineYeah. Right. So yeah. And it's just like, you know, we got, you know, a bad education problem on, you know, the on the vendor side, you know, just basically trying to shortchange, you know, the system. And then the customers don't know any better because, again, we talked about the incentive structure of, I just want this big deal, and they're saying I need a SOC two. I don't care the in between. I just wanted the cheapest, fastest, you know, way to get that, you know, type of thing. Well and I've I've seen so just I'd interested
Joe Wynnin if if you can speak at all to the clients that you have seen in your career on this, but I've seen a bunch of organizations where it's the sales team that's sponsoring the SOC report Right. Out of their budget as opposed to, like, the IT team as a for instance. And again, there's some incentives there, right, in terms of how fast you go and who's receiving it and stuff like that.
Justin LeaplineI'm just asking, are we bringing back the trustee certification on the website? Just slap it on your website. They're like, everything's good. Yep. Yep.
Rick YocumYou're gonna have to drink with that comment.
Justin LeaplineYeah. Right. That's what I'm gonna do next. I'm gonna invent the industry, bring back the trusty trusty two. You know? Right. But I'm but I'm just saying, like I'll do it for $4 instead of $5.
Joe WynnBut organizations have pressure internally to get their own SOC reports because they wanna they their sales teams wanna stamp it or they can't engage with their their, you know, downstream clients unless they have the thing. Right? Even if they're not gonna look at it. Right? It's it's a bar they have to clear.
Matthew J. SchiavoneI've definitely seen the push from the sales team. Whether or not I've ever seen it hit their budget, I can't speak to, but it's very interesting. Yeah. They the the pressures that have the report often come from the sales team, but
Justin LeaplineI don't think it would it hit their budget.
Joe WynnIs it? I've seen I I've worked at a it might not be terribly common. Yeah. But I have worked at a place where when I landed there, they were the sponsor of Yeah. The the stock reports and all that stuff.
Rick YocumYeah. Because it helps them close deals.
Joe WynnYeah. Absolutely. It's it's it's yeah. It removes friction. Yeah. And to and to some extent, I think at that place in particular, it was one of the distinctions at the time when I got there. They were doing a lot of what I would call checkbox compliance. Mhmm. It's like, well, who's paying for it? Who cares about it? They're not gonna read the thing. They're just gonna pass it forward. Right? This is the sales team thing. And then we eventually took it over and stuff like that, but I have seen it. Interesting.
Justin LeaplineThat is interesting. But is this a good breaking part? Love it. Yeah.
Joe WynnAlright. That that was definitely enough to drink.
Justin LeaplineYeah. Exactly. Let's refill here. Matt, some more. Please. So Matt was gracious enough. We don't often bring our guests force our guests to bring bottles, but we're we don't condone it as well. You good?
Matthew J. SchiavoneYep. I'll take it up.
Rick YocumThat's good.
Justin LeaplineSo, yeah, Matt brought us ard Ardbeg? Ardberg. It's there's no r.
Joe WynnOr is it Ardbeg? It's a I've pronouncing it wrong for years. Arbeg. Yeah. I've been pronouncing it wrong for years.
Justin LeaplineYeah. Arbeg. It's a five year old scotch. It has a pretty decent smoky taste, but not overpowering Yeah. Into this.
Joe WynnYou Sweet and smoky.
Justin LeaplineYeah. You definitely notice the the smoke onto this. It's the Wee Beasty Mhmm. Single malt scotch, the ultimate, non chilled filtered.
Joe WynnYeah. It's quite good.
Justin LeaplineSorry. Yep. Yeah. It's actually What's the proof? Proof is 47? I don't see it. 47. Yeah. Nice. I think that's it.
Joe WynnWhen if you follow the cocktail subreddit
Justin LeaplineForty seven four. Yeah.
Joe WynnYou will note that it was recently voted on Reddit for whatever it's worth on that subreddit. Like the The what? The the cocktail subreddit on Reddit.
Justin LeaplineOkay.
Joe WynnVoted it. I think most underrated scotch, and then there was a second category at one in as well for like I forget. Most versatile maybe or one of those. They they did a whole Okay. Thing.
Rick YocumI'd just say it was 47% alcohol or 47 proof? Percent alcohol. 94.
Justin LeaplineYeah. 94. Yeah. 94 proof.
Rick YocumYeah.
Matthew J. SchiavoneOkay. Sorry. I
Joe Wynnyeah. Which it's super smooth.
Justin LeaplineYeah. It settles really well.
Rick YocumWas gonna say, wouldn't be surprised if you told me it was 47 proof because I was only 23% because of how smooth it is.
Justin LeaplineYeah. So, yeah, you get a good smoky but not overpowering, you know, into that. And then it settles into a nice, you know, scotch whiskey flavor into that. So yeah.
Rick YocumYeah. Thank you very much
Joe Wynnfor bringing it.
Justin LeaplineThank you.
Joe WynnIt's great.
Justin LeaplineSo cheers, guys. Cheers. The table's so much bigger now. Yeah.
Rick YocumWe'll just have to pass it down. Quick
Speaker 7break to hear from one of our sponsors. If you own security, compliance, or risk, and it feels like you're always pushing a boulder uphill, I want you to know about CISO. CISO helps growing companies get order ready, reduce risk, and stay resilient without drowning in tools, endless checklists, or one time reports that quietly rot the moment the audit ends. This isn't shelfware. It's not drive by consulting. With CISO, you don't just get advice. You get hands on support from real security engineers, GRC specialists, and former CISOs who help you build, operate, and continuously improve your security program over time. Whether you're chasing SOC two, ISO 27,001, CMMC, HIPAA, or you're simply trying to get security under control so the business can move faster, CISO meets you where you are. Their managed VGRC model gives you enterprise level expertise without hiring a full internal team or reinventing the wheel. The focus is simple.
Clear priorities, practical controls, and measurable progress leadership can actually understand. Visit cisodellc.com and start the conversation. Security you can trust, compliance you can prove, and people you can depend on.
Justin LeaplineAlright. Welcome back to Distilled Security Podcast. Before we get into kind of our final topic here, Joe, I believe you had something to
Rick YocumWell, it's it's my monthly reminder about b sides Pittsburgh. It's coming up July 10. Tickets are still $20 until April 14. And being that today is the March, by the time this is out there, you're gonna be really close to that deadline. So get your ticket if you haven't already. Then they go up to $35 until June 12, and then I think it's in the 70 range. Mhmm. And right now, call for speakers closes April 17. So if you're listening to this, it's gonna be really close. Get that submitted.
Justin LeaplineWe'll have it out from the website. Here. Yeah. You know, type of thing. So you'll have a week. You have a week. At least a week. Yeah. Time.
Rick YocumYep. And and and maybe four days until the
Justin Leaplineprices Get on it. Don't delay. Yeah.
Rick YocumAnd then sponsor tables, they're going fast. We're getting it's gonna be a great event this year. We're getting a lot of sponsors to help us make that happen, but we still have tables left. Yep. So please, if you're a sponsor, get your your request in for sponsorship. And if you're one of our long long time attendees and you know some sponsors that you think will be doing a good job of helping us make this thing happen, encourage them. Yeah.
Justin LeaplineYeah.
Rick YocumSo thank you.
Justin LeaplineIt was at the end of last year, you almost had, like, somebody competing to get, like, the the platinum one. I remember at the after party, there were
Rick YocumYeah.
Justin LeaplineVendors are like, no. I'm doing it. No. I'm doing it.
Rick YocumYeah. Yeah. We've we we we've had a couple different new new perks, and I think the highest level out there, it's a little pricey, but it is, you know, well well worth the perks you can get.
Matthew J. SchiavoneYou get
Rick Yocuma And that one's still open. Yeah. And, you know, you really get to show your support to the community. Yeah. Like, all the money for this goes right back into the next b sides, planning this stuff. Everybody, all of us, do the we're volunteering to Wow. To make this happen.
Joe WynnAnd it's really why, like, for such a long time, the tickets are are really good price because if they were at cost, they're a lot closer to that $70
Rick YocumYeah.
Joe WynnOr more. Yeah. For the meal and the stuff you get.
Rick YocumYeah. For the quality of presentations you get all day and the quality of education you'll have and just for the atmosphere and and being able to be part of that, it is, you know, you wish to be charging twice as much as the highest amount.
Joe WynnYeah. But for the community.
Rick YocumYeah. Great.
Justin LeaplineAlright. So to kinda put a bow on all this stuff, we've been doing a lot of complaining, I feel like, in the first hour. Now what? Yeah. So, yeah, why don't we talk about some now what are we doing, you know, with this? What like, is SOC two dead? Are we waiting for changes? Is there something better out there? How do we modify our programs to make this resilient, you know, into issues and bad auditors out there? Like, what are what are some things on the table that we could be doing?
Joe WynnWell, I mean, I I would say, first and foremost, if if you have Dell as a piece of software or
Justin Leaplineor or even if you think not even them. Like, they they're not the only company.
Joe WynnThat's true. Absolutely right.
Justin LeaplineSo that's
Joe Wynnwhat I was gonna say. Or or you know that you're doing a level of checkbox compliance that might come back to haunt you. Right? You probably should think really hard about leveling up to a provider that'll, you know, kinda provide a more complete, you know, SOC review experience or whatever the ad test.
Rick YocumRight. Let's break this down a little bit. So you have you know, there there's a few components. You have the auditor Mhmm. And the auditor is gonna come in, and they're either gonna give you a quality or a non quality audit. And I think you can tell the difference based on, you know, just just under get understanding for how they test and what they do. And then you have and you don't have to use compliance automation tooling or GRC tooling. You can still have that same audit problem with you manually collecting all of your evidence Mhmm. And still having somebody not really pay attention to look at it, or you can then have a more quality audit with a more reputable auditor that you can trust. Right.
And so kinda break that apart. It's not just the software, and it's not just the audit. It's a lot about what are you doing in the middle Right.
Matthew J. SchiavoneAs
Joe Wynnwell. Well, and and, Matt, I mean, if if I was a potential client, right, and I'm coming from a place where I had maybe a not so good provider and I'm looking to move, but maybe I don't have the budget to do, the full on thing, are there certain levers I can pull? Or, like, what would be the the glide path towards moving from maybe a checkbox, you know, compliance place to a place where you're getting a slightly more thorough experience?
Matthew J. SchiavoneThat is that's a good question because, you know, we try to keep our fees competitive, but, you know, we produce perform a quality audit.
Joe WynnThere's gonna be a floor.
Matthew J. SchiavoneSo we try to in those instances, we try our our best to meet the customer at a competitive place and hoping they see the value in that. And Does
Justin Leaplinecontrol infrastructure matter at all? Like, if they're doing automated testing and you're getting all those reports and everything, like, instead of them doing all, like, manual controls or something like that. Like
Matthew J. SchiavoneIt does, but we've gotten burned in those instances because some clients think it's a set it and forget it. Okay. And they set it and then they forget it and we get in there and it leads to more questions than we have answers and then we're bothering the clients more than they anticipated because we were supposed
Justin Leaplineto It's more work. More work on
Matthew J. Schiavonethose. Yeah.
Justin LeaplineYeah. Yeah. Exactly. I don't know what you mean.
Matthew J. SchiavoneAnd a lot of these platforms, auditors advertise a hands free, hands off audit and we can't really operate in that way.
Joe WynnYeah. Nor yeah. That that's kind of the point. Right?
Matthew J. SchiavoneYeah. Right. We do our best to leverage those platforms as they're intended.
Rick YocumYep. But you still have to talk to the customer.
Justin LeaplineYeah. Yeah. And understand the infrastructure. Yeah. And it's not even like like the GRC automation tooling is one thing, but it's also like, what if I set up AWS, you know, security center to, like, monitor all my controls and map it back to a a standard and, you know, now I have real time monitoring for the instances. Hopefully, they're ScopeRight are into that. And now all you need to do is, on the technical side, download all the the stuff into there. You still have to look at change management and all that. But Right. I'm just wondering if anything, like, would affect like, if I'm strapped for cash, could I help put those automation controls into place where it makes your life easy from a testing standpoint?
Matthew J. SchiavoneYeah. Absolutely. Okay. But we you know, like I said, we've gotten burned.
Joe WynnSo we
Matthew J. Schiavonewe we have our floor. If we can if you can demonstrate to us in the, you know, the presale the sales cycle that, hey. These are our controls. They're operating as intended. We we typically don't get that deep because we're not really presented with this problem too often. Gotcha. But during that process, we do review the controls. We review the scope to make sure everything's aligned. Yep. We like to hear what went right with past audits, what went wrong, how can we improve. And if one of their, you know, things they liked is the auditors never talked to them, then then that's a that's a
Justin Leaplinered flag But, for
Matthew J. Schiavoneyeah, automation is helping, but we we just find ourselves in these instances where the clients have that platform and they're just not maintaining it. Right. Not And I get that. Yeah.
Justin LeaplineSo, like, maybe, you know, whatever it is in your contracts, like, year one, if you're working with a new client, that's a high risk, you know, because you're unfamiliar with the environment. Maybe year two and three, like, once you get familiar with it and, like, okay, they have a pretty mature environment we can rely on, maybe, you know, type of thing. Right. Give them some discounts or I don't know.
Matthew J. SchiavoneYeah. No. And and it helps for us when we're view reviewing the prior year report. You know, we understand the marketplace. Yeah. Yeah. We know who's issuing these reports. We know the price point. We know their tolerance for, you know, evidence
Justin LeaplineYeah.
Matthew J. SchiavoneAnd timeliness. So, you know, reviewing prior year reports helps us certainly.
Justin LeaplineYeah. Gotcha.
Joe WynnWell and I would imagine, you know, maybe, maybe not sometimes, but sometimes there might be scope levers you could pull. Right? If maybe you're getting a broad checkbox compliance thing, yeah, maybe you don't get certification of the entire environment or whatever. But, you know, maybe you only need this one thing. Yeah. And and you could do that a little quicker.
Justin LeaplineJust looks at their SaaS app, and that's all they care about.
Joe WynnYeah. Could probably do some I I would guess that. But also, don't think it's a bad answer to say, yeah, no. Sometimes, you know, you just people are gonna if you want humans to spend time reviewing the stuff, it's more of an education thing to say, well, a $5,000 sock report doesn't buy you compliance, it buys you liability. And if you wanna actually, you know, have the thing that you can stand by, well, you're gonna need to spend a little more money. I don't think that's a bad message necessarily, but to your point earlier, it's an education thing upstream.
Matthew J. SchiavoneSometimes they learn the hard way.
Justin LeaplineYeah.
Rick YocumYeah. I would imagine that when you start looking at all the effort it takes, all you're doing in some cases is trading effort from one area to another. And so your timing isn't really going down. So if you didn't invest in the, you know, whatever the cost is for your, compliance automation tool and you're doing everything manually, well, now you're spending hours of people's time going and gathering all the evidence, collecting it, putting it into a place so that your evidence repository, and those people could have been doing something else. So what do you do to fix that problem?
You go and you get the compliance automation tool. You go and get the GRC tool. Mhmm. You move off of spreadsheets and more docs into a system. And so what you're doing is trading, you know, hours of a person's time doing something for a system. Now, that system needs to have somebody configure the integrations. And when the integration breaks and it stops working, you need to go and work with whoever it was that was authorized. So you interrupt somebody else. They have the the person who has the ability to go and reconnect that service back to the, you know, the platform that it's gonna Mhmm. That's being monitored.
And then, you know, so that's a time. So Yeah. Really is like time trade offs. If you look at the whole cycle, you're just moving things to areas, and you try to get it working as efficiently as possible. So in a best case scenario, you know, you do away with spreadsheets, and now you have it on a system. And then you get that system to be able to create some automation so it's repeatable. And then you need to be able to do your own validation that that stuff keeps being reliable. Yeah. And then that might mean that well, now you're gonna go to the auditor. Well, the auditor sure, you are be able to feed the auditor all this stuff more quickly, maybe in a little bit more reliable way.
And I'm thinking about it, Matt, you you might have a different way of thinking about this. But as you're auditing, you have a number of hours you have to do. If all of a sudden, you were given access to a system that said, here's, you know, here's a thousand points in time that this control was operating effectively because it was automated. Mhmm. Now, you're gonna look at that and see Yeah. You know, now now you have some testing to do, like, well, you know, how is that good? And we covered set in the earlier half. And so now, your testing is you're still spending time testing something. I don't see where you're shrink where you're shrinking down the testing period. You're just shifting what you're looking at in order to say that's reliable as you go on to do something else.
What are you finding?
Matthew J. SchiavoneI think that brings up a good point with comparing ISO audits to SOC two audits where in ISO you have a required number of hours, man days. SOC two, there's no required amount of hours. So these firms, these a corp can just turn out these reports. And I mean, they were doing ISO certifications as well, so probably fabricating those time sheets and everything. But Joe The
Justin Leaplinecorp wasn't. Potentially, the other company was. Sorry. Was it the gen whatever? Yeah. Yeah. Gen Whatever. Yeah. Gen. Yeah. Yeah. Allegedly. Well, allegedly. Allegedly.
Matthew J. SchiavoneBut, Joe, to your point in a SOC two audit, we have to examine every control. And if it's you know, we're testing operating effectiveness, we have to sample based on the guidance. So sorry. I kinda lost my train of thought there.
Justin LeaplineDo you make up your own sampling methodology?
Matthew J. SchiavoneWe have PBC work papers Yeah. Thompson Reuters' checkpoint. But you document your own methodology and it into there. We justify if we deviate from their confidence levels. So 95% confidence level, 90% confidence level based on the population. Yep. Justify based on automated controls, level of risk. We can justify, but we have to support that, and that is one thing that peer review drills into.
Joe WynnWell, when you start to validate, I assume, the accuracy and completeness of the evidence being provided to you.
Matthew J. SchiavoneOf the population.
Justin LeaplineYeah. Yeah. Yeah. Yeah. Yeah. Yeah. So exact yeah. If control is highly automated, you got a thousand things, maybe you're only testing one, two, three, you know, of it knowing the reliance of it, you know, full Yeah. You have to test the process more. Yeah. Exactly. Yeah. It might get in reliable.
Rick YocumAnd that was my point. You're you went from just examining the evidence that used to be a screenshot that somebody would do and you might say, well, I have to do my walk through. Show me how you did that screenshot. Show me how you log in and and get that. Yeah. And Oh, I use that a common test you would do?
Matthew J. SchiavoneYes. And that brings up a great point with automated pipelines and software development. You know, legacy SOC two, we were looking at change tickets and making sure the process is tied out. Now we're a little bit more reliant on the processes and the controls throughout that process that might not be as legacy as a change Right. In
Justin LeaplineYeah. Good. Yeah. You got pipelines with rules around it. So you're testing those rules are enforced. When were they override you know, overridden, all that stuff and everything. Yeah. Yeah.
Joe WynnBut to your point, like, if it if it is like, oh, yeah. We use Vanta, Andrata or whatever for, like Ordrata. For, like, 80% of all these controls, it's like, okay. Well, now we need to test the access control to that system, and we need to test the change management of that system. So to your point, Joe, yeah, you're I mean, it's another shift left. It might help, but it just depends on how much it scales based on what it's displacing.
Rick YocumYeah. If you look at the total ROI and the economic value of it
Justin LeaplineYeah.
Rick YocumAnd you look at everybody's time involved and you look at all the costs for all those pieces of time Yeah. I'm not really seeing how one, you know, one level of effort is different than another.
Joe WynnIt's like
Rick Yocumit's like in physics. So if you're walking up a flight of stairs or you take a ramp, the energy expended is gonna be the equal amount from going from point a to point b.
Joe WynnRight.
Justin LeaplineRight.
Rick YocumAnd I'm thinking that you're not gonna see the overall. It's just are you paying somebody to do it and not hiring more people internally to get things done? Is it gonna cost more to have the more reliable audit? And what are you what are you gaining somewhere else?
Joe WynnRight. So now you're messing with factors that are like one rung out from the actual objective. Right? So the actual objective is a do compliance stuff. Okay. Great. But what's one rung out from that? It's what's the overall complexity of the environment? Alright. Am I making it more complex by introducing this thing that's not directly achieving functions but indirectly achieving all these functions at once? That's typically going to be a con of sort. Right? More complex is bad, often. And then on the other side, it's okay, but am I also instituting hygienic practices and a set of diligence and and setting a series of expectations that are going to build virtuous cycles? I have a friend named Stuart who who talks a decent amount about when you do certain things he's he does a ton of stuff very well, but he's, like, really focused on, like, sock stuff and all that.
Like, one of the things that he's seen when he's helped organizations like really level that stuff up is other hygienic practices in IT or other compliance functions will get a little better because the watchers are getting smarter.
Rick YocumOh, yeah.
Joe WynnAnd I was like, oh, that's super clever. And so but but again, you're one rung out from the direct compliance function of this thing. And I was actually wondering, like, from a SOC perspective and and because again, if you can't adjust the baseline of price, like, that much. Right? There's only so much you can do. You'll try to help, but quality work is quality work. It costs money to do quality work. That makes sense. Yeah. So, like, how do you if you're on the other side of the equation for the numerator, how much can we spend on this and you're trying to go from what used to be a low price tag to a higher price tag. How do you justify that? And you're okay. And now you need to talk about, I think, things like, well, what are the virtuous cycles this can set up? Or, you know, how how do how how can we start to see other teams perform better from a overall resiliency perspective because the line two stuff is happening in a more automated fashion or a or a more complete fashion?
Rick YocumWell, that one one of the things I was think you just maybe think about was we we have some customers who don't even want to get they're not doing the security because they need the certification or the out of station. They're doing it because they can just get better security.
Joe WynnThey wanna do good stuff.
Rick YocumAnd they and and and I love that. And so I always look at So they're
Justin Leaplinegetting a certification because they just wanna be forced to
Rick YocumNo. Some of them don't even get it. They just they're like they're putting in the compliance automation tools. They're getting better because they now have tools that can scan everything in like seconds and tell them where there's a problem.
Joe WynnIt's like auditing a class because you wanna learn not because you're getting the credits.
Rick YocumExactly. And and and then at that point, we're like, wow. Yeah. You're you're doing this so well. Now that you're actually are secure and you have good mature process in place, some of their customers understand that because of some of the evidence they can get. Mhmm. And because they have the meaningful conversations, they don't even always need to get that audit report. They pass the third party risk management with flying colors
Joe WynnRight.
Rick YocumBecause of what they're already doing. And they're like, the minute I'm gonna stop getting through this, and I need to get the audit, we'll pay for the audit. Right. But right now, not having the audit isn't preventing me from closing deals. Once it starts to prevent me from closing deals, then, maybe we need to shift from having as many That's cool. Security, GRC people being able to answer all these, you know, questionnaires. And we'll have in the SOC report, we'll have in the ISO certification lower our burden, our friction to closing deals, then we'll then we'll get it.
Joe WynnOh, that's such a cool point. Yeah. So what's the tipping point, you know, from just have a good program to, oh, I need the report to the stamp?
Rick YocumBecause what's gonna be worse for that customer is they they're developing the the tool. They're developing the software. They deploy that software to their customers. Some some of these in a way that they deploy them, every customer gets their own instance of it Sure. And the customer has full capability to scan this thing for Vault. Mhmm. And the minute that their tool or that their product is known to be deficient is a reputation problem. They're like, what's worse to us isn't any of this other stuff. It's reputation issue we'll have if we if one of our customers find that we didn't do as much as we could have Right. In order to prevent something and they get breached.
Mhmm. And so they're like, that'll kill us. That'll be the end. Yeah. So they're doing security for the purpose of doing good security because that's what they need to do.
Justin LeaplineWe did at Episode, it was a few months ago, but we put in a zero bug policy, you know, and we count security issues as bugs, you know, into that. Yeah. So any bugs, basically, have a shelf life at most a week, you know, into that more
Rick YocumSo any known any known bugs
Justin LeaplineAny known bugs.
Rick YocumYou you resolve them in a week.
Justin LeaplineYeah. Wow. So anything like that. More critical, the higher visibility, obviously, they are. But if it's like a CSS spacing issue, even that gets fixed within a week. Mhmm. Because, again, you're looking at quality, you know, and security issues are just quality issues.
Matthew J. SchiavoneYou know?
Joe WynnWell, it's either fix it faster, start maintaining a giant backlog that just gets bigger.
Justin LeaplineRight. And then all of a sudden, look back at the ticket a year later and we're like, why did I create that? Or, you know, or it didn't get flagged and fixed, you know, into that. It's just there's a lot of reasons for it that we did it, but it's it's worked out great. You know? And now anything that comes through, we're fixing it before it gets on production or if it sneaks into production. Actually, just this week, I I just hooked in that the late to the game, other people have done this. But any issues issued from Sentry automatically get triggered, and an AI agent automatically creates the fix, commits the PR, and then whole bunch of, like, things get reviewed. And, basically, the only thing for me to fix it is a hit except, like, to merge the PR.
Matthew J. SchiavoneYeah.
Justin LeaplineYeah. Like, that's the only thing.
Rick YocumAnd I So you're the human in the loop. You check it. The
Justin Leaplineargument's good. Human in the loop. Like, yeah. And I'm even at some point, I might even turn that off, you know, because we have a whole bunch of validation. We have end to end tests. We have our, like, our the relational security test.
Joe WynnOther automated stuff would fail.
Justin LeaplineYeah. All of it is tested on HPR going into into that. So we have all those tests, and all those have to be passing before I even have the option to hit merge, you know, into that. But, yeah, at some point, I'm I'm still getting comfortable with it, you know, type of thing. But at some point, I'll be like, now if it's a bug and you're confident, you fix it, yeah, do it, you know, type of thing.
Rick YocumWell, that's cool.
Justin LeaplineI see.
Rick YocumMatt, I saw you were gonna comment on something earlier.
Matthew J. SchiavoneI was gonna ask in those instances where the organizations just wanna do security, they wanna do it well, they still get through the third party vendor risk management process effortlessly. How what is assisting them through that process so easily? Is it just their ability to answer the questions? What what evidence are they Well,
Rick Yocumit's it's because they have a a strong second line of defense in place. So if you're familiar with the three lines of defense, the first line is doing the work. The second line is validating that the first line's work is being done properly, and that's all done even before an internal audit comes in. They do have audit, but it's more like an internal audit function. It's not important to them to go and get that external audit, that external validation because everything management says, we're pretty comfortable because we have the three lines of defense operating well.
And, you know, just just cheap plug for CISO. That's one of the things we do. We focus on being that second line of defense. So, Matt, you didn't set me up for that intentionally, but thank you. And so that that's what they have. They have basically a team of people who are super knowledgeable on what to look for, how to push issues both to the other teams, how to get things escalated, how to create the right awareness with top leadership so that top leadership understands, well, what is the risk? Is that a business risk? What will be the business impact if that thing doesn't happen properly or if that risk comes to fruition? Then it becomes like, you know, a nonconformity and
Joe WynnRight.
Rick YocumThe whole idea is before that nonconformity becomes an incident or even worse turns into a data breach, top leadership says, get us involved. We will make sure we stop doing the things we're doing, so focus on fixing this so we can get that risk back into tolerance and and and and stop it from becoming a nonconformity. So those are the things that they're doing right.
Joe WynnWell, and from, like, a prospect mechanics perspective, do those clients typically have something like a a standard attestation package that they'll use to send to their prospects? Like, so CISO is working with a client, say, and that client has clients then
Justin LeaplineOh, yeah.
Joe WynnDo they have, like, a standard
Rick YocumWell
Joe Wynnset of things often?
Rick YocumSome of the trust portals that are out there Yep. Are actually looking at real data in the back end on these systems Mhmm. And giving you actual real data Live or near
Justin Leaplinelive data.
Rick YocumYeah. And so when you see that it's working Mhmm. And you trust the system and you know there's a human in the loop
Justin LeaplineYeah.
Rick YocumThen it creates a little bit more reliability. So that's what you're getting.
Joe WynnYeah. The trust portal mechanics, super. Yeah.
Justin LeaplineSo you guys like them? Like what? Trust centers, trust portals?
Rick YocumOh, when they're done well.
Justin LeaplineYeah. Yeah. That was actually a thing in the Delve report in the first one was that when they first set it up, everything was green before they even entered any information into it. I was like,
Joe Wynnstarts green. Starts green.
Justin LeaplineBut, you know, I'd like one of the things so well, we're designing a a feature within Piskey to have a trust center Yeah. You know, into that. One of the things that's interesting is because when have you logged into a trust center and someone has been read? Right. Have you ever thought of that, you know, type of thing? Have you ever seen a control failing in their list of assurance controls and being tested? The most likely not, but that doesn't mean, like so most will remove it.
Rick YocumRight. Remove it. Yeah. I
Matthew J. Schiavonesee that.
Justin LeaplineKnow, into that so that everything will still show green, you know, but So what don't know what's there or not. You don't
Joe Wynnknow what was there yesterday.
Justin LeaplineRight. Exactly.
Rick YocumAnd so what you wanna do in those situations when you're the customer and you're doing your TPR, your third party risk management, If what you're relying on is their trust portal, then how are you logging into it? What are you capturing? What are your snapshots? Are you doing a screenshot of the trust portal? Is this risky enough vendor that you're using that you should need to look at them more frequently. So Right. You know, then have you compared last screenshot to this one? Right. Is it still have the things on it that you thought were important last time? If not, that's not the end of the story.
You don't just say, oh, yeah. I don't see the thing. It's now I need to have a conversation.
Joe WynnRight.
Rick YocumNow I need to call up and talk to their security team, understand what's changed.
Justin LeaplineBut it's hard to identify those. Like, they'll have, you know, four or five dozen controls already there in all the various groups and all that stuff. How do you know one disappeared? You know? Ear chat GPT. Tell me
Rick YocumYeah.
Justin LeaplineTell me what's different in the screenshots. Go to the way machine. Right. Well,
Rick Yocumno. No. If you're doing this, what's your evidence that you so if you're the GRC analyst reports,
Justin Leaplineyou know, that mainly you're going up to the trust portal to download their SOC two or ISO certification or whatever it may
Rick Yocumbe. But you also should look at the controls that are they're saying are operating effectively.
Justin LeaplineYeah.
Rick YocumAnd if they're on there, and it and if it is indeed the situation where it removes the thing that's not being done anymore
Justin LeaplineYeah.
Rick YocumAnd you actually cared about that thing, your TPRM process should be saying, oh, these are what I'm relying on. I'm relying on this report, or I'm relying on not only the reports I can download, but I'm relying on the controls that were displayed to me, screenshot it. And then next time,
Speaker 2what's changed?
Joe WynnI smell a new I smell a new application that just evaluates trust portals that are out there for changes over time.
Justin LeaplineYeah. Or maybe if you have your standard controls, it should go up to the trust portal and actually look at it and say
Joe WynnYeah. If you have
Justin Leaplineany requests to do a pen test once a year, and maybe they're a month late, and so that drops off their Right. Annual pen test requirement. And you go up and, like, well, I require a pen test once a year. Because maybe the control is meaningless to you. Maybe. Yeah. You might not care. Yeah. Exactly. Depends on what it is. But if you had maybe AI going up and saying, here are the controls I care about. Tell me if they're operating effectively in this trust portal here. And then you get kind of a comparison to say, I don't see this here, and it might be they just left it off and never had it, you know, or they're deficient at this. Right.
Rick YocumRight? And then that evokes a conversation.
Joe WynnYeah. Exactly. But that conversation part's critical. Right? Because if it's just if you're just always looking for green Right.
Justin LeaplineYou don't have time to do something about it when it's missing a red. It's just theater. You're just Yeah. That's often when when we talk about the bit site or security scorecard and stuff of that nature, it's it it might be an alright tool. Like, I really appreciate it. There was one customer I had it in place with, and Log four j came out, and we're able to see all the vendors that we worked with that actually had an active export that actually, you know, version of that that was actually publicly accessible. Obviously, it's not all of them, but we were able to initiate a communication and be like, hey. Go Oh, yeah. For sure. You know, type of thing, which was really handy.
But for the most part, the only thing I really use it for is to get kind of a finger in the air. Are you doing patch management well or not? You know? So if I see their public website riddled with old patches, then maybe their SaaS app that's behind the off, you know, block Right. Maybe they're not doing too well on patches either, you know, that I can't really see, you know, from the view of a public website, you know, type of thing. So that, again, initiates a conversation. Be like, you got struts open. You got this open. You got that open. You tell me you're doing patch management.
What's going on here? You know?
Joe WynnBut I I think a lot of GRC teams are are staffed for business as usual. Mhmm. And if business as usual means things are green and you don't actually have enough time in the day to react to when things are red, like, that's an issue. You're need to talk about that.
Rick YocumYeah. So, Matt, got a question for you. You know, what should companies be asking their auditors when they're looking to hire an audit company?
Justin LeaplineWhat what what Oh, like what what
Joe Wynnmakes a good audit firm?
Rick YocumYeah. Yeah. What would you say that you'd expect customers or potential customers to be asking?
Matthew J. SchiavoneExperience with certain technologies, industries, you know, you wanna make sure they have that background. I think their methodology for just collecting evidence, conducting walkthroughs, unfortunately, you you do want that hands on approach to a degree. There's value there.
Justin LeaplineSo the assessment methodology overall?
Matthew J. SchiavoneThe assessment methodology. Communication protocols, I think when you have centralized communications, regular communication touch points, I think that just kinda demonstrates a commitment to quality and keeping things on track and moving from an audit firm's perspective. If they're reluctant to do that or they don't offer that kind of regular touch points, I mean, I think it just indicates that they're just maybe, you know, not as committed to the engagement as
Justin LeaplineWhen you say communication, are you talking about, like, in between year, like, touch points and evidence collecting, or that's something separate?
Matthew J. SchiavoneBoth. Okay. I think throughout the engagement and then throughout the year. Yeah. I like to have quarterly between audits, even maybe sometimes monthly just say, hey. Have there been changes? Have you experienced anything? Do you need help with anything that wouldn't empower independence?
Justin LeaplineAny questions? Any evidence during that time?
Matthew J. SchiavoneWe typically don't unless we're engaged to conduct interim testing Okay. In which sometimes we are Yeah. To lessen the burden of field
Justin Leaplinework. Exactly. Yep. And I think that's actually a good I think out of this here, people should actually do more interim testing, you know, into that. I think that's a just a great practice to get into. Not only like, it can't be that much more expensive.
Matthew J. SchiavoneRight? Like Oftentimes, we don't charge more. It's just a matter of timing.
Justin LeaplineAnd then you're not getting the qualified opinion surprise. Like, if you catch it in first quarter and
Joe WynnYou can still fix it.
Justin LeaplineYeah. It doesn't turn into a big issue, you know, into that. So yeah.
Matthew J. SchiavoneI think the problem is people just don't like audits, and they don't wanna be burdened with it. And the other the other value of keep keeping in touch with the auditee throughout the year is there's numerous times throughout the year. We have those ongoing conversations, and they told us about a migration to a new platform or technology. And we have to remind them that they have to retain that legacy evidence before the migration because it's in that window.
Justin LeaplineRight.
Matthew J. SchiavoneHad we not done that or, you know, had they not realized that Right. When audit time comes, that's a scope limitation, and that's automatically a qualified opinion. Mhmm.
Justin LeaplineUnless they cut the scope cut the time frame. Yes.
Matthew J. SchiavoneYeah. Yes. Unless they cut the time frame.
Justin LeaplineSo Which how many people look at it and actually know the time frame into Well,
Matthew J. SchiavoneI don't know. I mean, how
Justin LeaplineYou you do. And I know everybody probably at this table, but we've talked about, like, they delete the findings out of it and customers Well look at
Matthew J. SchiavoneRue, that's a good that's a good third party risk management practice question. Are you comparing last year's time period and making sure it's concurrent Right. Continuous?
Joe WynnI was actually wondering that as a potential, like, cost shifting measure potentially is like, well, if you're trying to justify a higher price tag, could you do a bridge letter for six like, do hey. Do the full thing for a year, and then, you know, just have a six month bridge letter, and then you start the next one. Right? So you're more like eighteen months apart as
Justin Leaplineopposed twelve months. Letter for five years? Is that what Yeah. Right. Got one five years ago. Here's our bridge letter.
Joe WynnSo mark of a bad firm signing a four plus year bridge letter.
Matthew J. SchiavoneIt is management of the organization's responsibility to define the period. We have instances where people just do a nine month period every year. Yeah. January to October or January to September One. Every year. I think they want the break. Just to I mean, honestly, I don't know.
Justin LeaplineSo that they don't collect. I
Rick Yocumdon't know.
Joe WynnWe're getting tired
Justin Leaplineof all this evidence collection. Okay.
Joe WynnIf you're gonna break the rules, these are
Justin Leaplinethe three months. This is busy. These are our three cheap months. Yeah. Now, that being said, we, of course, have
Matthew J. Schiavonewe have internal processes to evaluate the risks.
Joe WynnOf course.
Matthew J. SchiavoneWe see, you know, a system shift in those nine month that next period. It's an elevated risk. What are they hiding? And we have to dig dig in the controls a little bit further. You know, we see six month periods. We see type ones every year.
Justin LeaplineYeah. Right. Yeah. And I can understand the abridged, you know, like, time frame, so especially when you're starting off, you know, or I mentioned that time, like, do we hurry up and get another one because of our first one? Yeah. Get a clean one? I mean, there's wiggle room, you know, into that. But, yeah, I haven't heard the nine month one that's on a ongoing basis. Because once you get settled, it's a annual review, typically.
Speaker 7You know?
Justin LeaplineShould be.
Matthew J. SchiavoneI mean
Justin LeaplineTypically. That's what customers are expecting for the most part. Yeah. Yeah. I wonder how many of those customers actually know that they're only getting a nine month one. I'd be curious.
Matthew J. SchiavoneI out of I'll say out of 500 reports, round numbers that I've issued, two or three times have I seen, like, a nine month rate.
Justin LeaplineYeah. Yeah. But I'm saying the the customers they hand that report to, do they even know that it's a nine month report, not a an annual report, you know, type of thing. Yeah. I don't know. Yeah.
Joe WynnYeah. Interesting.
Justin LeaplineSo trust centers, alright. If I like them. Programmed sufficiently. I like them as a communication point. Honestly, there's a lot of faults that I'd look at from a people running security programs. And I think one of the biggest things is that they're not espousing what they're doing more often, both internally and externally. You know? Oftentimes, as you know, security is the Red Hat stepchild in the back, like, don't make too much noise. We're required to do this. Just, you know, keep doing what you're doing.
Get out, like, all that. But, like, all the good stuff like, we see the the success stories every single day with our clients and customers and our companies that we're working with. That should be communicated out, you know, all the wins that we're getting, all the customers, like, we've we've won clients because we were superior when it came down to between two big contracts, we had better security, and that was a defining factor of why we got the contract. Right. You know, type of thing. And people don't realize that, that that's a selling point. So
Joe Wynnif The burden of explaining if you're if you're the the ultimate client choosing and you're in a highly regulated industry for for instance, the burden of explaining why you went with the less secure partner can be pretty heavy.
Justin LeaplineYeah. So you got two final ones. You want got one from Dell's and got one from Matt. You know? Which which one are you gonna go with?
Matthew J. SchiavoneQuestion on trust center,
Justin LeaplineYeah.
Matthew J. SchiavoneYeah. How can a prospect ensure that the trust center scopes appropriately? Because to your point earlier, if you're in a platform, you might not be covering all the AWS and Yeah.
Justin LeaplineAnd that's a big thing. They'll actually the little notes, you can actually customize and put in what you're covering into that. And, actually, I really like a lot of the things. Vanta actually has a really good subprocessor, third party, however you put it. They'll actually say what the company is, how they're using it, what the region they're using it in, and what for what service and all that stuff, and they update that list. And you can get notifications when they change. Same thing with the controls. You can say, here's the controls of, like, what we're doing and what services they apply to in, like, a an upper statement into that. Because it could be not the entire company. It Right.
Know, it's it's a public facing thing. So more than likely, it's gonna be customer. If you can apply it to the entire company, again, if you're a, you know, a a SaaS company, that's a lot easier, you know, done than if you're, like, a multiregional hospital system, you know, type of thing that's a little bit harder. Right. You know? And I probably wouldn't wanna list out all my controls because there'll be a lot of deficiencies, you know, in various areas, you know, type of thing. So yeah. So it all depends. Yeah.
Joe WynnOkay. But yeah. I it's funny. You were talking about internally and externally. One of my clients, we were just talking about on putting on the road map for the coming year, an internal trust portal that in two to three years, depending on how it goes, will likely become a portion of it will likely become an externally facing trust portal. But yeah. Gotcha. So a couple
Justin Leaplineof FastFire things and everything. Do you think there's room for, like, a consumer reports type thing for good companies out there or a BBB or a publicly run rating system for companies.
Joe WynnLike, under the other one?
Justin LeaplineNo. No. No. But, yeah, I guess it would be more for the security realm. Not necessarily the company itself, but, you know, like, you know, I got a, you know, a bad report from this company, you know, and here's all the details. You
Joe Wynnknow? Angie's list for security and compliance things.
Justin LeaplineIs there room for that? Is it to do think
Rick YocumI don't know. I I see a lot of companies signing NDAs in order to Mhmm. Get these documents. And even if you look at what you get from, say, Azure or AWS, you're you're you're able to get their SOC two report. But you signed an NDA, and then that it base it it pretty well says that this is for you to evaluate
Justin Leaplinein their term buried in their terms condition.
Rick YocumYou you're you're clicking through and you're Yeah. You're agreeing. So it's a legally binding agreement that for confidentiality, the stuff you're getting in. When they give you their SOC two and let you download it, you're obligated to protect that because that's what you agreed to in exchange for the service. Oh, that probably wouldn't work.
Joe WynnYeah. What's what's your what's your cease and desist tolerance?
Justin LeaplineYeah. And so Exactly.
Rick YocumAnd and so, like, I have an instance of a customer who their their customer's third party or their customers was asking them to provide the evidence for, you know, their security controls. And and I'm not sure why they asked for this, but they said, oh, well, if you're using cloud services, and you then you're probably getting their SOC two, just upload those to our portal so we can see that you're doing it. And they're like, no. We can't share AWS's SOC two with you because we signed an agreement or click through an agreement that said that we're the I'm not gonna. It's for us.
Yeah. It's not for us to pass on. And so that was explained and
Justin LeaplineIt's funny.
Rick YocumThat the customer
Justin LeaplineYou get them passed on quite a bit. You do. You do.
Joe WynnThat's a it's a good practice.
Justin LeaplineYeah. Let's talk to yeah. Here it goes. This is a Yeah. AWS made it. Yeah. Yeah.
Joe WynnIn fact, I've seen a lot of instances where people will just try to rely upon that as opposed to the thing that they built
Justin Leaplineon top of that. The time. Yeah.
Rick YocumOh, yeah. That does happen all the time. And that's just uneducated customers who's not they don't they don't even know what the SOC two is achieving for them. They just know they should ask for
Joe Wynnit. Yeah.
Justin LeaplineSo one of the things I think, you know, I I bring up that consumer reporting. One of the fixes that AICPA could do is more improve their website where they list some of their quality reporting and actually more publicly shame and or control their quality into that. Like, if companies were actually afraid of doing poor quality, you know, and actually being named, you know, more viciously, I think there'd be it would settle out some of these quick one week SOC two, maybe. I don't know. Ecosystems are weird, man.
Joe WynnIt's like it's like pushing on one part of a balloon and another part, you know, gets bigger because, like, I I don't know. The AICPA also is gonna have its own risks in terms of making its clients who are the AICPA's clients ultimately. Right? So who's paying the fees, all that stuff. And so you make it too hard on them or too aggressive, and now there's room for another one.
Justin LeaplineYeah. I know. And it's a balance. It's a Yeah. You know, into that. But I think to you know, the pendulum has swung very far this way.
Joe WynnThey do have to defend the rep. Back. Yeah. To defend the reputation of
Justin Leaplinethis Yeah. Exactly. So I think public shaming is probably the least, you know, into that. I mean, it and it would be for you have a whole bunch of serious bad stuff and make it a I mean, they do have a portal and everything, but it should be prominent that, like, when I go up and look for a new CPA, you know, I I should check this portal first Well you know, type of thing and seeing, like, what's their quality like, you know, into this.
Joe WynnWell, at the risk of g r's poor beleaguered GRC teams having to review yet another thing, Should it be normalized, or would anyone even do it if someone was like, yeah. Thanks for that SOC report. I want to see the last AICPA peer review of the firm that performed it. Like, should that be a default appendix in stock reports or something like that? Or at least a summarized, you're right, de identified version of something like that. Should there be a report card of the the whoever's Of the auditor. Yeah. Of the auditor. Yeah.
Matthew J. SchiavoneWe commonly include that in proposals.
Joe WynnYeah. But
Matthew J. Schiavonedoesn't make its way to the reports and really proposals is all.
Rick YocumSo that's another thing when I, you know, when I'm asking like what what should you ask your auditor is, you know, is it possible for me to see your last peer review?
Joe WynnAnd and what would we ask for? Would you ask for the peer review or is there a specific letter
Justin Leaplineor a specific statement? Can go get.
Matthew J. SchiavoneThere there's the peer review portal and then there's there's a letter that may or may not be uploaded to that portal. Okay.
Justin LeaplineSo So Yeah. But I think, yeah, I I think they need to improve that a little bit, you know, type of thing, where it's a lot easier to check the quality of who you're considering or who you're in bed with, you know, into that to, you know, validate it.
Joe WynnYeah. Or at least well known. I mean, there's a lot of people that wouldn't know about that portal. Yep.
Matthew J. SchiavoneI also think one of the AICPA's problems this is not a knock on them, but remember, it's accounting wide. So we're focused here on SOC two and security.
Justin LeaplineThat's absolutely true.
Matthew J. SchiavoneThey have a much broader scope of services that they're concerned with. They've labeled SOC two as a high risk because they have the so they have the enhanced oversight function. But, you know, we're fixated on probably just this very small segment of what the a v AICP is overall.
Justin LeaplineAnd that is true. And I've actually
Joe WynnGo ahead.
Justin LeaplineSeen some of the especially back in the SaaS seventy where security was kinda intertwined with some of that financial, you know, reliance controls and all that stuff. We actually in another security firm I was with, we got hired I was a security consultant under a CPA to test out the security control.
Joe WynnOh, yeah.
Justin LeaplineBecause they're like, I don't know. Right. Because they're tech controls. I know numbers. You know? Like, I'm checking their general ledger and, you know, some of the reconciliation stuff or, what about this passwords and all this stuff? Like, I still feel like it's kinda like that. You know? Like, your group is specialized, you know, into this, but another CPA firm might not have that skill set. You know, a three person office, they're like, we could do SOC two too. You know? Why not?
Matthew J. SchiavoneSure. And in our planning, we have to well, we have to document that we have the competence, we have the experience, or maybe we're out we're using outside resources
Justin LeaplineYeah.
Matthew J. SchiavoneBecause we don't. So those are all considerations that we have and document during the planning.
Justin LeaplineBut that doesn't I mean, you say that's all documented. There are other companies. I'm sure they're out there that they they think they're competent. You know? They saw a YouTube video out, you know, now that that they're competent, you know, type of
Matthew J. Schiavonething. Sure. Sure. But that's part of, you know, the peer review process. Do you actually have competent people? Yeah. Know? But again, to our point, the peer review process once every three years after eight or eighteen months after the first report. So But I love
Joe Wynnthe point you made about AICPA's overall remit being like accounting wide. That's super interesting. And I would guess, although I don't know this for sure, the peer review processes for these SOC reports align to the generalized peer review processes for high risk stuff in general. And I actually wonder if this is an opportunity to say, oh, you know what? This is specialized enough. It needs a slightly different process for tech side peer review stuff. Because I mean, how quick does technology change?
Justin LeaplineThat's Like, yeah.
Joe WynnInteresting. No. You're you're absolutely right. It's three year process for Anything high risk.
Matthew J. SchiavoneYeah. Well, not just high risk, any attestation, any assurance, you know. Yeah. It's all included in peer review. They've just labeled SOC two, SOC one as a higher risk area.
Joe WynnGotcha. So that that doesn't even necessarily mean it gets a higher frequency. It just No. Interesting. Yeah.
Matthew J. SchiavoneYeah. The sample selection rate is probably higher.
Joe WynnThat makes sense. Yeah.
Matthew J. SchiavoneOf course. But same frequency.
Justin LeaplineSo that's what that dictates. Higher risk means we're testing more companies and everything. Got it.
Joe WynnInteresting. Oh, that's good. Any So if I'm a security nerd, here's here's a question I
Justin Leaplinehave for just the panel. You are. If I'm a security well, yes. True.
Joe WynnBut if if I am in this situation, what do I do about it? I'm not a GRC guy. I'm just a security guy. Is there anything I should be doing as a CSO who relies on a chief compliance officer over there to to do the compliance work? Like, is there anything
Justin Leaplinefor there and as in another company?
Joe WynnWell, whatever. It could be virtual. It could be internally. But someone else is taking on taking on the third party risk stuff and all that stuff. I'm just responsible for the perimeter and the crunchy bits, right, of of technology security. Is there anything I should be thinking about or looking at internally? Because I because our audience is kinda made up of security nerds and GRC nerds, I think. So there's probably some security nerds being like, yeah, and and and bully on you if you've listened to two hours of this
Justin LeaplineYeah. Right. If you're a security guy. And you're talking about the security nerd. Are they getting a SOC two or some type of attestation?
Joe WynnI don't know. They they just they're they're they're being asked by their executives, do we need to think about any internal controls, or do we need to do anything in response to this?
Justin LeaplineI I honestly think that's absent in everything we talked about here. That's just a good security program. You know? You should be looking at the what controls you need to put in place to minimize the risk that you're most susceptible to and put those in place. Forget about any ISO, SOC two, anything along that lines. It doesn't matter, you know, at that point.
Joe WynnBut so does this potentially change the risk ranking of your vendors and who you're paying attention to and stuff like that?
Justin LeaplineI I I don't think the the vendor's risk is, you know, dependent on the vendors and how you're utilizing them.
Joe WynnThat's probably poorly put. Yes. The risk the general risk would be dependent on how you're using them, but your ability to rely on the reports coming out from them might be put in question. And so
Justin LeaplineDepending on the quality of the auditor, you know, because How do know that? To that. Yeah. And that's where I was like, there should be a validation, you know, into that. I think, you know, I don't think unless AICPA does something, you know, specifically to SOC two, I don't think there'll be a big market shift. There'll be a little bump, you know, right now where companies are like, I don't like, you know Mhmm. This one company and let's shift around. But in six, nine months from now, it's I think we're gonna settle back into the same thing, you know.
Joe WynnThere'll be a breach next month where we like
Justin LeaplineWell talking about that instead. Yeah. Exactly. You know, type of thing.
Matthew J. SchiavoneI I don't know. I
Justin LeaplineYou think it it's here to stay?
Matthew J. SchiavoneAm I I don't know.
Justin LeaplineI mean, it's it's
Matthew J. Schiavonestill thinking.
Justin LeaplineWe'll we'll we'll come back on
Joe WynnI like optimism. Yeah. Exactly. I like optimism.
Justin LeaplineI don't know. I look at everybody's memory so short, you know Right. Shortchanged right now. So it's like, well, this is a really big concern and then, you know, 2026 comes as like, I just wanted $5. Sock too. You know? Like The buyer of the SOC report
Matthew J. Schiavoneis not gonna change. The auditor's not gonna change. What's gonna have to change I think is the AICPA a little
Justin Leaplinebit That's what I'm talking about.
Matthew J. SchiavoneAnd mostly the user of the reports.
Rick YocumRight. If they start The ultimate customer of the report Ultimate customer. Isn't paying for it. They're getting it handed to them when they ask for their vendors
Matthew J. Schiavonereports. I wonder rejecting these and it's gonna shift.
Joe WynnYeah. And I wonder if I some of the
Justin Leaplinedon't see that. I don't know.
Joe WynnI think that the the driver of that would be could see if some big players if the right people get pissed off at big players, and we're talking about like the the Westinghouses or, like, or whoever, like, the the Northrop Grumman's, the, like, the really big, big people that, like, look at tons and tons and tons of these. Right? Really any probably Fortune 100. If they start going, yeah. You know what? No. We're gonna start accept reports. We're gonna start requiring ISOs. No. Just we're gonna start requiring a different we're we're not interested in SOC reports anymore.
Justin LeaplineYeah. I don't think they're
Rick Yocumnecessarily gonna fix the problem because you're just getting somebody else's report that could also be bad from a different standard.
Joe WynnYeah. It's true. It's it's slightly it's I don't know. It's managed differently though. Right? Like the whole ISO ecosystem like we talked about before
Justin LeaplineIt can be. More structured.
Rick YocumIt can be, but like we saw in the evidence, if, you know, if everything in this report is true, they were getting both ISO and
Joe Wynnthose reports. Yeah.
Matthew J. SchiavoneYeah. So if not, why else would these customers of Dell be scrubbing evidence that they were audited and supported by Dell? I think because they know that it's not gonna be accepted, there's a risk. They obviously don't want their name associated with
Justin LeaplineYeah.
Matthew J. SchiavoneWith with the incident.
Justin LeaplineI think the negative publicity with getting a cheap report Yeah. Today, you know, type of thing. But do you But if big players stop accepting a SOC too Right.
Joe WynnThat's an issue.
Justin LeaplineThat's a big thing to do. Can you imagine, like and what if you're already in bed with them? Like, it's it's not an easy thing to rip a vendor out depending on how embedded they are.
Joe WynnI have seen really big players push around small players quite a bit.
Justin LeaplineSmall players
Joe WynnIn terms of GRC world.
Justin LeaplineIt depends on how embedded they are, you know, type of thing. Oh, yeah. Sure. They're like, you told me an idea of SOC two. I was like, well, I have one from Delve. Well, we don't accept that.
Joe WynnYeah. But then I go Well,
Justin Leaplineyou didn't tell me that. I I have one. You told me I needed one.
Joe WynnWell, it's not gonna be no. We're just letting you know in, like, two years, it's not gonna be accepted anymore.
Justin LeaplineOkay. And then maybe they have if they give it a nice time frame, then, yes, so they they could switch. You know? Yeah. Yeah.
Joe WynnI think that's what shifts the market though. When the really big players are like, I don't know that we're gonna rely on this anymore.
Justin LeaplineI don't see that shifting. I don't see that yet, at least.
Joe WynnOh, I haven't seen anything to indicate that. I'm just suggesting like, you know, that that would be to me the turning point. If one or two big players start to shift on that Yeah. It's like Yeah.
Rick YocumThat's that's what would have to be true
Joe WynnYeah. In order for
Rick Yocumthis to happen.
Justin LeaplineAnd I think for that to happen, again, I think if AICPA did a little bit more on shaming the companies, you know, a little bit more prominent, then it would be easier for companies to look at the the blacklist or the naughty list, you know, into that and say, okay. Yeah. Anybody AICPA has had, you know, three egregious reports or one in the last 20, you know, you know, twenty months, we're not accepting, you know, or something like that. I don't know how No.
Joe WynnYou're right.
Justin LeaplineWork. But then you can obviously do some quality to say, we know there's some bad ones out there, and this is where we don't accept it. You know? Yeah.
Joe WynnWell, I think you're right. And not to do too much crystal balling here, but for if if a big player was gonna like, we all know, like, it you should be doing good security stuff, but one way or the other, this is like a liability game. Right? Mhmm. And you get third party attestations to shift risk and liability and all that stuff. And so if at some point it becomes, again, too difficult for some of these big, again, call them Fortune 100 players to defend taking SOC reports from a risk reduction perspective because the AICPA hasn't responded appropriately to something like this, then I think they'll just start to shift because they're pay they're spending so much in lawyer fees to defend why it's okay for them to accept SOC. He's like, yeah. Yeah. Never mind. We'll just make everyone change.
Justin LeaplineYeah. I mean, we see this in every industry, though. Like, PCI was the trust waves before they rebranded, you know, into that.
Joe WynnThat's true.
Justin LeaplineYou know? Was That's absolutely right. A cheap PCI player. They, you know, beat out the market by an easy $8.09, $10 a lot of times, you know, into that.
Joe WynnAnd then what so then did how did PCI respond to that? I mean, they they made their
Justin LeaplineThey audited them, but then they paid them a lot of money in sponsorships and a lot of their conferences and all that stuff, and they stuck around, you know, for the longest time till they got bought out by some Japanese company and got shifted around.
Joe WynnBut Yeah.
Justin LeaplineYeah. That they're they're they're still around for what's the company? I forget. The the owner shifted to another company. He basically did the same thing. You know?
Joe WynnIs PCI different, though? Because there's no other game in town. Right? So like
Justin LeaplineIt was mandated. Yeah.
Joe WynnThat's what I'm saying. Like, yeah, the the the brands are just like We're not. Yeah. Do it or don't take cards. Right? Whereas this, like, this is all about like, oh, can you ultimately trust
Justin Leaplinea
Joe Wynnvendor while you could do just your own questionnaires and say, yeah. We're not taking SOC two's anymore. Or you could take only ISO, or you could probably do a couple different things.
Justin LeaplineYeah. But, I mean, this I think the same thing will happen with CMMC once it gets, like, some legs and everything. There'll be a natural degradation of the audit firms. You know? They'll they'll get away with what they can get away with.
Joe WynnThe incentive alignment structure is built to be a race to the bottom unless someone enforces the quality.
Justin LeaplineRight. Exactly.
Matthew J. SchiavoneI don't know who and I well, I don't recall who, but today, I did see a CMMC in thirty days. So it's it's already started.
Joe WynnThey're called dive. We're we're gonna we're gonna come out
Justin Leaplinewith the CMMC in twenty nine days now. Yeah.
Rick YocumYou can't do.
Justin LeaplineCan't do it. You can't do it. You
Joe Wynnit. Audit.
Rick YocumYeah. There you go.
Justin LeaplineAlright, gents. Any last thoughts here? Anything like that?
Rick YocumYeah. Matt, how do people get ahold of you in order to see if you can give them a quality bid for their SOC two or
Joe WynnSOC two in the right number of days.
Matthew J. SchiavoneYeah. I appreciate that.
Justin LeaplineThat's just yeah. Yeah.
Matthew J. SchiavoneSorry. Matt.chevon@sickagecom. I'll probably be linked in a LinkedIn post.
Justin LeaplineYeah. And we'll have them in the the YouTube show notes and everything else. So we'll have your LinkedIn into that.
Matthew J. SchiavoneOr go to our sickich website, sikich.com, and you'll find me there under third party attestation. So thanks, Joe.
Rick YocumAlright. Yeah. Thank you. Thanks for coming.
Justin LeaplineYeah. And thank you everybody for joining us here. Don't forget to like, comment, and subscribe. This will be coming out in a well, it'll be coming out now because you're viewing it right now. So cheers, guys. Until next month. Cheers.
In Episode 30, Rick shares the biggest takeaways from two invite only executive conferences, Palo Alto Networks in Napa and CDW in Arizona, where the agenda was 85% AI. We cover scaling laws, frontier vulnerability...
In Episode 29, Doug Salah returns as DSP's first-ever repeat guest. We preview TRISS Pittsburgh 2026 (October 22nd, $118K+ in scholarships over 9 years), break down the real implications of the CMMC pause and class...
This episode covers the growing controversy around Flock cameras: what they are, how their centralized license plate reader network enables powerful searches across jurisdictions, the documented cases of abuse by law...
In this episode of the Distilled Security Podcast, we dive into AI-driven vulnerability management, a massive breach hitting 9,000 educational institutions, and the future of GRC engineering, including a live preview of...
In this episode, we celebrate our 2nd anniversary and Episode 24 of Distilled Security! We cover the Vercel breach, how a Roblox script led to compromised Google Workspace credentials via an unauthorized OAuth...