
244: How Theories Arise
An AI-generated episode based on a human-composed blog post (which is an excerpt from my 2026 book): How Theories Arise
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

An AI-generated episode based on a human-composed blog post (which is an excerpt from my 2026 book): How Theories Arise
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Imagine a massive school district, right? And they're spending, I don't know, millions of dollars to roll out this groundbreaking new learning software. Oh, yeah. We see this all the time. Right. And the superintendent points to this glowing peer -reviewed study that promises like a 20 % spike in student retention. But then, three years later, the budget is completely drained. And the test scores haven't moved at all. Exactly. They haven't budged an inch. And it turns out that the single study backing this entire massive initiative was, well, it was essentially a mathematical
fluke. Which is just devastating. I mean, the fallout from that kind of pressure is completely systemic. Yeah. And for you listening, whether you are designing systems, managing a team, or just trying to navigate this professional landscape that is utterly obsessed with the next big thing, this scenario is probably all too familiar. Oh, absolutely. The pressure to jump on these novel discoveries is immense. But building your actual strategy on unreplicated science, that is incredibly
dangerous. It really is. We tend to treat every newly published finding as this established law of nature, right? Rather than what it actually is, which is just a single data point. Just one data point in a very long, very grueling process of actually finding the truth. Yeah, and that disconnect between a novel finding and a reliable theory is exactly where organizations lose millions of dollars. Which brings us to the core of today's deep dive. I'm really excited for this one. Me too. So we are looking at an excerpt from a September
2026 blog post. It's titled How Theories Arise, written by Dr. Gary L. Ackerman, and it was featured on the Hack Science Education website. Right. And it's actually a piece of his 2026 book, Efficacious Educational Technology, which is out under a Creative Commons license. Yep. So our mission today is to track the life cycle of an idea. Like, how does a mere question transform into a fully established theory? And we should clarify, Dr. Ackerman defines a theory very specifically here. Right. Yeah, he means the actual rules,
like the validated playbook that we should absolutely follow when we're designing educational technology systems. OK, let's unpack this. So to understand why that million dollar software rollout you mentioned completely failed. We first have to establish how a theory is actually supposed to be built under, you know, ideal conditions. The gold standard. Exactly. The foundation doesn't just start with a random guess or a hypothesis. It starts with this really comprehensive mapping of existing knowledge. Like a literature review.
Yeah. Researchers scour the established literature to identify a very specific gap, you know, a missing piece in the architecture of what we already understand. So they aren't just throwing darts at a board. They're looking for the hole. Right. Then they formulate a research question aimed directly at that gap. And then they have to design a methodology to answer it. And this part is key. That method must be formally reviewed and approved before they ever interact with a single test subject. Because the methodology is basically the blueprint. And Dr. Ackerman
points out that for building these theories, the absolute peak of that blueprint is the double -blind random experiment. the holy grail of research. Exactly. And since we are looking at this through the lens of EdTech, it means randomly selecting a sample of, say, students or educators, right, and then dividing them into a treatment group that uses the new intervention and a control group that just sticks to the baseline. Yeah, that structure is specifically designed to isolate the variable. But what's fascinating here is the absolute necessity of blinding the individuals
who are conducting the statistical analysis. Wait, the analysts, not just the subjects? Right. We always think of blinding as keeping the subjects in the dark so there's no placebo effect. But the analyst crunching the numbers must also be completely unaware of which data set belongs to the treatment group and which belongs to the control group. Oh, wow. That makes me think of like a... high -stakes corporate taste test for a new soda formula or something. Okay, yeah. How so? Well, obviously the taste testers are
blindfolded. Right. But to make it truly double -blind, the person standing there with the clipboard handing out the cups and tallying the scores, they also have no idea which cup holds the new formula. Right. They just record cup A and cup B. Exactly. It prevents the person running the test from, you know, subconsciously smiling more enthusiastically when they hand over the shiny new product, which could totally skew the tester's reaction. That is a perfect analogy. And when you translate that taste test dynamic to data analysis, the stakes become entirely mathematical.
Because the analyst is making choices, right? Exactly. When a statistician is cleaning up raw data, they make dozens of micro -decisions, like how to handle outliers or which statistical model fits best. If that analyst knows that Group A got the expensive new software, they might subconsciously tweak things. Yeah, they might subconsciously choose parameters that make Group A look slightly better. Blinding the analyst locks human hope and bias completely out of the math. So the math becomes completely sterile and objective. But,
I mean, applying that sterile ideal to an actual school or a corporate training environment, that introduces a massive contradiction. Oh, absolutely. because a classroom is not a lab. Right. It's chaotic. It's filled with interpersonal dynamics, weird schedules, infinite variables. Dr. Ackerman even notes that a strict clinical double -blind experiment often just isn't possible in these real -world settings. Which forces researchers to completely adapt their methodology. If they can't isolate variables in a vacuum, they have
to capture the complexity of the environment using massive amounts of quantitative data. The numbers. Yes. They rely on descriptive and inferential statistics to test hypotheses, document trends, and really describe the populations. And, I mean, in 2026, the volume of quantitative data we have is just staggering. We are talking about digital systems and cloud computing tracking almost every single interaction. It's a gold mine for researchers. Yeah, like if a student logs into a learning management system, the cloud captures the exact
second they logged in, right? How many milliseconds they hovered over a specific button, how many times they clicked back. It's incredibly high resolution. It is. The sheer scale of cloud -based data gives us this crystal clear numerical picture of behavior. But having a billion data points can actually create an illusion of certainty. Really? How? Well, cloud computing gives us a precise record of the what. like what the user clicked, how long they stayed, their score. But numbers are completely blind to the why. Ah,
I see. Like, high dwell time on a digital module could mean the student was deeply engaged, or it could mean the user interface is completely broken and they were just staring at the screen in pure frustration. Right. The quantitative data looks exactly identical in both of those scenarios. Exactly. That limitation forces a complete shift in methodology. To figure out the why, researchers have to step away from the clean cloud metrics and dive into qualitative data collection. Which means working with text mostly, right? Yes. Analyzing individual surveys,
focus groups, dissecting meeting notes. We're reading through these massive sprawling transcripts of classroom recordings. And doesn't it also involve, like, trained observers? Yeah. observers sitting in the room, specifically tracking how many times an event of interest occurs, like a teacher having to verbally redirect a student who's struggling with a specific app. Okay, wait, I have to push back here. Wait, if qualitative data is just text and observation, isn't that incredibly subjective? It sounds like it, doesn't
it? Yeah, like if you and I read the same meeting transcript, I might think a teacher's tone is really collaborative while you read the exact same words and think they're being super defensive. How does reading a transcript translate into the hard science we need for a theory? That is the big question. And the defense against that subjectivity is a highly structured mechanism called coding. Coding? Like programming? No, qualitative coding. Researchers don't just casually read a transcript and jot down their feelings.
The text is systematically dismantled. And the crucial safeguard here is that the data is processed by multiple independent readers. Oh, so two different researchers are reading that same classroom transcript separately. Exactly. They read it separately and they apply specific tags or codes to different statements. So one researcher might tag a student's comment as interface confusion, while the other tags that exact same comment as lack of prerequisite knowledge. So what happens when they disagree?
Once they finish, They bring their coded transcripts together, and they are forced to confront every single discrepancy. Oh, wow. That sounds like an intense process of, like, intellectual friction. It is. They have to argue out their interpretations until they align. They debate the context and refine their definitions until they reach total consensus. That's fascinating. Yeah, they build a master codebook that clearly defines what constitutes interface confusion. This process of establishing
iterator reliability completely strips away the individual biases. So the subjective narrative is basically converted into documented agreed upon data points. Exactly. And those consensus driven codes allow them to write highly detailed profiles of the users. They turn the messy human experience into a structured architecture. OK, so let's look at where we are in the blueprint. The researchers identified a gap. They formulated a question. They gathered their data, whether through the gold standard of a double -blind experiment or massive cloud metrics or this rigorous
consensus coding. Right. They run their analysis, and boom, they discover a statistically significant result. The software works. So what does this all mean? Do we have a theory yet? Not even close. Really? Yeah. Finding the result is merely the end of the initial construction phase. They have the data, but they absolutely do not have a theory. Wow. Okay, so what is the next phase? Running the gauntlet of the publication process. They develop their hypothesis based on the findings, and they submit the entire package to a peer
-reviewed journal. Where other experts tear it apart. Exactly. People who have zero stake in the success of this specific study review it. They look for logical leaps, statistical anomalies, confounding variables the original team missed. And the original researchers have to defend it. Yep. They have to resolve every single concern raised by the review panel. It is a highly adversarial process. The study is only accepted once it survives that scrutiny. Okay, but here's where it gets really interesting. Because surviving peer review
and getting published, that is still not the finish line. Not at all. Publication is merely the starting line for the broader scientific community. Right, because Dr. Ackerman's framework emphasizes that a single published paper is never enough to establish a theory. Never. Before those findings become the rules we use to design systems, the results must be replicated, in similar settings and in different ones. And this brings us to the replication problem, which is arguably the biggest crisis in modern science, and it directly
explains that million -dollar software disaster from earlier. It really does. When a study produces a novel result, something Totally new and surprising. It's like a magnet for attention. Oh, yeah. Novelty generates headlines. It drives funding. It creates a massive pressure to just adopt the framework immediately. Because the incentive structure heavily rewards being first. But when independent research teams attempt to run the exact same experiment again, a shocking percentage of these novel discoveries fail completely. The original
results just can't be reproduced. Right. Due to data collection errors, analysis issues, or just hidden variables. It's kind of like software development, honestly. An internal dev team builds a new app, tests it in their closed environment, and it runs perfectly. They get the green light from management, which is basically their peer review, and they roll it out. But the second they push it to a million diverse users on hundreds of different devices, it crashes immediately. Because the real world exposes vulnerabilities that were literally impossible to see in the
closed test. That maps perfectly to the replication crisis. Yeah. An educational intervention might show spectacular results in an affluent suburban district. The peer reviewers examine the math. It looks pristine. But when another team tries to replicate it in a rural district with poor broadband, it completely fails. Because the original finding wasn't a universal truth. It was a localized phenomenon. Exactly. And beyond environmental variables, there's just statistical noise. If a study uses a small sample size, random chance
can masquerade as a profound pattern. And researchers under pressure to publish might lean on that noise. Right. Finding a significant result that's practically meaningless. That combination of small sample sizes and localized variables is exactly why replication is non -negotiable. The original study introduces a possibility. Yes. Replication tests its durability. Yes. If we connect this to the bigger picture. This is Dr. Ackerman's primary warning to practitioners.
You cannot build your decisions on a possibility. You just have to wait. You must wait. Whether you're an educator, an IT director or manager, you have to resist the siren song of the novel. A concept only transitions into a true theory after it has been repeatedly validated by multiple independent studies. Time and replication turn an interesting finding into a fact. Synthesizing Dr. Ackerman's framework really gives us a totally different lens for evaluating information. It really does. We've tracked this exhausting journey,
right? Identifying gaps, choosing between sterile math and rigorous coding, surviving peer review, and facing the brutal test of real -world replication. It just reveals how fragile a single scientific study truly is. The word groundbreaking should honestly be treated as a warning label. It really should. Which leaves us with a final thought to mull over. We've just seen how highly trained scientists with strict peer -reviewed methodologies can still be routinely fooled by statistical noise and hidden variables. The replication crisis
is very real. Yeah. So if the experts are constantly getting tripped up by a sample size of one, how often do you make major life business or educational decisions based on a single novel article you read online? Or worse, how often do you lock in your worldview based on a one -off personal experience that has never actually been replicated? Keep questioning the novelty, keep looking for the replication, and we'll catch you next time. Thanks for listening.
Nobody is credited on this episode yet.
An AI-created episode based on a human-composed blog post: Changing Vision of Technology Literacy
An AI-generated episode based on an AI-generated blog posts based on an old microblogging post: The Inquiring Technologist: Why Great Solution Design Begins with "Too Many" Questions
A brief AI-generated episode based on a human-composed blog post: Gathering Organizational Intelligence for Network Security
An AI-generated episode based on a human-composed blog post: Models of Technology-Rich Teaching
An AI-generated episode based on a blog post designed for students in a community college course. Concepts of Operating Systems