Episode 175: This week on Thinking Deeply about Primary Education, Jon Kay (Head of Evidence Synthesis and International at the Education Endowment Foundation) joins me for an exploration of evidence synthesis and the ways teachers can put evidence to work for them in the classroom. As part of the chat we discuss... How the EEF evaluates the quality of research How teachers can navigate the huge body of education research Common pitfalls to avoid when implementing an evidence-informed approach The most significant evidence-based strategies ...and much, much more. If you enjoy this episode, please support us by subscribing to our YouTube channel, leaving a review on Apple Podcasts, or making a donation via www.ko-fi.com/tdape. If you have any questions or comments, you can email us at thinkingdeeplyinfo@gmail.com or join our Discord server, where we have a special channel for unseen question submission.
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Welcome to Thinking Deeply About Primary Education, the podcast that makes time and space to think about pedagogy, teaching and learning professional development. Anything of interest to time, poorer but enthusiasm rich primary teachers. This week I'm joined by Jonathan Kerr. Hi, Jonathan, how are you? Hi, not too bad. Jonathan, we're going to explore evidence synthesis, which is something I know listeners will be extremely interested to hear about, but I thought it made sense for you to sort of explain who you are, where you're from and how you got here. Sure. Yeah. So, hi, I'm John Kay.
I am Head of Evidence, Synthesis and International at the Education Endowment Foundation. And a lot of my career history has been at the EEF. So actually on Sunday it was 10 years that I've been at the organization in lots of different roles. So sometimes supporting with more of our policy work, sometimes on our program evaluations. But largely most of my time has been focused on evidence synthesis and in particular a
couple of key resources that EF is responsible for. So probably the most well known is our teaching and learning tool kits, but also we produce guidance reports and and other evidence reviews that hopefully are informative for practice. 10 years is huge. How close to the very first cohort of members were you? Pretty close. So I never, I never feel like one of the originals. There's still a couple of people from from right at the start of the EF, but I think we've been going 13 years now.
So we were started in 2011. So, so pretty close and a lot of, a lot of changes, a lot of different areas and activity in those ten years. Yes, Stuart Canyon does a talk and he starts off by talking about what's happened since 2010. And it's always shocking. It makes you feel really old. So I mean, I think, you know, there's no one better place to explore the idea of evidence synthesis. And I thought I'd start by asking if you could share some insights on how the EEF evaluates the the quality and robustness of educational
research and what criteria are used to determine which evidence is most reliable and actionable for teachers. Yeah, sure. So, so I'll talk. Umm about the three key questions that the EF asks when thinking about evidence And I'll I'll go into bits that are a bit more nerdy and methodological, but I'll I'll also try and remain as accessible as possible and definitely interrupt me if I if I explain something in a way that isn't entirely accessible.
So. So the first question we ask ourselves if we were to find an individual study is could this study tell us anything about the question that we're interested in. So there we'd be thinking about has the study been designed in a way that could answer the question. And what's really important there is there isn't going to be a hard and fast rule that that some studies are good and some studies are bad, but the different methodologies we might consider better or worse at answering different questions. So if we were trying to answer a
question about the efficacy of a practice, then what we'd be looking for at EF is some kind of comparison group in that study. And that's that's mainly because children are always making progress as they go through schools. So quite often you'll see a study where someone is given a new maths intervention and you measure them at the beginning and you measure them at the end and you say, wow, all of the kids have improved in maths to which any teacher would say, well of course they have because
children are always progressing in maths and are receiving other bits of tuition and and support. And so one of the things we're we're quite strong on at EF is when we talk about efficacy we we like to see some kind of comparison group. It doesn't always have to be a randomized comparison group. I think EF does have this reputation as loving randomized control trials and and we've certainly funded a large number. But it's really important that there is some kind of
counterfactual so that you can understand is the intervention or the approach or the new thing that's being tried making a difference over and above the the regular practice within a classroom. But other questions we might have other methodologies that that we think are appropriate. So we also find quite a lot of qualitative evidence as part of our evaluations and and qualitative evidence is much better for understanding how things will be received in the classroom. So their feasibility, their
acceptability, whether there are any barriers to to their implementation. So that first question, what we're trying to understand is, is the methodology appropriate to the question that we want to report or that that we want to answer? Then the second question is you might have a fantastically appropriate methodology, but you might have a terrible study within that methodology. So the second question is really about what is the quality within methodology. And in particular, we often
frame it as are there risks of bias? So are there things that might be part of the study that might make us less likely to trust the results of it? So one risk of bias might be is the study independence. And there's lots of evidence that shows that when people evaluate their own work, you tend to see higher impacts. My favorite example is a lot of the evidence based on painting classrooms. So you might have heard people
say you know you have a a blue calming classroom and all of the kids will be well behaved. Quite a lot of that research was funded by large American paint companies. And so regardless of whether you think the actual study is any good, there might be some suspicions about about the kind of the level of independence there and whether you're only hearing about the positive results. But there are lots of other aspects to to methodology or or risks potentially to bias. So one might be around the size of the study, you you might get a less secure result if you only
test something with one or two kids than if you test it in 100 schools. And particularly when you think about whether something will be generalizable, you really want to have a a large number of of kids involved in the study to make sure that it's not just a A1 off. And then of course there's there's things like how many people drop out of studies like we talk about attrition, how comparable the groups actually are. If you did a study where I talked about having a comparison group, but if your comparison group looks really, really
different to the intervention group, then maybe it's actually the results you're saying or about differences in cohorts or or population rather than the intervention itself. So when we think about the the quality of the study within the methodology, we tend to go through a a list of different questions. I won't go into all of the the kind of methodological detail but those are those are the types of things we'd be asking. And then the final question is how relevant the study is and and people often get caught up
on relevance being did it take place in the same country and some things may be more transferable than others, that there are things that we've tested that have been really positive in America that then haven't translated at all. But then there are also lots of theories of education that seem to be applicable across lots and lots of education systems. You know, most education systems consider how to give high quality feedback to the children in their settings. Most systems will have some form of scaffolding the the content
of their learning and and and approaches like that. But it is really important to think about context, particularly when we interpret what we can say from results. And that isn't just geographic, but also might be about things like outcomes. So a trends that you often see in studies is studies measuring outcomes that are really, really close to the intervention and then making really, really big claims on the basis of those studies. So a classic example might be if I wanted to evaluate Duolingo
and and learning some new vocabulary for through an app like that, that's nicely gamified. What I could do is test that I learned my 10 words of the day, and then at the end I test whether I know those 10 words and I will perform better on remembering those 10 words than someone that has received no instruction whatsoever on those 10 specific words. You'll almost always get a really exciting, promising result. But does that tell you anything about learning Spanish language in general or does that just measure something that's so
close to the intervention that it's, it's not really giving us a, a, a big claim that we can make. And so we think really carefully about context not just in terms of country but in terms of is the study, is the study answering the question that that we we want to make a claim about. So we wouldn't try and make A a sweeping claim about improving key stage 2 results on the basis of a vocab very proximal measure and and together we think about those three things when we talk about quality.
So, so methodology study quality within the methodology or risk of bias and and then relevance as well. Fantastic. I love the the paint example because I Reg reasonably regularly had emails from a company and I don't know how best to describe and maintain their anonymity. But uh, the first thing I ask is can I have every study that was associated with this the claims that are being made here and they don't get sent to me So. And Chris such always talks about how lots of reading
research even though I think reading is reasonably robust and certainly compared with mathematics he always, he's always looking for comparisons within the within the trials because he finds that that's something that doesn't doesn't feature too heavily. And so yes, I think, I think teachers can definitely, there's lots of things they can look for themselves when they're engaging with the with the papers. I mean, what do you consider to be a large sample size? Like what do you what's your benchmark? Yeah, that's a, it's a good question because I think it does
slightly depend on the type of question that you're answering. So we've done some things that we call teacher choices trial where we're testing a decision a teacher might make in the classroom and we might have a smaller sample for that then say a big structural intervention where we're looking across the system. So EFS pipeline for an efficacy trial, which might be a kind of a smallish trial that EFS funding, we'd probably be looking for 50 intervention
schools and 50 comparison schools. And then as we go to the larger ends of our trials and look at things that might be ready for scale across the system, we might be talking about 100 schools versus 100 control schools. I should say though, that almost a lot of the power you get from your sample size comes from not the total number of pupils, but from where the comparison is derived. So you might get a similar. If you did have an intervention that you could test at the pupil level rather than at the school level, then 100 pupils each side
might be a perfectly valid comparison a thing. And this goes into kind of a very nerdy fractal I guess, but I think that I would look at to try and understand whether it's an appropriate sample size is the level of uncertainty around the final result. So people often talk about statistical significance of results. We at EF don't really like to use the the kind of the binary cut off of is something significant or not. But you can always see there's some uncertainty, statistical
uncertainty around any result and depending on how well powered or how appropriate the sample of the study is, that statistical uncertainty will be either reduced or or increased. So similarly, if you see a political poll, you might have a margin of error. All studies have have that similar margin of error. I think I remember seeing once there was maybe six months or so, there were times, but the margin of error was also 12 months or something like that. I can't remember what the exact example was. And so it almost meant that actually nothing was being said.
So I can see. Yeah. And it's really, it's really hard to interpret that because you know there is always going to be uncertainty and we shouldn't, you know, it's educated, you can't wait for the perfect result that gives you 100% certainty. We're always going to be talking about best bets and we're going to be talking about risk. You know, I I don't think we ever will get to the point where you can say if you do this you will get exactly this result. But we should definitely be cautious. If we're seeing really, really
uncertain results that include really, really positive results and also incredibly negative results, then that might change our risk appetite when we're thinking about doing something in a classroom. Nice. I mean, as the sort of lead person on this, do you feel any pressure with the final message that comes out of all this, this sort of analysis? I I should be very clear that I have a team around me, so I've got plenty of people to blame. But I I think that the important thing that that makes me
confident actually is and and we can we can talk about this when we talk about synthesis more generally when we do evidence synthesis, a key part of it is having a really clear pre specified process. That means that my opinion shouldn't come into it. And so almost what we get out in the tool kit, I don't really have a say in sometimes I'll be really surprised by it, but I won't have a say in it. And what we do is we specify a process for engaging studies and
we specify analysis processes before we see what any of the results are. And so sometimes I might be nervous about the way a result might be received or sometimes I might be nervous about did we get our process completely right. But I I try and stay out of the kind of that final how the result is because I think it's the the value of it has to be that it's process based rather than than what I think about metacognition or or what I think about feedback. Nice.
So yeah, so you've almost got like this. If if the system works, then, you know, there's less pressure on yourself, which is probably healthy. And if the system doesn't work, then I'm worried and then very accountable as well. I can teach you to effectively navigate and make sense of the what is essentially a very vast body of educational research, and particularly when findings may seem contradictory, counterintuitive in my experience or like you said, context specific.
Yeah and and I'm going to reflect my bias here and evidence synthesis is in my my job title. So I do have a a bias, but but I would when looking for research, I would always try and look first for reviews and evidence synthesis rather than single studies. And there's a few reasons for that. One is just convenient. If you find a really good systematic review, then rather than reading 200 individual studies, you can read one summary that hopefully is able
to deal with those contradictions. Secondly, it's more rigorous because if you look at any individual study, there's always the chance that that you have that outlier. And there there are outliers for lots of reasons. But a real risk is the things that tend to get most coverage or most attention are often the strangest results. So the things that might be hardest or hardest to find might actually be the most representative studies. And so I think that what a good
synthesis should do is pull evidence together. And I'm certainly not suggesting that every teacher individually does a systematic review and then meta analysis. But I think that's part of the role of organisations like EF is Also authors of systematic reviews is compiling evidence into a way that can hopefully be interpreted really easily by teachers or school leaders and giving them an overview of the evidence space. So, so when we do a A toolkit
strand for example, as I said, we we have this independent process where what we try and do is look for every single study at the beginning we'll lay out some criteria for inclusion. So we might say we only want studies with children age 3 to 18, but we only want studies that have an outcome that is an attainment outcome. But then we're trying to find every single study. So we're not just trying to find studies that match our opinion or studies that are done by
people we know or even studies that are in the most the closest context. So we'd be looking for studies from around the world and then we'd be gathering them together and trying to summarize the average impacts. And hopefully that gives a sense of where the overall evidence base is. Of course, the important thing then is to be able to understand what sits behind the average because maybe you want to know what is the impact just for schools in England. So what is the impact just in mathematics or just in in reading outcomes?
And so we also do a process once we've got all of the studies together, once we found the studies that we're interested in, of doing analysis that can compare how different factors explain the variation of those studies. So we do a process called meta regression or moderator analysis, and what that allows us to do is say all of these studies have very different impacts and all of these studies have slightly different characteristics. Now, what are the characteristics that seem to explain the variations in impact?
And sometimes that will be a pedagogical factor. So maybe something is more effective when it's delivered by teachers than by teaching assistants or digital technology. Sometimes that'll be a methodological factor that we've got a correct for. Maybe we're seeing much lower impacts when we're having more rigorous studies and we want to make sure that we're appropriately describing the security of the evidence base. But sitting behind that is is a process that hopefully gives teachers access to a much larger evidence base and and a much more rigorous and independent evidence base than looking for
an individual study. So if I was a a teacher trying to navigate and make sense of the vast body of of research, what I'm trying to 1st is go, is there a review Maybe that's the tool kit, maybe it's the guidance report, maybe it's just putting what topic you're interested in and the phrase systematic review because there is a vast body of literature out there. Obviously that won't always be the case. That hasn't always been a a review done and then there's probably a smaller number of single studies. And what I try and do is look
across the evidence base, not just take the two studies that look closest to your setting because context isn't everything, but to try and in a broad ranging approach, assess the quality of the studies, the relevance of the studies and the underlying methodology of the studies to see in combination what you think that message might be that. Makes total sense. I mean, I've got a favorite meta analysis I think. I think it's pronounced carbonyl. I think it was 2014. They explored the use of manipulatives in classes. How interesting A. Lot of those threads I used to
sort of like you say go down rabbit holes and things like that there. I mean I think I may have seen people talking about the use of artificial intelligence to conduct systematic reviews. Have you seen, has this come into your sort of peripheral vision or anything recently? That is, it's very much on trend at the moment and the big move that that we're trying to do, particularly if the toolkit is for the toolkit to be as living as possible. So what you want with your reviews is not only that they summarize the evidence base, but they continue to summarize the evidence base. So as new studies are published,
you want to really quickly add them to your evidence base so that you don't immediately have your go to resource being just evidence from five or six years ago. And So what AI is frequently seen as the solution to is how can we find the studies quicker and then how can we extract information from the studies quicker. So underpinning the toolkit. At the moment, there's a lot of people that I get to go on podcast and take credit for their work. But we have, you know, 20 research assistants that work with Durham University whose
jobs are to read studies and extract information from those studies so that we can collate them. And that I'm sure may sound enticing, but after you've been doing it day after day is not the most glamorous or interesting job. And it's the type of thing that we can work with AI to try and make more efficient so that we can use those people for the the harder and more interesting challenges around communicating evidence. And so at the moment on the Toolkit we always use bits of
machine learning. It's not generative AI, but we use machine learning to help us find studies more efficiently. So within the toolkit, the published toolkit, I think that's around 2500 studies. And what machine learning can do is say we can look at all of the information from these studies and suggest studies that look like them, so that we can find the studies that we're missing. So at the moment, there's a big project underpinning the toolkit that's using machine learning to find those studies and to screen those studies more efficiently.
The big win that everybody wants is to be able to get the machines to just extract all the information from the study so that you could really quickly run a meta analysis or or really quickly run a narrative summary. But I think we're a long way there. And I think the thing that worries me is, is the idea that ChatGPT or chat bot can summarize the evidence effectively already because they really can't. And and the thing that machines
are really bad at is what they tend to do is take the findings from individual studies and summarize them and try and amalgamate them. But the thing that Metro analysis allows you to do is get the quantitative trends. And so we were talking earlier about studies that may have a lot of statistical uncertainty and might be lacking in statistical power. Now if you have 10 studies that are all underpowered, give very kind of notional findings, they
maybe they all have a very small positive result. Those 10 studies together might tell us something really meaningful. And if we did a metro analysis, we might get a really statistically secure result by looking at all of the small effect sizes together. But each individual author might say there's no evidence of impact. So if you give that to ChatGPT, it will look at those 10 studies and at the end it will say we're really confident there's no evidence of impact. Because we found ten studies that all agree that there's no
evidence of impact. Because it it's not quite yet to the point where it can go through the processes of of taking out the statistical data and then manipulating and and combining it. I'm sure at some point it will get there and I'll become as expensible as as anyone else but but but we're not quite there yet. I totally see where you're coming from. I mean, Elon Musk seems to think that there's this point in the near future where sort of AI will know more than the combined sum of human knowledge. But I think there's something innately human in engaging with research.
And for me, I'm not. I'm not yet ready to, you know, I'd rather read the paper than ask a computer, but it thinks meta aggression sounds particularly complicated. And I mean is that almost the perhaps one of the reasons we need large bodies like the EEF because you've got the manpower to effectively negotiate that and whereas it feels like it's almost beyond the scope of individuals within the school system. I've I've learned Metro aggression and I I don't think it's it's purely the kind of the
might of the the EEF that has that has enabled that. But I think I think for the EF to be a useful role in the system what we want to do is the most burdensome, the most kind of time intensive bits of summarizing research. We want to do that in a way that is rigorous, independent and gives practitioners access to the findings. And so I think the thing that I'd say on synthesis is, regardless of the meta aggression, I think the feedback
strands in the toolkit has around 200 studies in it. Now I myself wouldn't want to sit and read all 200 of those studies and I doubt any individual teacher would want to sit and read all 200 of those studies because there are much better things that they can spend their time doing. But by working with a team of research assistants, extracting the data, we can do the really time intensive part that then gives teachers access to that evidence base of those 200 studies and understanding the average and what sits behind the
average and also interesting findings for implementation. And so it's really important that we empower teachers to engage with the research and challenge the research and be able to zoom in on individual studies and think about how to question information that's presented to them in the system. Because not all of it will be reliable. But I think the aim of what the EF wants to do is those really time intensive parts of interpreting the research is to to try and do as much of that as possible and and present it in as accessible a way as possible. Now you mentioned independence,
obviously we are in at the end of an electoral cycle. I think if we managed to make it through five years with our general action almost, will the EF continue on beyond that? Are you dependent on? No, no, we, we, I mean I think it's really important that we are completely independent of government. So the the EF funding model unlike lots of kind of things that were initially funded by government is that we were given our endowment as a lump sum.
And what that meant that we could do is firstly plan for the long term. So doing the studies that we fund takes a really long time, you know and an EF funded RCT from commissioning to publication is, is probably around three years. But most importantly what it means is we have no government involvement in the decisions that we make. So our our research agenda is completely decided by us and the government gets no say over what we publish. We send them the results.
We hope they're interested, but there is no world in which they can say do this or don't do this. So we will continue to exist. We have funding for for at least another 10 years and and the thing that that makes that powerful and I think makes the EF work is the fact that we are are not part of government machinery. And we don't have to go each year to negotiate with the Secretary of State and say you know give us some more money
because it's always those points of leverage that you might lose that independence. And I think there are lots of things that we all agree on but there are still lots of areas that are are really contested in education and and evidence should be a way of cutting through that rather than something that that becomes ideological. And I think a lot of the the methods that we put in place are also about protecting that independence. So that systematic review approach that I talked through, part of that is because it's the most rigorous. I think it's also really important that having a process
that's transparent and that is replicable means that people understand where where EF getting its results from. And you wouldn't want it if everything we put out seemed like the opinion of our CEO or the opinion of me or or of another team member. It always has to be very process driven and sometimes that can make it very technical or sometimes a bit opaque, but I think the transparency, independence and replicability of things that we think a lot about at the EEF think that is really.
Powerful. I mean that 10 years, you know, because I remember when I when I went to Shanghai in 2014, they talked about how they had a 10 year plan for education. And I know that the place of education in politics and vice versa is is reasonably contested and but in my mind it would be great if there were like this 10 year agenda and then we see how things are going and it almost feels like you've got this, you're doing that and you know it might not be like you said what the Secretary of State is, talented schools but actually this is more powerful because it goes beyond and political
influence. Yeah. And I think, you know it's really easy to get caught up on the the kind of policy side and we do have a policy team that that try and inform the DfE with our research, but actually the the powerful act that's going to make a difference in the outcomes we care about the schools. And so the thing that I think means that the EF can be a success and the thing that means that we're actually very unlikely to get political interference is because it kind of doesn't matter that conversation.
The conversation that matters is is the EF and research into schools, and that conversation can happen regardless of what's happening in the Department for Education. Absolutely. I'm really pleased to ask now because that's like really useful information and really important information. So we we've talked almost about the work of the EEF, what teachers might want to do if they want to effectively navigate research. What common pitfalls, or perhaps misconceptions, should teachers
be aware of when attempting to implement evidence based practices in their classrooms? So I think that the main thing is not to, not to trust us too much. So I think that it's really important that evidence is a best bet and it it it tells us something that's useful. If I was doing an approach, I'd want to know if that approach had been successful in 100 classrooms or unsuccessful in 100 classrooms. But I don't think it's ever going to be a prediction. And so I think that the key
thing in terms of implementing evidence based education is using teachers professional expertise when thinking about how to implement that evidence. And so doing a full implementation process where you think about what are the needs of the children in my classroom and not starting with a solution, it'd be really easy to go. There's a new evidence based thing. So I'm just going to do that. But but that isn't really how to do things. Well, thinking about what do I need, then looking at the evidence and hopefully looking
at a systematic overview of the evidence to find the best solutions. And then going through an implementation process that that brings the rest of the school with you. And I think that the hardest thing actually isn't finding the evidence. The hardest thing is how you then mobilize the evidence through the school. This is timely. We we recently published an implementation guidance report that that hopefully provides some supportive frameworks for how you might implement evidence based approaches.
I think the key lesson there across all of it is, is the main kind of enabling factor is the knowledge of the teachers of their classrooms and that it's really easy to get into a world in which evidence and professional expertise seem like they're in competition, but they're really not. One is hopefully supporting the other and and hopefully that's a positive conversation going between those two factors. And I think that the other kind of key things are not trying to do too many things at once. We've definitely, maybe this
goes back to the policy conversation. There's there's often a kind of impetus of in a particular region or particular school, let's throw 100 different interventions and and try and improve the school. But doing 100 things badly and then stopping most of them after six weeks, two months a year isn't really the way to make anything work. And I think the key lesson from the evidence is if you look through even the most positive strands in the toolkit there are examples where it has a negative
impact and and that should be intuitive, you know the most well evidenced approach in the world and if you really wanted it to fail, you could make it fail. I know I could make anything fail completely and have really negative impacts on on children and. And so I think it's that buy in that that professional application of the knowledge and that targeting appropriately is always going to come from the people in the classroom, you know the sense that I take. From the the recent report is definitely it's about you know how can we bring everyone along
together and and really make this impactful and it's almost like you know when you've got the breathing space and clean school then that's when these things go really well. So it's almost a case of make the breathing space for your teachers, you know, if you're and it it. Takes a really long time and it's like revisiting things and checking things have embedded so that it's not intervention after intervention but it becomes a part of the the school processes Is is really challenging because we can all get excited for the
first six months of something but it then becomes quite boring and not exciting and and that's the the tricky but I think one of the things I get asked about quite. Often is curriculum and choice of curriculum and my response is generally it doesn't really matter which one you have the best in the world. Like you say, it's very easy to make things feel, but what matters is, is that this is a 7 year journey and it's those kids who are near 1. Now you've got to wait to see how impactful this was over that time so I can, I can totally see where you're where you're coming
from. Now my last question I. Don't know if it's fair or not. It's almost like asking you to pick your favorite, your favorite children, and like what are the most significant evidence based strategies or interventions that have consistently demonstrated positive impacts on student learning outcomes and perhaps across educational settings and age groups if such a thing exists, I think simultaneously. Interesting and boring that the
things that have positive impacts are probably the things you expect to have positive impacts. So there is a large evidence base that shows that additional small group tuition provides positive outcomes and it probably isn't that surprising you get more targeted support particularly if it's well targeted to to kids that really need the support, it's likely to have a positive impact. Similarly, high quality feedback is generally has a really positive impact for examples of it having less positive impacts,
but generally it has a very positive impact. And I think one of the lessons in evidence synthesis is we get fewer shocking results. Actually a lot of the results of the things that that you might just expect to be the case because what we're doing is summarizing an evidence base that builds on professional knowledge that the practitioners often know really well. Then the interesting bit though is, is what sits behind those averages of those positive impacts. So you know feedback we know is really positive, but we've also seen examples where it's been
negative in unexpected ways. So the rise of kind of triple marking policies where workload was being really damaged you you have less time for planning lessons, you've had less time for quality teaching and there wasn't really an evidence base for that beyond feedback is good. And when we looked at the evidence base for those triple marking policies, it was pretty much non existent. And so again it's that combination of as research as we need to be good, EF needs to be good at looking behind the
average, anticipating those unexpected trade-offs and making sure that we don't just look at impacts, but we also look about how it's received by the profession, the qualitative staff, the barriers, the workload. But then as a profession that we think about how we implement things in ways that are most efficient and aren't creating those those kind of increases on on workload and then similarly small group tuition, you know that can be done really well or really badly. If small group tuition is very
tokenistic, not connected to classroom learning, then it won't have a big impact. And we've we've got a really strong synthesis in the tool kit that shows an overall positive impacts. But we've also evaluated specific programs where we've seen real mixed impact and some are positive, some are null, some are even slightly negative. I suppose in answer to your question it depends what you mean by consistently. There's nothing that is always positive but there are things that that trend towards positive
and a consistently consistently positive or more consistently positive and and for those I'd say, and I'm very much quoting from the toolkit here but but feedback metacognition approaches, although they also have their pitfalls you know they can be done really badly or really well small group tuition and also some of the approaches around reading. So appropriately targeted support for reading comprehension and combined with with phonological approaches, supporting of early reading. I think it's reassuring that
those seem. To be the things that T that schools are focusing their attention on. I mean, I once tried to follow the thread for triple marking to see if I could find the origin. I got to a 2003, you know, State of the Union, Ofsted Inspector or Chief Inspector report. How far back did you get in here? Do you remember? That is a good question. I mean we would have probably hit a blank wolf sooner than that because we'd have been looking one. When we pre specify what we want, we'd have been saying,
well what we want in order for us to be making any claim about efficacy is some kind of comparison group is some kind of attainment outcome. Not just people saying they think it works, but but some evidence of that actually changing outcomes. And once you set your parameters for things that would allow you to make the kind of claims people were making about triple marking, you very quickly find there's nothing. And so I think we, we wouldn't have done the deep investigation you did to see the more kind of
I guess political markers that have got us to the point of it being sad. But very quickly found that the actual efficacy of the approach was was not particularly strong for me it's probably one of the most important. Parts of the conversation over the last ten years or so is that is this idea that actually there are ways we can spend our time that don't involve this but can actually be more impactful. Yeah. And I think you know, we talked a bit about implementation. I think one of the things that that I'm proud of, stuff that the EF hopefully occasionally
does is support the implementation. So it's it's less on the synthesis side, but we've evaluated I think about 200 different programs in the English education system and only about one in four of those has a positive impact and evidence that, oh, you're a bit depressed by that. Surely EF should have a higher hit rate than that. You should be getting lots of positive results. But those negative results or no results are really important because they're not programs that EF have dreamed up. Those are programs that people
are selling to schools and that people are spending their time doing. And when you're finding that they're not having a a positive impact over and above just regular business as usual and people are spending money on them, that really becomes a de implementation challenge. And I won't name organizations for fear of legal repercussions if nothing else. But but but some of the things I'm proud to stop is where we see organizations cease to do as much business as they used to because they've had a negative
result in an EF trial because that is the sign that actually people aren't buying a product that that wasn't improving outcomes. And so I think you know business as usual in English schools is really good. So we should be sceptical that the new program or the new evidence based fad is going to fundamentally change outcomes because what's happening already is really good. And so I think that part of the power of evidence is cutting through those fads and and supporting the DE implementation
process when we might be doing extra things that that aren't having a positive impact. I think that's massive and I'm really. Grateful you didn't mention any names, because the podcast legal team isn't as big. And but yeah, you're absolutely. Yeah, because time is important. Budgets and schools are extremely important in the present. So you know that you're contributing to the reduction. In time. Waste, which I think is really, really important. It's been fascinating talking to you about. Sort of. The work that you do, the the
ways the teachers can benefit from it, Is there anything you think? It'd be great if every teacher in the country knew this and that. You might want to share with them a lot of the key things. That that I've already said. So I think that seeing evidence is something that can support but not remove professional expertise is really important. I guess there's a really nerdy one which is thinking about what a good evidence review is. Is is kind of my day-to-day work at the EF and I've talked a lot
across the podcast on the process that goes behind what we call a systematic review. But one of the things that really worries me is things that call themselves evidence reviews but but don't follow that systematic process. And so I think that one message I'd give generally to the profession is when engaging with things that call themselves evidence reviews being really, really critical of things that don't have some kind of systematic process.
Because what really worries me is evidence becoming a bit of a buzzwords. And the idea that everyone claims that everything is evidence informed and that everything is, we've got great research underpinning that whether it's a paint company or an organization trying to make claims about education in the system. And evidence reviews can either be a systematic process whereby we look at all the evidence in a non bias way and collate
findings or they can be an argument where you say here are my opinions and I'm going to find as many sources as possible to support those opinions. And I think that as we think about what it means to be evidence informed, it's really important that and this goes for for EF findings as as much as anyone else, that the teachers question, when people say this is based on evidence and think about what does that actually mean and what sits behind it. Is there a clear methods, are
there clear limitations that have been listed or is this someone trying to make an argument to to make me do something they want us to do. And I think the more that we can all work together on on questioning those actors, the the better the evidence informed system will be in the long run. Beautiful. Yeah, I'm totally with you and I. Think nerdy is definitely our our MO. And so I think it's a really, really positive way to finish because I know listeners will be interested. But also, you know, that healthy skepticism is definitely, you know, feeds the system and stops
Big Tobacco from making spelling interventions and that kind of thing. And I do remember my curveball. And it's curve. Because you may not have an answer, but I've been thinking recently about the parallels between medicine and education. Did anyone track what the experience of doctors was like? Because we existed in the world where it seems like all doctors have always been evidence informed, but that obviously happened over a not too distant past. Is are there any studies that I
could go to and go, oh, here's the experience, so then it becomes comparable with our research yet experiences and that kind of thing. That's. Really, that is a really interesting question and I don't know of any studies, but I definitely we for example with the What Work? Center for Education, but we often meet with the What Work Center for Health, NICE, the National Institute for Clinical Excellence. And there are a lot of parallels, but there are also places where we shouldn't underestimate how education is
ahead of healthcare. So I think that what healthcare has is lots and lots of strong funding of very rigorous evidence. Lots of RCTs that often come out of a regulatory environment where you need that in order to get your medicine onto the market and you know all of that regulation within a marketplace economy means there's lots and lots of research. But the thing that maybe education is is stronger on is that profession LED voice when
it comes to evidence access. And you know, NICE are actually very good at engaging stakeholders in the process. They'll often have patient groups, they'll have doctor groups. But what I found interesting is, is in some ways the actual conversation between profession and evidence base is more advanced in education. I don't think there's a research Ed equivalent in in medicine. There are obviously conferences, but they're often funded in by big conglomerates or are very linked to to academic
institutions rather than the profession itself. So I think, you know maybe there's a book in it for you to to to do the comparison. I can't think of a point of of reference at the moment or a study that has tracked it, but but there's there's definitely something interesting there about the way that different evidence systems adapt. And I think that you know in education we can learn a lot from medicine. I'm definitely not one of those people that says we just take the whole thing and assume it
applies to education because because they're clearly such different contexts and we should be really proud of some of the things we're doing in education that are actually ahead of the curve when it comes to an evidence LED movement. Nice. I mean that is definitely enough to get to get going and. Like I say, find a rabbit hole and then put off the work I should be doing and looking for interesting things to to read about. And I mean it has been an absolute pleasure and I know that listeners will really value the chance to hear about your work. And all set to do say thank you very much for joining me. Thank you.
It's been. Lovely. And to everyone at home until. Next time, thanks for listening.
Learn more about The Story of Maths and ACEL Pro - www.alta-education.com/tsom-overview and...
Learn more about The Story of Maths and ACEL Pro - www.alta-education.com/tsom-overview and...
Learn more about The Story of Maths and ACEL Pro - www.alta-education.com/tsom-overview and...
Learn more about The Story of Maths and ACEL Pro - www.alta-education.com/tsom-overview and...
Learn more about The Story of Maths and ACEL Pro - www.alta-education.com/tsom-overview and...