Today on the podcast we're taking a closer look at a recent study from the American Institutes for Research, also known as AIR, evaluating the effectiveness of Lucy Calkins' Units of Study, and unpacking the results. We're joined by Francie Alexander, chief research officer at HMH, and Dr. Pam Smith, Heinemann's Vice President of Sales and Customer Success.

Francie is an industry leader in the fields of Early Childhood Education, Literacy, and Intensive Intervention for striving reading and math students. Prior to her time at HMH, Francie held key positions in both state and federal education agencies, including serving as Deputy Assistant Secretary for the U.S. Department of Education's research branch, where she led Department-sponsored standards efforts and campaigns in support of libraries and the arts. Her work is also grounded in the classroom, as she has taught students from kindergarten to college.

Prior to joining Heinemann, Pam was an educator for 30 years, having taught grades 3-8 and served as a lead teacher, assistant principal, and principal. As an educator, she worked in districts implementing the Units of Study and has firsthand knowledge of the challenges and rewards of scaling up a new curriculum to transform teaching and learning in a building.

Here now is Pam in conversation with Francie on the takeaways of the recent AIR report.

Below is a transcript of this episode.

Pam: The American Institutes for Research, serving as a third-party in partnership with Heinemann, recently released a study of the Teachers College Reading and Writing Project, Reading and Writing Units of Study curriculum. What's the significance of this study, Francie?

Francie: I am so glad that we are looking at the science of what we already know has gone on in classrooms for decades with the Units of Study, so this is an exciting time. From the very beginning of the Units of Study, they were based on the best evidence that was out there, so they have always been scientific, but the significance of this study, this is the first time we've partnered with a third party, in this case, the American Institutes of Research, as you mentioned.

We looked out there for the best evidence when these programs came together, but now we are subjecting what we've done to the most rigorous science. We are looking at, what we call in that world, a quasi experimental design. So that means we're at the top of the science pyramid, if you would.

Pam: Can you tell us more about the methodology they used?

Francie: This study was a retrospective and it took place in two parts of the country, in the New York public schools, and then we went to a newer implementation for four school districts in Georgia around the Atlanta area. we looked back on evidence that was publicly available and we applied scientific analysis to it.

This is the kind of thing that is done in different kinds of tagging and coding and a very analytical way of looking at this data, and then the data was collected, and then all of these things had to be approached with utmost care and caution because you can't report results unless the science is of a level of rigor that people will understand the results and that they will merit being reportable.

Pam: That is really important to note, but let's get to what folks really want to know. And what folks really want to know is, what did we find out from this study?
Francie: You're right, everybody always skips to the end. And you'll notice if you're reading in a scientific journal, methodology comes first, always, but people always go to the end first because they want to see what the results were. So thank you for letting me go on about the methodology before we got to the results.

Francie: But the results actually should be proceeded by a drum roll, please. Because in the rigorous study as described, here's three things that I'd like people to understand about the results.

One is, in the first year, the Units of Study classrooms and the business as usual classrooms performed about the same. The second year is when the world needs to stand up and take notice. In the second year, the Units of Study classrooms outperformed, in a carefully controlled analysis, the business as usual. So that's something.

In education getting statistically significant results is not easily achievable, and what we learned about it in that second year, and I've shortcutted talking about our teachers college units of studies readers and writers, I shortcut that to Units of Study, so it looks like we're talking about materials, but I want to be really clear, we're talking about instructional resources, but we're talking about professional development too.

Clearly you can hypothesize why was there so much success across demographic groups, statistically significant. These are things we aim for an education. This is who we are in education, is attaining those kind of outcomes. But the significance is, we know that when the teachers have had time to attend professional development, collaborate with each other, that's when the results start to show, manifest themselves.

The third thing we learned, these effects are cumulative. That's where we want to be. We are looking at things longitudinally. We are taking the opportunity . I was a classroom teacher, and I understand fully that what comes first is changing your own practice and then understanding the changes you're making in practice. Of course, teachers are willing to change their practice.

When in the case of Units of Study, these materials have been in use for years, they were based on best evidence, so this is an evidence based, research based approach, that I'm going to be instituting. And then I think an important aspect of it is, and why the cumulative effects are on the positive side, is because one of my heroes in the world of research is John Hattie and he does meta analysis and then looks at what are the things that are most effective in education.

Francie: He talks about something called collective efficacy. When I read this report, when I saw from the no change year one, the statistically significant changes in year two, positive for those students using the Units of Study, and then the cumulative effects in year three, I thought about the interplay between the materials and the professional development and this idea of collective efficacy, meaning all of the teachers in these schools were working on this together, that the teachers weren't in an isolated faction, but they were intersecting and collaborating and talking about what works and what doesn't and improving their practice all along the way.

This bodes well for the continual use of Units of Study. So once you've established it in your practice, once you have become more expert in it, once you have shared and participated and collaborated with your colleagues, you continue to get positive effects.

Pam: Francie, I want to highlight a couple of things you said there. One is, it's very intriguing that the impact was recognized in year two. I know when I was in the classroom, anytime something new was brought in we were always told what it's going to take at least three years before we reap the benefits of implementing. So I wanted to highlight that.

The other thing I want to call out is the connection of professional development. And oftentimes when districts have to cut back on budgets, they choose to cut back on professional development dollars, and I think that's such a disservice to our teachers, but more importantly to the kids they serve.

Francie: In addition to being a classroom teacher, I like to boast I've taught from kindergarten through college, and so I taught prospective teachers too, and when I was on that part of teaching teachers, one of the things that was talked about was how long it takes to make a change and then, in terms of how long it takes to make a change, and then before you can measure the change. So it was quite encouraging to see that in the second year, we could start to observe achievement and in a statistically significant and in a positive way, because you know what I think that also does? There's a psychology to everything. Once you start to see your students gaining in their reading confidence and competence, you're going to want to deepen your practice and you're going to want more experience with this, and you're going to want to discuss with other teachers what's going on and what you're seeing. That gets that conversation going, and I think that deepens the commitment to what you're doing.

Then, talking about the professional learning part of it, and why do, and we've both done it, why you go into teaching? Because probably you love learning and you see being a teacher as part of your lifelong journey as a learner, and you want to be able to continue that.

I always thought one of the major gifts of teaching was to... My job was to keep learning, and so when we see when you support teachers learning, their ongoing journey, then that's only going to continue to improve results. So the professional learning is an important part of it and it also gets away from the notion that some people have, well, just hand somebody the resources and they just arrive on your doorstep. Without learning together about the resources, without collaborating with other teachers, you're not going to get the kind of results that the Units of Study can provide, and the promise of applying science to education in general.

Pam: If you had to share this with schools, districts, what are takeaways for them? What would you say to them? What should they take from this research study and apply in their settings?

Francie: First, of course, support the teachers who are making the changes, and so that is the commitment to professional learning. And by the way in so doing, when working with the school district, while teachers are making changes, we want our partners, our school leaders, to know what those changes are so they know what to look for in classrooms and they know how to offer support.

Having some of the professional learning be co-learning with teachers and school administrators. Again, everybody's in this work together, so I'd like people to take away from that.

Then I would like people to take away the understanding that it does take an investment of time, that this isn't something like turning on a tap of water and the water's there, this is something that you do have to put in the professional learning time, the time for teachers to talk and collaborate together, and the time to communicate with the community and others what's happening in the classrooms and in the school.

Pam: Francie, thank you so much. It's been such a joy talking with you.

Francie: My pleasure. Always good to talk to you.