When school districts invest in instructional technology, implementation is often measured through familiar metrics: How many teachers are using it? How often are students logging in? How many activities have they completed? How much time are they spending on the platform?
Those measures can tell leaders whether a tool is being used. They cannot necessarily tell them whether students are learning.
That distinction became especially clear to me while conducting a meta-analysis of Penda Learning, a computer-based science intervention. Across 13 independent school-based samples involving more than 33,000 students, we found significant gains in science achievement. But because results varied across schools, we were able to examine an arguably more useful question for education leaders: What conditions were associated with stronger outcomes?
One finding stood out: It wasn't simply the amount of time students spent using the intervention that best predicted achievement. It was mastery.
Students were considered to have mastered an activity when they demonstrated at least 80% accuracy. As the number of activities mastered increased, so did the effects associated with the intervention. Schools with low, moderate, and high levels of mastered activities showed effect sizes of 0.56, 0.89, and 1.40, respectively.
Mastery remained the strongest predictor of science outcomes even after engagement hours were considered. Time-on-task alone was not positively associated with better performance.
For district leaders, there is a broader lesson here that applies well beyond any single science intervention: Implementation should not be confused with usage. If we want evidence-based programs to produce evidence-based results, we need to pay much closer attention to what happens after the purchase.
Here are five ways district leaders can move beyond measuring usage and focus implementation on what matters most: student learning.
1. Define Successful Implementation Before Launch
Too often, districts decide how they will measure implementation after a program is already underway. Instead, leaders should define success before students ever log in. Adoption and participation matter, but these are starting measures, not end goals.
Ask more meaningful questions: Are students progressing toward mastery? Are they retaining and applying what they learn? Where are they struggling? Are teachers using the information generated by the program to adjust instruction?
The answers help distinguish between activity and impact. Digital programs make usage data remarkably easy to collect, but leaders should resist equating a dashboard filled with logins, minutes, and completed assignments with successful implementation. The metric that is easiest to measure isn't necessarily the one that matters most.
2. Measure The Quality of Engagement, Not Just The Quantity
Our findings illustrate why this distinction matters. The relationship between mastery and outcomes followed a dose-response pattern: as students mastered more activities, achievement effects grew substantially.
Rather than asking only, “How much did students use the intervention?” leaders should also ask, “What did students accomplish while they were using it?”
Two students can spend the same 30 minutes in a digital learning environment but have very different experiences. One may demonstrate understanding and progress while the other struggles without reaching mastery. Their time-on-task looks identical. Their learning does not.
When students spend significant time on an intervention without demonstrating progress, that should be treated as an instructional signal—not evidence of successful implementation.
3. Connect The Intervention To Instruction
Even a strong evidence base does not make an instructional resource self-executing. Technology works within an instructional system. It needs to connect to what students are learning, the standards teachers are responsible for teaching, assessment information, and the district's broader scope and sequence.
The studies in our analysis took place in real school environments, where teachers differ in experience, schedules vary, and local contexts affect implementation.
In one large implementation examined separately, teachers had a recommended schedule for using the science intervention, but support went well beyond usage expectations. They received professional development around curriculum integration and instructional alignment, along with ongoing opportunities to strengthen implementation and share instructional practices.
The lesson here is that leaders should think about implementation as instructional integration, not software deployment.
4. Support Teachers Beyond The Kickoff Training
Most educators have experienced some version of this: A new resource is introduced during PD, teachers learn how it works, and then everyone returns to their classrooms. That is adoption, but it isn't necessarily implementation.
Teachers need to understand not only how to use a resource but when and why to use it. They need support interpreting data, responding when students struggle, connecting digital learning to classroom instruction, and sharing practices with colleagues.
Variation shouldn't automatically be viewed as failure. It is information. If one school is producing stronger student mastery than another using the same resource, leaders should investigate what is different—whether it's instructional integration, teacher support, or opportunities for students to revisit concepts and demonstrate mastery. Implementation data become much more useful when districts use it to ask those questions rather than simply monitor compliance.
5. Treat Implementation As A Continuous Improvement Process
Perhaps the biggest mistake districts can make is treating implementation as something that happens only at the beginning of an adoption. It should be an ongoing feedback loop in which evidence continually informs practice.
District leaders should absolutely ask whether an instructional intervention has credible research behind it. But an evidence-based intervention doesn't automatically produce evidence-based outcomes. Research tells us what can happen. Implementation helps determine what does happen.
Conversations about edtech effectiveness need to move beyond adoption and usage. Are students mastering what they're learning? Do teachers have the support to act on the data? Are districts creating the conditions for the intervention to succeed?
Ultimately, the goal isn't to get students to spend more time using technology. It's to make the time they spend learning count.
