techLEARNING.com | Technology & Learning - The Resource for Education Technology Leaders

The idea of imposing a scientific research model on the elementary school reading classroom is striking fear into the hearts of many educators. Here, an expert demystifies and analyzes the various methodologies.

In order to qualify for funding under Reading First, all reading curricula, print- and technology-based, will need to show research-supported effectiveness.

"Technology is expensive and the electorate is getting restless. We—especially superintendents—must learn to measure the benefits of education." —Jon Bower

The gold standard of such research is a "true" experiment that requires random assignment of students to either an experimental treatment group (who receive the new instructional approach) or a no-treatment control group. A pretest is given to both groups, then the experimental students receive the new treatment, then both groups take the posttest. The test score data are subjected to statistical analysis (typically an analysis of variance) to determine whether, on average, the experimental treatment students demonstrated statistically superior achievement performance than the control students.

Despite the fact that the DOE seeks scientifically based research supporting the effectiveness of reading curriculum materials at the high level of true experiments, most K-12 research does not and will not follow a true experimental design because random assignment of students to treatments is not feasible in most schools.

Practical alternatives to true experiments are "quasi-experimental" designs. Quasi-experiments are like true experiments in that they involve comparison between an experimental treatment group and a no-treatment control group. However, in quasi-experiments, student subjects are not randomly assigned. Two respected quasi-experimental designs are:

It is highly likely that federal and state Reading First regulations and guidelines will accept this type of design as meeting the standard of scientifically based research.

In this design, the no-treatment control group would be an older group of students who received instruction before the new treatment was implemented, and the experimental group would be a younger group of students who received instruction after the new treatment was implemented. The historical academic performance of the older group (when they were the same age/grade as the younger students were during the treatment period) is compared to the current academic performance of the younger group.

While cohort designs are generally accepted among education researchers, we haven't read nor heard anything to suggest whether federal and state Reading First regulations and guidelines will or won't accept this type of design as meeting the standard of scientifically based research.

Additional federal criteria for scientifically based research stated for the NCLB Act Title I Reading First program (Section 1208) are as follows:

(A) applies rigorous, systematic, and objective procedures to obtain valid knowledge relevant to reading development, reading instruction, and reading difficulties; and

Many educational technology publishers are in the process of reviewing the research that has been completed on their products and services to date to determine whether the research is likely to meet federal or state guidelines eventually developed. Many are also preparing papers that establish the research base for the underlying pedagogy of their products. Other publishers are developing research plans.

The challenges that educational technology publishers face are that this research is expensive and takes one or more school years to complete.

Jay Sivin-Kachala is vice president of Interactive Educational Systems Design, Inc., which offers a variety of services related to the evaluation and development of educational software, multimedia products, and Web sites.

How Administrators Can Support Research in Their School or District

Put systems in place. District administrators should be prepared to bear major responsibility for program and materials selection, professional development, and ongoing student assessment and program evaluation under Reading First.

Line up qualified judges. Since the materials selection process will have to include examination of the research on each candidate product's effectiveness, there will need to be a district staff member or committee (or outside consultant) qualified to review the research and make decisions based on it.

Train your staff. Administrators will have to be clear about the implementation assumptions of the effective products they select, and then plan and implement professional development to ensure that teachers can meet the implementation assumptions (e.g., about the skills the teachers will need and the procedures they'll be expected to follow on a regular or periodic basis).

Develop a plan. Administrators will need to develop a plan for ongoing student assessment and program evaluation. To the extent that teachers will be required to play a role in student assessment and program evaluation, they will need sufficient professional development related to these procedures. For overall program evaluation, districts might create their own instruments or work with an outside research consultant or institute to develop and implement an evaluation plan. The evaluation should answer the district's key questions about the program (related to student achievement and other impacts). We recommend that it include quantitative analysis of student achievement data (using one of the designs described in "The Lowdown on Scientific Research and What It Means for Schools") and also qualitative data collection and analysis (e.g., from classroom observations, student and teacher interviews, surveys). The evaluation should be able to explain what impacts the program had, as well as the "how" and "why" of these impacts. The evaluation should also include recommendations for improving the program.-JS-K