School CIO: Who’s Afraid of Big Data?
2/26/2014 By: Dr. Steven Rubenstein
You shouldn’t be. Used properly, the numbers can improve education immensely.
First, a confession: I hate big data.
There are too many factors
influencing results to show anything
useful, I often protest. Results of
high-stakes test scores don’t measure
my abilities as a teacher. My students’
abilities and readiness to learn vary randomly from
class to class and year to year, so the data is skewed
and unreliable. I know more about my students
from daily observation than big data can teach me.
Big data is overwhelming, and since I’m untrained
in statistics, I have no idea how to interpret it. Data
can be used to prove whatever you want it to prove.
I suspect that many teachers and administrators
feel the way I do about our increasing
reliance on big data.
And yet I’m endlessly fascinated by data. I
teach AP Literature, and every year I pore through
my students’ scores on the AP exam. I compare
my students’ performance to that of my students
in previous years. I compare their performance to
their grades in class and their scores on practice
tests. I look at how many students improved
over the course of the year and whether that
improvement was reflected in the
final results. And I try to figure
out if I’ve really made
a difference and what
techniques and activities
introduced in a particular year
made an impact in the long run.
I know that even though the data is
sometimes mystifying and will never
provide me with absolute certainty, it
also supplies me with a good way to test
my assumptions about my teaching, reflect on my
practices, and strategize about improvements I
Big data also furnishes important information
about individual students, as well as the strengths
and weaknesses of particular schools and districts.
In the future, we’ll have ever more data points
available for analysis and interpretation, and so it
makes sense that we all start developing effective
systems for maintaining, reporting, analyzing, and
responding to our data.
Our district is making great strides with
our efforts to tame and use big data effectively,
but, like all districts, we are learning as we go
along. A key in the entire process is to create a
culture where the stakeholders—teachers,
administrators, counselors, parents,
and students—are all invested in
maintaining a continuing
conversation about the data and its implications.
To this end, districts need to create a strategy
to address four parts of the process: collection,
reporting, analysis, and plan of action. First, it is
extremely important that it is easy for teachers
and administrators to capture the data. Next,
stakeholders must have easy access to the data,
presented in a way that allows them to quickly
spot trends and patterns. Finally, frequent
opportunities need to be provided where
stakeholders look at the data, reflect on the
implications, and then plan next-step approaches.
The first part, the collection of data, sounds
easier than it actually turns out to be. A few
years ago, we had a real problem in our district:
we had a very expensive system to house all
of our student scores and demographic data.
However, few teachers were actually using the
system. Despite many trainings and professional
development sessions, many teachers did
not take the time to import their benchmark
assessments into the system nor did they run
reports to show the efficacy of what we were
doing. Our data was spotty and incomplete, and
so not particularly useful.
Why weren’t teachers using this fancy system?
Simply put, it was a lot of extra work. Teachers had
to export scores on benchmark assessments from
their gradebooks, follow a complex procedure
in the analytics tool to receive those scores, and
then import the scores into the system. Worse
yet, reports were difficult to produce—one wrong
selection in the report set-up, and no data would
be returned. In addition—and this may sound
dumb, but I know that I’m guilty too—teachers just
didn’t want to remember yet another user name
and password to access one more Web site.
Luckily, we had an excellent working
relationship with JupiterEd, the company that
supplied our gradebook system in our district
a couple of years earlier. The company already
had a strong student information system and had
been thinking about introducing an analytics
component. So we collaborated with them to
produce a system that would meet not only our
needs, but also the needs of other school systems
across the country.
The system now in place has many
advantages over what we’ve used in the past.
The analytics component is integrated with each
teacher’s gradebook. With this system, teachers
can simply connect an assignment or assessment
with the analytics module by clicking a checkbox
and then choosing the right analytics template.
Teachers don’t have to duplicate information,
and they can name their assignments and
assessments anything they choose.
At the same time, administrators receive the
results of the state testing in an electronic format,
which then can be imported into the system. In
our district, we’re importing high school exit exam
scores, annual state testing, CELDT scores, and
even fitness tests. The highest priority will always
be the data that’s used publicly to measure our
schools’ performance against that of other schools
and the data needed to measure our accreditation
goals, but we have a real interest in gathering as
much data as we can.
Once the scores are stored in the system,
there also needs to be an easy process to present
that data in clean and comprehensible reports.
With our system, teachers can run a variety of
reports about their students’ performance. For
example, they can compare student performance
on summative versus formative assessments, see
how their students measure up compared with
other students in the school and in the district, and
compare student performance from year to year.
Because our data warehouse system is part of the
gradebook, teachers can compare a host of data
points to performance on their own assessments.
Administrators can also view student scores
and print reports that track trends and even
compare schools and individual teachers. In our
district, this feature has been helpful in providing
support to the teachers who need to improve
their methods and choose intervention programs
for struggling students.
A system to warehouse and report on data
will be useless if nobody looks at that data. In our
district we pride ourselves on a culture where we
continually strive to improve, and conversations
about our data are now an important part of that
process. During every collaboration—whether it
is a staff development day, a department meeting,
or a grade-level planning session—we spend
some time looking at and discussing the data.
In order for our time working with data to be
productive, all the stakeholders need to be invested
in the process. While our district administrators
ultimately create the game plan and give us
directives, it is partly formulated as a result of
collaboration with teachers and counselors.
It’s also critical for every district to recognize
that there will be resistance to looking at data,
but rather than steamrolling over what might be
legitimate objections, it is important to address
them head on, so that we all understand how our
data might be useful in developing approaches
to improving student achievement. We also need
to address our natural confirmation bias when
looking at data. Study after study shows that
we pay attention to the evidence that supports
our worldview, and we discount or ignore the
evidence that challenges what we think we know.
Often when we receive data that contradicts our
views, we will find ways to explain why the data is
unreliable. It is precisely at those moments that we
have to resist the urge to go on the defensive and
instead attempt to learn from the data.
We also need to provide everyone involved
with better tools to understand the data we are
receiving. When comparing scores, our new
system does indicate whether differences are
statistically significant. However, we all need
better training and guidance on understanding
the more complex issues with determining
the significance of the data. In our district
we haven’t completely solved this issue, but I
suspect that ultimately all districts will need to
create administrator positions for well-trained
statisticians who would assume the role of data
guru and guide.
In the meantime, we’ve found it best to
position our explorations of data around questions
we all want answered. All the stakeholders should
start by figuring out what they need to know and
then locating the right data and determining the
best ways to examine that data to derive answers to
our questions. How do we know if we’re effective?
Are we reaching all the students? What methods
work best? Tapping into the curiosity of teachers
and other stakeholders (even students!) will make
the data less threatening and provide us with
opportunities to experiment with ways to improve.
Once we develop a new plan of action, the process
isn’t over. We will later look at how our new
approach worked, ask new questions, and continue
to refine our methods.
Because of our new abilities to gather
data, we’re entering a brave new world where
technology will give us great insight into student
learning. It’s important, therefore, that we create
systems and protocols that foster a culture that is
receptive to this new knowledge.
Dr. Steven Rubenstein is the English Teacher
& Technology TOSA (Teacher on Special
Assignment) at the Beverly Hills Unified School
Texas district uses real-time data to drive
|NWISD instructors use data from maps to manage campus populations at new and existing schools.
Challenge: Since 2003, the student population at Northwest (TX) Independent
School District (NWISD) has nearly tripled, averaging 1,200 new students each year.
With such rapid growth, the district had to find a way to conduct real-time data
analysis for academic needs, boundary planning, capacity analysis, and resource
placement. To effectively address this, school leadership sought a more effective
way to map out the boundaries and allocation of resources to meet student needs.
Solution: The district selected GuideK12, geovisual analytic software that visualizes
student data on an interactive map to allow for real-time analysis. “The dynamic nature
of the software helps us streamline the planning processes for everything from academic
needs, facilities, and boundaries to looking at wireless access points throughout
the district for the next phase of our 1:1 program,” says Dr. Edward Chevallier, assistant
superintendent of curriculum and instruction. “Being able to query our student data
will allow faster, more effective decisions on aligning resources with the needs in the
district, and [we can] better anticipate future needs to help us serve our community.”
By using GuideK12 to plan for the new building in 2015, the facilities, planning, and
construction department was able to quickly and effectively map out multiple boundary
scenarios. Seeing the data drastically improved the department’s ability to efficiently
manage attendance zones and campus populations at new and existing sites.
How a district used data to improve student performance on assessments
|Calvert County Public Schools can combine historical data with current data
from more than 330district benchmarks.
Challenge: In 2004, Calvert County (MD) Public Schools
wanted to provide quick and easy access to student data to support
informed decision-making across the district. Specifically,
administrators needed to be able to access students’ historical
performance on state tests and current progress on district
assessments so educators could improve teaching and learning.
So the district set out to find a system that would provide an integrated
solution for assessment and data management.
Solution: The district began using the Web-based Performance
Matters assessment and data management system in 2004 in
every school, across all grade levels. With the Web-based system,
administrators can combine historical data (including the
Maryland School Assessment (MSA) and High School Assessment
scores) with current data from more than 330 district benchmarks
administered from pre-K through 12th grade. The district
also includes students’ grades, attendance, SAT scores, DIBELS
scores, and other data in the system.
Since implementing Performance Matters, Calvert County elementary
and middle schools have achieved steady gains on the
MSA. “Before we began using Performance Matters, teachers
didn’t have data that showed exactly which objectives their students
were struggling with or what their misperceptions were.
But it’s the individual objectives and items that make a big difference
in student performance,” says says Matt Poteet, supervisor
of the department of instructional and informational technology.
5 Best Practices to
Create a Thriving
1 Set meaningful long- and short-term
One of the first steps to creating a high-performing,
data-driven culture is to set
specific, meaningful year-long goals.
2 Make it easy to collect and analyze
After the goals are set, close the intervention
cycle by measuring student growth.
3 Uncover and address the causes of
Making observations about the data is
often the first step in data analysis. One
way to do this is with sentence starters,
• I observed that…
• Some patterns and trends I
• I am most interested in…
These sentence starters help teachers
stay focused on what is happening
instead of why it is happening, and build
capacity to make insightful observations.
4 Allot sacred time for action planning.
A key part of developing a data
cycle that works is ensuring teachers
have time designated specifically for
analysis and action planning.
5 Create a culture of collaboration,
In addition to weekly data meetings,
teachers can use data in grade level collaborative
meetings to better address
student needs and share best practices
across a grade level or subject area.
School leaders can also use data to identify
areas for staff coaching and professional
Karina DiGirolamo is the Director of
Curriculum and Instruction for Equitas
Academy Elementary School in Los
Angeles. Her school uses a Web-based
system called Kickboard (www.kickboardforteachers.com).