Techlearning blog

Note: The grade levels and their interpretation that I refer relate to the UK and may be found at http://www.ncaction.org.uk/subjects/ict/levels.htm and http://www.ncaction.org.uk/subjects/ict/progress.htm respectively.

It's common enough to hear teachers saying that what's important is not so much the answer a student gives, but how they arrived at it. Hence teachers of mathematics exhort their students to show all of their working, and here in England we have devised an on-screen ICT test that will capture everything the student does at the computer, and translate the data into information in the form of grade levels. I know: I was employed by the QCA to oversee the development of the questions posed in the tests , and whilst there I developed what came to be known as the "rules base".

The rules base is, at its heart, a model based on Boolean logic. In simplistic terms, it says, "If a student does X, we don't really know what level she is working at. But if the student does X and Y, but chooses not to do Z, it starts to look like she is operating around a level 4. Let's see what transpires in the next part of the test."

Now, I always argued at that time, along with colleagues, that the rules base as translated into a set of logical algorithms for use by a computer could only take you so far: perhaps a level 5 at the most. After that, the behaviour encapsulated by the level descriptions is too complex for a mere machine to make an accurate judgement. Some colleagues in what was then the Assessment division (I was in the Curriculum division) disagreed. Others ventured the viewpoint that what really mattered was the process rather than the product, and that the computer program was very good at measuring, or at least inferring, that.

I was never happy with the idea that process is more important than product, and a simple analogy will explain why. Suppose your boss asks you to produce a report on the impact that the school's investment in educational technology has had on student attainment over the last 3 years. You interview people. You scrutinise data going back 5 years. You look at the published national statistics. And then you deliver your report -- on how the school's sports facilities have improved over the last 3 years.

Is your boss likely to say, "Well, this is absolutely not what I asked for, but the process you went through was excellent so I'm going to accept it. Good job."? Of course not. In this scenario, the process was excellent, the product was useless.

Returning to the idea of assessing students' capability in educational technology, a computerised test will never be able to assess the final product as well as a human being can, even at the lowest levels. All a test can really do is make an educated guess, a guess which becomes ever more guess-like as one ascends the grade levels. That is, clearly, why in England the once notoriously inaccurate (generally speaking) teacher assessment of ICT will still form part of the final assessment when the on-screen test becomes compulsory at age 14 in 2008. Each method informs the other, and so hopefully what will result is a fairly accurate assessment of the student's ability.

The question is: is "fairly accurate" good enough? In my experience of teaching, and also of interviewing job applicants, at the end of the day there is no substitute for asking questions in a viva or interview situation, and assessing the answers that are given, because it's not easy to bluff for 30 minutes or more. The way I see it, that approach should always be the final arbiter. (As an aside, that apporach would work a lot better than software in weeding out student plagiarism.)

In that case, what we need is not more off-the-shelf schemes of work and assessment materials, which I regard as equivalent to painting by numbers, but enough of an investment in teachers' professional development to create many experts. Is such an investment ever likely to be made? Who knows? All I can tell you is that I just saw the whole of London Zoo go flying past my window.