If you wanted to know how well New Zealand’s education system and Kiwi students were doing, you really wouldn’t want to start with NCEA data.

Getting a comparative assessment of students’ strengths was never the point of NCEA. The whole system seems designed to frustrate anybody wanting a simple statistic gauging a student’s overall performance.

But we need decent measures of student performance if we want to answer even fairly simple questions.

Knowing how well a school is doing depends on knowing how well the students are doing.

Parents need to be able to tell whether their kids are meeting expectations or falling behind.

And figuring out whether university degrees are worth the money depends on having decent measures of student performance at secondary school – if smart graduates enjoy higher earnings, it can be hard to tell whether to credit the degree or the student’s underlying ability.

The problem is not just that NCEA literally has thousands of standards among which students can choose – or that schools can direct students to take. And it is not just that students receive only a very coarse measure of performance – an excellence, merit, achieved or not-achieved.

Higher proportion

But on top of that, a much higher proportion of students receive an excellence in some subfields than in others. In languages, 43% of standards were passed with excellence; in economic theory and practice, only 19% received an excellence.

Since we fight with the data army we have, rather than the one we might want, we have to start with NCEA.

We went into the Statistics NZ data lab with what seemed like a simple question: Has the quality of students choosing education as a major been changing over time?

Declining PISA scores suggest a problem in maths achievement. And one plausible explanation was that fewer numerate students were choosing education as a career.

Tertiary funding had shifted toward science, technology, engineering and maths over other disciplines, and teacher salaries were not really keeping up with the cost of living in Auckland.

And some evidence was cause for concern. One 2012 New Zealand survey showed only 32% of undergraduate students studying to be teachers, and 56% of graduates, were able to add three-fourths to five-eighths. As I am drilling my own young children on basic addition of fractions, this is a bit of a worry.

Numerate teachers

At the same time, changes in how the schools teach maths demanded more numerate teachers. It is important that kids learn their times tables and teaching them is relatively easy. The newer ways of teaching multiplication will give students a deeper understanding of number theory but require teachers understand it, too.

But checking for any change in the quality of incoming teachers is not simple. To do it, you need a decent measure of how well students had performed in secondary school, both overall and in their maths classes. The cumulative number of credits a student had received is not enough. The number of credits in numeracy relevant subjects is not really enough, either.

And NCEA has nothing like a traditional Grade Point Average.

So our excellent econometrician went into the data lab and built a new measure of student performance in NCEA. Rather than count the number of credits, or of excellences, it weighed them. If half the students in a subject received an excellence, that counted for less than an excellence in a subject where less than 10% earned that top score.

Because we wanted the measure to be fit for broader purposes, the measure added up students’ scores across subjects rather than averaging them. If students worry that taking a harder course might hurt their average grade across all courses, they might shy away from the courses that might really stretch them. An additive measure means they cannot do worse for having tried.

Tracking students

We looked at NCEA Level 1 performance for every person in New Zealand who was born from 1988 through 1994. We tracked those students to compare the NCEA scores of those who went on to study to be teachers, those who went on to pursue other degrees, and those who did not continue through to tertiary study.

We found that while students pursuing teaching degrees were a bit weaker than students pursuing other degrees, both in maths performance and overall, that gap declined over time rather than widening.

Since our measure only looks at the flow of new trainee teachers rather than the stock of existing teachers, it could be that older teachers need better support in handling the newer maths teaching methods. But when it comes to the incoming crop of trainee teachers, if those kids aren’t all right, they’re at least getting better.

Better measures of student performance help research. Current measures of school performance are blunt – like the proportion of students completing an NCEA Level 2 certificate.

It is entirely possible to use better measures of student performance to produce far more nuanced pictures of overall school performance that account for the very real differences in students’ backgrounds. Those measures could help principals, school boards and parents tell whether their schools are really doing well.

NCEA might not produce the student achievement data we would most want. And we hope that the coming NCEA review will give some thought to producing better measures.

In the meantime, we can do a lot better with the data we’ve got. And we should.