New Zealand’s education system is designed not to measure performance.
How can we tell whether policies intended to boost student performance are successful, or how much academic achievement matters for later life outcomes, if we cannot measure it?
Our report, Score! Transforming NCEA Data, surveys existing student achievement measures and builds a new one useful for benchmarking student performance.
Performance measures matter.
Universities and employers need reliable measures for assessing applicants.
Parents and students need timely information to help them assess how things are going.
And sound policy also needs good measures of student performance.
The National Certificate of Educational Achievement (NCEA) secondary school qualification was introduced just over 15 years ago. It was designed to cater to students’ differing interests and strengths, providing multiple pathways to a qualification.
Since its introduction more students have indeed left school with a qualification.
However, you cannot tell much about student achievement by looking at NCEA completion rates or the number of credits achieved because all NCEA learning is considered of equal value.
The proportion of students achieving an Excellence grade can vary considerably across courses, from below 3% to well over 50%.
Treating all credits equally, disadvantages:
• Students who choose the more academically rigorous courses yet miss out on academic recognition;
• Researchers and policy analysts who use raw NCEA outcomes in their evaluations;
• Parents using NCEA completion rates as a measure of school quality;
• Employers and universities who need to make selection decisions among NCEA certificate holders.
Challenges with NCEA performance information
NCEA is made up of three levels of certificates usually achieved in the final three years of secondary school.
Students must demonstrate competence in given standards to gain credits towards a qualification.
Any awarded NCEA certificate can recognise credits from potentially hundreds of combinations of standards, from ‘Prepare and serve tea’ to ‘Investigate a given multivariate data set using the statistical enquiry cycle.’ And all the different ways of earning an NCEA certificate are deemed equally valuable.
In a separate report, Spoiled by Choice, analysing the NCEA system, the Initiative argues that this principle ultimately hurts students from disadvantaged communities the most.
In this Score! report, we build a more useful measure of student performance out of existing NCEA achievement data.
Weighted Relative Performance Index (WRPI)
Our WRPI metric considers how a student performed in a standard, relative to all other students who completed that standard.
To illustrate, a student might receive an Excellence in the ‘Perform a solo or duet dance’ standard. This standard is taken by another 1000 students in the country and 800 also receive an Excellence grade.
If only 350 out of 1000 students attempting ‘Apply the algebra of complex numbers in solving problems’ achieve an Excellence grade, this second standard is likely the more challenging of the two.
An existing ‘percentile’ student achievement ranking helps to solve this problem by taking an average of students’ relative performance. A student earning an Excellence in the dance standard is in that class’ top 80% of students, while the algebra Excellence puts a student in the top 35%.
Averaging that percentile score across attempted standards builds a good measure of relative student performance, but can unduly penalise students who push themselves with more difficult courses.
Our WRPI performs a similar calculation while avoiding penalising students for attempting more challenging standards.
The WRPI can help answer many questions
A better measure of educational performance can help researchers tracking the factors associated with better school outcomes as well as the importance of school achievement for later life outcomes. Coarser measures, like the highest attained NCEA Level, miss important variation in difficulty in standards within certificates.
In combination with information about students held in the Statistics New Zealand Integrated Data Infrastructure, the WRPI could be used in a more fine-grained measure of school performance.
We demonstrate the usefulness of our WRPI measure in a case study tracking the NCEA achievement of students enrolling in education degrees. We show that, if anything, the quality of students pursuing education degrees has been increasing. Consequently, declining teacher numeracy is unlikely to explain worsening student achievement in international measures of student numeracy.
Click here to download the two-page report summary of Score! Transforming NCEA Data.
About the authors
Dr Eric Crampton is Chief Economist with The New Zealand Initiative. He served as Lecturer and Senior Lecturer with the Department of Economics at the University of Canterbury from 2003 through 2014. He is co-author of several Initiative reports, including In the Zone and Decade of Debt. He holds a PhD in economics from George Mason University. He is author of the popular economics blog Offsetting Behaviour. He also teaches public economics at Victoria University of Wellington.
Martine Udahemuka is a Research Fellow at The New Zealand Initiative working on education policy. She is the author of Signal Loss: What We Know About School Performance, Fair and Frank: Global Insights for Managing School Performance and Amplifying Excellence: Promoting transparency, professionalism and support in schools. Martine has also published on migrant and refugee integration in New Zealand. She was a student success advisor and learning consultant at Massey University, and was with the Aid Programme at the Ministry of Foreign Affairs and Trade.
Martine holds a Master of Arts in Industrial and Organisational psychology with First Class Honours from Massey University.
