Leaning Towers?

Nick Pearce and that erstwhile Boy Wonder, Jonathan Clifton write:

Every three years the education world is thrown into turmoil as the media report findings from the latest PISA study. This time is no different, with reports that England is ‘lagging dramatically behind’ other countries and calls to align our entire exam system with that of PISA.

PISA is a phenomenal resource. First developed in 1998 with 28 countries taking part, it now covers 65 education systems, making up the vast majority of the world economy. It takes a reasonably large sample of 15-year-olds from each country and asks them to sit a common test. In this way, it provides a useful snapshot of the performance of children from different countries.

It is particularly useful given the current vogue for policymakers to cherrypick findings from international case studies – be it free schools in Sweden, charter schools in the US, the national curriculum in Hong Kong or the exam system in Singapore.

Unsurprisingly, newspaper editors focus on the first volume of PISA findings, which ranks the performance of different countries and how they have changed since the previous study. This shows that Asian school systems – such as those of Shanghai, Singapore, Taiwan and South Korea – outperform those in the west.

Australia and the US have seen a relative decline, while England’s system remains largely unchanged at OECD average. Interestingly, the Nordic countries which once dominated the tables now seem more precarious.

But these rankings have to be treated with a healthy dose of caution. A large number of the differences between countries are not statistically significant, and their scores are generally very similar, which means a change of just one or two points can lead to a country’s relative ranking fluctuating wildly.

There is also only so much that can be tested for reliably in a two-hour paper exam taken by a sample of students once every three years. And the way tests are matched with the curriculum of each country can explain some of the results (note, for example, that the UK scores much better on the other international test, ‘TIMMS’, which is designed differently and fits more closely with our own curriculum).

What is more useful to policymakers is the fourth volume of PISA findings. This analyses data at an aggregate level and give pointers about what is likely to drive improvement in a school system. Some of this is blindingly obvious: quite why the finding that ‘systems with larger proportions of students who arrive late for school and skip classes tend to show lower overall performance’ deserves a bold headline in the executive summary is unclear. But the majority of findings are useful for giving a sense of direction to policymakers.

These findings make uncomfortable reading for some of the vanity projects of the right. For those like Boris Johnson who support more grammar schools and selection PISA is clear that: ‘a school system’s performance overall is not better if it has a greater proportion of academically selective schools’. What’s more, it shows that selection is bad for equity and that achievement is the result of high expectations and hard work, not innate ability.

PISA is also dismissive of the claim that competition is the way to drive up standards, showing no correlation between school competition and attainment. In fact ‘only in the Czech Republic and Estonia do schools that compete with other schools for students in the same area tend to perform better, on average, than schools that do not compete’.

And those on the right who advocate allowing private providers to run state schools need to answer PISA’s conclusion that: ‘across OECD countries and all countries and economies that participated in PISA 2012, the percentage of students enrolled in private schools is not related to a system’s overall performance’.

But there is also uncomfortable reading for some of those on the left who advocate a return to schools being controlled and managed by local authorities.

PISA shows that ‘schools with more autonomy tend to perform better than schools with less autonomy when they are part of school systems with more accountability arrangements and greater teacher-principal collaboration in school management’. Giving schools freedoms over curricula and assessment can be a good thing – provided it is matched with sound collaboration and management.

PISA is also clear that simply calling for more money to be spent on education is not sufficient to raise standards. While high-performing systems do pay teachers more, there is a wide difference in which countries get the biggest bang for their buck. Careful thought needs to be given to raising productivity in education.

There is therefore a danger that international evidence is used to fit existing ideological positions, and both the left and right need to be honest about what PISA actually shows. There is also a danger that the media storm following the publication of PISA is used to justify kneejerk changes in education policy. We saw this yesterday with calls for England to prepare its students for the PISA test specifically, in a hope that we can teach children in a way that will boost our position in the rankings.

Rather than implementing reforms in response to panicked media reaction (and with the dubious claim that climbing PISA rankings is an end in its own right), we need a more mature response to the benefits that international benchmarking can bring to our education system. IPPR has called for a more systematic use of international evidence in education policy, on a timescale that is detached from the political cycle.

This has been done in some of the very countries that PISA itself lauds. Hong Kong and other east Asian countries, for example, have teams of educational specialists who rigorously compare everything from textbooks to exams.

Germany has also developed an effective approach by drawing inspiration from industrial benchmarking that took place in the 1980s, when manufacturers were keen to learn from their new competitors in Asia. It has established a whole institution – the Institute for Educational Progress – to carry out a similar sort of benchmarking in the school system.

Unlike these countries, England does not yet have an institutional framework for assessing and implementing international lessons on school reform. We need international benchmarks to be fixed into the English system, to ensure that evidence from other countries isn’t used in a selective way.

Without this, we will be left with ad hoc comparisons that are of little use.