Tuesday, March 1, 2016

More to the PISA scores

Many people try and use PISA test data to infer that there is a problem with Math education in their area.  I have seen this argument used in many blogs, papers, and social media outlets.

Initially, some people will show fancy graphs and try to convince others that the drop of PISA results has actually been caused by a change in curriculum.  Later they will conclude that unless we dig up some old curriculum these scores will continue to drop in the area of mathematics.

First, any arguments, at best, truly show that there is only a correlation between the type of math curriculum their area has and a drop in PISA data.  All arguments (which I have seen anyways) always fails to show causation.  What is the difference?

Correlation is when two or more things or events occur near or around the same time.  These things might be associated with each other, but are not necessarily connected to each other by a cause/effect relationship.

An easy example is when people get a cold during the winter months they usually end up with a runny nose and a sore throat.  These two events are correlated but we cannot conclude that a runny nose will actually cause a sore throat to occur.

Most arguments around PISA, tend to show some sort of data analysis and link it to a change in curriculum, and again this would be correlation, at best.  However, we can take a closer look at the data ourselves...

If we look at the Canadian results of PISA by province, we see that every single province dropped from 2009 to 2012, other than Quebec and Saskatchewan.  On the international level, Netherlands, Belgium, Australia, Denmark, New Zealand, France, and many other countries fell at similar rates to Canada.

If there is a Math Crisis in your zone, then there must also be a Math Crisis around the entire planet? Does this sound likely?  I would hope not!

Some "back to the basics" folk will ignore the fact that they are implying here is an International Math Crisis and simply tell us that:
PISA tells how well the math curriculum is in a certain area compared to other parts of the world.
Well here are some stats that show how false that statement is:

Shanghai had the highest score on the 2012 PISA test with an average of 600.24, while Australia had a much smaller average of 515.01.

What can you conclude?  That Australia's math curriculum is much weaker than Shanghai's?

If we look at students who were born in China and moved to Australia before ever entering school, their average on the 2012 PISA test was 614.77... 14 POINTS HIGHER THAN SHANGHAI!!!

This must mean that Australia's math curriculum is superior to Shanghai's?  See the problem?

Using PISA scores, alone, to determine the quality of education in a province, state, region, or country is similar to judging someone's ability to drive simply by only watching them parallel park. While this single test can make many great observations around education on a global scale, we need to realize that it is exactly that: a single test.

There are much more variables at play in an Education system than simply the results on a test that some countries value more than others.  If you attend school in Scotland and are called to write PISA, you will be forced to watch champions winning Gold Medals for their country and informed that you have the ability to "bring home the Gold for Scotland".

If you attend school in China and are called to write PISA, your name will be broadcasted and people will cheer you on as you walk into the testing room.  The test you will write will be similar to the test preparation you have received over the previous months.

If you attend school in Canada and you are called to write PISA, you will be quietly removed from a class, brought to a room to write a test you know nothing about and have had no formal preparation for.

So I ask again, does a drop in PISA scores really mean a math crisis?  I think not!

Lastly,  we need to understand that a lot of arguments against current math practices are actually attacking teachers and not curriculum.

Sam Sellar- Globalizing Educational Accountabilities
PISA Key findings