Saturday, 21 January 2012

The Bones Have Spoken: Is Value-Added Crystal Bollocks?

It's gospel, mate.
NICK GIBB  HAS HIS BALLS OUT TODAY! Calm down, major, his Crystal balls. Today's piñata is the Great Satan of Fischer Family Trust data, and the lesser demons of value-added and predicted grades, and boy am I going to beat the HELL out them.

He's- quite rightly- spoken out against the gamification of league tables, where schools, in an attempt to meet the success criteria dictated to them, put their shoulder to nothing but those criteria. Every teacher knows about this- intervention classes aimed at C/D borderline students; not entering the hopeless for final exams; press-ganging children into high-value BTECs for point score advantage, and so on. It's evil, but perhaps understandable when the stakes for schools are so high; let your slip show on the league tables, and you might as well load all six chambers of your gun with dum-dums and press the muzzle to your temple.

So what's the Funky Gibbon proposing? Stand easy citizens- schools will be exhibiting their Contextual Value Added scores from now on, not unlike a baboon, presenting its ghastly floral undercarriage. Gaze into the abyss, for it cannot be unseen.

Pick a Card...any card

Now I have a problem with CVA. Not me personally- my CVA is, thankfully, bulletproof, fireproof, and susceptible only to Kryptonite, and I only say that so you don't think I'm a bitter victim of its diabolic engines. I just don't think it's that useful. In fact, I think the way it's used, it's corrosive, and actively damages education.

In many ways picking a fight with predictions is an easy task, because I'm attacking the belief that we can tell what is going to happen about things that have not happened yet. Can you see where I might be going with this? Nobody can tell the future, not even with a great big telescope and all the data in the world. Not even then. While science has offered us many sweet meats and shiny trinkets in the fields of the natural world, it has yet to lift its petticoat in any meaningful way in the realm of something more stubbornly unpredictable: us. Human beings resist the reductive powers of physical determinism; we just won't do as we're told. This is why we are human, and not, say, a Meccano Set.

The problem is that schools are- now, at least- very much in the business of predicting the future. Why? Because..well, the simple answer is because Lucifer the Hoofed one rules this world, but that won't get me into the smart edu-blog clubs. The other answer is that we are required on an annual basis, to show that children have made good progress, not merely an A, or a B, but progress; that they have become smarter. The hope is that it's something to do with us. How do we show this? By comparing what they get to some notional projection of what they 'should' have obtained. And that's where the problems start.

If Little Davina or Limoncella start year 7 with a level 5 in English and Maths, then we know that's good- probably better than most of her peers. So you'd hope she'd leave with a bag full of A's at GCSE wouldn't you?

Your data manager, yesterday.
Most schools (and by most, I'm suggesting 100%) use Fischer Family Trust (FFT) or ALIS data to set targets for pupils. What am FFT? They're an organisation that sell data; they take the results of all the children in their data set, and then track the % of those children on, say, level 4 at the end of Key Stage 3, to see how many of them achieve an A or a B or a C at the end of their GCSEs. They then plot statistical projections of likelihood that Davina will get those grades.Sounds simple. On this level, it is. Then the problems start:

Here are SOME of the problems; I'll deal with more in a subsequent post:


1. Nobody but wizards can understand how CVA is calculated. Do you know how many factors are taken into account when constructing the predicted median for a school's grades? Here are, I think, the key ones:
  • pupils’ prior attainment
  • special educational needs;
  • English as an additional language;
  • pupil mobility;
  • age of pupils;
  • an ‘in care’ indicator;
  • ethnicity;
  • free school meals status; and
  • a measure of social deprivation.
These are then thrown into the tombola of certainty, the handle is turned, and the future is born, neat as an egg. When Einstein published his theory of Relativity, it was alleged (probably erroneously) that at that point, only a dozen men and women in the world could fully understand it. The number is even smaller for CVA calculus. It is a done deal; the High Priests have spoken, and we must genuflect to the wisdom of the ancients. I'm not saying they're lying or anything, I'm just saying I have a gut mistrust for any pupil prediction that can be obtained with NO KNOWLEDGE of the pupil whatsoever. This ties into my next point:

2. It's de-professionalises the whole role of the teacher. Excuse me? You want to say what one of my students is probably going to get this year....and you haven't even met them? You say that you DON'T sit in the classroom with them every day, working with them, talking to them, marking their books and correcting their mistakes? Well I do. I know my students. I can see their potential. I have my own data set I base predictions on, and it isn't one I can write down or reduce to an algorithm. It's in my mind, and my gut, and it's part of what being a teacher is about. My contempt for the practise of reducing children to points on a scatter graph is endless and bottomless. It could boil iron.


3. They're NOT predictions. I can't emphasise this one enough. EVEN THE FFT DOESN'T THINK THEY'RE PREDICTIONS. They are wise, the wizards of the Fischer Family. This is a common problem: often, research is published in any field, with sensible, delicate, cautious conclusions, only for the recipients (in this case, politicians, journalists and school leaders) to go OH BOY LOOK AT THESE COOL PROPHECIES WRITTEN BY THE WIZARDS IN THE MOUNTAIN. THESE ARE OMENS FOR SURE LOOK THE BONES HAVE SPOKEN.

Here's what the FFT actually have to say:

'The FFT Data Analysis project produces ESTIMATES of likely attainment. The estimates are calculated for each pupil and, from these, school and LA estimates are calculated. They are called estimates – not predictions or targets – because they provide an estimate of what might happen if your pupils make progress that is line with that of similar pupils in previous years.'
NOT predictions. NOT targets. That's because they're professionals, who are more than aware that we cannot tell the future. They're statistical guesses; they're probability fields; they are the statistical equivalent of saying that Summer will probably be hotter than Winter, but we don't know if it'll rain today or not. The comparison with the weather is appropriate: we know what water is; we know what heat is; we know what pressure is; but the enormous density of causal factors makes weather forecasting impossible for more than a few hours in the future. The Met office gave up long-term forecasts a long time ago. And humans, as I repeatedly point out, are a Hell of a lot more complex than a damp bag of warm gas in Brownian motion. (Most of them- I saw TOWIE once)
'What band should we use, oh omniscient one?'

So when the FFT says that a given child is estimated a B at GCSE, based on prior attainment data, once social and circumstantial factors have been accounted for, what does it mean?

Almost nothing. Almost nothing.

What it means is that many children with similar socio-economic and attainment levels achieved that grade. So what? Most children in Mozart's street didn't grow up to write The Magic Flute, but he did. Most children from Omaha, Nebraska didn't grow up to lead a black consciousness movement, but Malcolm X did. I taught a kid who scraped a C in bottom set RS, who scored a U in AS, and then an A at A2. The human spirit is a genie; it is absurd, noetic, a screaming eagle of ambition and indeterminacy. It is a ghost, a comet, a nuclear furnace of optimism and ambition and impossibility. It is also a disappointment; the anti-life, failure snatched from the jaws of victory. House prices can go up as well as down. That is what makes being alive so glorious and terrifying.

I have a knowledge of my children's predicted grades that approaches telepathy, because I know my subject and I know my kids. But every year I am knocked sideways by kids who exceed my expectations and those who ridicule them. Nobody can predict the future. Guesses are fine, but let's admit that's what they are.

Guesses.

Let's stop pulverising children with our bureaucratic assumptions about their potential. Can you imagine what it must feel like to be told by your teacher that your prediction is a D?

F*ck. That.

You know what my expectation of my children is? An A. For everyone. That's the target I set myself, and if I don't get it, well, I try again next year. I don't cry into my coffee, I just try again.

Here's a thing: what does it even mean to 'aim for a C, or a B'? Have you ever seen a kid revise, and try to get a B? It's nonsense. Kids try as hard as they can/ can be bothered, to get the best grade they can. If you set a child to run 100 metres, and they really bash their guts out on it, can you imagine asking them, 'What speed were you going for?' No. They just run. They just run. Target setting has become the fetish of 21st century teaching. It is another ravenous, ridiculous imported imaginary animal from the paradigm of the market place, where ambitions are plucked from the air- and they are- and called 'predictions', when they should be called 'hopes'.

When I worked for the Great Satan of commerce, every year our units were given sales targets to reach. Meeting them, would ensure we survived. Failing to meet these targets was an assured spell in the cooler. The targets were usually 'Last year's sales, plus 5%'. Brilliant. How long did you think it took for them to come up with that?

Aggregating outta Here

The market has infected the classroom, and I use the word infect carefully. It is a sickness that cuts down children, teachers and learning. It is already absurd that the economic model is predicated on infinite expansion, in a world that is plainly finite; it is doubly absurd to do so in a room full of children, nenulus and spectral a commodity as you can imagine, in an environment where they are learning, an abstract multiplied by an abstract. The Data Oracles pretend they are dealing in beans, when they are counting intangibles. They are trying to catch a fairy tale with an iron claw.

I have no idea what they have in their nets. But it isn't real.

I'm not finished with you, CVA *looks sternly at it*. Keep looking over your shoulder.

UPDATE: In response to some comments to this blog post, I've added another supplementary post to it here. I haven't changed this blog, because I think the discussion makes more sense if I jes' leave it as it is.

26 comments:

  1. You've nailed it thereTom, excellent reading. I was in a mid level leaders meeting the other day that was dominated by 'value added' talk while outside the kids were lining corridors booing and laughing at staff. All I could think was 'is this the job?'. This post is reasuring. Thanks.

    ReplyDelete
    Replies
    1. Glad it helps, somehow. Value added can polish my plums, frankly.

      Delete
  2. You've nailed it thereTom, excellent reading. I was in a mid level leaders meeting the other day that was dominated by 'value added' talk while outside the kids were lining corridors booing and laughing at staff. All I could think was 'is this the job?'. This post is reasuring. Thanks.

    ReplyDelete
  3. "Target setting has become the fetish of 21st century teaching."

    That's the only sentence in the whole piece I'd slightly disagree with. Target setting has become the fetish of Government, and thus of SLT. And Ofsted. Dear God, of Ofsted. I don't know a single classroom teacher (yes, OK, SLT do a bit of teaching in their spare time, but you know what I mean) who thinks it's anything but b**ll**ks. And yet we have to go along with this rubbish.

    ReplyDelete
    Replies
    1. HOW DARE YOU DISAGREE WITH ANYTHING I HAVE WRITTEN.

      No, you're right: this never was born of a teacher. It was invented in a laboratory, 'a child without a father' as Bentham said of Rights.

      Delete
  4. As someone who works with young people in the post-16 environment, I wholly concur. My little cherubs are apprentices. They are going to be working in the big bad real world and have to grasp that fact pretty quickly.

    Of course, some of them are fine and have the ability, maturity and attitude to be successful from the start. Some of them are never going to make the grade and, after being told that a BTEC First is "equivalent to" 4 GCSE passes, are stunned that they have been sold a crock and that life really *isn't* fair. But there are some who, after a bit of a rocky start - maybe it took them several goes to turn up at an interview, but they managed to secure a job and get to work almost on time and realise that spelling and punctuation is actually a big deal - go from tentative, toddling steps to being able to pick up the ball and run with it.

    And I, I need to know the inner workings of each of them. Which ones are just need pointing in the right direction and a gentle shove off, which of them need stabilisers for a few months and those for whom independent locomotion is never going to happen. I have to identify what support mechanisms we have and just how to apply them before anyone gets more than a bruised knee.

    At least, in my world, there is no FFT and no SVA, only the new SASE regulations and some idiot Ministers who don't understand the concepts of individuality and independence. But there are a bunch of newly fledged independent young people with qualifications and experience.

    ReplyDelete
    Replies
    1. Thanks for the insight- teachers need to be more than delivery men/ women.

      Delete
  5. I work in a "value adding" school. Coming from the land of Oz, I didn't get what this is all about. Thanks for lifting the scales from my eyes.

    ReplyDelete
    Replies
    1. Would that the scales didn't need lifting....Thanks.

      Delete
  6. I too struggle with the impersonal nature of this type of data, but there is at least one positive effect of the discussion it provokes: it challenges low expectations and shows kids what is possible. If treated with caution and used positively to motivate, it can be a powerful tool.

    ReplyDelete
    Replies
    1. Now that I do agree with. It can indeed- it can prompt greater effort in those that teachers have written off. Sadly, I often find it does the exact opposite.

      Delete
  7. Brilliant post, as ever. To anonymous above...

    For every positive effect challanging low expectations in some pupils there is an equal and opposite effect on some other pupils. The whole thing is a crock.

    ReplyDelete
    Replies
    1. 'No man is an island.' Thanks. Incidentally, is it Mr Iron D Animal? Or 'Iron'd Animal'? I often wonder.

      Delete
    2. All one word, and it rolls off the tongue as such :-)

      Delete
  8. Has my computer gone funny, or have you rejigged your site layout? If the latter, may I be the first to say that it's great? I know it's massively popular, but I find reversed-out text hard to read without eye-strain - and it's easier to find each of your blog entries now.

    Definitely Outstanding!

    ReplyDelete
  9. @ Sue

    Yes, I pressed the 21st century button at the top, and now it's showing 10001 on the CVA score or something. Glad you like it. I miss the German Bible in the background, but plus ca change.

    T

    ReplyDelete
    Replies
    1. As data manager for my school I hate setting targets and agree with everything you said but I was under the impression CVA was abolished last year:

      http://www.tes.co.uk/article.aspx?storycode=6086535

      Cheers,
      Alex

      Delete
    2. It certainly was- first proposed in the November White Paper a few months back, and now apparently a done deal. Still Value Added has taken its place, which is in many ways the same thing with a funny hat. See my next blog for development of this theme.

      Thanks for the comment.

      Delete
  10. Very good points, and isn't the use of ethnicity as a measure especially dodgy? Since you mentioned Malcolm X, it reminded me of his description of a conversation with one of his teachers, who told him (apparently meaning well) that he could never be a lawyer - 'that's no realistic goal for a ******'.
    (from http://www.spartacus.schoolnet.co.uk/USAmalcolmX.htm)

    Now obviously this is a far, far less extreme example, but it seems to be on the same continuum of low expectations.

    ReplyDelete
    Replies
    1. Huge agreement. Under the guise of balance, I found it ferociously patronising and in many ways quite sinister.

      Delete
  11. Thank you for saying what I have been thinking and feeling for a long time. I get shot down in flames at school for daring to question the data and targets that nobody knows how are created but everyone believes are true

    ReplyDelete
    Replies
    1. Dare to question it! If we don't, the bean counters win....

      Delete
  12. Thank you for saying what I have been thinking and feeling for a long time. I get shot down in flames at school for dating to question the data and targets that nobody knows how it was created but everybody believes is true

    ReplyDelete
  13. It takes a brave person to say 'the emperor has no clothes' in the current climate in schools, My school is totally obsessed with the whole thing of predicted grades, CVA, whose predicted what? how are they doing compared to FFT data? etc, etc. It is relentless. In addition to an assistant head who spends so much time on it, posts of responsibily have been created for each key stage plus for Eng, Maths and Sci to collate all the data, identify the predicted under-achievers etc. We have a directory called 'The War Room' in the staff shared area. There you can feast your eyes on many colour coded spreadsheets and the like. I have my doubts about whether it is adding to education in the broadest sense.

    ReplyDelete
  14. Is the War Room made of paper? Do you think....do you think it could BURN?

    This is satire. I am not implying that you should incinerate the War Room. I am merely mentioning it without prescriptive content. It's a thought experiment.

    ReplyDelete
    Replies
    1. We too have a war room! I tell all my children I am aiming for an A for them whatever formula target has been worked out. Effort matters, get used to it.

      Delete