Atten-HUT! Troops to Teachers sees battlefield promotions lauded- but is the science solid?

 I enjoyed Panorama tonight; I always do. There's something so intuitively respectable about the BBC's venerable investigative magazine that I would default to unqualified admiration even if it were to tell me that spaghetti grew on trees. This week: Troops To Teachers (TTT)- Michael Gove's drive to inject a bit of military discipline back into classrooms by aggressively recruiting and retraining ex-military servicemen. It apes the Troops to Teachers program in the US, launched 18 years ago after the first Gulf War, and since then it's seen over 15,000 men and women swap green berets for cardigans with leather patches (or whatever the symbolic equivalent is in America).

If you watched the program you would be forgiven for assuming the the program is an unqualified success; we were treated to the example of Lordswood Boys' School in England, which entertains no less than 1 in 12 staff from  military backgrounds, which shouldn't really be a surprise seeing as how the smallish Birmingham comprehensive has an assistant head who used to be in the Infantry, an ex-Sergeant from the Territorials, and a former sergeant major acting as a shooting instructor. Quite. Still, variety is the spice of life, and one thing that schools have to be praised for is diversity of strategies, trying different things, and adapting tactics to meet the needs of the local community. Looking at the prospectus and the Ofsted report, it seems a bit of a success story. Students like Hakeem Nawas spoke proudly of how it had transformed his self-esteem and motivation to be involved in Cadet activities, and Neil Macintosh, the aforementioned Assistant Head proposed that ex-military were 'more resilient...less down-hearted...and more robust.' As Mandy Rice Davies, said, 'Well, he would, wouldn't he.'

Actually I have no issue with this: in fact I admire many of the principles that inspire it. I particularly liked how the servicemen spoke about how they maintained order- they didn't have to raise their voices, they said. The students agreed. 'They just look at you,' one said. I know what he means. Screaming your head off is usually a sign that you've blown your stack, and for most kids it's better than telly. Speak silently, they say, and carry a big stick. I couldn't agree more. Who do you respect more- the small dog with the big bark, or the silent dog with the claw hammer behind his back? Exactly.

Troops to Hogwarts

Then we were off to Huntingdon Middle School, in Virginia's Newport News City (honestly- I wouldn't make up a name like that because you wouldn't believe it), where a clutch of ex-military had taken over their classes like Desert Storm. The story here was the same, it seemed- lines of biddable, disciplined and enthusiastic students queued up to enter classrooms, and we were presented with crocodiles of marching students who were noticeably not selling crack pipes to grandmothers or auditioning for The Wire. Glee, maybe.

Geoff Lloyd, the poster boy for this school's TTT project spoke proudly about bringing 'discipline into an undisciplined world,' and frankly, I couldn't agree more. His robust, direct attitude to being a responsible adult in a classroom full of students who need clear boundaries and someone they can rely on was more inspirational than a dozen Dead Poets' Societies or Dangerous Minds. I would put him on my fictional Heroes of Education list, but unfortunately he's a real person, so he'll have to content himself with a notional award instead.

So far, so good. As I say, I actually applaud many of the aims of this program. I think that what many of the ex-servicemen said made perfect sense- courage, responsibility, discipline and carrying your own water. Amen to that, brother.

And then- with the inevitability of the Sun rising- came the research. Because that's what we do whenever we want to justify something: we wheel out the academics who biddably endorse whatever is being flogged to us. And that's when it got interesting for me. William Owings of the Old Dominion University sat in an agreeable, respectable setting and enthusiastically waved the flag for the TTT program, his eyes twinkling as he did so. He twinkled a lot. 'Ex-military stay in the profession twice as long as non-servicemen,' we were told. 'Troops in the T2T program outscore all other teachers,' it was said. Owings also provided my favourite quote of the show- T2T had provided a 'stellar performance,' he said. Twinkle, twinkle.

Now that didn't strike me as the careful, cool, neutral perspective of the scientist, I thought. And as soon as someone starts to mention educational research, my spider sense starts to tingle, and frankly I start to sweat a bit. Because, as regulars to this blog will be painfully aware, I'm allergic to the way that some educational research is used to hustle strategies and big ideas that are composed, it is eventually seen, of equal parts moonshine and optimism. As a teacher of some years, I've been making a list of the Initiatives and Great Ideas that the hucksters of education try to flog us, and my hackles start to mambo whenever someone calls along and says, 'Hey, you guys! I have a great new idea for turning schools around! I just need your credit card number and your uncritical commitment...' I'm just funny like that.

So I did a bit of, rooting around on t'interweb. Just what IS the Old Dominion University, anyway? It sounds awfully grand. And it is, I am sure, a paragon of academic vigour, rigour and propriety, even if its mission statement does say that 'Our philosophy is simple: Knowledge should be productive. Research-driven solutions that make sound business sense.' Which isn't really a philosophy, is it? More of an admission that if something is worth something, it has to be worth money. Ah, it brings a tear to your eye, doesn't it? .

As I say, I'm sure it has the noblest intentions. It also has an interesting link to the Troops to Teachers program, as its website says: 'The state office for Virginia TTT is located on the ODU campus.' That's the state office. Of course, that doesn't suggest that the Old Dominion University might be a less than partial witness to the efficiency of the TTT program. I'm just saying, that's all. Isn't that a marvellous coincidence, though?

So I did what few civilians have done before: I had a peek at a couple of the papers quoted on their website as showing terrific, supportive data that confirmed the TTT program as a winner, and the ones that William Owings was quoting so freely on Panorama. You can find two of them here and here. They are, as most social science papers are, a thrill a minute, and I heartily recommend you print them off and read them on the way to work tomorrow. Unless you drive. Or listen to Coldplay while you read them.

The 2005 study was, broadly speaking, a survey of Teachers who had gone through the program, and of their supervisors. It asked if they felt that they had been appropriately trained to approved standards. It also asked supervisors of these teachers if they felt they were as good as, or better than teachers of similar experience who hadn't come through the program. The answer was strongly in favour in both cases. How many were surveyed? A fair few. Over 2000 teachers and their supervisors were sent surveys. That's not a bad study by any standards. Except that the response rate was 65%. We don't know why the other third didn't reply. We don't know what attempts were made to convert those no-shows. We don't know on what basis the surveys were sent. We can probably assume that surveys weren't sent, or at least answered, by teachers who had dropped out of the program.

And it's details like that, that make this kind of research so hard to value meaningfully. Big numbers are good, but without transparency about who answered, what their motives were, and show inaccuracies were avoided, the purity and reliability of this kind of data is always going to be hard to measure, let alone accept. I'm certainly not impugning William Owings, or any of his co-writers, but these are substantial, significant impediments to the development of social scientific research credibility.

Another problem is that this paper relies on perceptions- 'how well do you feel ...' questions. These questions fall short, IMO of the clinical precision and neutrality of the genuinely inquisitive, and stray into the territory of market research. When did you stop beating your wife? Who's to say that the TTT candidates were actually trained properly? What's to prevent the supervisors betraying their own inclinations, preferences and prejudices through their own opinions. Nothing. Nothing at all. This isn't the same as measuring the temperature at which mercury boils- it's like interviewing a series of marathon runners at the finishing line and asking if they feel out of breath.

The paper does acknowledge some of this. Actually, it seems to acknowledge all of this:

'the study does not provide evidence of T3’s self-reported or actual teaching behaviours. Neither does it provide empirical observations of school administrators watching T3s’ actual teaching behaviours. Nor does it provide evidence of students’ learning gains as a result of working for a period of defined time with T3s as compared with other teachers of similar experience. Further study of the actual teaching practices from T3 self-report or assessment of their students’ measured achievement, although very complex and difficult studies to undertake, would provide important information about T3s’ quality as well as feedback about how to strengthen T3 preparation.'

In other words, we know it's all just opinion and self analysis. But we don't think it's a problem. Of course, opinion and subjective experience have a place in analysis; but it's not the same place as objective, viewer-independent data. It doesn't prove anything more than the people who responded felt the way they felt. It's not corroboration that these teachers are better: it is what it is.

The other paper I looked at, from 2010 (and also by our hero from Panorama), focused on TTT candidates who went on to become Principals. This time it was 107 subjects; ah, boo, much smaller. Their supervisors (I didn't even know Heads had supervisors) overwhelmingly (90% plus) said that they thought such principals were better on a variety of scales than similar, non TTT Principals. Yes, you may also find it unsurprising that supervisors, who I assume are involved in the selection and support of these principals, overwhelmingly thought that they were doing a jolly good job, and hadn't they made excellent decisions hiring them? Again, we don't know the conversion rate, the response rate etc.. I'm sure it was fabulous, given that 107 is a very small number. Still, the data comes out rather well, doesn't it?

So is there nothing concrete at all to support the view that TTT candidates have a, if you will, tactical advantage over their civilian counterparts? Not a bit of it. Here it is:

'In a 2008 Florida study comparing measured academic achievement of elementary,
middle, and high school students taught by TTTs, results indicate that compared to all
teachers, students served by Troops teachers performed about equally well in Reading and
achieved a small but statistically significant advantage in Mathematics. In comparisons
where each Troop teacher was individually matched to another teacher, teaching the same
subject in the same school, with approximately the same amount of teaching experience,
students served by Troops teachers achieved substantially and statistically significantly
higher in both Reading and Mathematics (Nunnery, et.al, 2008; Nunnery, et. al., 2009).'

Call me a gutless limey cynic, but 'equally well in Reading' and 'a small but statistically significant advantage' in Maths doesn't exactly strike me as cause to start popping the champagne for the cause yet. Incidentally, the Nunnery paper mentioned above by Owings is co-written by.....yes, William Owings. And it wasn't published in an academic journal, but, as the report says, 'submitted to 'Educational Administration Quarterly
October 2008'. I can submit a poem written on bog paper to the Sunday Times. Does that mean I can say it was printed?   Have a look at the front page. It's got a lovely 'Troops to Teachers' logo all over the front. I'm don't have a Ph.D. in this exact subject, but I suspect that means they might have something to do with the report....

(I stopped reading it at that point, because I don't believe in an afterlife, and I value every precious minute I possess.)

In fact, so do the previous two papers I mentioned, both of which are prefaced by the sentence, 'A Report Prepared for Mike Melo, Director, Virginia Office of Troops to Teachers,' and 'Report to Dr. William McAleer, Executive Director, Troops to Teachers, Pensacola, Florida.' So all of the reports mentioned were written for (can I presume commissioned?) the TTT itself. Hey, waitaminute.....

It's not that I'm against the idea of ex-military training for schools: good luck to 'em, I say. And I think that there might be something in the idea that men and women who have experience with leadership, developing self-discipline and oiling rifles might have something useful to teach children (sniping, for instance). But it doesn't do anyone any good to use research like this that seeks to support proposals with empirical claims that can at the very least be contested as meaningful or verifiable in any real sense. Michael Gove needs to look elsewhere for better arguments, and maybe we might start to take research based policy more seriously.