How To Read An Opinion Poll

The Peter Hitchens classic (emphasis added), if a little generous and unduly self-deprecating:

If I were a billionaire, I'd set up a polling organisation and keep my involvement secret. I'd then offer frequent polls at bargain rates to newspapers and broadcasters. And they would be very different to the ones that are being published at the moment, as we shall see in a minute.

Polls are now the best way to influence public opinion, largely because they're treated (much like the BBC) as impartial oracles of the truth by most people who read them. As readers of the excellent political thrillers of Michael Dobbs (serialised on TV with the incomparable and much-missed Ian Richardson playing the ultra-cynical politician Francis Urqhart) will know, it’s not quite that simple.

Dobbs has one of his characters say (roughly) "The thing you must realise about polls is that they are not devices for measuring public opinion - they are devices for influencing it". And, as a former heavyweight in what used to be Tory Central Office (he was Norman Tebbit's chief of staff when Lord Tebbit was party chairman), Mr Dobbs should know how these things are done. My guess is that political professionals use polls to float ideas, and massage them to try to create swings in opinion from nothing, or amplify small swings into bigger ones, taking advantage of humanity’s regrettable herd instinct and desire to be on the winning side. But what do I know?

There are many other ways in which polls can tweak opinion. The most significant is their power to decide what the question should be, and how it should be asked, and at what point in the questionnaire a particular issue should be raised. The same tricks are useful in a referendum, and in a way a poll is a sort of rolling referendum, which could be used - if someone so choose - to stampede or at least herd British public opinion in directions desired by the elite. I reckon a skilled pollster could devise a survey which could lead respondents into saying in large numbers that the Tory Party was played out, and needed to be replaced. But this is not an answer that's currently much in demand, given the tedious sheep-like uniformity of political journalism, so the required questions are not being asked.

Then there's the question of how you slice the results. This is the most significantly interesting thing about the polls at the moment, supposedly showing a rampant Tory Party chasing Gordon Brown out of office. This is the story a lot of papers want, some to keep the flagging, desperately dull soap opera of Westminster alive (because vast numbers of political reporters make their living out of it), others because they seek a swift transition from Blair One (Anthony Blair) to Blair Two (David Cameron) which of course means the equally rapid erasure of Gordon Brown, the Unsmooth, Unspinnable UnBlair.

As it happens I am an admirer of YouGov and I know and like Peter Kellner, one of its moving spirits. I think they are an excellent polling organisation. And I have only chosen their Sunday Times poll for my examination below because it is the most recent to hand as I write this. I could have said exactly the same thing about all the others. What you must understand is that the pollsters obtain the data (usually consulting with their clients on the questions) and then pass it on to the client. They're not responsible for how the results are displayed.

But even so I would plead that it really is time that pollsters (in their raw charts) and newspapers (in their accounts) gave much more prominence to the enormous group of people who for various reasons will not support the main three parties. One way of doing this would be to lump them together into one, which would immediately show how important they were, and how different public opinion is now from the way it was 25 years ago.

Take last Sunday's poll [in December 2007]. The Sunday Times displayed on its front page a simple, stark bar chart - a blue streak showing the Tories at 45%, a chunky red rectangle showing Labour at 32% and a much smaller yellow rectangle showing the Liberal Democrats at 14%. The story also highlighted a 20% Labour lead among women voters. All these figures are correct and true. But there is quite a lot they leave out.

A much fuller version inside (on page 12, a left-hand page - seen in the trade as much less prominent than a right-hand page, which usually catches the eye first) is the first mention of parties beyond the big three, who are shown at 9% (though my version of the YouGov figures gives 10%. No matter. I'm sure it's just one of those things). 'Don't knows' are not mentioned in the page 12 party political charts at all, though they do get a mention in surveys on "How worried are you about a recession?" , "Do you think house prices will go up or down" and "Will you spend more or less on Christmas than you did last year?". In these, the 'Don't Knows' are a minor group, at 5%, 8% or - on house prices - 10%. As soon as direct party politics enters in, the numbers of the uncommitted shoot up. A fairly chunky 13% answered 'not sure', to a question about whether Gordon Brown is more or less competent than Anthony Blair.

Funny, then, that this set of charts never mentions the number of people who said "Don't know" in response to the "How would you vote?" question. Because this was a thumping 16%. And they were joined by another 7% who said they would not vote, essentially the same thing. That's 23% of the entire sample, nearly a quarter. And I'm not done yet, because the "Headline voting intention" figures used in the pretty bar charts exclude the 'Don't Knows' and 'Wouldn't Votes'.

Does this matter? Do fish breathe through gills? You'll need a calculator here. And you can go to the "Timesonline" website to get hold of the full YouGov figures, as I did, and see if your sums come out the same as mine.

The survey, conducted on the 13th and 14th December, polled 1481 adults, 709 men, 772 women, spread across all age ranges, social classes and national regions (excluding Northern Ireland) in what I assume must be a scientifically designed way. I am using the weighted figures, rather than the raw figures, on the same assumption, that the weighting is scientific.

They found 507 planned to vote Tory, 360 planned to vote Labour, and 159 Liberal Democrat.

Now remember that colourful front page bar chart -Tories 45%, Labour 32%, LibDem 14%?

But....

My calculations show that 507 is 34.23% of 1,481
They show that 360 is 24.3% of 1,481
They show that 159 is 10.73% of 1,481.

Which means (see below for workings) that the raw figures are:
Tories 34.23%
Labour 24.3%
Lib Dems 10.73%
None of the above (other parties, don't know, or will not vote) 30.72%

Rather different, isn't it, from that bar chart? Both are true. But one tells you a lot more.

The 'Others' (shown in the YouGov workings as 10%, but in the newspaper's page 12 chart as 9%) would then presumably (on the basis that each percentage point in the headline figures represents approximately 11.2 people) add up to 112 British subjects.

That means that out of 1,481 people surveyed, only 1,138 (76.8%) had their voting opinions recorded in the paper - and that was on Page 12. On Page One (the results picked up by broadcasters and other newspapers), the 'Others' were not mentioned, so only 1,026 (69.27%) had their opinions recorded in the eye-catching coloured chart. That means that 455 people were not included in the front-page headline figures. That's 455 out of 1,481- 30.72% of the total surveyed, almost as many as say they plan to vote Tory.

Can it possibly be right to ignore such an enormous multitude of the disenchanted? Remember, pollsters have been recording an alleged Tory surge for ages. But it has mysteriously failed (so far) to manifest itself in real life [it still hasn't - I'll come back to that if people want me to]. There has never been any actual evidence of it in any real poll, from Bromley South and Dunfermline to Ealing Southall. A serious examination of last May's local elections likewise fails to show any significant surge, despite extravagant claims, swallowed by too many in my trade, that such a surge had taken place.

So isn't the great army of 'others', 'don't knows' and 'won't votes' exactly the section that a truly curious person might want to highlight, and to question further about what was worrying them? Yet the polls - and the media - just set them to one side and pretend that this is business as usual. As it happens, the fact that such a large part of the sample is discarded could at least partly explain the extraordinary, unprecedented volatility of the polls since last summer. It takes far fewer voters to change their minds to produce a major percentage change. Surely that's important in itself?

There's also been a certain amount of fuss about how this poll shows that 'Worcester Woman', whoever she may be, is switching to David Cameron. As the Sunday Times put it "It appears to be women voters in particular who are deserting Labour. The Tories enjoy a 20-point lead among female voters, compared with six points among men".

Well, again, that's perfectly true. But female 'Don't Knows' also outnumber male 'Don't Knows' by 21% to 11%. Female 'won't votes' outnumber male 'won't votes' by 9% to 4%. That suggests a lot of women aren't actually being seduced by Mr Cameron, or anyone else, the sensible creatures. Also, in the 18 to 34 age range, the Tory lead is only 2% (38 over 36), whereas in the 55+ age range it's 26% (51 over 25), and in the 35 - 54 range it's 43% to 36%. Without the Tory concentration in the over-55s, there wouldn't be much in it, and I wonder how that breaks down between the 55-65s and the 65 and overs. There are also similar class divides (interestingly, the poll surveyed 682 in the lower social classes C2DE, and 797 in the upper ABC1 social classes, which says something about the class make-up of modern Britain. The C2DEs were - unsurprisingly - less Tory than the ABC1s. But they also recorded 20% 'Don't Know' and 9% 'Won't Vote'. Whereas the ABC1s recorded 13% 'Don't Know' and 5% 'Won't Vote'. I think we should consider the possibility that the Labour vote is still soft, but that doesn't necessarily mean that these votes will fall into the hands of the Tories, or remain neutral when an election comes. I'd also like to see some polling on people's intentions for tactical voting, which transformed British elections when it finally happened in 1997.