Opinion Polls; The Questions Say More Than The Answers
August 5, 2015
by Martin Odoni
The old sitcom Yes, (Prime) Minister can be dangerous when taken completely seriously, but it has a habit of hitting nails squarely on heads more often than not. One sequence in the first season of its incarnation at 10 Downing Street offered a very fair and necessary warning about the dangers of accepting the results of opinion polls completely at face value. It is not necessarily that the pollsters will alter the answers. The far greater danger is their tendency to massage the questions, framing them in such a way as to push the person being asked into a particular mindset.
Take these two questions which are both attempting to assess whether the Welfare State is a good or bad institution; –
“Is it good that vulnerable, sick or poorly-educated people receive financial support from those in society with a significant share of the wealth, or born to greater privilege?”
“Is it acceptable that feckless, idle layabouts are constantly subsidised in their workless lifestyles by hardworking, aspirational people who make a genuine attempt to get on in life?”
The first question is clearly framed in such a way as to draw a sympathetic response. It emphasises the vulnerability of those who depend on welfare, and the possibility that many of their problems are not of their own making, while also highlighting the possibility that many who are well-off may be where they are by accident-of-birth, and therefore have done little or nothing to merit their wealth. Whereas the second question is clearly designed to draw a lot of dismissive answers, by presuming that a predominance of work-shyness and indolence are the main causes of poverty, and that industriousness and aspiration are the main causes of wealth and success.
We need to keep this in mind when reading the average poll in the Daily Mail or the Sun, as they are among the worst newspapers in the country for having a very ‘anti-pauper’ outlook, and so would prefer an ‘anti-pauper’ answer. As a result, they are liable to ask something akin to the second question while taking soundings in the street, but when publishing the results, to print something more in keeping with the first.
And yes, it is entirely likely that the Mirror and even the Guardian will resort to transposed versions of the same trick when polling the public on the same issue.
Polling when done by newspapers, in short, is often just a dirty trick. (More on this subject is discussed in episode 2 of To Play The King, approx 14 minutes in – paid subscription.)
However, what this assessment does not allow for, which I think may happen quite often, is inadvertent massage-of-questions. This suggestion of mine could cause the odd doubtful look, but I do think ordinary, bungling human nature is as capable of making a mistake when wording an opinion poll as it is when wording anyhting else. (TYPO DELIBERATE – just to make the point in reverse.) Opinion poll questions are vulnerable to lack of nuance, after all, and so are human minds.
This possibility came to my thoughts today when studying the results of an opinion poll conducted independently, Jon Cruddas tells us, of the Labour Party’s higher echelons, in a bid to find out why Labour lost the General Election in May. The findings are interesting, and the first conclusion drawn is that the defeat was not a result of the party being perceived as ‘Tory-lite‘. Instead, it was perceived as being ‘Anti-Austerity’, and this was seen by many in the public as irresponsible.
Now, this conclusion may indeed be an accurate reflection of public perceptions, and I do not wish to take up time here assessing whether the perception itself is true (it is not), or whether being pro-Austerity is the same as being ‘financially responsible’ (it most definitely is not). Instead I want a closer look at the question asked in the poll that led to this conclusion. It reads; –
‘Agree or Disagree: We must live within our means so cutting the deficit is the top priority.’
Now, the question itself is somewhat leading, and presents two slightly separate concepts as one. But also, the interpretation of the answer by the Labour researchers leaps to an assumption.
Firstly, the very term live within our means is an economically-meaningless platitude, much loved by Conservative Party ‘toom-tabards‘ when trying to justify cutting spending that is perceived as not benefiting the well-off. In reality, our ‘means’ are limited by the amount of real resources that are available, not by the amount of money the Government owes, and also in reality, there is no danger of insolvency. But as most people-in-the-street appear to have no grasp of these details, we can forgive the term being used in the question.
But secondly and more importantly, cutting the public deficit and favouring Austerity appear, by the way the answer has been interpreted, to be treated as one and the same. Of course they are not the same, and so the opinion poll has been worded as a false dilemma – in effect, it accepts that if the deficit is too high, we must either cut spending or face the consequences of excessive debt.
Whether we think the present deficit really is too high or otherwise, it simply does not follow that an excessive deficit can be brought down by spending cuts, and spending cuts alone. In fact, spending increases that are targeted intelligently can cause the deficit to shrink more reliably, by fiscal multiplication feedbacks e.g. hiring more public sector workers can lead to more productivity, as well as higher income tax receipts and Value Added Tax through the workers’ subsequent personal activity. There is no indication whatever in Cruddas’ assessment that this point-of-view has even been considered in the poll, let alone been asked of those questioned. For instance, the poll could have asked a supplementary question to the one above, something akin to this; –
‘If you agree with the above, do you think that reducing the deficit is best achieved through a) spending cuts, or b) targeted investment?’
But no, there is no indication that such a question has been asked, and the way the majority answer is assessed by Cruddas seems to assume that this is already part of the previous question.
That is maddening, not only because it might lead to Labour getting the wrong idea about why they lost the Election, but also because it might have been a useful way of testing just how economically-literate people around the country really are. My suspicion is that this is a genuine mistake, and not just Blairite confirmation bias against the rise of Jeremy Corbyn, at least not consciously, for the simple reason that a great many people in the Labour Party appear to be no more economically-literate than the general public, an impression that is scary enough in itself.
One thing I can say about the poll result is that it both confirms and denies my recent suggestions at the same time. It clearly underlines, as I have pointed out before, that Labour’s only hope of getting back into power any time soon is to destroy the myths of what caused the Credit Crunch seven years ago, if they are to win back public trust on running the Economy. It also seems to indicate that a move to the left, as I have called for, is not going to be received well by the public, although that depends on there being a public prejudice that leftism-and-anti-Austerity-equal-bad-economics, something that the poll does not really check.
Whatever the case, I would suggest they needed to word the poll more carefully and comprehensively if they wanted a reliable answer.