Sunday, October 19, 2008

Polling problems

A few days ago we saw that rare thing in Welsh politics: the results of an opinion poll on Welsh voting intentions, taken in September by Beaufort for Plaid Cymru.

Asked how they would vote in both a general and assembly elections, the results for those certain or very likely to vote were thus:

Plaid Cymru described the poll results as showing "...that it is Plaid as opposed to the Conservatives who are benefiting most from the decline in Labour support in Wales."

However, at this point we should heed the words of Welsh poll expert Professor Richard Wyn Jones:
“Opinion polls in Wales are strange things. We don’t have many of them, but what we know is that for whatever reason they tend to overestimate support for Labour and Plaid Cymru, and underestimate support for the Conservatives in particular. We don’t know why that is – perhaps it’s something to do with the way samples are taken, or perhaps because of the traditional anti-Conservative hostility in Wales there is an unwillingness on the part of some to disclose they are Conservative voters."

Indeed, my good friend Cai Larsen elaborates on the unreliability of Welsh polls, particularly Beaufort ones, on his blog.
"Beaufort's record is far from perfect when it comes to predicting election results in Wales.

For example, during the weeks before the 2007 assembly election there were two polls by the company.."

And here they are, put side by side with the actual result, demonstrating perfectly what Richard Wyn Jones had to say on the subject:

Cai continues:
"Now, when a polling company is consistently incorrect there is a reason - and that reason is defective methodology.

"I know nothing about Beaufort's methodology - and to be fair it's quite possible they have thought carefully about their methodology by using British polling methods - but it's impossible at the moment for the methodolgy to be effective. The reason for that is that there isn't enough polling here in Wales, and that polling isn't tested against real elections often enough. It isn't possible for all-UK methodology to work in Wales - the profile of voters is different here.

"An effective methodology evolves over time. For example MORI made a mess of predicting the result of the London mayoral election. Their interpretation of how this had happened is that people working in the public sector are more likely to respond to an opinion poll than somebody working in the private sector - and those people are more likely to vote Labour of course. Perhaps MORI's interpretation is correct, perhaps it isn't - but the point is that it will be proved or disproved in due course. If this additional consideration makes the poll more correct next time, it will remain as part of the methodology. If it does not, it won't be used again.

"I welcome the fact that Beaufort is polling in Wales, and I welcome the fact that Plaid Cymru are commissioning polls - but before Wales can have a system of credible polls there needs to be much more polls in order for a culture of effective polling to develop. The reason why that is not happening is because the Welsh media isn't greatly interested in commissioning polls. Until this changes I won't be able to put much faith in any Welsh poll I'm afraid."

In the comments thread to Cai's post, Alwyn ap Huw adds some additional details concerning Beaufort's methodology:
"Beaufort's problem is that the company doesn't do surveys of political opinion. The company does markeing surveys; the questions concerning voting intentions are a supplement to a questionaire on shopping and the profile of a likely shopper is a very different one to the profile of a likely voter.

"Plaid Cymru used the Beaufort company to hold surveys before the 2005 general election - surveys which showed that Plaid would easily regain Anglesey and which failed to predict the danger of losing Ceredigion. As the company had given Plaid such disastrously incorrect information on that occasion it has to be asked why Plaid continues to use the company. Money, it seems is the answer; a supplementary question on a Beaufort marketing survey costs a lot less than asking MORI to hold a real opinion poll."

To close I should also mention that the team behind YouGov this year set up its own online gatherer of Welsh opinion called WalesView. It has yet to publish any results. This post should give you some idea of the difficulty of the task facing YouGov in this field, but they should be congratulated for making the effort.

You can find more details on the results of the Beaufort poll here

1 comment:

mister_choos said...

I read somewhere about a similar problem in the US with black candidates. when asked, people don't want to appear racist, so say they will vote for the black candidate, but end up voting for the other guy. It cited a Californian governor election I think where the black candidate was well ahead in the polls, but got humped on polling day. This could, of course, be the McCain camp clutching at straws.

Oh yeah? So what has happened for the last ten years, exactly?

Over at the ASI, they are posting some of the winning entries of the Young Writers on Liberty. One does not want to put such keen minds off,...