On Measuring Constitutional Preferences in Wales
The question of Wales’ constitutional settlement has recieved increased attention in recent months. In particular, questions of constitutional preferences in the ITV Wales Barometer polls are more frequently becoming headline news, both within and outside of Wales. Depending on who you listen to, you might think that support for independence is surging or the Welsh public are fed up with devolution and want to scrap the whole thing.
Given its growing prominence, I thought it was worth looking at the different ways surveys (and specifically the Barometer surveys) actually measure these preferences and what we should (and shouldn’t) take from the results.
Multiple Option Measure
Historically, the most common way that constitutional preferences have been measured in Wales is a multiple choice question that has been asked repeatedly for several years by opinion pollsters. These list a number of options survey respondents can pick ranging from ‘Independence’ to ‘No devolution’. This measure has the benefit of being part of an extensive time-series as it has been asked repeatedly for a decade. Given that polls in Wales are relatively rare, it is often tempting to emphasise the findings of one poll. However, trends over time probably tell us far more about people’s attitudes than any single snapshot poll can.
The two graphs show the results from two different polling ‘houses’: ICM and YouGov.
Two things stand out. The first is the absence of any ‘surge’ in support for any option in this question format — some gradual increases and decreases, but no sudden or dramatic changes.
The second is the substantial difference in results between ICM and YouGov, the two major polling houses. Given that we do not know which set of polls are a better representation of the broader Welsh electorate, we should be very cautious about making any grand claims from these data.
Referendum Style Questions
The next type of question asked is a more limited questions which mimics a hypothetical referendum (borrowing the wording from the 2014 Scottish Independence Referendum). These are useful for forcing respondents to pick between two options. They can also be a useful point of comparison for other polls asked in other sub-state areas like Scotland or Catalonia.
One important caveat here: people are generally not great at imagining how they would react in a hypothetical future situation provided in a survey. Although a referendum on the issue of independence is a conceivable event in Wales (abolition less so, but still conceivable), any actual referendum would likely see very different results.
Thermostatic Measure
The theromostatic measure starts with the understanding that most people don’t know or particularly care about the specific constitutional settlement of Wales (given how frequently it changes, who can blame them?). However, people might have a general sense of whether there is ‘too much’ or ‘too little’ devolution in Wales.
This question is great at showing the true range and diversity of views on this question as shown below on the two density graphs. It also allows us to combine the measures to establish which respondents want more or less powers than they currently pereive Wales to have currently (covered previously here). They are, however, not the easiest questions to discuss without visual representation, and as such, again, do not lend themselves to media coverage.
Trade-off Questions
Finally, the latest ITV Wales barometer also included a number of hypothetical ‘trade-off’ questions, forcing respondents to choose between independence and abolition. In my view, these questions don’t actually provide any useful information. This is not a choice that voters will ever face, nor is it a choice that makes any sense. Instead they force people into one of two extremes, with respondents unlikely to put much cognitive effort into answering a question that does not allow them to give their real view.
A Final Caveat
We know that constitutional issues are not that important to many people in Wales. It is far from the most pressing issue for people in their day-to-day lives and, unlike in Scotland, it does not yet act as a cleavage around which political parties organise. With that in mind, its worthwhile to end on a note about how people go about answering survey questions on topics that are rarely if ever at the forefront of people’s minds (as below chart shows).
Filling out a survey can be a pretty strange thing to do. You are asked to provide opinions and attitudes on a very wide range of topics and issues, some of which you might not be particularly keen to share, and others you will not have even considered until you are asked the question.
Our attitudes towards any given issue can be thought of as a kind of structure made up of “existing evaluations, vague impressions, general values, and relevant feelings and beliefs” (Tourangeau et al., p. 194). On all but the issues most central to our lives, we hold opposing considerations that could sway us either way. The salience of these considerations determine the attitutudes and preferences we report when answering surveys (Zaller and Feldman, 1992).
Tourangeau et al. put it nicely:
“ On any given occasion when we think about an issue, some subset of these contents will come to mind. Depending on which considerations we retrieve and the exact requirements of the task at hand, we may simply reiterate an existing evaluation, update it, or extend it to cover a new aspect of an issue; or we may make an entirely new judgement about the issue” (pp. 194)
For questions on a topic like constitutional preferences in Wales, it might not take much to sway an individual’s reponses one way or the other. As the below chart ilustrates, the context in which a survey is taken can substantially change how important people judge different issues, especially when those issues are not considered to be particularly important.
In the future, I would urge more caution when drawing conclusions from survey data on these issues. Whilst news and social media will understandably gravitate towards more exciting headlines about ‘surges’ for one side or another, the data rarely support such conclusions.
