A ship with an august history is lost at sea. A dense fog has encompassed the ship, and the crew suspects that the navigation instruments are damaged. No one seems to know the right direction to take in order to avoid encounters with reefs, shallow seas and bad weather. The crew decides to vote between only two options: north or south. 51% want to head south and 49% choose north. The captain says that the majority has spoken and decides to head south. Suddenly, a rock comes into view, and the crewmembers are now eager to change direction and head north. But the captain says such a change would be a betrayal of the election, and therefore they should continue heading south no matter what the cost.
This story brings up some interesting questions: In an unfamiliar setting, should we choose a new path when the fog is dense? Would it make sense to ask the crew to vote again when new information comes to light? With such a narrow margin, should the 51% have the right to rule the fate of the ship?
Having designed and conducted hundreds of statistical surveys over the past ten years, I’ve developed the ability to determine whether the outcome of a survey will be dependable or not by assessing its design. Survey results tend to be more reliable when survey questions are easily interpretable. Do you smoke or not? Do you wear glasses or not? Do you have life insurance? All these are questions that people can answer in the blink of an eye, with great certainty.
However, the use of surveys, questionnaires or referenda to measure complex and multifaceted opinions can result in unclear conclusions that are not indisputably in favour of any particular action. The 2016 Brexit referendum question had a long list of hidden premises and consequences but merely asked respondents to choose between two simple options. How could voters give a meaningful answer when no one knew the true consequences of either option? The vote result might seem to be scientifically accurate, but upon closer inspection, it is not the true will of the electorate. The purpose of voting systems is not to muddle our thoughts on the issues addressed, but in this case, that is what was done. Let’s re-examine the situation to ascertain what message the Brexit referendum conveyed.
Undependable results can occur when the question asked has one or more of these attributes:
- Oversimplification. An oversimplified question can be interpreted in a multitude of ways, and depending on how it is interpreted, the answer could vary. The phrasing of the question could influence the answerer’s reading of it while also creating additional questions in the respondent’s mind.
- Unclear premises and consequences. Questions with unclear premises and consequences are difficult to answer because not enough information is provided. In order to answer the question “Do you want to travel to China?” we need to know more. Price, travel time, accommodation and travel partners are just a few of the factors that must be revealed. Otherwise our answer might change from “yes” to “no” as we learn more about the preconditions involved.
- Voter’s well of knowledge is biased. When authoritative individuals in the media are – deliberately or not – distributing false “facts” and biased viewpoints, survey/election/referendum results might serve only the interests of a small minority and lead a whole nation on a path of ill-judged policies. This occurs in the absence of a clear, unbiased description of the matter at hand.
The volatility that is built into survey outcomes is potentially harmless when those surveys are taken periodically to measure changes over time. But asking just once about a complex scenario and using the result as a foundation for important decisions can lead to trouble. We could end up with misleading guidance and incorrect conclusions disguised as science.
Therefore, we must pay closer attention to the interpretation of survey, poll, election, and referendum results. It is crucial that we know when the moment is right to act firmly and make important decisions that are based on the results and analysis of such surveys. Those who conduct and analyse these studies must also become more skilful in translating with full integrity their actual meaning. Unfortunately, it is a common practice to misinterpret an outcome in favour of a particular interest. Misreading on purpose serves motivations other than public interest.
One example of such a scenario is the Brexit referendum, where 51.9% of those who voted supported the UK’s withdrawal from the EU. Those who favoured withdrawal called the result the clear will of the British nation. But what conditions must exist in order to inspire such certainty in this interpretation of the results? Is 51.9% a margin large enough to say that Brexit was the will of the nation? Did the voters have a clear view of the premises and consequences of Brexit? Did all voters have unbiased information on the question being asked? If any of these concerns linger, the true will of the British people can only be determined after the fog that obscures the issue has cleared. Otherwise, we might discover that the light at the end of the tunnel is the headlight of an oncoming train.
The following are three scenarios that can dilute the democratic legitimacy of surveys as well as referenda. We must watch out for them, especially if their results will be the basis for important decisions.
THREE SCENARIOS THAT DILUTE RESULTS FROM SURVEYS AND REFERENDA
Scenario 1: The case of oversimplified questions.
When oversimplified questions are asked in a survey, multiple topics lurk behind the seemingly straightforward phrases used. Some examples include “Which do you prefer to eat, fish or meat?” (Depends on how it is cooked); “Do you like Hillary Clinton” (As a person or as a president? Many don’t like everything about her but prefer her to Donald Trump); and “Are you interested in traveling to Tenerife?” (Depends on the price, accommodation, travel partners, duration, time of the year and more).
The Brexit referendum question in 2016 was: “Should the United Kingdom remain a member of the European Union or leave the European Union?” Voters were given only a binary choice, to remain or to leave. Structuring the query in this oversimplified manner goes against basic principles of survey question design, as reducing a complex matter to only two options will lead to a misleading conclusion that can easily change as more information becomes known.
In 1951, Kenneth Arrow (1921-2017), winner of the 1972 Nobel Prize for Economics, proposed his famous impossibility theorem, which points out the danger of selecting the best option out of many. Arrow’s theorem states that a clear order of preferences cannot be determined while adhering to mandatory principles of fair voting procedures. The theorem illustrates a situation where it is impossible to judge the clear winner when there are more than two options.
Look at the following example, where voters are asked to rank their preference of three candidates: A, B, and C:
- 45 votes A > B > C (45 people prefer A over B and B over C)
- 40 votes B > C > A (40 people prefer B over C and C over A)
- 30 votes C > A > B (30 people prefer C over A and A over B)
Candidate A has the most votes, so that should be a clear winner. However, if B was not running, C would be the winner, as more people prefer C over A. In that case, A would have 45 votes and C would have 70 votes.
Although only two options were offered in the Brexit referendum, there were actually a multiplicity of questions, options and consequences to consider. It is unclear how UK voters would have chosen if all of the relevant questions were asked.
The problem of oversimplified questions is obvious not only in survey interpretation, but also in online searches. Google, the most used search engine, strives to “understand” (as much as an algorithm can understand) its users’ oversimplified questions to maintain its top position. Google’s announcement in February 2018 stated, in part:
“Many of us come to Search with questions about intricate topics. When we ask about “heat from the centre of the earth,” for example, we could be asking about what causes it or how hot it is. And if we ask about “ears popping on a plane,” we might be looking for the cause or for remedies. To provide more actionable results to nuanced queries like these, today we’re starting to roll out multifaceted featured snippets.”
Google, at the forefront of today’s cutting-edge technology, has figured out that one of the most important challenges they face is deciphering the meaning at the core of user questions. Google understands that a single question can be linked to several topics. Therefore, the answer to such a question can be, in fact, a collection of answers. Oversimplified questions tend to lead to simple answers that might not address what the inquirer wanted to know.
When is a referendum result democratically legitimate? Is it when voters can make informed decisions? Yet we have often seen complex issues reduced to oversimplified questions in referenda. Even people with profound knowledge and expertise cannot adequately answer such questions as asked due to the high level of uncertainty involved. In such a case, voters must make an important decision regarding the future with very little knowledge about the options offered. Generally, it takes a long time and careful consideration to make a determination on such crucial issues as a life partner or a focus of study. We should be equally deliberate when establishing the type of society we want to live in.
To avoid the trap of using oversimplified questions in important surveys, elections and referenda, we must use explicit (literal, factual) questions and forgo implicit (interpretative, inferential) questions. The 2016 Brexit question is a clear example of an implicit question that can refer to various issues. Its oversimplified nature is the main reason why the result has caused the nation to be divided since the referendum was held.
Voting results are one thing, and government decisions on action are another. Unfortunately, there are too many examples around the world where legitimate referendum results have not been honoured. A constitutional referendum held in Iceland on 20 October 2012 asked voters if they approved of the six proposals included in the draft of a new constitution. All six questions were approved by a large majority of voters, but since then, three successive governments have failed to apply the referendum results. Dissatisfaction over this issue has been widespread amongst the people of Iceland for the past six years.
In the case of the Brexit example, with an oversimplified question and only two response options, the government has much room for interpretation and conclusions, a situation a democracy should try to avoid. Limiting the options to only two in a multifaceted matter essentially forces us to use a non-democratic strategy akin to flipping a coin.
The case of oversimplified questions is therefore a case of homogeneity vs. heterogeneity. While heterogeneity is in place, when designing questions, we must be fully aware of the fact that we can end up with mixed results that don’t portray a clear conclusion. This state should be avoided by all means.
Scenario 2: Unclear premises and consequences
How should one answer a question if premises and consequences are unclear? The usual way is to wait until the fog clears. It is hard to say why the British government asked about “leaving” or “remaining” with phrasing that left voters in the dark in terms of concerns such as:
- If we leave, will we stay alone or adopt the EFTA framework?
- If we leave, will net contributions to the EU decrease enough to allow for tax cuts?
- If we leave, will the UK’s trade be worse off in general?
- If we leave, will the UK’s real per-capita income level decrease?
- If we remain, does that mean everything will be unchanged?
- If we remain, is it true that two-thirds of British manufacturing jobs be dependent on demand from Europe?
- If we remain, is it true that we will gain back £350 million per week from the EU?
- Whether we remain or leave, how does a soft Brexit differ from a hard one?
It is easy to call to mind premises that should have been made clear before June 23rd, 2016. Even now, more than two years after the referendum, some still have difficulty in answering the Brexit question because uncertainty regarding the above-mentioned concerns still exists. Polls indicate that a referendum now would show opposite results. Which results should be used to change or keep the fundamental structure of the UK vis-a-vis the EU?
Let’s say that the UK ends up leaving the EU on the basis of a hard exit, hard borders, a flood of companies leaving the UK, a weakened pound, lower GDP and falling real wages. Most likely, that would not be the end result any voter would like to see. It would be a reversal of the fortune promised by the leave campaign and the contrary of what many voters believed would be the consequence of choosing to leave. This is exactly what Kenneth Arrow’s theorem demonstrates: Voting on an issue with hidden premises and consequences can leave us in a situation for which no one would vote. In our times, we should be able to avoid such misuse of elections and referenda.
Scenario 3: Statistical significance, narrow majority and biased voters
The fundamental question is: Do we want to use referenda and elections to determine the true will of a nation or do we want to portray the imaginary will of a nation? Therefore, does the 52%-48% split in the Brexit referendum truly imply a majority for leaving the EU? What can statistics indicate about assumptions drawn from the Brexit referendum?
Statistical methodologies do not support the conclusion that the British government has drawn from the Brexit referendum. The 52%-48% split does not represent any significant difference compared to the even split (50%-50%) and should not be used to declare a majority for either side or campaign. Claiming that majority of UK voters are in favour of leaving the EU is not supported by any statistical calculation mathematics can support. In fact, the Brexit referendum mathematically signals only an equal size between the two campaigns. This conclusion can be derived by quantifying the difference between two groups, remainers and leavers (calculating Cohen’s d effect size).
By only looking at raw results, we know that 17,420,742 voted to leave but 16,141,241 voted to remain. As we know, these voters did not represent all of the UK nation because almost 13 million voters did not show up. Therefore, we have to come to a conclusion on how we judge if a small majority is enough to reflect the true will of the nation. As we know from regular surveys, we have a simple measurement, called statistical significance that can help us to judge if a small majority is enough based on the sample size. But effect size is a far more sophisticated methodology to reveal if a majority is large enough to claim victory. By calculating effect size we can find out if the 1,269,501 votes in favour of leave, did truly imply a real difference between the two sides. For Brexit results, the effect size turns out to be negligible and therefore, the Brexit referendum did not confirm majority for either of the two campaigns. To claim a victory for either side, a split of 60%-40% would be somewhat minimum, assuming similar behaviour of voters as came true in the 2016 referendum. Polls taken before the referendum revealed similar splits and therefore reinforce the fact that majority cannot be claimed for either side. It only reflects the fact that the 52%-48% split is of negligible importance.
History reveals many examples of narrow election margins, when two options received an equal or almost equal number of votes. In the 1994 Quebec general election, incumbent Liberal Michel Charbonneau tied with Roger Paquin, both receiving exactly 16,536 votes. Consequently, a new vote was held 42 days later, and Paquin won by 532 votes. This case is an example of the preservation of basic democracy and legitimacy by way of allowing voters to discuss options further, gather more information, and re-think the situation.
Also, there is a need to address the impact of bias and discussion in the media that relies on incorrect or false information. This became evident in the months before the Brexit referendum in 2016. Fake news and untruthful “facts” were given by authorities. Liam Fox said a post-Brexit free trade deal with the EU would be the “easiest in human history”. That did not turn out to be true.
One of the most prominent claims made by the Leave campaign was that the UK would take back £350 million a week once it had left the EU and the sum would go to the NHS. This claim was said to be “a clear misuse of official statistics” by the UK Statistics Authority because after the rebate was calculated, the figure was revealed to be closer to £250 million. However, the question remains whether the money will go the NHS or not. A poll by Ipsos MORI published in June 2016 found that nearly half the British public believed the claim.
Alan Johnson claimed 2/3 of manufacturing jobs were dependent on Europe after looking at an outdated analysis conducted by the Centre of Economics and Business Research. More recent analysis revealed that the figure is more likely to be closer to 15 percent.
The case of the Brexit referendum diluting accuracy of results
All three of the above-mentioned scenarios diluting the legitimacy of democratic results occur in the Brexit referendum. Here is a summary of the problem and why it is very hard for the British government to take actions based only on the results from June 2016:
- OVERSIMPLIFIED QUESTIONS AND UNCLEAR OPTIONS: It is clear that the Brexit referendum asked the British nation an oversimplified question, giving only two options for a complex issue. That alone should have led to a decision to hold another referendum in which the options were clarified. To achieve a result less diluted than the original question, a new referendum could explicitly ask each voter to choose to either “leave” or “remain” based on a detailed agreement that could be reviewed at the time of voting.
- UNCLEAR PREMISES AND CONSEQUENCES. The Brexit matter in 2016 was full of unclear premises and ill-defined consequences. Forcing a nation to select to “remain” or “leave” in such circumstances is not only ethically unfair but also likely to lead to a pathway that is less than ideal for the UK. To honour minimum scientific standards, the British government should have analysed the situation and found out that the Brexit issue was obscured in a fog so dense that a simple choice between A and B was inadequate. Would anyone ask another to drive at night, in the rain without headlights on? Wouldn’t you ask to delay that trip until visibility clears?
- STATISTICALLY, NO MAJORITY CAN BE CLAIMED FOR EITHER SIDE. The result of the referendum was narrow to say the least. According to standard statistical methods, a split of 51.9% to 48.1% only demonstrates a spurious difference, not sufficiently distinguishable from a split of 50-50 to claim a majority for either side.
J.K. Rowling recently tweeted: “If you truly want the will of the British people to be implemented, you’ll be happy to have a second referendum to confirm what their will is. If you’re afraid your lies won’t fly twice and that breaking electoral law might be much harder a 2nd time, not so much.” What Rowling is calling for is true democracy.
When Theresa May says that a second Brexit vote would be a betrayal, she is demonstrating a total lack of understanding of the basics of statistics, surveys and referendums. Looking at the first Brexit referendum as the true will of the British nation is naive. May’s statement only makes one think that the government is perhaps not so interested in understanding that will. May’s disallowance of a second referendum to allow voters to express their opinions, now after they have much more information and a clearer view on the consequences of Brexit, is the real betrayal.
Our examination of this issue leads us to questions that some might not find easy to answer: How do we employ, carry out and interpret elections and referenda in order to maximize the effect of democracy? Is it an example of democracy to select one statistically ambiguous snapshot in time to be chiselled in stone? And when dealing with a 72.2 % turnout, far from 100%, is it truly democracy if the wishes of those who did not or could not make it to a polling station are ruled out? Is a referendum an agreed-upon method to demonstrate the ideals of democracy? And how do we operate our democracy when the electorate is not very well informed? What percentage of Brexit voters fully realised the true consequences of leaving or remaining? All these questions are roaring out loud, and we have just begun to consider the solution to this problem.