Automatic polling using Computational Linguistics: more reliable than traditional polling?

Other authors: Jacqueline Bechet, Fabio Celli, Carmelo Ferrante, Marc Poch, Hugo Zaragoza, Giuseppe Riccardi

 

Section 7: Social Media

The outcome of the 2016 EU referendum did not only spell disaster for the incumbent prime minister and the Remain campaign. It also amounted to a PR disaster for the commercial pollsters, with YouGov, Populus, ComRes, ORB Ipsos-Mori and Survation all failing to correctly predict the outcome. Of the larger pollsters, only TNS correctly ‘called’ the outcome, although still underestimating the LEAVE vote.1 This follows hard on the heels of similar failures in both the 2010 and 2015 General Elections.2 Public faith in commercial polling has, in other words, taken another serious blow.

the final prediction produced on our own site, SENSE-EU, estimated the vote correctly to within one-tenth of a percentage point

By contrast, automatic poll prediction sites using Computational Linguistics (CL) techniques to automatically extract information from social media (methods also known as Natural Language Processing, or text mining), by and large correctly predicted that LEAVE would prevail. Indeed, the final prediction produced on our own site, SENSE-EU,3 estimated the vote correctly to within one-tenth of a percentage point: 51.79% for LEAVE and 48.21% for REMAIN. While every new methodology is rightly treated with a degree of suspicion and while it is premature to expect traditional polling to disappear, there are grounds for both campaigners and the media to take CL techniques seriously in the future.

How do sites using CL methods work? The characteristic common to all is that they analyse a massive numbers of posts on social media websites such as Twitter or Facebook.4 The posts about topics of interest are found (the simplest way to do this is by looking for tweets with a particular hashtag, such as #Brexit, although most CL sites use more sophisticated methods). What happens next depends on the site. The most popular approach is to use so-called sentiment analysis to classify the relevant posts according to their positive or negative sentiment towards that topic. By contrast our own site, SENSE-EU, uses a different method based on the assumption that posts on social media tend to be part of a conversation: after classifying these posts as as favouring LEAVE or REMAIN, it classifies responses to these posts as being in agreement or disagreement with the statement in the post. These methods, of course, are by no means perfect: they often incorrectly assess the sentiment of a post, or whether the poster actually agrees or disagrees (e.g., when the poster is being ironic); nevertheless, the sheer volume of data analysed means that by and large CL-based automatic poll prediction sites tend to be a pretty accurate gauge of opinion towards a topic.

In the aftermath of the referendum, YouGov attributed the error in their predictions to higher turnout in LEAVE-oriented areas. Other pollsters will doubtless provide their own accounts of the debacle that will be worth considering. In the meantime, we think that there may be three reasons why CL sites may produce more accurate predictions than traditional polling:

One reason is the sample size. Traditional polls typically interview at most 1,000 to 2,000 individuals. By contrast, CL-sites process a minimum of 200,000 posts by tens of thousands of people per day and their predictions are generally based on at least 800,000 posts. Aggregation of this order produces compelling evidence.

Traditional polling asks for the people’s behavioural intentions or opinions, whereas CL-sites try to infer opinions that motivate behaviour. Modern cognitive science has established that direct questions about opinions and behavioural intentions may produce unreliable and invalid responses. Asking subjects to fill questionnaires is only used when more indirect methods cannot be applied, such as measuring the time it takes to perform a task, or eye-tracking. CL represents just such an indirect method, since it focuses on opinions that are some distance from the behaviour.

CL sites may well cover posts coming from a wider range of geographical locations.

These are just speculations and require further study. To date, neither polling organizations nor the media have paid much attention to CL methods, but the results of this and previous elections suggest that these methods, with all their limitations, produce reliable forecasts. At the very least, campaigners and the media alike should consider using CL methods to compare with the polls.

1. See the final predictions reported at: http://ukpollingreport.co.uk/

2. Market Research Society, Report of the Inquiry into the 2015 British general election opinion polls (London: MRS, 2016).

3. SENSE-EU was developed by the EU project SENSEI: http://www.sensei-conversation.eu. The primary developers were Fabio Celli, Carmelo Ferrante and Giuseppe Riccardi for UNITN, and Marc Poch and Hugo Zaragoza for Websays.

4. See http://www.nytimes.com/2016/06/25/business/brexit-talk-on-social-media-heavily-favored-the-leave-side.html for an interesting analysis of the discussion about the referendum on social media.