For whom the Polls Toll
Article for Walkley
Magazine, (Journalists’ Union) December 04 – January
How did the opinion polls perform in the Australian federal election? A
better first question is: how was the media's reporting of them?
In a perfect world, political opinion polls wouldn't get within cooee of the
front page outside of election campaigns. They wouldn't make the morning news
and Radio National's Peter Thompson wouldn't ask Michelle Grattan why the
government “got a bounce” last fortnight.
Still in this sensible parallel universe, polling bosses would be seen and not
heard (because they don't make insightful political commentators) and no-one
would ever begin a poll report with “an election held last weekend would have
Quantitative opinion polls aren't that precise. But the process that pays for
them pretends they are. They cost a bundle and so are given pride of place. Once
they're there, everyone involved goes along with the charade.
So how should we treat them? Like this: they're the best tool we have to
anticipate election outcomes, and they're pretty good. Importantly, it's the
trend that's useful, not the individual data, although as we get closer to
polling day each survey becomes a better “predictors”.
Take this year's Queensland state election. The published surveys all pointed to
a huge win for Premier Peter Beattie. And a big win was what happened. We all
expected it, and we all gave our reasons, but it was really only because of the
polls showed one side fifteen to twenty points in front.
Polls have been known to do star turns, like Newspoll in the 1999 Victorian
state election. Throughout that campaign, The Australian's pollster, like
all the others, had the Kennett government headed for comfortable re-election.
But in its final survey, published on election day, Newspoll alone showed the
ALP opposition slightly ahead. Which was how Victorians then voted. A class act.
Roy Morgan Research stole the limelight at the 2001 federal election, but
for all the wrong reasons (getting the result horribly wrong).
And the polls did ok at the November US presidential election. They anticipated
a close result, with Bush slightly ahead, and that's what happened.
The problems begin when we take polls too seriously.We all know about the three
percent error margin. A sample size of about 1,000 that reports, say, 48 percent
support for a party, has a margin of error of about three percent, so the
“true” result might be anywhere from 45 to 51 percent. But that's not the
half of it. This is with a 95 percent confidence interval, which means, on
average, one opinion poll in twenty is outside that error margin; we just don't
know which one.
Another obvious problem: when was the last time the Australian Electoral
Commission surprised you with a phone call explaining there's an election on,
and who will you vote for? The survey process is highly artificial, although it
becomes less so as election day approaches.
And there are always “undecideds” who make up their mind in the polling
So how did the polls, as opposed to those who interpret them, do in the
October federal election? The answer has two parts.
Under the Australian preferential voting system, the two party preferred
(2pp) votes are more important than the primary ones, because 2pp is how a party
wins a seat at least, if not always an election.
Just looking at primary votes, Australians on October 9 gave the Coalition
46.8%, Labor 37.7% and Greens 7.2. If we see these as the “true” figures,
then throughout the campaign every pollster over-stated ALP primary support a
little, most underestimated the Coalition's and all trended in the right
direction as election day neared. Their final surveys produced similar primary
votes: Newspoll had the Coalition on 45 and Labor on 39; Morgan said 45.5 to
38.5; ACNielsen (Fairfax papers) 49 to 37; and Galaxy 46 to 39. All correctly
had the Greens on 7 or 8 percent. None was a million miles from the “true”
result, and Galaxy and Morgan vied for closest.
But after preferences they went everywhere. Still with Coalition support first, 2pps
in those final polls were Newspoll 50:50, Morgan 49:51, Nielsen 54:46 and Galaxy
52:48, which covers everything from a narrow Labor win to a Coalition landslide.
The true 2pp on election day was 52.7 to 47.3 in the government's favour, so
from being equal most accurate in primary votes, Morgan went to the back of the
back after preferences. Galaxy remained the best performer.
How could this be?
Once upon a time the major parties had high levels of primary support and it
didn't much matter how pollsters treated minor party (and independent)
preferences. Compared with the imprecision of polling itself, preference
distribution wasn't important. But this has changed in recent years,
particularly since the rise of the Greens (whose preferences overwhelmingly
favour Labor ahead of the Coalition) in 2001.
The evolution of Newspoll's preference strategy is a case in point. They
used to simply ignore preferences outside of election campaigns and report only
primary support. So from November 2001 to early 2003, the hapless Labor leader
Simon Crean had to watch while a Coalition primary vote lead of several
percentage points was written up as the government comfortably ahead, when
actually Crean's position after preferences would have been competitive,
sometimes even in front. Newspoll being the most watched poll, these perceptions
rippled through the commentariat. (Crean also presided over some truly dire poll
In early 2003 Newspoll began calculating a “notional” two party preferred,
based on the total preference flow of minor party votes at the most recent
election. But because the make-up of those non-major parties had changed, this
generally under-stated Labor's two party preferred support - by perhaps one
percent. Then from January this year they did what they usually do only in
election campaigns - ask those respondents who intended voting for minor parties
and independents who would get their second preference, and extrapolate to get
2pp. For complicated reasons, this probably overstated Labor's two party
preferred support, again by about a percent.
Of the other outfits, Nielsen and Morgan always ask for full preferences, and
new kid on the block, Galaxy, published in News Ltd tabloids, uses, like
Newspoll used to, the previous election preference flows but calculates per
minor party, rather than in aggregate.
Here's an irony: had Newspoll persisted with its 2003 method, its final poll
published on October 9 would have shown, like Galaxy's, the Coalition ahead 52
to 48. That is, a first preference reading that was much too kind to Labor would
have almost been compensated by preference distribution that worked the other
Nielsen's primary vote in its final poll overly favoured the Coalition, and
their 2pp did too. But Morgan is the real mystery; with excellent primary vote
data but woeful 2pp. How their preferences went that way is anyone's guess.
You could rank the pollsters, in their ability to “predict” the outcome, in
this order: Galaxy, Nielsen, Newspoll and Morgan.
But the main lesson you should take is: by all means read the polls. You can
only talk to so many taxi drivers, and anyway, they read the polls too. But keep
the things in perspective.