Corporate Research Associates poll is completely meaningless

The Chronicle-Herald, CBC and Metro have completely botched their reporting of a Corporate Research Associates poll that supposedly showed a collapse in support for the provincial NDP.

In truth, the NDP may or may not be losing support, but the polling data doesn’t support either conclusion.

At issue are the responses to one of three poll questions announced by Don Mills, president of CRA. (The questions were part of a larger poll conducted by CRA, but Mills has not released those results.) The second of the three questions reads:

If a provincial election were held today in Nova Scotia, for which party would you vote?

CRA provided the following table of results:

The accompanying press release reads:

Less than four in ten decided voters prefer the NDP (37%, down sharply from 46% three months ago), while support for the Liberal Party increased (35%, up significantly from 26%). Support for the PC Party is stable (24%, compared with 22%), while four percent prefer the Green Party (compared with 5%). The number of Nova Scotia residents who are currently undecided, do not plan to vote, or refuse to state a preference is stable at 43 percent (compared with 40%).

[…]

These results are part of the CRA Atlantic Quarterly®, an independent, quarterly survey of Atlantic Canadians, and are based on a sample of 1,208 adult Nova Scotians, conducted from May 11 to May 31, 2010, with results accurate to within + 2.8 percentage points, 95 out of 100 times.

Now, the thing that should jump out at us here is that 43 percent of those polled didn’t have an answer. CRA deals with this by simply excluding those people— 519 out of 1,208— from the results, and published percentages related only to “decided voters.” While technically those percentages are true, they don’t say what CRA or other local media are implying they say.

More honestly, we should cast those responses as percentages of the total sample, that is, the entire 1,208. If so, we’ll get these results*:

519 have no answer- 43%
255 would vote NDP- 21%
241 would vote Liberal- 20%
165 would vote PC- 14%
28 would vote Green- 2%
1 would vote Other- 0%
(*There is a rounding error of one respondent, because CRA gave only whole number percentages.)

Consider that there is a margin of error of 2.8 percentage points, and you see that you can make no determination at all about whether more people vote NDP or Liberal, and the difference between Liberal and PC voters could be as small as 0.4%.

I’d like to back this analysis up to the past survey results but I can’t because CRA doesn’t provide sample sizes for past polls. Worse still, for the pre-election poll of May 2009, CRA threw in which way undecided voters were “leaning,” so we’re comparing apples and oranges in the table—it’s worst than meaningless, it’s misleading.

Now, let’s look at media coverage of the poll. First is Alex Boutilier at Metro (Boutilier is a friend, and generally a pretty good reporter, but he blew this one) who completely ignores the undecided responses and reports:

The poll, conducted by Corporate Research Associates, found only 37 per cent of respondents would vote for the NDP if an election were held today — down from 53 per cent in November and 46 per cent in February. 

The latest figure is also only two per cent more than the Grits, who came in at 35 per cent.

Boutilier left out “of decided voters,” and so his reporting is completely, utterly, wrong.

Next, is David Jackson, the provincial reporter at the Chronicle-Herald, who simply rewrites the CRA press release, with none of his own analysis:

The NDP still leads at 37 per cent of decided voters but the two-point margin over the Liberals was within the poll’s margin of error.
And the level of satisfaction with the government stood at 43 per cent, down eight points. Dissatisfaction was at 48 per cent, up seven points.

But even there, Jackson is technically correct, but doesn’t seem to know what he’s saying. Yes, the margin of error is 2.8 percentage points for the entire polling sample, but for a sample size of 689 decided voters—were you polling only them—the MOE would be more like 3.7 percentage points. But CRA didn’t poll only decided voters, so bringing the margin of error into it only shows that Jackson doesn’t understand what he’s writing about.

Lastly, there’s an unnamed reporter at CBC, who likewise parrots the bullshit 37/35/24 numbers, but at least uses the word “decided” and mentions the very large “didn’t say” response:

Asked which party they would support if an election was called today, 37 per cent of decided voters backed the NDP, down from 46 per cent in February. In August 2009, 60 per cent chose the NDP.

The Liberals are close behind with a 35 per cent approval rating, up from 26 per cent three months ago. That’s up from 18 per cent last August.

The Tories remain in third place with 24 per cent approval up from 22 per cent in February. They were at 16 per cent in August 2009.

About 43 per cent of those surveyed said they don’t know how they would vote, refused to say or don’t plan to vote.

However, the CBC still missed an important part of this story. As my colleague Bruce Wark, a former CBC journalist himself, tells me, “One of the keys to polling is randomness as captured by the ‘response rate.’ Every time a polling firm calls someone who refuses to complete the survey, randomness is compromised. Or, in other words, high refusal rates skew the results as the polls end up recording the opinions of people who are willing to be polled. The CBC’s journalistic standards and practices requires the reporting of response rates, but this is never done.”

It may seem axiomatic that the NDP is losing support, but this poll doesn’t tell us that, and nor does the series of CRA polls listed in the release.

If the CRA poll tells us anything at all, it’s that a hell of a lot of people either don’t know how to cast their votes, or aren’t saying.

Correction: This story’s sub-headline was edited for clarity July 9, 2010.

This article appears in Jun 3-9, 2010.