One of my favourite American blogs is Nate Silver's fivethirtyeight.com. Silver is a Chicago writer and owner of sports media company Baseball Prospectus and uses his blog to examine political polling.
"Both baseball and politics are data-driven industries," explains Silver. "But a lot of the time, that data might be used badly. In baseball, that may mean looking at a statistic like batting average when things like on-base percentage and slugging percentage are far more correlated with winning ballgames. In politics, that might mean cherry-picking a certain polling result or weaving together a narrative that isn't supported by the demographic evidence."
In his daily postings, Silver follows the uses and misuses of polling. Last year, for example, Silver exposed the polling firm Strategic Visions, LLC, which had apparently been creating bogus polling results that were in turn used by various right-wing politicians to promote their perverse agendas.
Similarly, I have long been very skeptical of local polling results that showed, for instance, that HRM residents overwhelmingly approved of the Commonwealth Games bid or were supportive of "tax reform."
For one, while polling sells itself as a scientifically sound enterprise, as Silver regularly shows, polls can be formatted in a way that leads to pre-determined results. Some questions are asked, other questions are not. The questions can be framed with certain context, or not.
But more, even the best polling can only reveal what people think at a certain point in time, as they have a certain amount of information. They might be vaguely supportive of something called "tax reform," but as they gain more information, look at the details of the specific "tax reform" proposal and explore arguments they may not have considered---that is, as they learn---their opinions could change.
Or, people might not learn. Their opinions could be ill-informed and irrational.
Which brings me to the citizen survey now being conducted by HRM. The 24-page survey, sent to 26,601 households "is scientifically reliable, with a margin of error of 2.4 percent, 19 times out of 20," claims a city press release. If it's "scientifically reliable," that means it will produce an accurate reflection of what people think, right?
Because the survey takes 45 minutes to complete, the vast majority of people simply chucked it in the recycling bin---only 1,500, or about six percent, of the 26,601 surveys mailed out have been returned. People who did fill it out are therefore those who are already most interested and engaged in city affairs, and not reflective of the population as a whole.
Michael Pappas, the city staffer overseeing the survey, is an affable fellow, and means well. He's following "best practices" principles established in the business world. Problem is, city government isn't a business. A business has a narrow self-interest in selling and promoting a narrow selection of products or services, whereas government's job is to be open to all concerns and respond as best as possible.
To be fair, some of the survey questions are open-ended, allowing respondents to bring their unique concerns forward. But many questions can be leading or reflect staff's interests. (I've posted the survey at thecoast.ca/bites.) A question about economic development, for instance, asks respondents to rate 16 possible strategies, all of which are large staff projects ("tax reform," a new convention centre). Not on the list are other potential economic development strategies, like green building rebates, or a more equitable distribution of city pay rates.
Similarly, many questions uncritically reflect staff assumptions: "attracting major events (e.g. large concerts and national sporting competitions)" is listed as a way of "ensuring economic prosperity for HRM," even though that claim is hotly debated outside City Hall. I, for one, am convinced that hosting the Commonwealth Games would have bankrupted the municipality and led to decades of impoverishment.
In short, the survey, like all surveys, reflects its creator's biases. But responses to it will inevitably be held up as "proof" that this or that proposed policy has public support, and should therefore be pursued.
Very often, however, that claim will be like looking at batting averages, instead of on-base percentages.
posted by KYLE SHAW, Oct 25/16
Because Atlantic Canada deserves great journalism. comments 0
posted by KYLE SHAW, Oct 12/16
You'll be surprised how easy it is to vote for president. comments 4
posted by KYLE SHAW, May 1/15
Here’s to the next thousand. comments 4
posted by KYLE SHAW, Apr 23/15
Know credit comments 0
posted by KYLE SHAW, Oct 2/14
posted by ADRIA YOUNG, Sep 25/14
Outraged by short sentences, one woman has taken up the task of improving Nova Scotia’s penalties for sexual assault. comments 3
Is this the Coast's attempt at a coy Betteridge-bait headline? Any headline that asks a…
And where do you sign up to be straddled?
Damned if they do and damned if they don't. Bash on, it ain't going away!