CBC ombudsman: Polling pitfalls

By Esther Enkin, CBC Ombudsman

By Esther Enkin, CBC Ombudsman

The complainant, Barry Kiefl, who runs Canadian Media Research and used to work here at CBC, had some concerns about some features on Power and Politics. He thought other programs were not living up to CBC’s polling policies either. He was right in at least two cases. Power and Politics is not living up to the rigors of CBC policy on polling in a couple of their regular features. I suggested CBC lay out some standard processes for polling approval and presentation.

COMPLAINT

The basis of your complaint was published on your blog. You asked CBC News to respond to the issues you raised in the posting. You were concerned that CBC News’s standards were slipping because you thought that some program segments on the CBC News Network program Power and Politics were violating CBC’s own policies on polling. You felt the material being presented in these segments was being presented as if it met the criteria of bona fide polls, and could therefore be characterized as representative of public opinion. And in the case where polling techniques were used, you thought the policy was not being followed because there was inadequate explanation of the methodology behind the results being presented.

One of the features you were concerned about is entitled “Political Traction.” This segment features Jaime Watt, the Executive Chairman of Navigator, a communications firm. The purpose of this weekly presentation is to compare the issues that are most discussed in Ottawa with those that are prominent in the rest of the country. The Power and Politics website describes the purpose of the feature as a chance to take “a closer look at the top public affairs issues being discussed in Ottawa to evaluate how well these issues resonate among everyday Canadians.”

Related content on J-Source:

Watt’s team at Navigator assesses what Members of Parliament, including questions asked during Question Period, editorialists and political and public affairs programs are talking about each week. Through various analyses, these are assigned a numeric value. The top issues are given a ranking and then compared to the frequency of “conversation” in a broader Canadian context. The broader picture is measured via key word search in social media, as well as a scan of public affairs and political programs and national editorials. The program host, Evan Solomon, and Jaime Watt spend some time outlining the issues and measuring how the preoccupations of Ottawa are mirrored, or not, in the rest of the country.

You felt that the nature of the discussion led viewers to believe the data is representative of Canadians, and it cannot be because it does not use conventional polling methodologies to make that claim. You point out that the “tracking” percentages are actually based on analyses of blogs, online comments and twitter. You thought the methods used should be much more prominently featured:

“There is nothing wrong with someone analyzing tweets and online comments but CBC should not be implying that this is representative of Canadians and therefore possibly leading viewers into thinking that Navigator has conducted representative polling. We know that relatively few people tweet and research, including some done by CBC, determined that those who bother to write newspapers or call talk shows, etc. are generally not representative of either the audience or the population. Power and Politics should be more forthright about the methodology used by Navigator.”

The second weekly Power and Politics feature that concerned you was the “Nanos Number.” This program segment is devoted to interpretation of public opinion research and its meaning and impact on the political scene. The “Nanos Number” is given at the conclusion of the segment as a way of highlighting the significant development or trend of the week. Your concern about this segment is that the methodology is not presented prominently enough. You thought that CBC should be a lot more transparent about the Nanos surveys and provide information about the methodologies used in the polls, both on air and on the Power and Politics website. You also had your doubts about some of the methods used and questioned whether CBC’s own Research Department had approved its use:

“Survey results on some occasions appear to come from a conventional telephone poll, while on other occasions, results are based on what Nanos calls an “RDD Crowdsource random survey of 1,000 Canadians… recruited by telephone through the proprietary Nanos Crowdsource sample and administered a survey online.” Crowdsourcing has been used to fund entrepreneurs but it is very unclear how it has been used here. The Nanos Number segment, featuring the amiable and professional Nik Nanos, usually provides little or no information about the methodologies, other than an illegible footnote partially obscured by program graphics. One must go to the Nanos website to learn how the surveys have (presumably) been conducted. So, once again, this seems to be in violation of the CBC’s journalistic policy.”

The third feature that troubled you was the daily “Ballot Box” question. Before the show even goes to air, the programmers post a question on the website and then promote it throughout the show. The question is topical and people are asked to vote, generally between two choices and a “don’t know” category. They can participate by going to the website, through the program’s Facebook page or by using a QR code displayed on the TV screen. The results are referenced throughout the program. The numbers are expressed both as a percentage and the actual number of votes for each position.

You see this as a violation of CBC policy because it is a completely non-scientific poll and in no way represents Canadian public opinion, but you think it is being presented as such. You point out that CBC’s own policy on the matter clearly states that “Since it does not fulfill any of the criteria set out in polling policy, the questions and the results are not characterized as polls. We report the results by giving the number of votes cast for each option. We do not give the results as a percentage, as we normally do with bona fide polls.”

The “Ballot Box” does express the results as percentages both on air and on the web site. Your concern is that this presents the results as a bona fide poll: “In my view this lends an air of polling legitimacy to The Ballot Box, which is not warranted, as any professional researcher and experienced journalist knows.”

In subsequent correspondence you mentioned the misuse of these online surveys on other radio and television programs as well.

MANAGEMENT RESPONSE

The Executive Producer of Power and Politics, Amy Castle, responded to your concerns. She rejected the contention that Power and Politics’ three features did not live up to CBC’s own standards.

Specifically, she disagreed with you that Mr. Solomon and Mr. Watt left the impression that the data they reference during the “Political Traction” segment implied or left the impression they were discussing public opinion based on polling results:

“Rather, we are clear on the show and on our website that Jaime Watt is tracking the political conversation in Ottawa and across Canada. The goal of the segment is to discuss which political issues are trending in the Canadian conversation, and to highlight the differences – if any – in the conversation being held in Ottawa compared to the national conversation”.

She pointed out that the methodology for the Traction feature was available on the CBC website and that in the interests of transparency she would ensure that a mention of where to find the details behind the data would be given on air.

She stated that the Nanos Number feature also abided by CBC journalistic policy. She explained “pollster Nik Nanos draws on a number of polls from reputable sources and we are always clear where the polls come from.” She added that all polls referenced in the feature meet CBC criteria for bona fide polls. Information on methodology, sample size and margin of error are available on the website as well as on a graphic on air. Ms. Castle informed you that she is working with the program’s graphic designers to find an on-air format that is easier to read.

She also addressed your concern that Mr. Nanos used crowdsourcing in the creation of his polls. She explained that “in fact the term crowdsource is a Nanos tradename and should not be confused with a process. The Nanos polling methodology includes a sample of random land and cell lines where people are randomly selected to do a study online. His methodology is robust and meets our strict CBC standards.”

She believed that the “Ballot Box” feature also met the criteria of CBC policy. She pointed out that the results are never referred to as a poll, and that while it is against the policy to express the results as a percentage, the absolute numbers appear on the screen as well. She undertook to revise the presentation online so that it too gave both percentages and numbers.

REVIEW

CBC Journalistic Standards and Practices has three policies on polling which apply here. They are all similar in demanding rigorous standards and a review of methodology by the corporation’s research department. For polls that CBC itself commissions, the policy states:

CBC commissions polls to reflect Canadian opinion on important issues of the day. The polls are commissioned, designed and the results are interpreted with the oversight of the CBC's research department.

To help our audience place a poll in context, we provide relevant information about the size of the sample and methodology along with the results. Where appropriate we report the margin of error.

It goes on to say because of the impact public opinion polling can have on public policy debates or elections, “the commissioning of polls requires the approval of the head of research and the Director. The final design requires sign off from the head of research or delegate.”

To continue reading this column, please visit ombudsman.cbc.radio-canada.ca, where it was originally published.

Related content on J-Source: