Monday, April 25, 2016

This poll needs an asterisk

You don't have to be a fantasy baseball team manager to understand the role of statistics in baseball. With a quick review of a player's batting average, on base percentage or earned run average, one can determine how effective a player has been over the course of his career. These statistics even provide a way to compare players who never met on the diamond — or even lived at the same time. The exceptions are players who excelled in the steroids era and justifiably many in baseball put an asterisk (*) next to those numbers to note that the statistics may be out of whack because of the players alleged use of performance enhancing drugs.

Statistics — in the form of public opinion polls — are critical in politics too. While I am not asserting that anyone at the esteemed Taubman Center at Brown University is on drugs, their polling data is in dire need of an asterisk because each and every time they conduct a poll, their methodology is so out of whack with accepted industry practices that their results cannot possibly be accurate. They recently released a poll that covered the presidential primary as well as several state political issues ranging from legalization of marijuana to something they called a “toll tax on large trucks.” While I cannot disagree with their results because I have no better data to compare them to, their methodology (i.e. how they go about gathering the data) is so suspect that I can’t put any credence in their findings.

While there are many problems with the most recent Brown poll, the first is that they called “registered voters.” A good political pollster will only call “likely” voters. The difference? People who are “likely” voters have usually voted in at least 2 of the last 4 elections, making them more likely to show up again. A registered voter may or may not have ever voted in his or her life, but he or she has most certainly been to the DMV (where they were registered). When you ask registered voters about how they may vote in an election, your results will not be as accurate because they may never actually participate in the election (note that only 40% of RI’s eligible voters showed up to vote in 2014). When asked if they will vote, many registered voters will say yes even if they do not plan to show up, so asking them directly is not helpful.

The “poll” put Hillary Clinton up 9% over Bernie Sanders with 16% “undecided.” Chances are good that the “undecided” number actually represents people who have no idea that there is a presidential primary this week because people who are likely to vote this week know who they prefer. In fact, Brown could only scrounge up 164 people who said they were going to vote in the Republican primary — creating a huge margin of error in the Republican results. Even worse: Brown put out a statement last week saying that they were delaying the release of their poll because they were struggling to complete enough interviews due to “voter fatigue.” A well-executed poll is like “Fight Club” (there is no fight club) and a good pollster would not poison his or her sample by letting potential interviewees know that they might get a call.

Why does it matter? As a relatively small media market, there are very few public opinion polls released each year, so when one is released, it takes on an outsized significance. Just like hearing “9 out of 10 dentists recommend Crest” on TV, people are influenced by polls. If their candidate is out in front or very far behind, they may choose to stay home on election day. Polls are also used by campaigns to fundraise and build momentum and it truly unfair to have false information floating around in the ether. I do blame the RI news media for promoting it — the poor quality of Brown polls is well-worn topic among reporters, but the results still land on the front page of the Journal, making the state’s paper of record complicit in putting bad information out to the public.

The hard part is that there’s so rarely a “good” poll that runs concurrently for an apples to apples comparison, so political geeks are left to bemoan the methodology while putting an asterisk next to the numbers and reminding people to vote regardless of what they hear. So, * VOTE!

No comments:

Post a Comment