Political journalists give lavish attention to polls and often slide past those footnotes about a “95 percent level of confidence” and “margin of error plus or minus 4 percentage points.”
Thus, numbers are never exact and a three-point edge could actually be seven points ahead or one point behind. Not to mention that even the best poll is merely a time-bound snapshot of reality (and it always pays to check how questions are worded).
After Election Day 2020, complaints about misleading political polls were as loud as usual, perhaps even louder. Take the bizarre Maine Senate race data. Polls consistently showed incumbent Susan Collins would lose, and she was behind 6% in the one closest to Election Day. She won easily by an 8.6% margin.
Or consider recent surveys about whether Americans plan to get COVID-19 shots. The Wall Street Journal notes that “yes” answers have ranged from 20% to 63% depending on the methodology, the wording and, especially, whether respondents were offered more than two options.
These days the “response rates” among those in randomly selected samples are so low it’s tough to tell how representative the people are.
That’s among the major problems raised by eminent Princeton sociologist Robert Wuthnow in his 2015 book “Inventing American Religion: Polls, Surveys, and the Tenuous Quest for a Nation’s Faith.”
Gallup data about U.S. religion are important because the firm has asked consistent questions across so many decades. But 21st Century religion writers rely heavily upon the Pew Research Center (where Alan Cooperman, a former Washington Post colleague, is director of religion research). Pew’s expertise often provides all-important distinctions between white “mainline” and white “evangelical” Protestants, and between white and Hispanic Catholics.
Pew is changing the way it surveys religious behavior and attitudes, so the media will want to be aware of why and how.
Of major importance is how this affects writing about trend lines over time that uses past surveys. The technicalities on that and other aspects are explained in the full report here, but The Guy will cover some of the basics.
Like most polling organizations, Pew traditionally conducted live telephone interviews with representative samples of people selected by “random digit dialing” (RDD). But this is expensive and, worse, those response rates are dismal because so many people are reluctant to answer calls from unknown numbers. Also, the explosive growth of mobile telephones — with many people dropping the use of landlines altogether — have changed the telephone-poll game.
Therefore in 2014 Pew began using its American Trends Panel (ATP), a large, nationally representative group of some 13,500 respondents, selected by phone or mail, who answer online questions one or two times a month. (The similar “Gallup Panel,” with about 100,000 members who mostly respond online, has operated since 2004 alongside that company’s other types of polling.)
Pew began evolving away from RDD, and since 2019 most U.S. polls have used the new ATP. The shift from “interviewer-administered” phone calls to “self-administered” internet answers has had a significant effect, portraying a less religious population, With ATP, higher percentages are “nones” who do not identify with any particular religion and lower percentages regularly attend worship services.
How come? The Pew folks figure some people may present themselves as religious to another person on the phone as more socially desirable. This is not necessarily dishonest. A person with a tenuous link or mere background with Christianity might identify that way in some contexts or not in others without even thinking about it. Feelings may even “depend on what happened that day.”
ATP has its problems. Some may not participate consistently because they’d rather not answer online, though Pew provides internet access to randomly selected panelists who lack it. Importantly, internet-wary people tend to be older, and older people tend to be more religious. Thus “ATP may underestimate the country’s religiosity.”
Enter Pew’s National Public Opinion Reference Survey (NPORS), designed last year due to the gap between RDD and ATP religion results, though it’s used for many topics. This method employs “address-based sampling” (ABS) instead of phone numbers, and lets participants respond either online or by paper and pencil. The payoff is a 29% response rate, well below the rates decades ago but 20 points better than with RDD or ATP.
Importantly, the early NPORS results suggest ATP may have overstated the population of “nones.” On the other hand, NPORS agrees with ATP that regular worship attendance is lower than RDD surveys indicated. Full number comparisons are listed in Pew’s memo on this.
Which brings us, finally, to “weighting” of poll results. Pew has decided that for the foreseeable future it will reshape ATP poll numbers to fit the religious affiliations profile found from NPORS. The Pew team notes that such religious weighting is used by e.g. the National Science Foundation’s Cooperative Election Study. https://cces.gov.harvard.edu/
ybvdadminuser
