Facebook Twitter Tumblr Close Skip to main content
A Project of The Annenberg Public Policy Center

How Not To Use a Survey

Republicans cite a new survey to claim health insurance premiums are up 90 percent in New Hampshire because of the Affordable Care Act. But that figure is based on just one insurance broker in the state.

The 90 percent figure is included in a survey of 148 insurance brokers by Morgan Stanley, to help guide investor decisions about stock purchases. Their responses “point to significant acceleration in small group and individual [market] rate increases,” the report states. And, its author’s concluded, “We expect the increases are largely due to changes under the ACA.”

Near the end (on page six), the report provides a chart of “State Specific Commercial Rate Increases.” The chart shows a 90 percent rate increase in the individual market in New Hampshire, and a 15 percent increase in the small group market. But there’s one other telling piece of data in the chart indicating the number of respondents: 1. That’s right, the rate increase is based on the response of one insurance broker in New Hampshire. The broker reported a 90 percent increase in 2014 — 10 percent over last year’s increase.

New Hampshire is one of two outliers among the survey results. Delaware was the other, with the chart showing a 100 percent increase in individual market rates — also based on the response of just one broker in Delaware.

Morgan Stanley declined to comment when we asked whether it was advisable for anyone to be citing state-specific data based on such small sample sizes. However, in small type at the bottom of the chart, the report cautions: “We consider the aggregate trends much more useful than the trends among the individual insurers, where the number of observations is necessarily smaller.”

Despite this warning and the thin backing for the New Hampshire numbers, the 90 percent figure quickly became political fodder.

The Republican Party of New Hampshire put out a press release claiming that, “An independent study released today shows that health care premiums are ‘skyrocketing’ and have increased 90 percent in New Hampshire because of ObamaCare.”

The release includes a statement from Jennifer Horn, chairwoman of the New Hampshire GOP, citing the survey as evidence that the Affordable Care Act is “an unmitigated disaster.”

Horn, April 7: This survey confirms that ObamaCare is an unmitigated disaster that is raising the cost of health care for the middle class. Senator Jeanne Shaheen, Governor Maggie Hassan and Representatives Annie Kuster and Carol Shea-Porter have enthusiastically championed ObamaCare and they are directly responsible for imposing it on New Hampshire.

Former Massachusetts Sen. Scott Brown, who announced April 10 that he will challenge Shaheen, also released a statement:  “Today’s study showing that insurance premiums are going up 90 percent in New Hampshire because of Obamacare is the latest sad reminder of the consequences of this broken law. This is what happens when politicians put the interests of their national party ahead of the priorities of their state and vote 99 percent of the time with President Obama.” Brown took a similar shot via Twitter.

The 90 percent rate hike figure also was cited in a New Hampshire Union Leader news story and editorial, and in Forbes. None of them noted the fact that the 90 percent figure for New Hampshire was based on one anonymous insurance broker.

Robert Santos, chief methodologist at the Urban Institute and president of the executive council at the American Association for Public Opinion Research, dismissed the Morgan Stanley survey as “typical financial marketing material where they pull together some data and create insights.” But without any explanation of its methodology or margin of error, he said, it has no scientific validity. That’s true of the aggregated nationwide results, he said, but especially the state results which are based on so few responses.

“Anyone would be on very tenuous ground in trying to make a state-specific inference,” Santos said.

And in the case of New Hampshire, Santos said, where there was just one respondent, unless that person was the exclusive broker for the entire state, it’s no better than “calling one person up on the phone and asking their opinion.”

Moreover, the 90 percent figure is wildly different than what the New Hampshire Department of Insurance is seeing.

“Premium rates in New Hampshire, in the aggregate, did not go up 90 percent,” Danielle Kronk Barrick, director of communications for the New Hampshire Department of Insurance, told us via email. “An independent actuarial analysis modeling the Affordable Care Act impact on New Hampshire policyholders found that, on average, New Hampshire  policyholders will realize an 8 percent rate decrease after subsidies.”

She’s referring to a 2012 analysis performed by Gorman Actuarial LLC, which concluded (on page 8) that in 2014 “after the application of federal premium tax subsides, the Individual Market will experience a 9 percent premium decrease overall.” So far the numbers underlying those projections are holding up, said Kronk Barrick. “We reviewed and approved the 2014 rates, and they were in line with Gorman’s projections,” she stated.

New Hampshire’s insurance commissioner, Roger Sevigny, was reappointed for a third term last summer by Democratic Gov. Maggie Hassan, a supporter of the ACA.

Kronk Barrick cautioned, however, that “individual experiences may vary significantly.”

Indeed, as the Gorman report noted, when subsidies are factored in, “34 percent of the market will receive on average a 63 percent premium decrease.” However, the report adds, “even after the subsidies, 30 percent of the market will still experience significant increases, averaging 57 percent.”

Those getting decreases are generally people who earn below 400 percent of the poverty level; and those seeing premium increases are generally higher income, younger, and healthier. We have been saying for quite some time now that readers ought to be careful about generalizations made from both sides about rate increases or decreases due to the ACA, because the results will vary widely among individuals.

It’s too soon to know if those Gorman projections will play out precisely as forecast, but they are at least based on a data-driven actuarial analysis, not the word of one insurance broker.

Update, April 24: The conservative group Americans for Prosperity released a TV ad that includes the bogus claim that health insurance premiums are up 90 percent in New Hampshire. Text in the ad attacking Sen. Shaheen states, “Health Care Premiums up 90% in New Hampshire.” The ad cites Forbes as the source, but that’s actually a Forbes blog post about the Morgan Stanley survey. The post does not mention that the New Hampshire figure is based on the word of one insurance broker in the state.

— Robert Farley