A New Health Care Project Won Awards. But Did It Really Work?

The program seemed like a fantastic idea at first, says Manoj Mohanan, an assistant professor of public policy and economics at Duke University.

It's called the WHP-Sky Program. The idea behind it was to transform health care in rural India, where doctors are scarce. WHP hoped to set up franchises where patients could get electronic advice from doctors with degrees instead of less-qualified health workers.

The program got a lot of support, including $23 million in funding from the Gates Foundation (which is a funder of NPR and of this blog). After the 2011 launch, the Gates Foundation commissioned a team of independent scientists, including Mohanan, to review the project. They found that after its first three years of its operation, WHP-Sky seems to have had zero impact on India's rural landscape.

Doctors are scarce outside of India's bustling urban centers. Qualified health professionals are loathe to leave the comforts of city life, says Abhijit Banerjee, a professor of economics at MIT. "Doctors just don't like practicing in rural areas. They find it inconvenient to live there." That leaves over 70 percent of the country's population — some 876 million people — without any doctors in easy reach, Banerjee says.

Instead, Indians in rural areas rely on individuals who have no formal medical degree, making it illegal for them to practice general medicine in India. "They are usually people with a little health experience," Banerjee says. "They worked in a pharmacy. Or in some cases they were the janitor in a public health facility. Some have some kind of training, maybe [studied] as a pharmacist or a nurse. When we looked in 2001, a fifth had no high school degree."

In 2011, the WHP-Sky Program started setting up digital hubs in some of these informal clinics so rural patients could video chat with doctors in urban centers. The WHP-Sky Program trained local informal care providers to run the connection. In theory, people would be able to get qualified health care as long as they lived near one of these internet or telemedicine hubs.

And WHP wanted the program to be self-reliant and scale up to become the dominant health provider in rural India, according to Prachi Shukla, World Health Partner's India director. So, they designed the WHP-Sky Program as a franchise business. Informal health providers would brand themselves as a "SkyHealth" clinic, and pay a franchising fee — about 50,000 rupees or $750 dollars — for the fee and equipment. Patients would pay a little extra — somewhere between 50 and 200 rupees or roughly 75 cents to $3 — to be able to electronically commune with a doctor.

A couple of years after its startup, the WHP-Sky Program started collecting honors like the 2013 Skoll Award for Social Entrepreneurship and the Schwab Foundation Award for being an innovative concept.

"It's a rather compelling idea. Use technology to leapfrog an otherwise large problem in global health," Mohanan says. "A lot of international donor agencies and agencies giving the awards were keen on recognizing the opportunity. The problem was we hadn't the data yet."

When WHP-Sky got their first award in 2013, Mohanan says he and his team still had no clue whether or not the program was actually working.

The results of that evaluation were published this month in the journal Health Affairs. Mohanan says he and his team found two main failures of the WHP-Sky Program — one involving the clinic owners, the other involving the patients.

It turned out many informal health clinics just had no interest in joining a franchise because their businesses were already thriving. They didn't see the need to pay a fee to join a brand, Mohanan says.

In 2014, about three years after its launch, WHP-Sky had only managed to sign up 6 percent of informal clinics in their pilot districts. That adds up to roughly 9,000 clinics, about 10 percent of informal providers in the area.

"A related failure [is] that the people who wanted to join were not the leaders," Monahan says. Instead, the informal health providers who did sign up tended to be newer, less experienced providers who needed to improve their business. "The providers that are already doing well with lots of patients, what do they stand to gain [by joining the franchise]?" asks Meenakshi Guatham, a health policy and management researcher with the London School of Hygiene and Tropical Medicine who did not work on the study. "They don't need a brand."

As for failure number two, there seems to be no indication that patients were attracted to the SkyHealth brand, Mohanan says.

The study found that WHP-Sky clinics were treating roughly 3 percent of the population, despite enrolling 6 percent of health providers in the target area, a state called Bihar. Banerjee, who was not involved with the study, says that's not too surprising considering that it's hard to convince someone to switch caregivers if they already trust their health care provider.

On top of all that, the team checked to see if WHP-Sky clinics tended to treat two common childhood illnesses, pneumonia and diarrhea, correctly more often than the regular clinics did. Mohanan says there doesn't seem to be any difference between the two types of health care providers. "I'm not sure the patient is receiving any better care," he says. "So that, after all this time, money and investment, we don't see any significant changes [in health care]."

But World Health Partners doesn't see it that way. "We respect the research. We don't want to contradict it," Shukla says. "But there were very positive trends [recently that] the paper doesn't talk about." WHP-Sky has been supplying basic medicines like antibiotics, zinc and oral rehydration salts to over 10,000 pharmacies, and Shukla thinks that's led to a drop in childhood pneumonia and diarrhea cases. She says more informal care providers become aware of proper treatment for diarrhea in the time the WHP-Sky Program has been running, too. "There could be other factors related like sanitation, but we believe we've contributed to this."

Mohanan says it's true that there's less pneumonia and diarrhea in the area, but he didn't see any evidence that the positive changes in Bihar came as a result of the WHP-Sky Program. "In some areas of Bihar they see big changes, and they start believing they are the drivers of the change. That's just not the case," he says. "They happen to be in this place as these big changes come."

The paper's results don't mean that the project was a failure or a waste of money though, Gautham says. "We shouldn't dismiss any program in informal providers because it's such a difficult area," she says. "There's still a whole lot about the program that we don't really know, and maybe there are ways to improve it."

A different project that Banerjee worked on simply provided 150 hours of medical training to informal health care providers. After the course, Banerjee found that informal health providers performed significantly better than before. Informal providers who didn't get any training managed cases correctly about half of the time. Providers who were trained managed about 60 perent of cases correctly. By comparison, doctors managed cases correctly two-thirds of the time.

Mohanan thinks perhaps the WHP-Sky project could add a similar course to improve health outcomes in their clinics. "If they give a training like what Abhijit [Banerjee] did, I think the program would have done enormously differently," he says.

The Gates Foundation wrote to Goats and Soda that they're revising the WHP-Sky model and reviewing Mohanan's findings to guide future investments.

Shukla says WHP-Sky was always supposed to be something of an experiment.

"This is a learning project, and it was test [of] what hasn't ever been tested before," she says. "Whether the project has failed or made a good impact, the lessons we've learned from the project are very crucial."

For one, Shukla says that people are pretty content with what they get from their informal health care providers, so future projects should enable providers to offer treatments and services that currently aren't available, like prenatal care.