Billions Go To Victims Of Disaster And Disease. Does It Really Help?

The news is filled with stories of people in need. Perhaps they've just lived through an earthquake. Or they're war refugees. Or they're facing a deadly epidemic like Ebola.

Your heart goes out to them. And you're gratified that your government and other major donors are stepping in with a massive infusion of cash and other aid.

But how do we know if all that help is actually, well, helpful.

Too often, the national or multinational donors and aid groups providing the assistance in a disaster don't check with the people on the receiving end, says Nick van Praag, who's held high positions at the office of the United Nations High Commissioner for Refugees, the Aga Khan Development Network and the World Bank. "The so-called beneficiary is treated rather like the proverbial child — seen and used for fundraising purposes but not really heard," he says.

Or perhaps a relief organization does consult the beneficiaries — but only at the start, when they're designing their response to a disaster. And perhaps they just talk to a focus group that yields only anecdotal information. After that, there's often little monitoring of a program's impact as it rolls out, says van Praag. Instead aid groups and donors generally wait until a program is concluded to commission an in-depth evaluation.

Those final reviews may also be too narrow in scope, he adds. "They essentially monitor outputs — this number of tents erected, this number of latrines dug. They want to know, did you do the things you said you would do rather than evaluating whether any of those things ended up being useful."

Van Praag stresses that none of this is due to callousness on the part of aid groups. "It's not that people [on the giving side] don't want to know what the beneficiary thinks, it's that they haven't really developed the tools to do so."

That's where van Praag's Ground Truth program comes in. Launched in 2012 under the auspices of a small nonprofit called Keystone Accountability, Ground Truth aims to provide continuous feedback on a relief program's effectiveness by surveying the people the program is supposed to serve as frequently as once a week. The results are then tracked in real time. "It's like having radar in the cockpit of your aircraft that you're constantly checking so you can adjust if there's anything in your way," says van Praag.

Ground Truth kicked off with two small-scale pilot projects: one tracking the response to floods in Eastern Pakistan, the other examining the earthquake recovery effort in Haiti. The surveys were funded through grants from the Ikea Foundation and other sources.

This fall, Ground Truth got its first job from a donor directly responsible for disaster relief: The British government hired the group to track local perceptions of the international community's response to the Ebola epidemic in Sierra Leone. About 350 people from different regions of the country — along with a similar number of front-line aid delivery workers — are polled using text messages sent to their cell phones. They're given a small amount free phone time as an incentive to participate.

The decision to hire Ground Truth "signals an important new frontier in disaster relief," says W. Courtland Robinson, Deputy Director of the Center for Refugee and Disaster Response at the Johns Hopkins Bloomberg School of Public Health. And while he's not aware of another organization quite like Ground Truth, Robinson says the group's work fits into a growing trend in global health of trying to incorporate more information from people in the field — at least partly fueled by the growing availability of cell phones. Organizations are experimenting with mapping areas of need based on texts sent by people who live there. Researchers are sending field workers out with smartphones to track deaths in a family over time and other measures of health. "The spread of cell phone coverage across remote areas of the world is revolutionary, and we're kind of just catching up to the opportunities in public health," says Robinson.

While cell phones are a major part of Ground Truth's model, they're not crucial. In Pakistan, the group sent surveyors door-to-door, because many people didn't have cell phones in the area where the project was focused. In Haiti, they used a hybrid arrangement: The surveyors went door-to-door but were equipped with smart-phones that let them send in results directly from the field.

Ground Truth's survey design borrows heavily from the approach the private sector uses to measure customer satisfaction. For instance, many for-profit companies put great stock in consumers' answer to a single question: "On a scale of zero to ten, would you recommend this product to a friend?" The percentage answering yes is known as a company's "net promoter score." And "companies like Apple and Nike are as aware of their net promoter score as they are of their share price," says van Praag.

Similarly, Ground Truth tries to come up with surveys that are at once brief and revealing. The number of questions is kept to about five. But like the net promoter score the questions are intended to get to the heart of whether a particular disaster response is achieving its larger goal — to crystallize overall effectiveness in just a few data points.

In Sierra Leone they're attempting to do this by asking just six questions once a week to a representative sample of ordinary Sierra Leoneans chosen randomly each week. The first five questions get at highly specific issues:

• Are people worried about the way they're dealt with at the checkpoints set up to monitor travelers for Ebola?

• Do people trust the information they're getting about how to prevent the disease?

• Are families able to make a living these days?

• Does a lack of food and water make people worried about being put in quarantine?

• Are people ready to welcome Ebola survivors back to their communities?

The final question cuts to the chase: "Overall, is the Ebola response making progress against the spread of the disease?"

"Sometimes the most obvious question is the best one," says van Praag, laughing.

This abbreviated approach doesn't necessarily give a full picture of how a project is playing out. But van Praag says that when a response is not working, the surveys can at least provide early warning — "like the canary in the mine shaft" — so that aid groups can investigate further: Is the work that's being done just not getting communicated to the public? Or is there some fundamental problem with the response?

For instance, in Pakistan, Ground Truth was tracking a project that was supposed to better prepare the affected area for future floods. After several months, the survey found, many people still felt unprepared. Following up on that finding, says van Praag, it emerged that the project had made a lot of progress on badly needed engineering work on infrastructure but had been slow to improve the ability of individual homes to withstand a flood. "It gave an incentive to people running the program to get things moving faster on that front," he says.

In Sierra Leone, the survey results are encouraging. They have largely reflected the government and international community's overall progress toward reducing the number of new Ebola cases.

But the findings have also pointed to some room for improvement. For example, while worries over harassment at checkpoints have steadily dropped in most parts of the country over the last several weeks, more than the 70 percent of respondents in the east remain uneasy. Similarly, there is still a great deal of concern that people in quarantine lack sufficient food and water. And those worries have been rising in the north, where much of the outbreak is currently focused.

Van Praag doesn't particularly want Ground Truth to continue conducting surveys like this in the long run. Instead, he hopes the initiative's example will spur aid groups to incorporate this kind of continuous surveying into their own work. Ground Truth would then transform into an advisory group, developing best practices and disseminating them. "We would become stewards of the methodology, if you like," he says.

At that point, van Praag adds, his group will confront the bigger challenge: getting organizations to actually modify their programs in response to real-time feedback. Robinson agrees: "That's where the institutional politics comes into play."

One way to get over the challenge is if major donors start requiring a culture of continual course correction. But van Praag says the most effective pressure could come from the beneficiaries being surveyed. After all, theirs should be the voice that is hardest to ignore.