Which Foreign Aid Programs Work? The U.S. Runs A Test — But Won't Talk About It

It was the summer of 2013 and Daniel Handel had just moved to Rwanda. He was unpacking boxes in his new house, when his wife walked over with her laptop and said, 'You have to listen to this radio story!' The piece she played him was by NPR's Planet Money team, and it profiled a charity that was testing a bold idea: Instead of giving people in poor countries, say, livestock or job training to help improve their standard of living, why not just give them cash and let them decide how best to spend it?

Handel was fascinated — because for some time he had been mulling this exact question. He was an economist with USAID — the U.S. government's main agency for distributing foreign assistance. And although he'd only been with USAID for two years, Handel had already grown concerned over what he saw as an agency habit of spending enormous sums on programs to help poor people, only to boost their incomes by a fraction of the program's per-person cost. At this rate, Handel had often thought to himself, might it be better to just hand over the money to people directly? Now it turned out there was a charity doing precisely that. (Indeed the charity's name is GiveDirectly.)

Handel wasn't ready to propose completely scrapping traditional programs in favor of cash aid. But at the very least it seemed reasonable to make sure that any given traditional program produced more benefits than simply giving people an equivalent amount in cash. In other words, he wanted to use cash aid as the benchmark by which all other forms of aid are judged.

So Handel went to his bosses at USAID's Rwanda office and proposed an experiment: Take one of USAID's typical programs and test it against cash aid. His initiative has since grown to encompass six experiments in four countries. He is currently overseeing these tests from a new position, senior adviser on aid effectiveness at a USAID research unit in Washington, D.C.

USAID would not grant permission for Daniel Handel or any other agency official to do an interview with NPR. The account of his wife playing him the radio story and his subsequent experiences launching the cash-benchmarking experiments were described to NPR by multiple sources, including former USAID colleagues who worked extensively with him on the experiments.

It will take several years for most of the findings to become available. But on Thursday, the government released the results of the first study in the series: An evaluation of a program to improve child and maternal health in Rwanda by teaching families about nutrition and hygiene. The experiment found that the program met none of its main objectives. (Offering people the equivalent amount in cash — about $114 — also did not improve their nutrition or health. But providing a much larger cash grant of about $500 did make some difference.)

Supporters of such "cash-benchmarking" exercises are heralding this particular one as a milestone. For years, anti-poverty advocates and researchers have complained that the U.S. government doesn't do enough to make sure its aid programs actually work. "The number and quality of impact evaluations at USAID has risen over time, but it's still really small," says Amanda Glassman, who heads the think tank Center for Global Development. Now, she says, USAID's nascent cash-benchmarking experiments suggest the United States – which is one of the world's largest donor countries — could be on a path to greater accountability.

But the many hoops that Handel has already had to jump through may speak just as powerfully to the challenges of truly transforming how USAID does business.

As does the reaction to the study's findings within USAID itself.

Asked why USAID would not authorize an interview with Handel or any other official to discuss the cash-benchmarking experiments, an agency spokesman declined to comment but provided a statement. It said, in part, "cash-benchmarking is not relevant to most of USAID's programs." The statement further noted that this particular experiment was "very limited" and commissioned "under the Obama administration," and concluded that "there is not sufficient evidence that this approach can be applied to development programs on a broader scale" even as USAID "is consistently looking for ways to develop, test and validate new and innovative approaches."

A "Quixotic" Quest

Joaquin Carbonell, one of Handel's former colleagues interviewed by NPR, recalls being a little surprised when he first laid eyes on him. The two had started collaborating on the cash-benchmarking effort while Carbonell was based at USAID's Washington, D.C. headquarters and Handel was still in Rwanda. They spoke regularly over the phone for about a year before they finally met in person.

"I was expecting this big person and here's this five-foot-seven red-headed guy in a three-piece-suit," says Carbonell, who is now a fellow at the Harvard Kennedy School of Government. "He's so full of energy and he was pushing this thing with such verve, he's just larger than life in my mind. He likes to say that he's from New Jersey, so he doesn't take crap from anybody."

It was a useful quality as Handel embarked on a quest that Carbonell says was "quixotic from the beginning."

He did, at least, have some key allies. Handel's bosses at USAID Rwanda were very much on board. As were the leaders of the division where Carbonell worked — the Global Development Lab, a unit within the agency that helps fund and evaluate innovative initiatives.

The lab also provided Handel with some crucial connections. Another colleague there happened to know that Google's charitable arm was interested in funding further experiments with cash aid by GiveDirectly, the charity featured in the Planet Money radio piece Handel had heard. GiveDirectly, which was founded by four American graduate students of economics, was an appealing potential partner for Handel because it had already worked out an efficient system for identifying recipients and delivering cash to them through mobile phones. At that point its work was limited to Kenya. But Handel quickly hammered out a plan to collaborate with the charity on launching a cash aid venture in Rwanda that could be tested by USAID as part of what became a $4 million venture.

From that point, however, the process ground to a crawl as the team began a months-long series of briefings of USAID higher-ups and members of Congress.

Traditional-style aid programs with much larger budgets are often pushed through in far less time, says Carbonell. "But when you talk about giving money to people straight up, with no conditions, people at USAID look at you kind of like you're a crazy person."

To be sure, he says, there are reasonable questions about what the evidence suggests cash grants can actually accomplish. In fact it's the subject of a lively debate among experts well beyond USAID.

Berk Ozler, lead economist at the World Bank's Development Research Group, is one of the cash-aid skeptics. Over the last decade, notes Ozler, cash aid has evolved from a niche idea to one of the hottest research topics among poverty researchers. The result has been a pile of studies suggesting that in many cases cash grants can provide a more effective safety net than other forms of aid. In the process, "all of a sudden the narrative [on cash aid] became, 'This is the panacea.' "

The trouble, says Ozler, is that most of these early studies necessarily focused on short time frames. But when it comes to the ultimate goal of many aid programs – permanently lifting people out of poverty – "unfortunately the recent slew of papers [on cash aid] that have been coming out are not promising."

Still, Ozler says, this doesn't mean traditional programs are any better. So he's all in favor of using cash aid as a benchmark to ensure that all programs, whatever their type, are as cost-effective as possible.

So why did the idea cause jitters at USAID? Carbonell says a lot of the concerns expressed to him while he was there seemed to be less about substance than about optics.

"In this country we don't like giving poor people money," he says. There's "an inherent sense" that they can't be trusted to spend it wisely. No matter, Carbonell adds, that the research strongly indicates that poor people don't spend cash aid on vices like tobacco and alcohol.

"We don't want to be seen as just giving handouts to poor people," he says. "And that is a very deeply ingrained fear at USAID."

It's also the reason, "there was all this language wrangling that happened. The lawyers [within USAID] were saying, 'Well, we can't call these cash transfers. We've got to call it household grants.' It was a year of just bureaucratic jiujitsu to get this thing to a point where everybody had signed off."

Lucky Timing

By then it was August of 2015. And in a certain sense the timing was lucky because it just so happened that USAID Rwanda was about to roll out a major nutrition and health program in cooperation with the Rwandan government. The country has made tremendous strides in improving its population's health in recent years. But one stubborn problem that remains is malnutrition among children. Large shares of Rwanda's kids are underweight and short for their age – a sign of serious nutrition deficits that cause health and cognitive problems into adulthood and that often contribute to keeping people in poverty.

The new program would be sending USAID-funded teams from village to village, identifying at-risk families and educating the parents about practices that can make a difference – how to plant a more diverse range of foods to feed their kids, for example, or how to follow basic hygiene routines that prevent diseases like diarrhea, which contribute to poor growth in kids.

"They teach people about hand-washing practices. How where people go to the bathroom relates to the likelihood of getting sick afterwards, things like that," says Andrew Zeitlin, an economist at Georgetown University who was on the team that USAID commissioned to compare the program with cash aid.

Zeitlin adds that there was a lot to be gained by assessing whether this particular program was effective. "It's a nationwide program. And it's in sync with the way lots of aid agencies and national governments in neighboring countries and around the world try to tackle these problems."

The experiment that he and his collaborators set up was a rare head-to-head, simultaneous comparison of cash aid and a traditional program. A pool of families from nearly 250 villages was selected based on the typical criteria for determining who was at risk of malnutrition, then randomly assigned to one of four groups. Those in the first were the "control" and received no help. Those in the second group were visited by the teams from the nutrition and hygiene program. Families in the third group were given small cash grants by GiveDirectly equivalent to the per-person cost of the nutrition and hygiene program, which ultimately averaged out at $114. In the final group, families got a much larger cash grant of around $500 – a figure chosen because this was the amount that GiveDirectly estimated was more likely to make an impact.

Then independent evaluators employed by another partner, the research group Innovations for Poverty Action, monitored a host of measures over the course of a year: How much did the children weigh? How much food were they consuming? How diverse was their diet?

"It's a real risk."

Even as the study was being set up, it seemed so promising that officials back at USAID's Global Development Lab in Washington – specifically its Development Innovation Ventures, or DIV, department – agreed to help Handel launch similar experiments in other countries. Soon they hired him to oversee the effort.

Anne Healy was working at the U.S. State Department when word filtered to her of this, as she puts it, "stealth mode" plan to introduce cash-benchmarking at USAID. Healy was so intrigued she decided to pursue a job as head of DIV.

"It was really an exciting moment to be able to say, 'Okay, this is not just one experiment happening in one place with one program. This actually has the potential to be a global initiative at USAID and really, with that, the potential for transformational change.' "

But as Handel and members of the DIV team traveled through about 30 countries pitching the USAID country missions on the idea, they ran into a new obstacle. Over and over again, Healy says, staffers at the country missions would say, this is politically risky.

"It was a theme that every mission had to grapple with," she says. "If the results of a cash benchmarking experiment demonstrate that our non-cash USAID program was either lower impact or less cost effective ... that would open up USAID to criticisms from Congress and the American public."

Such a reaction would be unfair, adds Healy, who has since left USAID for the nonprofit Evidence Action. She argues that a disappointing assessment shouldn't lead to the conclusion that foreign aid is wasteful and pointless. It's just a sign you need to figure out how to improve the program.

But Healy doesn't fault USAID staffers for worrying about a possible backlash from Congress or the public. "It's a real risk," she says. And she adds, the risk has only increased since the election of a president who has complained that the U.S. spends too much helping people in other countries instead of looking after its own, and whose proposed budgets have included deep cuts to foreign aid.

"I think it's a particularly challenging time for government agencies to pursue evidence-based policymaking," says Healy.

Yet for all the qualms, by the fall of 2017 the USAID offices for three more countries – Liberia, Malawi and the Democratic Republic of Congo — agreed to set up cash-benchmarking experiments. And the Rwanda office agreed to a second experiment testing the effectiveness of a job-training program.

Carbonell says he and Handel made a point of setting up these experiments on a timetable that ensured they would be well underway before the results from the study of the Rwanda nutrition program came in. There seemed a good chance that the program would compare poorly to cash aid on, at the very least, cost-effectiveness given how little overhead cash aid requires. "We didn't want to give people the opportunity to get cold feet," says Carbonell. "And you know these kinds of findings could really force a strong reaction from the leadership within USAID."

"Our hearts sank."

In February of this year, the results of the Rwanda study came back. Zeitlin, the independent researcher, recalls his reaction as he and his collaborator, Craig McIntosh of University California San Diego, combed through the data: "I mean, our hearts sank."

A year on, the children who had been targeted by the nutrition and hygiene program were no more likely to eat a better or more diverse diet, and no less likely to be malnourished or anemic than children who had gotten no help at all.

The program's focus on trying to change behaviors is one of the world's major strategies for ending malnutrition. And, at least in this example, it had failed to achieve any of its primary goals.

But within USAID a pushback was brewing. Over the ensuing weeks experts on nutrition and other relevant areas questioned just how illuminating the study really was. Among the complaints, says McIntosh, "there is a general sense that the time frame of the study being only one year is problematic."

Running the study over a longer period wasn't an option because it would have required leaving the families in the control group without help for that much longer. And this was a program that Rwanda's government wanted to extend to the entire population.

The World Bank's Berk Ozler, who has reviewed the study, agrees the time horizon was less than ideal. "It's a good study. But there is that limitation," he says.

But McIntosh says it's worth noting that when it comes to child nutrition programs time is of the essence.

And the fact that the larger grant of $500 did improve children's health, albeit slightly, suggests it's not impossible to make a difference in a short period of time.

He and Zeitler say perhaps the answer is to shift the focus from trying to change people's behavior in favor of getting them additional resources.

Michael Faye, co-founder of GiveDirectly, the charity that ran the cash-aid arms of the experiment, emphasizes another takeaway. Giving people no-strings cash grants lets them pursue their own priorities. And so cash aid tends to point up when those priorities might be different from the goals of the donors who fund traditional programs. In this case, for example, people who were given the cost-equivalent grants used much of the money to pay down their debts. Perhaps in their view that was a more urgent need than giving their children more food.

It doesn't mean they are right, says Faye. But if donors are going to override recipients' preferences, they should at least recognize that's what they're doing. "For a long time the donor's vote has counted for almost everything," says Faye. "In people's lives their vote should count for something."

As for USAID's take, it remains unclear what, if any, material changes the agency is planning to its nutrition efforts based on the study's findings. They were published on a government website this week in accordance with a statutory requirement that the results of government-funded studies of foreign aid must be made public within 90 days of their completion.

Healy says she's at once concerned and hopeful. As hard as it was to launch the cash-benchmarking experiments, "the true test will be how USAID acts on these results."

Copyright 2018 NPR. To see more, visit http://www.npr.org/.