Michael Clemens on the Millennium Villages Project:

The only independent evaluation of the MVs is currently planned to proceed in three waves: baseline, year 3, and year 5 of the project. That is, the evaluation has no publicly-stated plan to proceed long past the end of the five-year intervention in each village, before deciding whether or not it would be right to vastly scale up the intervention all across Africa. A recent research paper starkly showed how inadequate such a stance can be.

That paper, by Shaohua Chen, Ren Mu, and Martin Ravallion, is a lesson in humility (ungated version here, published here). It studies the Southwest Project in China, a village-level development package intervention executed in 1,800 rural villages in the late 1990s. Like the Millennium Village intervention it targeted the poorest villages, lasted about five years, and cost hundreds of thousands of dollars per village. It sought to permanently reverse the fortunes of those villages with a broad-based package including roads, piped water, power lines, upgrading schools and clinics, training of teachers and health-care workers, microcredit, and initiatives for raising crop yields, animal husbandry, and horticulture.

Right before the end of the Southwest Project intervention, five years after it started, the project seemed to indeed be reversing the fortunes of the treated villages. Income in those villages grew by 20% more during the project than in similar villages in the same area that had not received the intervention, and savings grew by 100% more.

Then the intervention ended and—fast forward five years—all those effects on income and savings disappeared. Ten years after the five-year project began, average income and savings in the villages that got that massive package of interventions were indistinguishable from income and savings in villages that did not.

Notably, incomes in both the treated and untreated Chinese villages in the Southwest Project area increased greatly during the span of the project (1995-2000) and for years thereafter. The reason this happened is because the Chinese economy was being transformed during this period, not because of massive village-level package development interventions.

The point here is not that the Southwest Project was the same as the MVP; it wasn’t. The point is that short-term evaluation is plainly inadequate. It is obvious that the MVP is going to have short-term impacts. The size of the intervention is the same order of magnitude as the size of the entire economy of each village; that is, the MV intervention is roughly 100% of local income per capita (see the bottom of this post for this calculation). Indeed, it would be astonishing to not see short-term effects with an intervention that gargantuan. The three-year evaluation results that the MVP plans to release this year simply won’t tell us much.

The only interesting evaluation question is in the long term, for three reasons: 1) because unlike short-term impacts the answer is not obvious, 2) because long-term change is the stated goal of the MVP, and 3) because other village-level package interventions have shown that short-term effects and long-term effects can be completely different from one another.