Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Journal of Globalization and Development

Ed. by Stiglitz, Joseph / Emran, M. Shahe / Guzman, Martin / Jayadev, Arjun / Ocampo, José Antonio / Rodrik, Dani

2 Issues per year

Online
ISSN
1948-1837
See all formats and pricing
More options …

Evaluating Antipoverty Transfer Programmes in Latin America and Sub-Saharan Africa. Better Policies? Better Politics?

Armando Barrientos
  • Corresponding author
  • University of Manchester, Brooks World Poverty Institute, Manchester, UK
  • Email:
/ Juan Miguel Villa
  • University of Manchester, Brooks World Poverty Institute, Manchester, UK
Published Online: 2015-05-01 | DOI: https://doi.org/10.1515/jgd-2014-0006

Abstract

Two broad explanations can be offered for the incidence of impact evaluations in antipoverty transfer programmes in developing countries. The first, and arguably dominant, explanation suggests this is a consequence of a shift towards evidence-based development policy. A second explanation emphasises the complementary role of policy competition and political factors in motivating evaluations. The paper assesses the relevance of the latter in Latin America and sub-Saharan Africa through (i) a comparison of evaluation practice and (ii) the analysis of a new database of flagship antipoverty transfer programmes.

Keywords: antipoverty transfers; impact evaluation; Latin America; poverty; sub-Saharan Africa

JEL: D72; H43; I32; I38

1 Introduction

The growth of antipoverty transfer programmes in developing countries has been a distinctive feature of development policy and practice in the last decade. Programmes providing direct transfers in cash and in kind to households in poverty have sprung up in all developing regions, first in middle income countries but more recently spreading to low income countries. Combined with policies enhancing growth and supporting the provision of basic services, antipoverty transfer programmes have the capacity to make a significant contribution to the global reduction of poverty and vulnerability. The incidence of impact evaluation, uneven such as it is, marks out cash transfer programmes among development interventions. Two broad explanations can be put forward to account for the relatively more intensive use of impact evaluation in antipoverty transfer programmes. One explanation emphasises the influence of an epistemic shift towards evidence-based development policy (Banerjee and Duflo 2011). A second explanation emphasises the influence of policy competition and political factors, the need to persuade reluctant policy-makers and electorates of the benefits of effective antipoverty transfers (Levy 2006). The paper throws light on the relevance of the latter explanation in the context of a comparison of the incidence of impact evaluations of antipoverty transfer programmes in sub-Saharan Africa and Latin America.

There is a strong technical case for supporting rigorous monitoring and evaluation of antipoverty interventions (Glennerster and Takavarasha 2013). Government agencies responsible for antipoverty policy furthermore have a duty to ensure programmes are effective, are based on available knowledge, and include protocols to learn from their implementation. Monitoring and evaluation processes are essential to test innovative programmes or programme features. They are an important tool to develop implementation capacity and to enable adaptation to changing environmental conditions. Monitoring and evaluation are essential instruments for improving government effectiveness. The technical case for evaluating antipoverty transfer programmes is strong but account must be taken of associated costs and methodological challenges (Barrett and Carter 2010).

In the context of antipoverty transfer programmes, impact evaluations also have a policy competition and political dimension which can contribute to the explanation of their incidence. The shift to evidence-based policy explanation of the growing use of impact evaluations does not translate directly into effective political demand factors.1 Criticising a dominant focus on assessing mean impacts, Heckman et al. (1997) note that these findings are unlikely to be of interest to voters, whether they are self-interested or altruistic. For self-interested voters in median voter models, “the mean is irrelevant unless it coincides with the median” while for altruistic voters, a concern with the worst off suggests an interest in the specific impact of interventions on them as in the Rawlsian maximin. Explaining the incidence of impact evaluations in antipoverty transfers requires examining their interactions with policy and political processes as the source of effective demand factors. In countries where elites are resistant to their introduction and scaling up, as has been the case in many countries in sub-Saharan Africa, impact evaluation findings can have powerful demonstration effects. As discussed further below, experience in multiple countries suggests that rigorous programme evaluations can help to overcome opposition to antipoverty transfer programmes. Opposition to antipoverty transfer programmes can emerge from many quarters, including from agencies involved with competing development programmes, from interest groups seeking to protect their position and influence within government, from politicians keen to use public policy as a means to strengthen electoral support, and from sub-national governments seeking to prevent the centralisation of power. Well-designed impact evaluations can provide crucial ammunition to those advocating antipoverty transfer programmes and support pro-poor coalitions. Where antipoverty transfer programmes are in place, programme agencies use impact evaluations as a means of protecting themselves and their budgets from day-to-day interference from politicians and bureaucrats. Impact evaluations can also strengthen medium-term sustainability by firming up public perceptions of programme effectiveness.

The approach adopted to examine the factors influencing the incidence of evaluation in antipoverty transfer programmes is twofold.

First, we examine the contrasting experiences of Latin America and sub-Saharan Africa as regards the evaluation of antipoverty transfer programmes. In Latin America, it is broadly accepted that the rapid spread of human development conditional income transfer programmes – also known as conditional cash transfer programmes – has been facilitated by the rigorous impact evaluation protocols implemented in Mexico’s Progresa/Oportunidades and the findings thus generated (Rawlings and Rubio 2005; Fiszbein and Schady 2009). In sub-Saharan Africa, on the other hand, it has been argued that the spread of antipoverty transfer programmes has been slowed down by the limitations of monitoring and evaluation processes (Garcia and Moore 2012). A comparative discussion of evidence from these two regions can thus throw new light on the influence of policy competition and political processes. We find that political processes contribute to our understanding of evaluation incidence in both regions, but with diverse outcomes due to contextual factors. The Latin American experience supports our initial expectation that the high incidence of impact evaluation in antipoverty transfer programmes is positively associated with the strength of policy competition and political resistance to antipoverty transfer programmes. This should have applied with greater force in sub-Saharan Africa, but our review of the experience there suggests that policy competition and political resistance encouraged instead a focus on pilot programmes as “demonstration” tools diverting pressure on programme agencies to include rigorous evaluation protocols.

Second, we develop and estimate a basic model incorporating the main predicted factors influencing the incidence of antipoverty programme evaluation, using a new dataset of flagship programmes in developing countries supplemented with variables capturing government effectiveness, policy competition and political resistance to transfers. Keeping in mind the limitations of the data, our analysis does not attempt to establish causality, but instead throw light on the relevance of the main explanations offered for the relatively more intensive use of impact evaluations in antipoverty transfer programmes over time.

The structure of the paper is as follows: Section 2 discusses demand factors arising from policy competition and political processes associated with antipoverty transfers. Section 3 provides a comparison of evaluation practices and influences in Latin America and sub-Saharan Africa. Section 4 reports on an empirical analysis of the incidence of evaluations in antipoverty programmes. The final section draws out the main conclusions.

2 Policy Competition and Political Factors in Programme Evaluation

This section provides a brief review of the literature on the influence of policy competition and political factors in programme evaluation.

Policy competition among agencies can generate demand for programme evaluation. They include multilateral and bilateral donors engaged in poverty reduction in developing countries, bilateral donors and the voters and taxpayers they represent, domestic programme agencies, and domestic political parties and constituencies. It can be hypothesised that programme agencies will have strong incentives to implement rigorous impact evaluations (Levy 2006; Niño-Zarazúa et al. 2012). Resistance might come for instance from competing programme agencies vying for limited budgets. In this case, programme evaluation will be more likely the greater the competitiveness of the policy environment, and the number of programmes and agencies involved could serve as a proxy for policy competition. Resistance to antipoverty transfer programmes might come also from strategic imperfections of political processes. To the extent that the introduction of antipoverty transfer programmes could lead to, or require, rules-based resource allocation and/or greater openness and accountability, these programmes will attract opposition from elites benefiting from discretionary and/or closed decision-making processes. To the extent that the introduction of antipoverty transfer programmes could lead to, or require, greater openness and accountability, these programmes will attract opposition from elites benefiting from closed decision-making processes. In both contexts, programme agencies will have a stronger preference for rigorous programme evaluation the more closed or discretionary the decision-making processes are.

Preferences for programme evaluation among multilateral donor agencies could be driven by two main factors: the degree of policy competition within the respective agencies and the incentives for policy innovation embedded in their management structures (Pritchett 2002). Policy competition within donor agencies makes rigorous programme evaluation an important tool to secure resources and influence. Rigorous programme evaluation will be more likely in conditions of stronger policy competition and weaker incentives for innovation within multilateral agencies. The point about policy competition extends to contexts in which several multilaterals and bilaterals operate side by side. Multilaterals and bilaterals often have differences in approach, objectives, capacities, and influence. In the context of antipoverty transfers, differences in approach translate into advocacy and support for specific target groups and policy instruments. For example, the extent to which antipoverty transfers should focus on children, or whether restrictions are appropriate. In aid dependent countries in sub-Saharan Africa, donor influence and donor competition have been important in facilitating or constraining programme evaluation (Devereux and White 2010; Niño-Zarazúa et al. 2012). The structure of incentives for innovation could be powerful drivers of programme evaluation (Pritchett 2002). For bilateral donors, political accountability to politicians and tax-payers in the donor country will, other things being equal, encourage evidence-based policy and by extension programme evaluation. Indeed, influenced by such factors, several evaluations of antipoverty programmes in Latin American were introduced well ahead of the so-called “randomista revolution” and the widespread shift toward evidence-based development policy. In Mexico, for instance, the implementation of Progresa in 1997 followed a pilot designed to assess its effectiveness (Skoufias et al. 2001). Progresa was pivotal in the adoption of similar experimental or quasi-experimental protocols in the impact evaluation of Familias en Acción in Colombia, Bono de Desarrollo Humano in Ecuador, and Red the Protección Social in Nicaragua, which were all introduced in the early 2000s with the assistance of the World Bank and the Inter-American Development Bank (Niño-Zarazúa 2011).

Public attitudes to international aid in donor countries can strongly influence the quantity and orientation of foreign aid (World Bank 1998; Chong and Gradstein 2006). Perceptions of the relative effectiveness of antipoverty interventions among the general public and political elites are also likely to have implications for their adoption and sustainability. Information on effectiveness provided by programme evaluations can therefore influence public opinion and political support for programmes (Graham 2002; Lindert and Vinscensini 2008). In aid-recipient countries, public perceptions regarding the effectiveness of antipoverty transfer programmes are important in aligning public and political support for programmes, especially from voters and taxpayers. To an extent, public perceptions are shaped by outcome and process indicators (Lindert and Vinscensini 2008). Evaluations can also protect programmes and their budgets from undue interference from governing coalitions and from opposition from parliamentary groups.

In sum, policy competition and engagement with political processes and stakeholders can influence the incidence of antipoverty transfer programme evaluation through a variety of channels, supporting the working hypothesis that programme evaluations will be more likely the greater the predicted opposition to antipoverty transfer programmes. Our analysis below finds support for this hypothesis in the Latin America context, but not in the sub-Saharan Africa context. In the latter, policy competition and political resistance to transfers encouraged a focus on pilot programmes. To use of pilots as “demonstration” tools lifted pressure on programme agencies to include rigorous evaluation protocols.

3 Comparing Antipoverty Transfer Programme Evaluation in Latin America and Sub-Saharan Africa

Latin America has experienced a remarkable expansion of antipoverty transfer programmes since the turn of the century. Brazil and Mexico are the pioneers in this respect. In Brazil, the new 1988 Constitution established the right to a minimum guaranteed income, which led to three main policy developments: the incorporation of rural informal workers into the existing social insurance scheme for private sector workers on favourable semi-contributory terms in 1993; a social assistance pension scheme for older people and people with disabilities in extreme poverty established in 1996; and a human development conditional income guarantee programme directed at families with children in extreme poverty (Bolsa Escola) which grew out of municipal initiatives into a federal programme in 2001. The latter was consolidated into Bolsa Família in 2003, together with other transfer programmes. In Mexico, a human development conditional income transfer programme called Progresa, was introduced in rural areas in Mexico in 1998 to address intergenerational poverty persistence. Progresa was evaluated and subsequently extended to urban areas and became Oportunidades (now known as Prospera).

In the new century, the majority of countries in Latin America have followed Brazil and Mexico in introducing antipoverty transfer programmes. Human development conditional income transfer programmes show the fastest growth, reaching 134 million people by 2012 or around a quarter of the population in the region (Stampini and Tornarolli 2012). Non-contributory pensions have also expanded, leading to a rapid improvement in income security for the population over 65 years of age (Rofman et al. 2013). There is considerable diversity and innovation in the design of antipoverty transfer programmes across the region, as well as variation in reach and effectiveness. On the whole, the spread of antipoverty transfer programmes has lagged in lower middle income countries with poor capacity and fiscal space (Barrientos and Santibañez 2009). In upper middle income countries, on the other hand, the consolidation and institutionalisation of social assistance has grown apace. Welfare institutions in Latin America had been described as “truncated” because they reached at best a fraction of workers in formal employment leaving the rest of the population without protection. The emergence of social assistance has extended social protection to excluded groups.

In contrast to conditions in Latin America, antipoverty transfer programmes in sub-Saharan Africa have progressed at a much slower pace. Historically, South Africa and Namibia have relied on transfers in cash as a means of addressing poverty and vulnerability (van der Berg 1997; Lund 2008). Over time, social assistance grants have expanded in their reach and by 2010 one in every two households in South Africa had at least one member in receipt of a transfer (Woolard et al. 2012). More recently, Lesotho and Swaziland have introduced antipoverty transfer programmes for older people and children along similar lines as in South Africa and Namibia. Southern African countries are unique in sub-Saharan Africa in having established large scale antipoverty transfer programmes.

The situation is substantively different in other African sub-regions. There, antipoverty transfer programmes remain limited in scale, scope and institutionalisation (Garcia and Moore 2012; Niño-Zarazúa et al. 2012). With few exceptions, they consist of pilot transfer programmes, heavily dependent on donors’ financial support and technical assistance. In East Africa, the largest programme is Ethiopia’s Productive Safety Net Programme, providing guaranteed employment in food insecurity areas and direct transfers to households without members of working age. It currently reaches around 11% of the population. Kenya’s Orphans and Vulnerable Children Programme reaches around 100,000 households. Mozambique’s Food Subsidy Programme has been in existence in different forms since 1991. It currently reaches around 125,000 households with older people and people with disabilities in acute poverty. Zambia, Malawi, Tanzania, and more recently Uganda and Rwanda have introduced pilot social transfer programmes. In West and Central Africa, several countries are introducing social transfer programmes on a very small scale. Ghana’s Livelihood Empowerment Against Poverty Programme reaches <30,000 households. Nigeria’s In Care of the Poor Programme was intended to reach 1000 households in 12 districts, in a country of 160 million and with high poverty incidence.

Several explanations can be offered for the relatively slow progress of antipoverty transfer programmes in Africa when compared to Latin America. The incidence of poverty is significantly greater than in Latin America, and resources to address it are more limited. Of particular interest in the context of this paper, however, are constraints deriving from the political and policy environment.

3.1 Programme Evaluation in LA and SSA

In Latin America, the recent expansion of antipoverty transfers has been facilitated by the application of experimental and quasi-experimental methods of programme evaluation. The evaluation of Mexico’s Progresa has acquired a paradigmatic status. In advance of the introduction of Progresa in 1997, the Mexican government commissioned the International Food Policy Research Institute (IFPRI) to design and implement the evaluation of the programme (Skoufias 2005). The selection of Progresa participants was done in three stages. First, an index of marginalisation identified the rural communities with the highest levels of deprivation in seven states. Progresa was initially implemented only in rural areas of these seven states and was restricted to communities with <2500 inhabitants. Second, a census in rural areas provided information on the socio-economic status of households in these communities enabling a ranking of households. Third, community validation provided a check on the households selected for participation and led to the inclusion of additional households. Administrative constraints meant that the programme was rolled out first in 1998 in a majority of selected communities, with some were left for a later date. This gradual implementation created conditions approximating an experimental setting, enabling the identification of a treatment group of communities incorporated into the programme in 1998 and a control group incorporated by the end of 1999.2 Difference in difference estimates of impact provided strong evidence of the positive effects of Progresa on participant households. This impact evaluation set a standard for other antipoverty programmes in the region, and elsewhere.

The Progresa evaluation strategy can be contrasted with the other large-scale pioneer flagship programme in Latin America. Brazil’s Bolsa Escola emerged from municipal policy innovation in 1995 in three municipalities. The key innovation was based on the view that for antipoverty transfers to have an impact on poverty it was necessary to combine transfers in cash with conditions in children schooling. In 1997, the federal government offered counterpart funding in an effort to extend the programme to poorer municipalities facing resource limitations. In 2001, Bolsa Escola became a federal programme and in 2003 it was consolidated into Bolsa Familia integrating four other cash transfer programmes, including the Programme for the Eradication of Child Labour (PETI) initiated in 1996. Research into Bolsa Escola at the sub-national level found positive impacts. Research into PETI supported by multilaterals also found strong positive effects (Yap et al. 2002). The Brazilian Audit Court examined PETI and provided a strongly supportive evaluation of the programme (Brazilian Court of Audit 2003). Nevertheless, the evolution of Bolsa Escola and its consolidation into Bolsa Familia was not dependent on the results of impact evaluation studies.3 Political opposition to Bolsa Escola and PETI was very limited, and there was broad support for the use of direct transfers as a means of addressing poverty among Brazilian politicians, especially within the governing coalition. In fact the proliferation of cash transfer programmes in the early 2000s reflected broad-based support.

Evaluation protocols in later antipoverty transfers in Latin America have employed a full range of approaches. Juventud y Empleo (Youth and Employment) in the Dominican Republic, aims to raise the productivity of youths in poverty by providing job training and a cash stipend during the course and internship. The experimental setting ensured that difference in outcomes across the two groups could be attributed to the programme (Card et al. 2011). Soares and Britto (2007) used administrative data to evaluate Paraguay’s Tekoporá, an integrated anti-poverty social transfer. They compiled baseline information on beneficiaries and non-beneficiaries and later administered a follow up survey to support quasi-experimental difference in difference estimates of impact. Levy and Ohls (2010) evaluated the PATH social transfer in Jamaica using a regression discontinuity design. Finally, Attanasio et al. (2010) evaluated the impact of a public nursery programme in Colombia operating since the 1980s relying on instrumental variable estimation to assess nutritional effects on participant children. Impact evaluations relying on observational data are common in the region. Borraz and González (2009) employed propensity score matching methods to evaluate Uruguay’s Ingreso Cuidadano (Citizenship Income). Participants and non-participants were matched according to their probability of participation in the programme. The Ingreso Ciudadano was phased out in 2007 and replaced by other interventions.

In contrast, categorical antipoverty transfer programmes, non-contributory pensions, and disability transfers in particular, are seldom evaluated in Latin America. Categorical antipoverty transfer programmes are rarely supported by donor funds. Instead, they are sanctioned by legislation before their implementation and lack conditions. And because they have weaker implications for work incentives, they tend to attract less critical attention from voters and taxpayers. To the extent that individuals are eligible through personal characteristics like age, there is less discretion on the part of programme agencies. Often, their grounding in social or citizenship rights has precluded demand for their evaluation.

In sub-Saharan Africa, Garcia and Moore (2012) find that only a minority of transfer programmes have any evaluation components. Evaluation components feature in the design of the programmes in only a handful of cases.4 Ethiopia’s Productive Safety Net Programme, for instance, included relatively stronger evaluation components, as well as the participation of IFPRI in its design and implementation (Gilligan et al. 2007; Gilligan et al. 2008). Kenya’s Orphans and Vulnerable Children Programme also includes an evaluation component, implemented by an external consultancy firm, Oxford Policy Management. However, even in these two exceptional cases, impact evaluations suffered from deficient implementation, especially as baseline survey data was collected after the start of these programmes.

In contrast to the Latin American experience, the weakness and non-existence of evaluation of antipoverty transfer programmes in sub-Saharan Africa constitutes a puzzle for the argument presented in this paper. Domestic capacity to produce such evaluations might be a potential explanation, but the presence of donors and the low transactions costs associated with the diffusion of international technical advice and support would militate against placing too much stress on this point. Surprisingly, the high incidence of pilot programmes in sub-Saharan Africa also has not been accompanied by the spread of impact evaluations. Pilot programmes are expected to include strong evaluation components because their raison d’etre is to generate knowledge on the feasibility and effectiveness of a scaled-up programme. In Africa, however, first generation pilot programmes have seldom incorporated strong evaluation procedures (Davis et al. 2012). More recent programmes have paid more attention to evaluation, although this is some way away from the kind of experimental evaluation implemented in Mexico’s Progresa and elsewhere in Latin America. The evaluation of antipoverty transfer programmes in 7 countries5 under the Protection to Production initiative is largely a donor-funded and donor-supported initiative with engagement of the relevant agencies in the countries concerned, as opposed to a domestically driven initiative. The findings from these evaluations are beginning to emerge in the public domain.

As in Latin America, programme type is an important predictor of evaluation. Garcia and Moore (2012) find that, in a sample of transfer programmes in sub-Saharan Africa, experimental evaluation is significantly more common among conditional transfer programmes than among unconditional programmes.

3.2 Sources of Policy Competition and Political Resistance to Antipoverty Transfers and Evaluation Incidence

The Latin American experience illustrates well the influence of policy competition and political factors, alongside the rise of evidence-based development policy, in explaining the high incidence of impact evaluations in. As suggested above, political processes have contributed to shaping the demand for impact evaluations of antipoverty transfers in Latin America in several ways. In the paradigmatic case of Mexico’s Progresa, for instance, Levy (2006) documents the use of impact evaluations to address opposition to innovation from competing agencies in a context of budgetary restriction. Progresa was introduced in the midst of a financial crisis in Mexico, which meant strong competition for diminishing resources with existing programmes and therefore opposition from existing policy constituencies. There was also a sense that the flagship antipoverty transfer programme Pronasol had been compromised by political interference. Proponents of Progresa hoped that rigorous evaluation protocols would help protect the programme. Argentina’s Asignación Unica por Hijo is another example of a transfer programme with strong evaluation processes facing strong opposition from competing programme and policy and political constituencies. Brazil’s Bolsa Escola, on the other hand, did not face strong opposition within government and its organic spread across municipalities ensured a degree of political support. The absence of strong opposition in this case contributed to a relative neglect in setting up rigorous evaluations of the different transfer programmes later to be consolidated into Bolsa Familia.

The contribution of programme evaluation to building an evidence base for antipoverty transfer programmes is also apparent in Latin America. The evaluation findings from Mexico’s Progresa were a significant influence in the decision to extend the programme nationwide and to urban areas by the incoming Fox administration in Mexico, and in other countries. The document approving a loan for US$211 million in 2000 for Colombia’s Familias en Acción by the World Bank and the Inter-American Development Bank noted that programmes “with conditional subsidy grant mechanisms similar to the one proposed here have been carried out in other countries of the region (Mexico, Honduras and Brazil), and have been evaluated as among the most successful social programs” (IADB 2000: p. 38). An impact assessment of Colombia’s Familias en Accion was included as one of the conditions for the loan, replicating Progresa’s approach. According to Attanasio et al. (2005), the initial restriction of the operation of the programme to no more than 200 out of the 1100 municipalities in Colombia was intended to allow the selection of “control” municipalities in a quasi-experimental impact evaluation. The influence of multilaterals has been important in disseminating the results from evaluations in successful programme to justify their adoption elsewhere (Borges Sugiyama 2011). The documentation for Guatemala’s Mi Familia Progresa andthe Dominican Republic’s Solidaridad explicitly acknowledges this influence.

The presence of a successful impact evaluation could prove insufficient to guarantee the sustainability of a programme. In Nicaragua, an incoming government decided to phase out the Red de Protección Social in spite of the positive outcomes of its impact evaluation and the support of multilaterals (Maluccio and Flores 2005). The evaluation of Chile’s Chile Solidario and the subsequent decision by the government to replace it in 2012 with the Ingreso Etico Familiar (Ethical Family Income) also raises concerns with the appropriateness of the Progresa model of evaluation in different settings. Chile Solidario was a multidimensional antipoverty poverty programme aimed at overcoming social exclusion among households in extreme poverty. The programme was designed to provide participant household with a designated social worker who ensured access to all existing public programmes addressing poverty, including transfers. The evaluation studies of Chile Solidario followed impact evaluations methods with observational data and identified comparison groups through, inter alia, propensity score matching techniques (Galasso 2006, 2011; Guardia et al. 2011). The results pointed to marginal improvements. The difficulty is that, given the design of the programme, participant households differed from equivalent non-participant households only in having had access to intermediation and not in programme entitlements. Non-participants were also entitled to the range of cash and in kind transfers. Some features of Chile Solidario have been reproduced in other programmes, including Paraguay’s Tekoporá and Colombia’s Red Unidos.

Policy competition and Political processes have been an important factor responsible for the incidence and scope of programme evaluation in sub-Saharan Africa. Leaving aside Southern African countries with longstanding social assistance, the spread of antipoverty transfer programmes has been strongly advocated by bilaterals and multilaterals, with domestic elites initially reluctant to embrace this agenda. Multilaterals and bilaterals have a strong motivation for pushing antipoverty transfers. By the 1990s, repeated and regular food crises in Eastern Africa had transformed emergency food aid into a regular feature. The case for replacing annual rounds of emergency assistance with regular forms of support for food insecure households was obvious to donors and researchers (Barrett and Maxwell 2005). Social transfers provided an attractive and more effective option (DFID 2005). Political resistance to the introduction of cash transfers among domestic elites has been well documented (Beales and German 2006; Devereux and White 2010; Niño-Zarazúa et al. 2012). Resistance from domestic elites to direct transfers in cash as a means to reduce poverty has a variety of sources. The pervasiveness of food aid as the principal source of support for food insecure households encouraged clientelistic forms of politics (Munemo 2007). Rules-based antipoverty transfers threaten to undermine existing clientelistic instruments. Sub-national disparities and patrimonial politics also militate against accepting rule-based transfer programmes. Resistance to social transfers has been strong among Finance Ministers in the region for, arguably, sensible reasons. Given current aid modalities, donors generally commit financial support for the start-up of social transfer programmes but are reluctant to commit to medium- and longer-term financial needs. This imposes a large measure of uncertainty over the longer-term financial sustainability of antipoverty transfers. Potential electoral ratchet effects can lead to strong pressure on policy-makers to increase the scale of programmes in ways that prove unsustainable.6 In low income countries where a significant proportion of the population are in poverty, or perceive themselves to be in poverty, voters are bound to exercise electoral pressure on their representatives to extend transfers to them with obvious implications for the longer-term financial sustainability of the programmes. There is also a widely shared view among elites in sub-Saharan Africa that antipoverty transfers will generate dependency and that cash transfers may encourage corruption (Beales and German 2006). In addition, elite perceptions of the strength of informal assistance share a greatly exaggerated assessment of these networks. These factors are important in explaining resistance to direct transfer in cash as a means of addressing poverty, but the list is not exhaustive.

In sub-Saharan Africa, the standoff between the donors’ push for the adoption of direct transfers in cash as an alternative to food aid on the one hand, and the domestic elites’ resistance to them on the other has been resolved in several countries through the implementation of pilot projects. Pilot projects enable donors and domestic agencies to experiment with new forms of delivering aid, while limiting their implementation to localised areas. Pilot projects should in principle help provide information on whether transfer programmes have been designed and implemented correctly, and also provide necessary learning and training for the relevant agencies. Pilots facilitate an effective scaling up of programmes, and are an important component of evaluation processes. In practice, limited donor harmonisation and outright competition among donors in a context of government resistance to antipoverty transfer programmes has contributed to a rush to implementation. Resistance and delays in getting government approval for the implementation of pilot programmes has often forced donors to seek to implement pilot programmes before collecting baseline data or set in place rigorous evaluation protocols. The fact that pilot programmes lack the very tools needed to generate knowledge on their feasibility and effectiveness is regrettable.7

Most importantly for the purposes of this paper, a focus on pilot programmes as “demonstration” tools effectively reduces the pressure on programme agencies to include rigorous evaluation protocols in the design and implementation of antipoverty transfer programmes. If neither the donors nor the domestic elites have confidence that the pilot programme will be scaled up, the demand for rigorous evaluations is seriously weakened.

3.3 The Role of Political Processes in Evaluation Incidence

The paper has identified several influences on the incidence of antipoverty transfer programme evaluation in Latin America and sub-Saharan Africa, especially in the context of policy diffusion supported by multilaterals. An epistemic shift to evidence-based policy is apparent in low and middle-income countries. Programme type can also influence the incidence of evaluation. In Latin America and sub-Saharan Africa, rights-based, categorical, antipoverty transfer programmes are seldom evaluated.

However, the main hypothesis put forward is that policy competition and political resistance influence the incidence of antipoverty poverty programme evaluation. It is particularly relevant to a comparison of Latin America and sub-Saharan Africa. In Latin America, policy competition and political resistance to the introduction of large scale antipoverty transfer programmes from competing programme agencies and as a response to the threat of day-to-day political interference, has contributed to a high incidence of program evaluations. In sub-Saharan Africa, limited harmonisation and donor competition on the one hand, and domestic elites’ reluctance to embrace the agenda on the other, have resulted in the proliferation of pilot programmes. Contrary to what we might expect based on the Latin American experience, in sub-Saharan Africa, policy competition, resistance from domestic elites, and the proliferation of pilot programmes have not led to strong evaluation practices. Why?

The nature of the interaction between donors and domestic political elites contribute to an explanation. This interaction generated pressures for donors to rush to implementation, with insufficient attention paid to evaluation protocols. Donors have been more focused on using pilot programmes as a tool to engage and persuade domestic political elites, and their own colleagues in multilaterals and bilateral organisations, of the advantages of transfer programmes, than as a means to assess their effectiveness in possibly adverse conditions. The crucial point is that pilot programmes were not designed with the expectation that they would be scaled up, as was the case in most Latin American countries. Pilots have been used largely as “demonstration” devices, as a means to gain commitment from reluctant elites.8 These demonstration effects were also intended to work within the relevant multilateral and bilateral donor agencies, many of which lacked a strong evaluation ethos (Pritchett 2002). Where the implementation of antipoverty transfer programmes has been sub-contracted to non-government agencies in sub-Saharan Africa, they have an interest in ensuring control over evaluation. The fact that pilots were highly localised, often not fully embedded in government structures and domestic politics, and not expected to grow to scale combined to lift the incentives for rigorous evaluation, as would otherwise have been predicted under conditions of agency competition and political opposition.

4 Correlates of Antipoverty Programme Evaluation Incidence

This section reports on the findings from an exploratory analysis of correlates of the incidence of antipoverty transfer programme evaluation in Africa and Latin America. It aims to explore more systematically some key correlates of evaluation incidence identified in the previous section. The analysis relies on a working dataset constructed of 121 flagship antipoverty transfer programmes in both regions, extracted from key sources (Fiszbein and Schady 2009; Barrientos et al. 2010; Garcia and Moore 2012).

To organise the analysis on the factors associated with programme evaluation, consider the following equation:

EVij=f(EPj,PTik,Rij(C1,,C4)) (1)(1)

where EVij indicates the probability of the presence of impact evaluation in programme i in country j. The incidence of programme evaluation is a function of EPj, which indicates strength of evidence-based practice at country j. This is assumed to depend on government capacity and effectiveness. Note that this variable can stand for impact evaluation supply factors, in the sense that evaluation practices lead to improvements in evaluation capacity, for example Progresa’s evaluation practices encouraging the eventual establishment of Mexico’s evaluation agency CONEVAL (Consejo Nacional de Evaluación de la Política de Desarrollo Social). PTik denotes the type k of programme i. For our purposes, it will be useful to distinguish two types of programmes. In terms of the incidence of evaluation, some programmes (k=1) can be described as deontological9 or rights-based, in the sense that entitlements derive from acknowledged rights. In deontological programmes, consequences are of secondary importance with the implication that they are associated with a weak demand for evaluation. Social or non-contributory pensions and unconditional cash transfers often fit this description. A different set of programmes (k=0) can be described as teleological. In teleological programmes interventions are implemented to secure specific objectives, and consequences do matter. If programmes fail to reach their objectives, then continuation is not justified. Teleological programmes are associated with stronger demand for impact evaluation. Finally, Rij denotes the strength of resistance against programme i in country j. In the discussion in previous sections we considered, with examples, the proposition that programme evaluation was in some cases a response to political resistance to the introduction or scaling up of antipoverty transfer programmes. In the analysis below, we focus on four external and internal sources of resistance to antipoverty transfer programmes Cm(m=1,2,3,4). C1 denotes resistance associated with competition among external agencies. C2 denotes resistance associated with competition between external agencies and domestic policy-makers. C3 denotes resistance associated with competition among internal agencies. C4 denotes resistance arising from the interaction of internal agencies and policy-makers and/or electorates.

We cannot observe EVij directly, instead, we are able to observe a binary variable, EVij*, which takes a value of 1 if the programme does count on an impact evaluation and 0 otherwise. Our econometric model assumes a linear form for (1), estimated as the following stochastic equation:

EVij=β0+β1EPj+β2PTik+βcmRij(C1,,C4)+eij (2)(2)

We employ a new dataset of flagship programmes in developing countries including variables that capture programme information, the presence of impact evaluation, the presence of multilaterals, and other relevant information. To identify empirical counterparts for evidence-based practice, policy competition and political factors in the model, we supplement the programme dataset with information from well-established data sources. Information on economic, government, and politics variables at country level were taken from the Quality of Goverment (QoG) 2013 dataset (Teorell et al. 2013) and matched to the programme data. The QoG data merged with the programme data corresponded to that of the start year of the programme.10 Table 1 below provides information on labels and descriptive statistics.

Table 1

Variable Description and Statistics.

The selection of antipoverty programmes for inclusion in the dataset requires explanation. The dataset focuses on “flagship” programmes understood as larger, more institutionalised, programmes, but this is not a clear-cut criterion. In sub-Saharan Africa, antipoverty transfer programmes pilots tend to be small in coverage. Most middle-income countries have a large number of programmes addressing poverty (143 in Chile in 2002, over 80 in Bangladesh in 2010). Our dataset focuses on the most significant antipoverty transfer programmes in the corresponding countries. The dataset concentrates on cash and hybrid transfer programmes only. Whilst we attempt to have an inclusive and accurate sample, there is irreducible uncertainty over the population of programmes.

The dependent variable indicates whether a programme has been evaluated. As shown in Table 1, only 49% of the sampled programmes showed any kind of evaluation. The variable was coded based on documented evaluations (reports, working papers, published papers). For some programmes the evaluation was designed and managed by the programme agencies. In other cases, the evaluation was managed by donors. In practice, evaluation by external researchers often overlaps these modalities. In the paradigmatic case of Progresa, for example, IFPRI was commissioned to undertake its evaluation. As long as the documented evaluations made use of quasi-experimental or observational data and applied appropriate econometric techniques, the dependent variable records the presence of evaluation.

It was not possible to find a variable capturing directly the strength of evidence-based practice at country level. We use an index of government effectiveness (Wdgi_gee) as a proxy for evaluation capacity and propensity. The documentation describes this variable as combining into a single index “… responses on the quality of public service provision, the quality of the bureaucracy, the competence of civil servants, the independence of the civil service from political pressures, and the credibility of the government’s commitment to policies. The main focus of this index is on “inputs” required for the government to be able to produce and implement good policies and deliver public goods” (Teorell et al. 2013: p. 123). This is likely to be positively correlated with evaluation capacity and propensity.

The programme types identified in the dataset are reasonable counterparts of the programme types described in the model. Conditional cash transfer programmes are typically teleological. Political resistance to them, based on their specific features and on expectations that they would not work in sub-Saharan Africa, has been well documented. Social pensions, and to a lesser extent unconditional transfers, are generally perceived as closer to deontological or rights-based type programmes. In-kind transfers and employment guarantees are longstanding, especially in sub-Saharan Africa.

Recall that in (1) the Rij function is dependent on sources of political resistance factors. It is challenging to find variables capturing these influences on the incidence of evaluation. Recognizing that they are very imperfect indicators, we use the following variables: The engagement of multilaterals or bilaterals in the development, funding or management of antipoverty transfer programmes (Donor) is taken to indicate external agency influence, C1, expected to be positively correlated with the incidence of evaluation. Resistance due to competition between donors and domestic policy-makers C2 can be captured by aid dependence (Aid_gdp), likely to be positively correlated with the incidence of evaluation. Competition among domestic intra-government agencies, C3, is captured by the number of leading antipoverty programmes in one country (Nprog). Our programme database collects information on flagship antipoverty programmes, the presence of several programmes in a particular country can serve as a proxy for inter-agency competition. Finally, competition between domestic programme agencies and policy-makers and the electorate C4 is proxied by two variables, the index of legislative competitiveness (Dpi_lipc) and the Vanhannen index of political participation (Van_part). The Index of legislative competitiveness captures the extent of single party dominance. Higher values for this index reflect less single party dominance. The Vanhannen index of participation measures the proportion of the population actually voting in elections. Higher levels of legislative competitiveness and stronger electoral checks are expected to associate with a higher likelihood of incidence of impact evaluations. Finally, given the discussion in the previous section of the different contextual factors at play in sub-Saharan Africa as opposed to Latin America, we include a dummy variable for the region.

Table 2 presents the results from the estimation of five linear regression model of the incidence of evaluation. The model in column 1 includes only the dummy variable for sub-Saharan Africa. The model in column 2 includes the variables capturing evidence-base policy, the index of government effectiveness and the presence of donors. The model in column 3 highlights the influence of variables capturing policy competition and political factors generating resistance to antipoverty transfers. The model in column 4 combines the variables highlighted in the model in column 2 and the model in column 3. The model in column 5, our preferred specification, adds interaction effects aimed at capturing the way in which the sub-Saharan Africa context shapes the influence of policy competition and political factors. Models in columns 2–5 include dummies for programme types, with conditional cash transfers as the reference category.

Table 2

Regression Estimates.a,b

Broadly, the estimates provide some support for the influence of political competition on the incidence of antipoverty transfer programme evaluation discussed in the previous section. Starting with the model in column 2, the government effectiveness measure is positively associated with the incidence of evaluation. The engagement of donors with the programme also has a positive influence on the incidence of evaluation but the estimated parameter is not significant. The predicted correlation of different programme types on the likelihood of incidence of programme evaluation, distinguishing “deontological” and “teleological” programmes, is fully supported by the estimated parameters. The reference category is conditional cash transfers. The parameters associated with social pensions and unconditional cash transfers are negative and significant. The parameters associated with in-kind transfers and employment guarantees are also negative, but only the in-kind transfer parameter is significant at the 5% level. The estimates suggest that the likelihood of programme evaluation is significantly lower for deontological programmes and for programmes which effectiveness is reasonably well understood. The sign and significance levels of the estimated parameters associated with programme type remain largely the same across the model specifications.

The model in column 3 highlights the variables proxying competition and potential resistance to antipoverty transfer programmes. Potential policy competition arising from the presence of several flagship programmes positively influences the incidence of programme evaluation at 5% significance level. The parameter associated with political competition in parliament indicates a positive and marginally significant influence on programme evaluation. A higher ratio of aid to GDP, however, weakens the incidence of programme evaluation, again at a marginal level of significance. Interestingly, when these variables are included in the model, the SSA dummy becomes larger but marginally significant at a 10% level. The results suggest that after accounting for the effects of policy competition and political resistance, the likelihood of programme evaluation in SSA countries becomes stronger in comparison to the likelihood of programme evaluation in Latin America. Regarding our hypothesis, this suggests that policy competition and political resistance to transfers influence the incidence of programme evaluation in the region. Comparing the results in columns 2 and 3 suggests these factors are perhaps of greater relevance to the incidence of programme evaluation in SSA than evidence-based factors.

The model in column 4 combines the variables proxying evidence-based factors and policy competition and political factors. With the exception of political competition, the parameters retain their sign although their statistical significance is weakened. Note that here the SSA dummy is not robust enough, as its significance vanishes when accounting for both groups of variables. This result suggests that accounting for policy competition, political resistance and evidence-base factors renders the likelihood of programme evaluation in SSA not significantly difference from the likelihood of programme evaluation in Latin America.

The model in column 5 incorporates interactions of the sub-Saharan Africa dummy with all the explanatory variables. The aim is to assess the ways in which the regional context shapes the influence of these variables on programme evaluation. The parameter associated with the SSA dummy is now much larger and has changed in sign but it is not significantly different from zero, confirming the lack of robustness of this variable. Government effectiveness, the engagement of donors and the number of programmes, as expected, influence positively the likelihood of programme evaluation. The parameters are now estimated with greater precision. The parameters associated with the interacted variables are of particular interest. Indeed, the parameters associated with government effectiveness and donor engagement show a negative sign and marginal significance in when interacted with the SSA dummy. Taken together, these results are in line with our argument that the involvement of donors and governments in introducing pilot schemes in the region might have undermined the incentives for programme evaluation. On the other hand, among the variables capturing policy competition and political factors, only the variable capturing electoral turnout has a significant and positive influence on the likelihood of programme evaluation when interacted with the SSA dummy. The latter suggests that direct citizen participation in democratic elections increases the demand for evaluated programmes in SSA countries, in contrast with what is observed in Latin America. Accounting for programme competition and legislative competitiveness make no difference in SSA and Latin America. It is likely that the limited size of our sample, and potential country heterogeneity, could have contributed to these results.

Using the approach suggested by Gelbach (2009), we further decompose the influence of variables proxying evidence-based factors and policy competition and political resistance factors on the changes in the estimated parameter associated with the SSA dummy. Table 3 provides the main results. Policy competition and political resistance contribute the largest component in the total change. This suggests that the likelihood of programme evaluation in sub-Saharan Africa is stronger when account is taken of the specific influence of policy competition and political factors in the region, which are otherwise of less relevance in a Latin American context.

Table 3

Decomposition of Changes in SSA Dummy.a

To sum up, the results emerging from estimating models in columns 1–5 in Table 2, taken as a whole and keeping in mind the limitations of the data, provide some support for the relevance of the two main explanations put forward for the intensity of evaluation in antipoverty programmes: a shift to evidence-based policy and resistance to transfers associated with policy competition and politics both contribute to an explanation of the higher intensity of such evaluations. Programme design features matter in this respect too. In particular, these findings fail to reject the main hypothesis in the paper, that policy competition and political resistance to antipoverty transfers are relevant factors in explaining the incidence of programme evaluation.

5 Conclusions

The article has examined the incidence of impact evaluations in antipoverty transfer programmes in Latin America and sub-Saharan Africa. A feature of the growth of antipoverty transfer programmes in developing countries has been the attention paid to the evaluation of their outcomes. Two broad explanations were offered to account for this feature. One explanation locates it within a broader trend towards evidence-based development policy. A second explanation points to the influence of policy competition and political factors. The paper focused on the latter. The scarce literature on this issue helped map out key influences and a working hypothesis, namely that the incidence of impact evaluations responded to agency competition and political resistance to antipoverty transfers. The analytical approach adopted was twofold. First, a comparison of the factors influencing the incidence of impact evaluations of antipoverty transfer programmes in sub-Saharan Africa and Latin America threw further light on the role of policy competition and political processes. Second, a new database of flagship antipoverty transfer programmes from the two regions was employed to examine empirically the key influences and the main hypothesis.

What explains the incidence of antipoverty transfer programme evaluation? The review of the incidence of impact evaluations in antipoverty transfers in Latin America and sub-Saharan Africa concluded that government effectiveness, agency competition, political resistance, and programme type are important factors influencing programme evaluation. Why has programme evaluation been weaker in sub-Saharan Africa? Whereas in Latin America the factors identified above strengthened effective demand for evaluations, in sub-Saharan Africa the nature of the interactions between donors and domestic elites undermined demand for evaluation. Strong agency competition and political resistance from domestic elites would have signalled greater attention to programme evaluation, but a focus on pilots as tools to generate demonstration effects and low expectations that programmes would eventually scale up undermined demand for rigorous programme evaluation. The empirical analysis broadly supports these conclusions.

Rigorous impact evaluation of antipoverty transfer programmes has the potential to make a significant contribution to better policies. Monitoring and evaluation protocols are crucial to facilitate improvements in government effectiveness. The evolution of antipoverty transfers in Latin American countries has strengthened effectiveness in antipoverty policies and across arrange of public policy. Mexico’s CONEVAL, a specialist government agency charged with the evaluation of human development programmes established in 2005, reflects a growing institutionalisation of evaluation processes in Latin American countries. An important insight contributed by this paper is that, at least in the context of antipoverty policy, policy competition and political factors have a strong influence on the incidence and quality of evaluation processes. The analysis in the paper focused on the influence of these processes on the incidence of programme evaluation but, indirectly, the findings suggest that the information generated by impact evaluations has the potential to improve the working of political processes associated with antipoverty policies, and also the potential to strengthen pro-poor coalitions and public support for effective antipoverty policies. This is evident from developments in Latin America in particular. The findings in the paper suggest that, to the extent that antipoverty transfer programmes become embedded in domestic policy and politics in sub-Saharan Africa, attention to the evaluation of antipoverty transfer programmes will also lead to better politics.

Acknowledgments

We are grateful to four referees, the Journal Editor, and the Editors of the Special Issue for the comments and suggestions offered. They have greatly improved the paper. The errors that remain are ours alone.

Appendix

Table A1

Programme Data

References

  • Attanasio, O., E. Battistin, E. Fitzsimons and M. Vera-Hernandez (2005). How Effective are Conditional Cash Transfers? Evidence from Colombia. London: University College London.Google Scholar

  • Attanasio, O., E. Fitzsimons, A. Gomez, M. I. Gutiérrez, C. Meghir and A. Mesnard (2010). “Children’s Schooling and Work in the Presence of a Conditional Cash Transfer Program in Rural Colombia,” Economic Development and Cultural Change. 58(2):181–210.Google Scholar

  • Banerjee, A. V. and E. Duflo (2011). Poor Economics. A Radical Rethinking of the Way to Fight Global Poverty. New York: Public Affairs.Google Scholar

  • Barrett, C. B. and D. G. Maxwell (2005). Food Aid After Fifty Years: Recasting its Role. London: Taylor & Francis.Google Scholar

  • Barrett, C. B. and M. R. Carter (2010). “The Power and Pitfalls of Experiments in Development Economics: Some Non-random Reflections,” Journal of Economic Perspectives and Policy, 32(4):515–548.Google Scholar

  • Barrientos, A., M. Niño-Zarazúa and M. Maitrot (2010). Social Assistance in Developing Countries Database Version 5. Manchester: Brooks World Poverty Institute.Google Scholar

  • Barrientos, A. and C. Santibañez (2009). “Social Policy for Poverty Reduction in Lower-income Countries in Latin America: Lessons and Challenges,” Social Policy and Administration, 43(4):409–424.Google Scholar

  • Beales, S. and T. German (2006). Situation Analysis of Social Protection and Cash Transfers in Africa. London: Development Initiatives and HelpAge International.Google Scholar

  • Behrman, J. R. and P. E. Todd (1999). Randomness in the Experimental Samples of PROGRESA (Education, Health, and Nutrition Program). Washington, DC: International Food Policy Research Institute.Google Scholar

  • Borges Sugiyama, N. (2011). “The difussion of Conditional Cash Transfer programs in the Americas,” Global Social Policy, 11(2–3):250–278.Google Scholar

  • Borraz, F. and N. González (2009). “Impact of the Uruguayan Conditional Cash Transfer Program,” Latin American Journal of Economics-formerly Cuadernos de Economía, 46(134):243–271.Google Scholar

  • Brazilian Court of Audit (2003). TCU Evaluation of the Child Labor Eraducation Programme. Brasilia: SEPROG.Google Scholar

  • Card, D., P. Ibarrarán, F. Regalia, D. Rosas-Shady and Y. Soares (2011). “The Labor Market Impacts of Youth Training in the Dominican Republic,” Journal of Labor Economics, 29(2):267–300.Web of ScienceCrossrefGoogle Scholar

  • Chong, A. and M. Gradstein (2006). Who’s afraid of foreighn aid? the donors’ perspective’. Washington, DC: CESIFO.Google Scholar

  • Davis, B., M. Gaarder, S. Handa and J. Yablonski (2012). “Evaluating the Impact of Cash Transfer Programmes in sub-Saharan Africa: An Introduction to the Special Issue,” Journal of Development Effectiveness, 4(1):1–8.Google Scholar

  • Devereux, S. and P. White (2010). “Social Protection in Africa: Evidence, Politics, and Rights,” Poverty and Public Policy, 2(3):53–77.Google Scholar

  • DFID (2005). Social Transfers and Chronic Poverty: Emerging Evidence and the Challenge Ahead. London: DFID.Google Scholar

  • Fiszbein, A. and N. Schady (2009). Conditional Cash Transfers. Reducing Present and Future Poverty. Washington, DC: The World Bank.Google Scholar

  • Galasso, E. (2006). “With their Effort and One Opportunity” Alleviating extreme poverty in Chile. Washington, DC: Development Research Group, The World Bank.Google Scholar

  • Galasso, E. (2011). “Alleviating Extreme Poverty in Chile: The Short Term Effects of Chile Solidario,” Estudios de Economia, 38(1):101–127.CrossrefWeb of ScienceGoogle Scholar

  • Garcia, M. and C. M. T. Moore (2012). The Cash Dividend. The Rise of Cash Transfer Programs in Sub-Saharan Africa. Washington, DC: The World Bank.Google Scholar

  • Gelbach, J. B. (2009). When do covariates matter? And which ones and how much? Tucson: University of Arizona.Google Scholar

  • Glennerster, R. and K. Takavarasha. (2013). Running randomized evaluations. A Practical Guide. Princeton, N.J.: Princeton University Press.Google Scholar

  • Gilligan, D. O., J. Hoddinott, A. S. Tafesser, S. Dejene, N. Tefera and Y. Yohannes (2007). Report on 2006 Baseline Survey, Ethiopia Food Security Program. Washington, DC: IFPRI.Google Scholar

  • Gilligan, D. O., J. Hoddinott and A. Seyoum Taffesse (2008). “The Impact of Ethiopia’s Productive Safety Net Programme and its linkages,” Journal of Development Studies, 45(10):1684–1706.CrossrefGoogle Scholar

  • Graham, C. (2002). Public Attitudes Matter: A Conceptual Framework for Accounting for Political Economy in Safety Nets and Social Assistance Policies. Washington, DC: The World Bank.Google Scholar

  • Guardia, F. H. d.l., A. Hojman and O. Larrañaga (2011). “Evaluating the Chile Solidario Program: Results using the Chile Solidario Panel and the Administrative Databases,” Estudios de Economia, 38(1):129–168.Google Scholar

  • Heckman, J. J., J. Smith and N. Clements (1997). “Making the Most out of Programme Evaluations and Social Experiments: Accounting for Heterogeneity in Programme Impacts,” Review of Economic Studies, 64(4):487–535.CrossrefGoogle Scholar

  • Heinrich, C., J. Hoddinott and M. Samson (2012). The Impact of South Africa’s Child Support Grant on Schooling and Learning. Pretoria: SASSA.Google Scholar

  • IADB (2000). Social Safety Net Program (CO-0247). Washington, DC: Inter-American Development Bank.Google Scholar

  • Levy, S. (2006). Progress against poverty. Sustaining Mexico’s Progresa-Oportunidades Program. Washington DC: Brookings Institution Press.Google Scholar

  • Levy, D. and J. Ohls (2010). “Evaluation of Jamaica’s PATH Conditional Cash Transfer Programme” Journal of Development Effectiveness, 2(4):421–441.Google Scholar

  • Lindert, K. and V. Vinscensini (2008). Social Policy, Perceptions and the Press: An Analysis of the Media’s Treatment of Conditional Cash Transfers in Brazil. Washington: The World Bank.Google Scholar

  • Lund, F. (2008). Changing Social Policy. The Child Support Grant in South Africa. Cape Town: HSRC Press.Google Scholar

  • Maluccio, J. A. and R. Flores (2005). Impact Evaluation of a Conditional cash Transfer Program: The Nicaraguan Red de Protección Social. International Food Policy Research Institute.Google Scholar

  • Munemo, N. (2007). “Political incumbency and drought relief in Africa”. Ed. Eds, Social Protection for the Poor and Poorest: Concepts, Policies and Politics. London: Palgrave.Google Scholar

  • Niño-Zarazúa, M. (2011). Mexico’s Progresa-Oportunidades and the Emergence of Social Assistance in Latin America. Manchester: Brooks World Poverty Institute.Google Scholar

  • Niño-Zarazúa, M., A. Barrientos, S. Hickey and D. Hulme (2012). “Social Protection in Sub-Saharan Africa: Getting the Politics Right,” World Development, 40(1):163–176.Web of ScienceCrossrefGoogle Scholar

  • Pritchett, L. (2002). “It Pays to be Ignorant: A Simple Political Economy of Rigorous Program Evaluation,” The Journal of Policy Reform, 5(4):251–269.CrossrefGoogle Scholar

  • Rawlings, L. B. and L. Rubio (2005). “Evaluating the Impact of Conditional Cash Transfer Programs,” World Bank Research Observer, 20(1):29–55.CrossrefGoogle Scholar

  • Rofman, R., I. Apella and E. Vezza, E. (2013). Mas allá de las pensiones contributivas. Catorce experiencias en America Latina. Buenos Aires: The World Bank.Google Scholar

  • Skoufias, E. (2005). Progresa and its Impacts on the Welfare of Rural Households in Mexico. Research Report. Washington: International Food Policy Research Institute.Google Scholar

  • Skoufias, E., B. Davis and S. de la Vega (2001). “Targeting the Poor in Mexico: An Evaluation of the Selection of Households into PROGRESA,” World Development, 29(10):1769–1784.CrossrefGoogle Scholar

  • Soares, F. V. and T. Britto (2007). Confronting Capacity Constraints on Conditional Cash Transfers in Latin America: the cases of El Salvador and Paraguay. Brasilia: International Policy Centre for Inclusive Growth.Google Scholar

  • Stampini, M. and L. Tornarolli (2012). The Growth of Conditional Cash Transfers in Latin America and the Caribbean Did they go too far? IZA Policy Paper No. 49. Bonn: IZA.Google Scholar

  • Teorell, J., N. Charron, S. Dahlberg, S. Holmberg, B. Rothstein, P. Sundin and R. Svensson (2013). The Quality of Government Dataset, version 15 May 2013. Gothenburg: The Quality of Government Institute, University of Gothenburg.Google Scholar

  • van der Berg, S. (1997). “South African Social Security Under Apartheid and Beyond,” Development Southern Africa, 14(4):481–503.CrossrefGoogle Scholar

  • Woolard, I., K. Harttgen and S. Klasen (2012). “The History and Impact of Social Security in South Africa: Experiences and Lessons,” Canadian Journal of Development, 32(4): 357–380.CrossrefWeb of ScienceGoogle Scholar

  • World Bank (1998). Assessing Aid. What Works, What Doesn’t, and Why. Oxford: Oxford University Press.Google Scholar

  • Yap, Y. T., G. Sedlacek and P. F. Orazem (2002). “Limiting Child Labor through Behavior-Based Income Transfers: An Experimental Evaluation of the PETI Program in Rural Brazil. Washington, DC: Inter-American Development Bank.Google Scholar

Footnotes

  • Arguably, the shift to evidence-based policy has had a stronger influence on supply factors, through research and technology innovations which lowers the significant costs associated with impact evaluations. 

  • Comparisons of the treatment and control groups of communities confirmed the absence of significant bias or contamination in the evaluation sample (Behrman and Todd, 1999). 

  • An evaluation of Bolsa Familia by a consortium including IFPRI collected the first round of data only in 2005. 

  • The recent evaluation of the Child Support Grant in South Africa is another example of a well-designed impact evaluation and strategy implemented well after the start of the programme (Heinrich et al., 2012). 

  • Ethiopia, Ghana, Kenya, Lesotho, Malawi, Zambia and Zimbabwe. 

  • Preferences for social pensions among sub-Saharan Africa governments can be explained in part because the associated financial liabilities can be calculated with some precision, and also because civil servants themselves are uncertain of their pension entitlements. 

  • Second generation pilot programmes are paying more attention to impact evaluation. See the Protection to Production initiative website at http://www.fao.org/economic/PtoP/en/. 

  • Garcia and Moore (2012) argue that pilot programmes can in theory address both political and technical issues. The influence of these factors is also at play in contexts where programme evaluation has been relatively stronger. Arguably, the stronger evaluation protocols in place in Ethiopia responded to the competition among donors being compounded by the strong opposition to unconditional cash transfers and strong preference for employment guarantees by the government, and that a rapid scaling up was seriously considered. In Kenya, on the other hand, evaluation protocols owe much to donor competition around the design of the programme, especially target groups and the role of conditions. 

  • In ethics, deontological theory holds that choices are to be morally assessed in terms of duty and moral norms, whereas teleological or consequentialist theory holds that choices are to be morally assessed in terms of the states of affairs their bring about. 

  • In a handful of cases missing variables were filled in with information from subsequent years up to 4. The programme data is available in the Online Appendix. 

About the article

Corresponding author: Armando Barrientos, University of Manchester, Brooks World Poverty Institute, Arthur Lewis Building Oxford Road, Manchester, M13 9PL, UK, e-mail:


Published Online: 2015-05-01

Published in Print: 2015-06-01


Citation Information: Journal of Globalization and Development, ISSN (Online) 1948-1837, ISSN (Print) 2194-6353, DOI: https://doi.org/10.1515/jgd-2014-0006.

Export Citation

©2015, Armando Barrientos et al., published by De Gruyter. Copyright Clearance Center

Comments (0)

Please log in or register to comment.
Log in