CRS Uses the PPI to Test How Variations in Project Implementation Affect Poverty Outreach >

Catholic Relief Services
•05/29/17
• 1 Comment

An interview with Samuel Beecher, Research Officer, Catholic Relief Services

Note: This blog post was updated in January 2018 to reflect analysis corrected by the EFI team, which indicated that the PPI results were indeed consistent with (rather than possibly overestimating) the poverty levels identified through other methods in the Financial Diaries project.

Catholic Relief Services (CRS), founded in 1943 by the United States Conference of Catholic Bishops, seeks to preserve and uphold the sacredness and dignity of all human life, foster charity and justice, and embody Catholic social and moral teachings to promote human development in the United States and abroad. To achieve these goals, CRS engages in the program areas of Emergency Response and Recovery, Agriculture, Health, Education, Justice and Peace Building, Partnership and Capacity Strengthening, and Microfinance.

In partnership with the MasterCard Foundation (MCF), CRS started the Expanding Financial Inclusion (EFI) in Africa program in 2014. The goal of EFI is to expand financial service access to vulnerable households to improve their resilience. To this end, the project aims to create 19,200 new Savings and Internal Lending Communities (SILC) groups with 502,320 members using the Private Service Provider (PSP) methodology in four countries in Africa—Burkina Faso, Senegal, Uganda, and Zambia—by September of 2017. This methodology involves identifying, training and certifying entrepreneurial people as PSPs, who then go on to help members in their community to form and manage savings groups. You can read more about it here.

To date, EFI has formed more than 20,000 groups with 543,000 group members. EFI’s monitoring and evaluation team uses the PPI to measure the depth of poverty outreach among SILC group members and comparison households in the community. Across the four countries, over 75% of these households live on less than $2.50, and between 30-80% live on less than $1.25 per day. The poverty rates for the SILC group members in the four project countries as measured by the PPI are below.  

Country
% Below National
Poverty Line
% Below
$2.50 / Day 
% Below
$1.25 / Day
Burkina Faso 40% 85% 54%
Senegal 53% 76% 29%
Uganda 28% 83% 47%
Zambia 71% 97% 86%

1. When did your organization start using the PPI, and why? What was the need you were hoping to address?

EFI started using the PPI to collect comparison group data at the start of the project in 2014, sampling approximately 50 villages in each of the four implementation countries. The majority of organizations implementing savings-led microfinance programs have a poverty focus, but most face obstacles in reaching the poorest households.  

2. How does your organization use the PPI? To measure poverty outreach? To improve social performance (targeting or product/service design)? To track changes?

EFI is using the PPI to measure poverty outreach, comparing the mean poverty rate of SILC group members to other households in their communities. EFI also uses PPI to determine whether the PSPs are deepening their poverty outreach over time and to test whether variations in the project implementation strategy influence and improve poverty outreach. 

3. Does your organization collect PPI data directly from households, or do you get PPI data reported to you from partners/investees?

EFI works with a network of ten partners in four countries to carry out the training, implementation, monitoring, and evaluation of the project. Initially, EFI hired consultants in all four countries to conduct PPI surveys on a randomly selected group of households in a randomly selected subset of project villages before SILC group mobilization began. Since SILC groups began forming, CRS has trained partner staff members to administer the PPI to new SILC group members in the previously sampled villages using a mobile device with the support of TaroWorks™.

4. Describe the logistics of collecting and using the PPI at your organization.

The EFI project randomly selected 222 villages distributed evenly across the four project countries. CRS hired consultants in all four countries to conduct baseline PPI surveys in the home villages of a randomly selected subset of PSPs before those PSPs began SILC group mobilization. The goal was to have approximately 1,000 reference households in each project country. When administering the interviews, the consultants initially used paper interview sheets and project staff entered the data into an Excel spreadsheet. At a later date, this data was merged with the Salesforce platform.

EFI has offered technical support to its partner implementing organizations in the administration of PPIs using the TaroWorks™ application, exporting the data from Salesforce to Excel. CRS partner staff working in the same project areas administered the same PPI survey to all the households who had at least one member in a SILC group in the PSPs’ home village. To date, approximately 20,000 SILC group members have been administered the PPI survey in the 222 villages and an additional 10,000 PPIs will be administered before the end of the project. The PPIs are collected on a rolling basis: newly formed SILC groups are interviewed within the first two months of group formation.

EFI staff has worked in partnership with Microfinance Opportunities (MFO) to analyze the PPI data. The PPI is a major component of EFI’s monitoring and evaluation plan as it is integrated with all the other research tools and activities which include: Financial Diaries; a standardized reporting system for savings groups called SAVIX; and a tool which CRS designed called the Agent Management table, which tracks PSP performance.

5. Describe the data analysis you conduct on PPI results.

EFI focused on two questions central to the monitoring and evaluation of the EFI project. They are:

  1. What is the mean poverty level of SILC group members, relative to their communities at the start of the program and over time?
  2. What are the effects of the PSP delivery model variants in terms of poverty outreach?

CRS has used the 27,000 PPI surveys (of which 21,424 were SILC group members) collected to compare the poverty rates of non-SILC, comparison households to the poverty rates of all SILC group member households. The PPI results showed that EFI has met its goal: on average, across all the countries and partners, about two-thirds of SILC members were in the lower half of the income distribution in their village.

We also identified the sequence in which SILC groups in each sampled village were formed and then analyzed the data to see whether households in groups that formed later were more likely to be poor than those in groups formed earlier. These analyses allowed us to answer the first research question above. Due to the small sample size of villages with more than 101 groups formed, the figure below and the analysis performed includes villages with 10 groups or fewer.

Figure 1: Share of SILC Members above Reference Median by Group Cohort

Looking at cohorts 1 through 10 using the village-by-village comparison of SILC members’ poverty rates to the baseline median, we see that the data suggests an increase in depth of poverty outreach in the later cohorts—the average percent of SILC group members in the poorest two poverty quartiles increases the later that the group is formed in the village. For example, on average, the cohort of first groups, when aggregated together had 63% of group members in the poorest two poverty quartiles. By the 5th cohort, the number was up to 73%, and 79% by the 10th cohort.

6. Discuss the difficulties you faced (or still face!) when using the PPI. How are you solving them?

Operational challenges

Once the consultants finished administering PPIs to reference households, the project equipped field supervisors with smart phones. The TaroWorks application was downloaded to the devices and field supervisors were trained on how to use the application and troubleshoot simple configuration issues. However, the field supervisors faced multiple issues in using the phones and partner staff lacked capacity to fix these problems. Similar problems with hardware have occurred many times in the project and resulted in delays in PPI administration. Identifying, purchasing and delivering spare parts in a timely manner also proved challenging. After two years of issues with the first brand of mobile devices purchased, CRS chose simply to purchase a different brand rather than continuing to order spare parts for the original phones. 

Another challenge which CRS faced in the project was merging data to Salesforce that had been collected on paper sheets. When merged, all of the data was given the Village Name “[Country] Baseline.” CRS worked with MFO to restructure the database and disaggregate the Reference Village Names so that a village-level analysis could be performed in all four countries. 

Technical challenges

CRS faced issues specific to the PPI’s construction and results, particularly around whether the tool is sensitive to variations in poverty in rural African communities. One of CRS’ main goals in using the PPI as a poverty measurement tool was to examine whether PSPs using normal and pro-poor delivery models for SILC groups achieved different levels of poverty outreach. A problem emerged in late 2016 when it became clear that many respondents to the PPI surveys in Zambia in the ‘Expanding Financial Inclusion Financial Diaries’ study had the same, very high poverty likelihoods, yet the Financial Diaries data, a mid term evaluation, and information from a questionnaire2 administered alongside the PPI suggested that there were real differences in the poverty levels of households with the same scores.

To assess the PPI’s sensitivity, EFI triangulated PPI scores with household rankings and categorizations based on the wealth ranking exercise and detailed income and expenditure data from the Financial Diaries study.

The results showed that overall, the PPI tracked broad trends in poverty in the Financial Diaries population but appeared to struggle to differentiate among households that are almost all living below the poverty line. Such insensitivity in differences in poverty has two major programmatic consequences. First, it suggests that the PPI should not be used for targeting (at least in Zambia) when the goal of targeting is to extend the depth of outreach among the poor, not simply to reach the poor. This suggests that the PPI is best complemented with other tools, which tools would depend on the objectives and the budget.3 Second, the data presented in this report suggest that the PPI should not be used to measure progress from one level of poverty to another—it should not be used to measure progress within poverty, but only more general progress out of poverty—because the PPI is not precise enough to determine whether a group of households’ poverty had really changed over time. EFI and CRS discussed these issues with IPA and know that there are efforts underway to retool the PPI to take into account the issues raised in this report. (A brief and the full report of this triangulation exercise are forthcoming on the efiafrica.crs.org site.)

EFI has also trained partners to perform simple analysis to find average poverty rates of households with different demographic characteristics.

7.   Has the way your organization has used the PPI evolved, or does your organization have plans for it to evolve in the future? If so, why and how?

Because of the results of the triangulation exercise in Northern Zambia (described above), we would recommend that CRS supplement the PPI with other tools to verify sensitivity in measurement of poverty depth where this is programmatically of high importance.

 


[1]  The number 10 intentionally selected as the sample size of villages with more than 10 groups formed was low and the results were therefore deemed insignificant. 

[2] The additional questions in the survey included indicators in food security, education, and financial inclusion.

[3] The PPI guidance advises that greater sensitivity can be achieved by combining tools but does not advise against only using the PPI for targeting

1 Comment

Comments

Thanks Samuel for your informative blog post. I address two issues below that speak to the technical challenges you described. These technical concerns stem from the premise that the PPI cannot distinguish between households that are below the 1.90 USD PPP a day poverty line. If this was indeed true, then the PPI would not be a useful tool to target ultra-poor households in low income countries like Zambia. Further, it may also follow that the PPI would not be able to measure small changes in poverty for these households. However, at least for the new PPIs that are being created at Innovations for Poverty Action, this premise is not true. The PPI is typically calibrated to between 12 and 18 poverty lines that we hope will be used across a wide range of use cases. For your concern, what is relevant is that in low income countries, the PPI is calibrated to a very low poverty line – the 1.00 USD PPP a day line. Therefore, relevant to your first technical challenge, for a low income country like Zambia, the PPI is able to distinguish between households that are considerably below the 1.90 a day line. As a consequence, the PPI is also able to capture changes in group poverty for ultra-poor households (your second technical challenge). Every new PPI for a low-income country will have points and look up tables calibrated to the 1.00 a day USD 2011 PPP line. In addition, the PPIs are also calibrated to the 20th percentile line, which may also be a useful tool when targeting ultra-poverty, should an organization prefer to focus on a relative measure of poverty as opposed to the international poverty lines. Yes, we concur the PPI could be complemented with other tools, such as geographic targeting, which is widely used by several other organizations that aim to reach ultra-poor households. Thanks for using the PPI, and for your careful analysis. We look forward to continue to work with you in the future. -Varun Kshirsagar, PPI Technical Lead