Evaluations of Unfinished Business ELMA's pediatric HIV initiative in Africa — ELMA Philanthropies

Evaluations of Unfinished Business ELMA's pediatric HIV initiative in Africa

< BACK TO E&L HOMEPAGE

Evaluations of Unfinished Business - ELMA’s Pediatric HIV Initiative in Africa

July 8, 2020

June Lee
Director, Evidence & Learning
ELMA Philanthropies

Stroke-Gray.png
Photo: Eric Bond/Elizabeth Glaser Pediatric AIDS Foundation

Photo: Elizabeth Glaser Pediatric AIDS Foundation / Eric Bond

When The ELMA Foundation began its work in 2005, pediatric HIV was one of the most critical issues facing children in sub-Saharan Africa.

In response, the early years of ELMA’s HIV investments focused on addressing barriers to treatment, including the cost and formulation of pediatric antiretroviral medications, access to diagnostics, training health workers, and building political commitment and leadership.

Despite some important gains, by 2013, only 23% of the 3.2 million children living with HIV (91% living in Africa) had access to treatment. ELMA therefore sharpened its focus on pediatric HIV through the Unfinished Business initiative, which aims to advance the global UNAIDS 90-90-90 targets for children and adolescents.

ELMA funds complementary partners toward the same set of goals: increasing the testing, identification, and treatment of infants, children, and adolescents. Implemented in Eswatini, Kenya, Lesotho, Malawi, Mozambique, South Africa, Uganda, Zambia, and Zimbabwe, these programs put a spotlight on pediatric HIV in high-burden countries. ELMA commissioned a series of evaluations to assess the impact and sustainability of the Unfinished Business initiative and to inform our ongoing grantmaking.

Unfinished Business Sustainability Evaluation

Evaluation Partner: Rabin Martin (2018)

Malawi, South Africa, Uganda, and Zambia

Impact Evaluation of Unfinished Business Programs Implemented by Elizabeth Glaser Pediatric AIDS Foundation (EGPAF)

Evaluation Partner: Itad (2018)

Kenya, Eswatini, and Zambia

Impact Evaluation of the Unfinished Business Initiative in Uganda

Evaluation Partner: Itad (2019)

Uganda

Impact Evaluation of the Unfinished Business Initiative in South Africa

Evaluation Partner: Itad (2019)

South Africa

 
Stroke-Gray.png

Lessons Learned About Evaluations

ELMA commissioned this series of evaluations to enhance grantmaking and assess the impact and sustainability of the Unfinished Business pediatric HIV initiative. Impact evaluations (described above) focused on programmatic outcomes, the contribution of ELMA support to increasing pediatric HIV testing and treatment, enabling factors for positive impact, and barriers to demand for HIV services. Because ELMA was one of many funders in a complex landscape, it did not seek attribution of results, but instead sought to understand how funding contributed to positive change. Evaluations were summative, conducted at the end of the grant term to help ELMA understand the impact of the grants, inform decisions about follow-on funding and strategy, and inform the field as a whole. These evaluations provided valuable lessons and insights into program design as well as ELMA’s evaluation practice, which are described below.

A highly collaborative approach to evaluation is valuable but is time-intensive. ELMA found that a co-creation approach to evaluation was invaluable for gaining grantee insight into the findings and engendering buy-in for the evaluation. ELMA and its evaluation partner, Itad, brought grantees into the evaluation process early, starting a theory of change workshop to ensure alignment on the pathways to change being evaluated. This was followed by a participatory evaluability assessment to determine evaluation questions, and co-creation workshops post-evaluation where findings were presented and interpreted with input from grantees. This approach helped to ensure that the views of grantees are well represented while maintaining objectivity in the evaluation. The co-creation process, however, necessitated multiple points of participation and was time-intensive for grantees, evaluators, and ELMA program and E&L staff. These were worthwhile trade-offs, but they required a high level of commitment and engagement from participants.

The evaluations could benefit from greater focus and be more forward-looking. ELMA’s E&L and program staff crafted the scope of the evaluation collaboratively, but were not able to avoid the pitfall of wanting to know everything about everything and including too many evaluation questions (the Terms of Reference for one of the evaluations had 18 evaluation questions!). While an evaluability assessment helped to narrow the scope, the evaluation could have benefited from further prioritization and focus, and be more heavily driven by a specific learning agenda or targeted decisions. The broad scope of the evaluations resulted in a wide array of data gathered, represented, and discussed, and at times led to information overload. 

Where robust program monitoring data are available, duplicating efforts with an evaluation is unnecessary. In some instances, the evaluation questions could be answered with robust program monitoring data (especially on testing, identification, linkage, and retention), but the ELMA team also wanted an objective review of the data. Itad was tasked with compiling the program data, doing additional analyses, and folding them into the evaluation. ELMA and Itad envisioned a fairly straightforward process, but despite Itad’s best efforts to help grantees understand how to prepare the data, it was a challenge to obtain the data at the correct level of granularity and in the correct format. With competing priorities and multiple data requests, grantees did not always respond to data requests in time, which required additional mediation from ELMA and resulted in evaluation delays. While ELMA thought it was valuable to get a second pair of eyes on the data, the duplication of efforts and delays that resulted were not worth the effort. In short, where program-monitoring data exists to address important questions, funders and evaluators ought to reconsider the utility and efficiency of including those questions in an external evaluation or focus efforts in validating the data.

Be mindful of the timing of evaluations to avoid evaluation fatigue. Most of the Unfinished Business grants wound down at approximately the same time and the evaluations thus overlapped with one another. Because the evaluations emphasized a heavily-involved co-creation and collaborative approach, as described above, evaluation fatigue set in among grantees, ELMA program and E&L staff, and stakeholders who sometimes responded to multiple interview requests from different evaluators. While ELMA was mindful to space out the fieldwork among evaluators, some overlap in timing was unavoidable and became burdensome for participants.

Have a deliberate strategy for knowledge management, dissemination, and socialization of the findings among stakeholders. A great deal of effort went into ensuring that the evaluations were high-quality and had buy-in from stakeholders during the course of the evaluations. While we thought about audiences for the evaluation, there was less deliberate planning on mechanisms and forums for dissemination and socializing the findings among stakeholders (including other partners and the government). As a result, formal discussions about dissemination did not take place until the evaluations were nearly completed, leading to an extension and expansion in the evaluators’ scope of work and budget. ELMA E&L and program staff worked together to define external audiences for dissemination efforts. They also engaged with evaluators to facilitate workshops and meetings to disseminate findings from the evaluation, create knowledge products for different audiences, and ensure there is follow-through to act on the evaluation. Dissemination efforts have been well received, and ELMA is now more mindful of building in time and resources for a dissemination plan as part of the evaluation design.

Photo: Eric Bond/Elizabeth Glaser Pediatric AIDS Foundation

Photo: Elizabeth Glaser Pediatric AIDS Foundation / Eric Bond