Evaluations of Unfinished Business ELMA's pediatric HIV initiative in Africa

< BACK TO E&L HOMEPAGE

Evaluations of Unfinished Business - ELMA’s Pediatric HIV Initiative in Africa

July 8, 2020

June Lee
Director, Evidence & Learning
ELMA Philanthropies

Photo: Elizabeth Glaser Pediatric AIDS Foundation / Eric Bond

When The ELMA Foundation began its work in 2005, pediatric HIV was one of the most critical issues facing children in sub-Saharan Africa.

In response, the early years of ELMA’s HIV investments focused on addressing barriers to treatment, including the cost and formulation of pediatric antiretroviral medications, access to diagnostics, training health workers, and building political commitment and leadership.

Despite some important gains, by 2013, only 23% of the 3.2 million children living with HIV (91% living in Africa) had access to treatment. ELMA therefore sharpened its focus on pediatric HIV through the Unfinished Business initiative, which aims to advance the global UNAIDS 90-90-90 targets for children and adolescents.

ELMA funds complementary partners toward the same set of goals: increasing the testing, identification, and treatment of infants, children, and adolescents. Implemented in Eswatini, Kenya, Lesotho, Malawi, Mozambique, South Africa, Uganda, Zambia, and Zimbabwe, these programs put a spotlight on pediatric HIV in high-burden countries. ELMA commissioned a series of evaluations to assess the impact and sustainability of the Unfinished Business initiative and to inform our ongoing grantmaking.

Unfinished Business Sustainability Evaluation

Evaluation Partner: Rabin Martin (2018)

Malawi, South Africa, Uganda, and Zambia

The evaluation examined the sustainability of ELMA’s Unfinished Business initiative in the context of the pediatric HIV ecosystem. It identified the gaps and opportunities for increased program sustainability in Malawi, South Africa, Uganda, and Zimbabwe. The three domains of sustainability examined were: (1) governance, leadership, and accountability, (2) financing (buy-in and funding), and (3) program capacity (staffing, skills, tools, and measurement).

Key Findings

Each country presented different strengths and weaknesses. Overall, the most promising areas of sustainability were in national plans and policies, many of which had been updated or created to reflect a greater focus on pediatric and/or adolescent HIV. Human resource capacity and data systems emerged as ongoing challenges to sustainability in all four countries. The program enabled the institutionalization of critical tools and practices. These include training that has improved aspects of provider capacity, government adoption of key tools, and quality assurance and data processes that have helped implementing organizations assess and improve performance. Service providers in all four countries, however, faced the continued need for additional training and mentoring support to improve comfort and competence with pediatric and adolescent HIV care. On a broader level, reliance on donor funding in Malawi, Uganda, and Zimbabwe often adversely affected coordination and alignment of priorities (government vs. donor), and attempts to transition activities to national government have had mixed success.

ELMA has been supporting many of the pathways to sustainability that the evaluation has identified, including increasing healthcare provider skills, strengthening data systems, improving service delivery integration, supporting government stewardship. To ensure sustainability, intentional transitioning of support to government should be (and are) a critical part of future initiatives.

Impact Evaluation of Unfinished Business Programs Implemented by Elizabeth Glaser Pediatric AIDS Foundation (EGPAF)

Evaluation Partner: Itad (2018)

Kenya, Eswatini, and Zambia

From 2015-2019, ELMA supported the Elizabeth Glaser Pediatric AIDS Foundation (EGPAF) to increase pediatric HIV testing across all ages and improve linkages to treatment in Eswatini, Kenya, and Zambia. To assess the outcomes of this program, ELMA partnered with Itad for an evaluation.  The evaluation assessed the programmatic outcomes and challenges in scaling up pediatric HIV testing and treatment.

Key Findings

As of the completion of the evaluation in November 2018, over 986,800 children and adolescents had been tested, and over 8,600 had been initiated on anti-retroviral therapy across Eswatini, Kenya, and Zambia. The evaluation concluded that with ELMA support, EGPAF accelerated testing in children and adolescents and contributed to better prioritization of pediatric HIV in policy and practice in all three countries. The evaluation identified important strengths of Unfinished Business:

• EGPAF is well respected as a technical leader in pediatric HIV, has worked effectively with the Ministry of Health in each country, and plays an active role in technical working groups.

• Stigma continues to be a significant barrier to full coverage of pediatric HIV counseling, linkage, and retention.

• Adolescents continue to be a key age group that remain difficult to reach, and a multi-sectoral approach that extends beyond the Ministry of Health (e.g., to Ministries of Education) is needed for an effective and sustained response.  

Impact Evaluation of the Unfinished Business Initiative in Uganda

Evaluation Partner: Itad (2019)

Uganda

In Uganda, ELMA supported a coalition of five partners (Baylor Uganda, Clinton Health Access Initiative, Elizabeth Glaser Pediatric AIDS Foundation, Infectious Diseases Institute, and Mildmay Uganda) to accelerate pediatric and adolescent HIV treatment, and to improve the quality of services in 19 high-burden districts. This evaluation assessed the outcomes this Unfinished Business initiative to understand contributors to positive impact and barriers to demand.

Key Findings

The evaluation found year-to-year increases in HIV testing in Unfinished Business-supported districts, with over 1.08 million children under 15 years old and over 947,000 adolescents ages 15 to 19 tested. Unfinished Business partners (Baylor Uganda, Elizabeth Glaser Pediatric AIDS Foundation, Infectious Diseases Institute, and Mildmay Uganda) contributed to testing rates through the use of screening tools, and innovative and effective outreach and testing strategies, including testing at key entry points (with highest yields in TB clinics), index testing, and outreach to targeted populations or hotspots. Among infants, Unfinished Business districts had high testing rates for initial HIV tests (96%), but there was substantial loss to follow-up on second HIV tests, and greater loss for 18-month tests. Unfinished Business districts had higher positivity rates compared to non-Unfinished Business districts, but positivity rates have been declining in both Unfinished Business and non-Unfinished Business districts, even with more targeted testing.

By the end of the program, treatment initiation was high for children under age two (93%) and more moderate for other ages. The Unfinished Business program made significant contributions to linkage-to-care through training specialized facilitators and counselors. Retention presents a mixed picture across implementing partners and age groups. Retention was not central to this phase of Unfinished Business, but partners have recognized the importance of improving retention strategies, which is now a focus in the second phase of the initiative.

Patients were generally happy with the quality of care they received; long wait times were the primary complaint. Adolescents had mixed perceptions of how they were treated at facilities, with some feeling that they were judged and unwelcome. Training of health care providers, quality improvement processes, and support supervision were credited for improving quality of care, although the evaluators also noted that there were no objective measures that could be used to verify quality improvement.

Sustainability (of activities, institutionalizing important practices such as data review, increasing government ownership and support) emerged as an ongoing challenge for Unfinished Business. ELMA has identified this as a critical goal of the second phase of Unfinished Business to ensure that activities end with direct adoption or institutionalization. 

Impact Evaluation of the Unfinished Business Initiative in South Africa

Evaluation Partner: Itad (2019)

South Africa

In South Africa, ELMA supported implementing partners to improve strategies to identify infants, children, and adolescents with HIV and link them to care. This Unfinished Business initiative took place in five priority districts in Johannesburg and KwaZulu-Natal. This evaluation assessed the outcomes of the program and provided insights that inform future investments in pediatric HIV.

Partners for Johannesburg Health District: Wits Reproductive Health and HIV Institute (WRHI), Anova Health Institute, Right to Care, CARE, HIVSA, with WRHI as the lead on data management. Partners for KZN: Zoë-Life, Maternal Adolescent and Child Health (MatCH), Health Systems Trust, AIDS Foundation of South Africa (AFSA), Networking HIV & AIDS Community of Southern Africa (NACOSA), National Association of Child Care Workers, Mothers to Mothers (m2m), with Zoë-Life the lead on data management  

Key Findings

The evaluation revealed important contributions that Unfinished Business partners have made to pediatric HIV testing and treatment, though progress has been uneven across areas of care. Retention and viral suppression remain more challenging. In addition to putting a spotlight on pediatric HIV and increasing the number of infants, children, and adolescents identified and linked to care, the Unfinished Business initiative has added to efforts in other critical ways:

• The program worked and consulted with government at all levels and built consensus on the importance of pediatric and adolescent HIV treatment and care.

• Program partners developed and used effective tools to screen for patients who should receive HIV tests, linkage tools to increase infant ART initiation, and tools that help to retain children and adolescents on anti-retroviral therapy. A more consistent approach to the use of tools across partners is still needed to help amplify their impact.

• Human resources account for much of the achievements in Unfinished Business, and the program helped to strengthen the necessary health cadres for this work. Community health/outreach workers were essential for locating children and families for testing, initiating them on treatment, and retention in care. Child and youth care workers, adolescent peer counselors, nurse mentors, counselors, and linkage officers have been important through the entire cascade of care. Institutionalizing the work of these cadres into the health system will continue to be a focus of Unfinished Business.

• Unfinished Business partners also contributed to the improved testing and care of adolescents, who continue to be a key population. Adolescent peer educators, support groups facilitated by community organizations, and psychosocial support emerged repeatedly as being important for referral for testing, and adherence to treatment.

• The Unfinished Business program improved data management among partners and promoted data use for decision-making. Daily data capture and review helped to improved linkage to care. Weekly data reports and monthly data reviews became more routine. However, coordinating data collection and reporting through the Department of Health data network remains a challenge in some locations.  

Going Forward

While the program has made impressive strides, the following are areas to be bolstered to sustain gains.

• The actual package of care for children and adolescents still varies widely by area. Harmonizing testing and treatment approaches that have worked, scaling targeted strategies, and consolidating high-impact practices for a wider rollout is an important part of the second phase of Unfinished Business.

• Demand for HIV services are constrained by personal and structural barriers. Overcoming these barriers to increase the awareness and uptake of services will become increasingly important for last-mile efforts.

• Patients’ perceptions of quality of care were mixed. Some were happy with the services they received, whereas others noted the long wait times, lack of youth- and child-friendly spaces, and staff attitudes as barriers.

Lessons Learned About Evaluations

ELMA commissioned this series of evaluations to enhance grantmaking and assess the impact and sustainability of the Unfinished Business pediatric HIV initiative. Impact evaluations (described above) focused on programmatic outcomes, the contribution of ELMA support to increasing pediatric HIV testing and treatment, enabling factors for positive impact, and barriers to demand for HIV services. Because ELMA was one of many funders in a complex landscape, it did not seek attribution of results, but instead sought to understand how funding contributed to positive change. Evaluations were summative, conducted at the end of the grant term to help ELMA understand the impact of the grants, inform decisions about follow-on funding and strategy, and inform the field as a whole. These evaluations provided valuable lessons and insights into program design as well as ELMA’s evaluation practice, which are described below.

A highly collaborative approach to evaluation is valuable but is time-intensive. ELMA found that a co-creation approach to evaluation was invaluable for gaining grantee insight into the findings and engendering buy-in for the evaluation. ELMA and its evaluation partner, Itad, brought grantees into the evaluation process early, starting a theory of change workshop to ensure alignment on the pathways to change being evaluated. This was followed by a participatory evaluability assessment to determine evaluation questions, and co-creation workshops post-evaluation where findings were presented and interpreted with input from grantees. This approach helped to ensure that the views of grantees are well represented while maintaining objectivity in the evaluation. The co-creation process, however, necessitated multiple points of participation and was time-intensive for grantees, evaluators, and ELMA program and E&L staff. These were worthwhile trade-offs, but they required a high level of commitment and engagement from participants.

The evaluations could benefit from greater focus and be more forward-looking. ELMA’s E&L and program staff crafted the scope of the evaluation collaboratively, but were not able to avoid the pitfall of wanting to know everything about everything and including too many evaluation questions (the Terms of Reference for one of the evaluations had 18 evaluation questions!). While an evaluability assessment helped to narrow the scope, the evaluation could have benefited from further prioritization and focus, and be more heavily driven by a specific learning agenda or targeted decisions. The broad scope of the evaluations resulted in a wide array of data gathered, represented, and discussed, and at times led to information overload. 

Where robust program monitoring data are available, duplicating efforts with an evaluation is unnecessary. In some instances, the evaluation questions could be answered with robust program monitoring data (especially on testing, identification, linkage, and retention), but the ELMA team also wanted an objective review of the data. Itad was tasked with compiling the program data, doing additional analyses, and folding them into the evaluation. ELMA and Itad envisioned a fairly straightforward process, but despite Itad’s best efforts to help grantees understand how to prepare the data, it was a challenge to obtain the data at the correct level of granularity and in the correct format. With competing priorities and multiple data requests, grantees did not always respond to data requests in time, which required additional mediation from ELMA and resulted in evaluation delays. While ELMA thought it was valuable to get a second pair of eyes on the data, the duplication of efforts and delays that resulted were not worth the effort. In short, where program-monitoring data exists to address important questions, funders and evaluators ought to reconsider the utility and efficiency of including those questions in an external evaluation or focus efforts in validating the data.

Be mindful of the timing of evaluations to avoid evaluation fatigue. Most of the Unfinished Business grants wound down at approximately the same time and the evaluations thus overlapped with one another. Because the evaluations emphasized a heavily-involved co-creation and collaborative approach, as described above, evaluation fatigue set in among grantees, ELMA program and E&L staff, and stakeholders who sometimes responded to multiple interview requests from different evaluators. While ELMA was mindful to space out the fieldwork among evaluators, some overlap in timing was unavoidable and became burdensome for participants.

Have a deliberate strategy for knowledge management, dissemination, and socialization of the findings among stakeholders. A great deal of effort went into ensuring that the evaluations were high-quality and had buy-in from stakeholders during the course of the evaluations. While we thought about audiences for the evaluation, there was less deliberate planning on mechanisms and forums for dissemination and socializing the findings among stakeholders (including other partners and the government). As a result, formal discussions about dissemination did not take place until the evaluations were nearly completed, leading to an extension and expansion in the evaluators’ scope of work and budget. ELMA E&L and program staff worked together to define external audiences for dissemination efforts. They also engaged with evaluators to facilitate workshops and meetings to disseminate findings from the evaluation, create knowledge products for different audiences, and ensure there is follow-through to act on the evaluation. Dissemination efforts have been well received, and ELMA is now more mindful of building in time and resources for a dissemination plan as part of the evaluation design.

Photo: Elizabeth Glaser Pediatric AIDS Foundation / Eric Bond