DGHI’s Evidence Lab Creates Evaluation Toolkit for Healthcare Innovators

Joy Noel and Jennifer with Afya Staff

(From left) Sam Gwer, Willah Nabukwangwa and Elizabeth Ombech of Afya Research Africa discuss the evaluation tools with Joy Noel Baumgartner and Jen Headley of the Evidence Lab.

Published April 24, 2017, last updated on April 21, 2020 under Research News

Thousands of social entrepreneurs are hard at work around the world trying to solve the most pressing challenges in global health, but they often lack a clear approach to evaluate their enterprise’s impact. The Duke Global Health Institute’s Evidence Lab, in collaboration with the Social Entrepreneurship Accelerator at Duke (SEAD) program, has developed a set of five easy-to-use evaluation tools to address this challenge and help make social entrepreneurs more effective.

The toolkit—which is free and publicly available—contains resources and templates to help innovators better understand the links between data and impact statements, strengthen their own statements about health outcomes, effectively communicate their organization’s impact and make strategic, data-driven decisions about how to scale up their enterprise.

These tools help innovators evaluate their products or services in an affordable, iterative way that aligns with the pace and structure of the social entrepreneur world.

Jennifer Headley, evaluation research specialist

The tools include:

  • Monitoring Organizational Reach and Influence – Includes a modifiable template that helps innovators track their influence across various domains, enabling them to plan strategically sound outreach activities.
  • Measuring Household Out-of-Pocket Health Expenditure – Explains how to measure out-of-pocket costs, provides suggestions for lean data methods and strategies for addressing common challenges entrepreneurs face in establishing and communicating evidence that their services’ out-of-pocket costs are lower than other similar services. 
  • Strengthening Access and Quality of Care Patient Data – Helps entrepreneurs assess and articulate how their enterprise may improve access to healthcare or provide higher quality of care. Includes tips and guidance to strengthen data collected at a facility related to access and quality of care. 
  • Considerations in Leveraging Data for Expansion – Promotes better understanding and utilization of existing data resources that could inform expansion plans, including tips on using publicly available datasets and when to consider self-collected data.
  • Economic Impact: Finding the Right Analysis for your Evaluation Needs – Provides an overview of seven different types of economic analyses, such as cost-benefit analysis and budget impact analysis and guidance on when to use each type of analysis. 

The toolkit was field-developed with nine SEAD innovators in East Africa and India whose experiences, data gaps, challenges and opportunities informed the focus and framing of the content. As part of the development process, some of the SEAD innovators in East Africa and India have had an opportunity to pilot and provide feedback on the tools. So far, the response has been overwhelmingly positive. 

“We have used the ‘Monitoring Organizational Reach and Influence’ tool and found that it really helped us be more deliberate about our work and how we structure our time within various domains of influence,” said Sam Gwer, executive director of Afya Research Africa, one of the SEAD innovators.

Image
Joy Noel Baumgartner (second from left) discusses the new tools with healthcare innovators at a symposium in March.

Healthcare Entrepreneurs—and their Funders—Need Agile Evaluation Tools

To survive in a competitive healthcare environment, social enterprises need to demonstrate that their products or services result in positive outcomes for the people they’re serving. But traditional evaluation approaches, such as randomized controlled trials, are often not feasible given the dynamic context in which these entrepreneurs operate.

“In part because the product or technology is usually evolving quickly, it’s not feasible to maintain the static, controlled environment required by a rigorous impact evaluation,” said Jennifer Headley, evaluation research specialist with the Evidence Lab. “These tools help innovators evaluate their products or services in an affordable, iterative way that aligns with the pace and structure of the social entrepreneur world.”

In addition to enhancing entrepreneurs’ ability to evaluate and communicate their business’s results and potential, the tools can also help funders and donors understand what kind of information is reasonable to request from healthcare innovators regarding their impact. 

“We often get requests from innovators for assistance conducting relatively resource-intensive cost effectiveness analyses,” said Headley. “But after deeper conversations, we realize that what they need is likely something more simple, like a cost analysis. Their confusion often stems from misused economic terms coming from their donors or funders. Our economic tool also serves an educational purpose for them.” 

Funded by USAID, Tools to Be Widely Used

Funded primarily by USAID’s Higher Education Solutions Network through the SEAD program, the development of the toolkit was informed by the needs of the 25 SEAD innovators in East Africa and India. Headley notes, though, that many of the challenges the toolkit addresses are universal among healthcare social enterprises in low- and middle-income countries.

Headley and Evidence Lab director Joy Noel Baumgartner anticipate that USAID will share these tools with other healthcare social enterprises supported through their programs. Innovations in Healthcare, a frequent collaborative partner with the Evidence Lab, will also disseminate the toolkit to their network of 70+ healthcare innovators globally. And Headley and Baumgartner have presented the toolkit at a number of conferences and will continue to spread the word about these resources.

“Working with numerous social enterprises through SEAD and Innovations in Healthcare, I hear very similar stories across these networks about the challenges of collecting and utilizing data to strengthen statements about their global health impact,” said Baumgartner. “These capacity-building tools are the distilled lessons and guidance that the Evidence Lab communicates one-on-one with many innovators. With their public release, we hope more healthcare entrepreneurs will have at their fingertips tools to do more evaluation themselves and better knowledge about when they may need to reach out for additional expertise."

The tools can be accessed here.

What’s Next for the Evidence Lab

Established in 2014, the Evidence Lab conducts high-quality, objective evaluation using rigorous and innovative research designs and methods, with a focus on evaluation in real-world settings. Working with external partners as well as Duke faculty and entities, they also focus on building capacity in individuals and other organizations to conduct this type of research effectively.

The Evidence Lab has ongoing impact evaluation projects across sub-Saharan Africa with a range a partners including smaller social enterprises, large NGOs and universities. Project topics include quality of care for maternal and child health services, early childhood development programming for HIV-affected families and strengthening delivery of reproductive services for adolescents. 

The Evidence Lab is open to new evaluation research partnerships that move the field of global health forward by generating evidence on the impact of programs, services and technologies.    

Learn more about the Evidence Lab’s approach, services and leadership.

To inquire about collaborating on a project, please contact Joy Noel Baumgartner.