How Do We Measure What Really Counts_md_sept09

FORUM 2009

How Do We Measure What Really Counts? Session discusses assessing the impact of complex development interventions and applying results. BY HILARY NALVEN, SENIOR PROGRAM ASSOCIATE FOR STRATEGIC IMPACT, INTERACTION

I

NTERNATIONAL DEVELOPMENT actors confront two challenging demands: the need to respond to complex development problems with multi-faceted interventions and, at the same time, to demonstrate impact. In response, InterAction’s Evaluation and Program Effectiveness Working Group hosted the Forum session entitled How do we measure what really counts? Assessing the impact of complex development interventions and applying results. The workshop was moderated by Carlisle Levine, Senior Technical Advisor for Advocacy Evidence Systems and Impact Assessment at CARE USA. Panelists shared their experiences successfully measuring the impact of empowerment programs and using evidence to influence decision-makers. Ms. Levine noted that, while there is

38

MONDAY DEVELOPMENTS September 2009

not a single definition for impact evaluation, this session would use the one that centers on assigning attribution by establishing a credible counterfactual, i.e. evidence that without a specific action some change would not have occurred. Howard White, Executive Director of the International Initiative for Impact Evaluation (3ie), put forth an objective, “to counter the view that [organizations should] only measure what is easy to measure and not what matters.” To successfully do this, one must first clearly define the concept that is to be measured. While at the World Bank, Mr. White measured women’s empowerment using indicators at the national, intermediary and local levels and allowing for them to remain culturally specific. There are a number of surveys such

as the USAID-supported Demographic and Health Survey that collect various measures of women’s agency, the questions varying by country. A promising new approach is that of “vignettes”, which provide culturally specific reference points through short descriptions on three or four people in a familiar local setting. Respondents are asked to rate their own circumstances relative to these vignettes. This ranking, although difficult and costly, allowed the World Bank to accurately quantify how empowerment had changed over time. Mr. White emphasized that impact evaluations require costly data collection and are not always appropriate. Therefore, it is important to identify when to complete impact evaluations. Namely, decisions should be technically, practically, and politically feasible, and: “be issues driven, not methods driven, pilot projects, or important projects [those that are large-scale or expected to have substantial impact]; test ‘conventional wisdom’; and representative of an agency’s activities.” When impact evaluations are not advisable, consistent performance monitoring can be carried out. Dr. Anju Malhotra, Vice President for Social and Economic Development at the International Center for Research on Women (ICRW) concentrated on four points. First, it is important to try to measure complex concepts such as justice, equity and poverty alleviation. Second, classifying issues as soft or hard is a false dichotomy; most have both hard (tangible) and soft (intangible) measures. Third, we should measure the “what” and the “how” of development interventions. Finally, evaluators and implementers must be closely connected. Dr. Malhotra believes women’s empowerment is one of those complex concepts that can and should be measured. She defined women’s empowerment as “the ability to make strategic life choices,” and clarified this further stating, “having a cow does not mean empowerment, rather it is the ability to use the cow to sell milk and make money.” She compared measuring women’s empowerment to the complex issue of poverty—something people have been trying to measure for years. For example, like poverty, women’s

FORUM 2009

empowerment looks different in various contexts and may take decades to see progress. Furthermore, the overall picture and the sub-parts of complex interventions must be considered. Specifically, since women’s empowerment consists of large and small changes over the long-term, it should be broken up into “slices.” A key question is, “What slice do we take on?” and, based on this, develop appropriate indicators and evaluation design. ICRW concentrates on the “empowerment slice”, using as measures the actors involved, the context of the project, and the levels at which ICRW works. Both panelists addressed how to use evaluation findings to influence decision-makers. Dr. Malhotra hypothesized that evaluating mid-level issues might be more conducive to influencing change, as they identify more actionable, feasible solutions (e.g. while women’s empowerment is broad and complex, child marriage is a mid-level issue). They advised that evaluations must be well designed and technically rigorous. It must address relevant questions for both program participants and policymakers, and include all actors from the beginning to ensure necessary buy-in. Notably, findings can be context specific; to generalize results we need a wide-ranging body of evidence. Finally, those in power must be educated about the issues. After hearing the presentations perhaps a more accurate title for this session would be, “you can measure what really counts,” with an addendum, “but it isn’t easy.” MD

Further reading Measuring Empowerment in Practice Structuring Analysis and Framing Indicators by Ruth Alsop, Nina Heinsohn Comparing Incomparable Survey Responses: New Tools for Anchoring Vignettes by Gary King and Jonathan Wand Measuring Empowerment: CrossDisciplinary Perspectives by Deepa Narayan

Managing Risk and Litigation Exposure

Panel offers ways to improve security and limit liability. BY SHANNON DAVIS, SECURITY ASSOCIATION MANAGER, INTERACTION

D

O NGOS NEED TO BE CONcerned with litigation? Ray Biagini, a partner at McKenna, Long, and Aldridge, says yes. Yan Bui, Account Executive at Clements International, and Victor Ferreira, Senior Consultant at Control Risks, discussed ways in which an NGO is at risk and the measures it can take to protect itself from litigation at the InterAction workshop Increasing Risk and Litigation Exposure. There is a rising trend of NGOs facing litigation due to the increased risk NGO staff encounter in hostile environments. Although there are not many reported cases, the presenters believed there are an increasing number of NGO legal liability lawsuits, typically taking the form of a negligence (tort) claim. John Schafer, Senior Security Coordinator at InterAction, offered a hypothetical scenario: two NGO staff members, member Alpha and member Bravo, leave for a trip to Haiti. Alpha received security training, the coun-

try risk assessment, and information about the organization’s policies and procedures for staff travelling overseas. Bravo received very little information and no training. Upon arriving at the airport, Alpha was picked up by prearranged driver. No pre-arrangements had been made for Bravo so he used a local driver and was attacked en route to his destination and taken captive. After 37 days, Bravo was released from captivity, suffering from hepatitis C, body injury, malnutrition, and psychological ailments. Is the NGO at fault? The answer is yes. Using this example, the presenters discussed the steps NGOs can take to protect their staff. Victor Ferreira noted that the organization has a responsibility to provide structures and tools to ensure the safety and security of its employees, otherwise referred to as the organization’s duty of care. InterAction member organizations are required to comply with the Minimum Operating Security Standards, or MOSS, which can limit exposure to negligence claims. NGOs should provide risk assessments to staff working in or travelling to specific regions or countries that may be dangerous. Staff members should also be asked to review and possibly sign some form of written documentation that acknowledges their understanding of the risks they may encounter. NGOs should consider having some form of insurance to help mitigate risk. However, even with insurance coverage, NGOs should be proactive in ensuring that all staff receive security training. If the NGO has a security infrastructure, this helps to reduce the premium rates as well as liability. Security standards should be continuously monitored and communicated according to the potential risks that staff may encounter overseas. MD

MONDAY DEVELOPMENTS September 2009

39