We used the widely adopted Organization for Economic Cooperation-Development Assistance Committee evaluation criteria (OECD-DAC) - adapted for humanitarian contexts by the Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) - to define the following evaluation domains: relevance/appropriateness, connectedness, coherence, coverage, efficiency, effectiveness, and impact [
9].
Relevance/appropriateness: “…relevance is concerned with assessing whether the project is in line with local needs and priorities (as well as donor policy) [and] appropriateness is the tailoring of humanitarian activities to local needs…”. ([
9], p., 20) In addition, others have suggested expanding the definition to include other considerations including the needs of vulnerable groups, relevance in the face of evolving needs, and appropriateness to crisis contexts [
10,
11].
Connectedness refers to, “…the need to ensure that activities of a short-term emergency nature are carried out in a context that takes longer-term and interconnected problems into account”. ([
9], p., 20)
Coherence is defined as, “[t]he need to assess security, developmental, trade and military policies as well as humanitarian policies, to ensure that there is consistency and, in particular, that all policies take into account humanitarian and human-rights considerations. ([
9], p., 21)
Coverage is defined as, “[t]he need to reach major population groups facing life-threatening suffering wherever they are”. ([
9], p., 21)
Efficiency measures, “…the outputs – qualitative and quantitative – achieved as a result of inputs. This generally requires comparing alternative approaches to achieving an output, to see whether the most efficient approach has been used”. ([
9], p., 21)
Effectiveness measures, “… the extent to which an activity achieves its purpose, or whether this can be expected to happen on the basis of the outputs [;] implicit within the criterion of effectiveness is timeliness”. ([
9], p., 21) Finally,
impact considers, “…the wider effects of the project – social, economic, technical, environmental – on individuals, gender- and age-groups, communities and institutions. Impacts can be intended and unintended, positive and negative, macro (sector) and micro (household)”. ([
9], p., 21)
CRM extracted data into a framework including both descriptive domains (e.g. setting, outcomes, recommendations) and the OECD-DAC evaluation criteria [
9]. CRM and LB assessed the quality of included papers using the Quality in Qualitative Evaluation framework and the NIH Quality Assessment Tool for Observational Cohort and Cross-sectional Studies [
12,
13].