As part of developing the TOR, the manager needs to provide the focus for the evaluation, based on its anticipated users and uses. The scope of the evaluation describes what is included and what is excluded. Defining the scope provides an opportunity to think through what is important to obtain from the evaluation and what is possible.
The scope of an evaluation looks at issues of timing (when in the life of the programme is the evaluation being conducted), timeframe (should this evaluation cover a specific timeframe in the life of the life of the programme, such as since a previous evaluation) and geography (where the programme has or has not operated / provided services).
The scope also looks at the thematic coverage and the key issues to consider and thus is further defined by the specific evaluation questions that will be answered by the evaluation. Answers to evaluation questions provide evaluation users with the information needed for the purpose of the evaluation in order to make decisions, take action or add to existing knowledge. In this process the manager should develop and gain consensus on the evaluation questions, including the GE / HR aspects, with stakeholders.
The evaluation should clearly spell out the evaluation criteria against which the programme can be assessed (according to UNEG Standard 3.6). Evaluation questions should be formulated with a gender equality and human rights perspective, and the evaluation should present findings accordingly.
The UNEG Standards and OECD Development Assistance Committee (DAC) provide guidance for evaluation criteria. Most commonly used criteria include:
Example Questions: How well do the programme objectives target the identified rights and needs of male and female beneficiaries? What rights does the programme advance under CEDAW, the Millennium Development Goals and other international development commitments?
Relevance:The extent to which the objectives of a development intervention are consistent with the evolving needs and priorities of beneficiaries, partners, stakeholders – within country and global contexts.
Relevance provides information about the relationship of the most important GE and HR issues present in the situation to the objectives of the intervention. It should also assess whether the programme design was guided by international Conventions, principles and protocols (e.g. CEDAW, UDHR) and whether it followed adequate GE and HR analysis.
Example Questions :To what extent have the objectives been achieved, and do the intended and unintended benefits meet fairly the needs of disadvantaged women? To what extent have the capacities of duty-bearers and rights-holders been strengthened?
Effectiveness: The extent to which the development intervention’s objectives were achieved, or are expected / likely to be achieved.
Effectiveness determines whether the expected results of the intervention in terms of improvement of GE and HR have been achieved. In cases where these results were unclear or not explicitly stated in planning documents, they may have to be reconstructed by exploring the programme’s theory of action.
Example Questions: Could the activities and outputs have been delivered with fewer resources to the target populations without reducing their quality and quantity? How has the programme maximized partnerships in the delivery of the programme? Have UN Woman’s organisational structure, managerial support and coordination mechanisms effectively supported the delivery of the programme?
Efficiency: A measure of how economically resources / inputs (funds, expertise, time, etc.) are converted to results.
Efficiency addresses if the expenditure on promoting rights and building the capacity of duty bearers and rights holders achieved the expected results or if funds could have been spent more wisely. Given that many of the behavioural changes implied in GE / HR take a long time to be achieved, the analysis of efficiency should consider short term process achievements (e.g. participation and inclusiveness) and long term results (e.g. actual changes in the enjoyment of HR by the target population and equality between women and men). A direct relationship between resource investment and long term results should be carefully established.
Example Questions: To what extent have efforts been successful in stopping harmful and discriminatory practices against women? What is the evidence that the programme enabled the rights-holders to claim their rights more successfully and the duty-holders to perform their duties more efficiently?
Impact: Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.
Impact assesses whether and how right holders have been enabled to exercise their rights and duty bearers are complying with their responsibilities. It also looks at changes in access to and use of resources, decision-making power, and work burden for women and for men. Assessing impact is complex because there may be multiple causes for processes and events, a long time span before an impact is seen, etc.
Example Questions: Is the programme supported by national and local women’s organisations? Do these organisations demonstrate leadership commitment and technical capacity to continue to work with the programme or advocate for change?
Sustainability: The likelihood of a continuation of benefits from a development intervention after the intervention is completed or the probability of continued long-term benefits.
Sustainability provides information about whether rights achieved by rights holders, and the benefits accruing from those rights, can be maintained over time. It should also look at whether redistribution of resources, power and workload between women and men is likely to persist after the intervention. Sustainability should assess whether accountability and oversight systems have been established between duty bearers and rights holders, as well as whether capacity has been built of rights holders to claim their rights and duty bearers to fulfill them.
The manager needs to think about how these criteria influence the questions identified and/or how the identified questions are expressed. Additional examples of GE / HR evaluation questions, aligned within these evaluation criteria are provided.
These criteria are a starting point. It is important, throughout the development of evaluation questions to consider issues of:
Example Question: Did the programme benefits affect equally men and women?
- Equality and non-discrimination
- Empowerment
- Accountability
- Social transformation
- Participation and inclusion
Example Question: Does the intervention’s theory of change include attention to GE and HR? Did the budget designate sufficient resources and level of effort to address the inclusion of disadvantaged or marginalised groups?
The manager will need to include evaluation questions that relate not only to the outcomes and impacts of the programme, but to the planning and design and the monitoring and implementation stages of the intervention. Questions relating to the planning and design stage of the intervention provide an assessment of how well the GE and HRBA goals and processes were incorporated into the planning documents of the intervention being evaluated.
Example Questions: Did the implementation make systematic and appropriate efforts to include women and men, and/or reach out to disadvantaged groups? Were monitoring data (disaggregated according to relevant criteria such as gender, age, ethnicity, location, and income) collected and used to adjust implementation?
Questions related to monitoring and implementation provide an assessment of: how well the intervention succeeded in involving women and men, and rights holders as well as duty bearers; how well the program was monitored and if this included GE /HR dimensions; and issues related to the process and effects of implementation. Additional examples of questions related to the analysis of design, planning and implementation, as well as the analysis of outcomes/impact are available in the UNEG Handbook for Integrating Human Rights and Gender Equality in Evaluations in the UN System (forthcoming).
Programmes in which the focus on promoting GE and HR is not explicit or is absent (where the intervention is gender and rights “blind”) are more challenging for the development of GE/HR responsive evaluation questions. But it is possible, and important, to develop good questions by applying the principles of GE and HR. Indeed, it is important to assess the effects of all interventions on GE and on HR, no matter the nature, focus or original intentions of the intervention. When managers come to naturally think in terms of GE / HR sensitivity, such questions become central to all programming and evaluation.
The scope defined in TOR should be realistic. It needs to be feasible given the budget and time available for the evaluation.
The manager can brainstorm with stakeholders on the evaluation scope to identify what questions they would like answered by the evaluation. The programme proposal, logic model and ongoing monitoring information and data are sources that can be used to guide the development of evaluation questions. The Baseline Report is also an important information source, as it identifies what was important and measured at the outset of the programme.
The initial list of questions may be very large. The manager needs to work with stakeholders to consolidate and prioritize these into a manageable size. Generally, 3-5 key questions related to each of the selected criteria will provide for a more focused evaluation. Too many questions will make the evaluation unfocused and possibly not do-able. The time and money available for the evaluation is also impacted by the scope and number of evaluation questions.
The scope should take into account other existing or planned evaluations of the same subject so as to harmonize efforts and avoid duplication. The relationship between the planned evaluation and other related evaluations should be described, including how information from these other evaluations may be used.
Tips: Where baseline information is not available, see if and how the evaluation might assist in recreating this. Revisit the scope and your list of evaluation questions (to prioritize and modify) as you develop the evaluation approach and methods.
