The main objective of the final evaluation is to assess the short and medium-term impact of the project intervention on the lives of individual beneficiaries and their families, achieved through the delivery of a package of different emergency response activities and trainings for national partners. The evaluation shall also look at in how far the interlinkage of activities (MPCT and protection) affects (positively but also negatively) affects the wellbeing of the target group.

The final evaluation is expected to

• Provide an assessment of the overall impact of the CARMA4 project against the specific and overall objectives as well as results as defined in the project’s logical framework.
• Deepen insight into assumptions and approaches underpinning the project, and how these compare to the priorities and concerns of beneficiaries.
• Assess the relevance of the project with regard to design and coherence, including the intervention logic and its assumptions.
• Assess the extent to which the project was effective in delivering the planned outputs and outcomes.
• Assess the extent to which the project has managed to achieve a positive impact on the lives of the project’s beneficiaries (disaggregated by women, men, girls and boys).
• Provide recommendations for possible future interventions of the same type.
• How did the consortium adapt to unforeseen challenges (including changes due to take over of new authorities etc.)

This final evaluation will be conducted in line with the revised OECD/DAC evaluation criteria of 2019[1].

EVALUATION QUESTIONS

Below a preliminary list of guiding questions, which will be agreed on and finalised in the inception report.

Relevance (25%)

• To what extent were project design and approach appropriate to the situation?
• To what extent are the outcomes and indicators of the project still pertinent?
• Are the planned outputs of the project appropriate to meet the outcome and the overall goal (impact) – as part of the analysis of the logframe matrix / project strategy?
• Are the assumptions and risks still valid?
• To what extent have multiple sources of information (including affected people and communities, local institutions, etc.) been consulted when assessing needs, vulnerabilities and context?
• To what extent are the interventions (activities, targeting, transfer modalities) appropriate: culturally, social-economically and environmentally?

Effectiveness (15%)

• To what extent has the programme already produced its expected outputs or will be likely to achieve them? To what extent has the programme already achieved its outcomes or will be likely to achieve them? What were the major factors influencing the achievements or non-achievements?

• To what extent is the programme likely to achieve real impact (change) for the target groups?

• How is project-generated (additional) disposable HH-income utilised or invested by target HHs (e.g. education, food security, others, etc.)?

• Did the programme contribute to capacity building as planned? To what extent is knowledge of the project teams exploited and contributing to the achieved results (benefits of knowledge management)?

• Was the main target group involved in the project planning phase? Are the statements of the main target group on the attainment of goals identical with the opinions of the actors having provided humanitarian assistance (e.g. employees of the respective organisation)?

• What have been the key challenges and/or risks hampering implementation of the project? How have these been addressed by the programme management? Which steering and/or mitigation measures have been taken? Have these been the right ones? How effective was the decision-making process?

• Were (local) coordination structures established? How did the organisations harmonise and coordinate their interventions with other partners? Was there a lead agency nominated? Which factors hindered and supported the coordination? Has the coordination led to more efficiency and impact of the intervention?

• Where activities properly designed to address the needs of the target group and accordingly delivered in an effective manner? Where MPCT effectively distributed (also given the challenges faced due to the change of authorise within Syria)? Was the set of protection activities effective in releasing urgent protection risks among the target group?

  Efficiency (15%)

• Was the project implemented in the most efficient way in terms of time and resource allocation, planning, coordination, and communication?

• To what extent has the project been managed and implemented as planned? In case: what issues occurred and why?

• Are constraints and risks regularly identified and analysed, and plans adapted accordingly?

• Was the transferring modality implemented in the most efficient way (cash through FSPs) compared to alternative modalities or transfer mechanisms?

• Were the budget and available financial resources realistic for the achievement of the intended objectives and outputs (to include staffing, training, capacity building)?

• To what extent were all items/equipment purchased and used as planned under this programme?

• Are programme management and decision-making processes within the programme team and steering committee efficient? What role does participation play in decision-making processes within the programme team?

• Are there clear processes in place to support monitoring and use of the monitoring-results for management and decision-making?

• In the CARMA project, what are core issues and benefits of working with multiple/different partners in order to create impact?

  Impact (25%)

• Are the results/outcomes of CARMA in line with the overall and specific objectives as formulated in the project’s logical framework and specified in the proposal? If any, what were unintended impacts of the programme?

• What has happened as a result of the project? What do beneficiaries consider the most significant changes brought about by the project in their lives (immediate impact)? How do beneficiaries evaluate the impact of the CARMA intervention in proportion to their overall needs?

• What were the positive and negative, intended and unintended effects?

• How was the money (cash grants to beneficiaries) used? What was the money used for (or in which sectors)? Were the investments of cash short-term or long-term?

• Which impact have protection activities had on the target group? Was CfP/in-kind support utilised for the intended purpose and which impact can be seen on the target group?

• How where capacity building measures integrated into work practices of national partners?

• To what extent has the programme affected HH dynamics, female participation and/or inclusiveness in HH decisions? Who is in charge of the cash on HH level? How is it distributed on HH level and who is spending the cash (gender differences)?

• Have the cash transfers affected HH behaviours, such as use of coping strategies, changes in food consumption, changes in education or health practices? Is the transfer value sufficient to meet project objectives (specifically, meeting HH basic needs)?

• Did beneficiaries consider the vulnerability criteria fair and transparent and understand the selection process? Are any groups missed out in their opinion?

• What evidence is there of the impact that the project has had to date on persons of concern with special needs, women, the relations between the IDPs, returnees, and the host communities and the local economy? Which changes are evident and attributable to the project?

• Has the project reached the key groups who are at greatest risk? Have all of those in need of protection received protection during the interventions?

• To what extent has the CARMA intervention contributed to wider sector learning and/ or harmonisation?

  Sustainability (20%)

• To what extent will the benefits of a programme or project continue after the project has been completed? What are the major factors which influence the achievement or non-achievement of sustainability of the project? What steps are already and/or need to be taken in order to secure sustainability of the programme?

• To what extent did CARMA contribute to strengthening existing systems or to systemic change (e.g. Damascus-based Cash Working Group and Interagency referral system)?

• To what extent did CARMA contribute to building capacities and expertise of national partners?

• What are key considerations to keep in mind when developing a second phase of the project? What are key considerations to keep in mind when developing an exit strategy for the project?

• To what extent are the target groups and counterparts able to adapt sufficiently to external changes and shocks? How stable is the situation in the surrounding field regarding social justice, economic efficiency, political stability and ecological balance?

METHODOLOGY

It is expected that the final evaluation will employ quantitative as well as qualitative data, making use of different methods, such as analysis of documents, structured interviews, focus group discussions (if feasible), semi-structured interviews (face-to face or remote modality), etc.. All data collected needs to be disaggregated by sex and age of beneficiaries; be sensitive to gender disparities: seek to include persons with disabilities, and be prepared to probe about unintended outcomes of the project.

The methodology to appear in the proposal should include but not limited to

• Review of key project documents including: proposal, project reports, PDMs, previous evaluation reports from CARMA1, CARMA2, CARMA3 and other related documents.
• Overview of any changes based on PDM findings.
• Review of CARMA4 process and outcome monitoring data as well as other data collected in the course of the CARMA intervention (e.g. primary data for outcome and process monitoring, etc.)
• Review of the effectiveness of feedback, complaints and response mechanism.
• Interview key staff associated with the project and other stakeholders with each meeting documented (face-to-face or remote modality).
• Meet and interview project beneficiaries at community level, structures and document all discussions (only if feasible).
• Undertake physical verification visits to project sites or structures to affirm level and results exhibited (only if feasible).

Consultant(s) will be asked to specify methodology for the final evaluation in their technical proposal submitted with the application.

Given potential political instability and uncertainty in the region at the time of field presence, this plan may need to be adapted flexibly and stakeholder meetings may have to be planned and organized in a way to meet legal requirements and minimize exposure to risks. Further remote modalities for meetings can be utilized and CARMA4 consortium members can provide support as needed.

DELIVERABLES

• Inception Report
• Final Draft Evaluation Report
• Final Evaluation Report
• Raw Data
• Validation workshop
• Final Workshop

PROCESS

Contracting: First phase to conclude administrative work

Desk Review: The evaluation team studies all necessary project documents including project proposal, reports, access to monitoring data, third-party reports and studies.

Inception Report: The evaluation team uses a data collection planning worksheet or a similar tool based on the agreed methodology. The first interviews take place. Phase is concluded with the delivery of the inception report: see deliverables. Eventual field trips shall only take place upon official approval of the inception report by the contractor.

Evaluation/ Field phase: Data needs to be gathered, analyzed and interpreted. Consultant(s) are expected to make use of existing project data and generate new data as agreed in methodology (e.g. through interviews, FGD, etc.) either through face to face modality or remote modalities adapted accordingly.

Validation Workshop: Presentation of key findings (feedback workshop) at the end of the office visits / field trips.

Draft Final Evaluation Report: Submission and presentation of draft final evaluation report, inclusion of comments from partners and evaluator.

Final Evaluation Report: Submission of final evaluation report.

Final Report Workshop: Presentation of final report findings, key learning points and recommendations.