Terms of Reference for End of Project Evaluation
Project: Urban Early Warning Early Warning Action (UEWEA)
Grant Information
Project Area
Nairobi _ Kibera, Mukuru and Korogocho slums
Locations
Kibera, Mukuru and Korogocho slums
Grant Amount
£800,000
Contract/Project Start Date
15th November, 2015
Contract/Project End date
31st March, 2018
BACKGROUND INFORMATION
The use of standard food security and nutrition indicators in the urban often depicts a picture of normalcy and does not highlight the extreme vulnerability that there is within the urban areas across the world. This is because the population living in the urban is highly heterogeneous in terms of wealth and high mobile. The large population mass often hides the few pockets of persons experiencing extreme vulnerability. Recognizing that standard indications of measuring food security and nutrition status, Concern worldwide with support from Consortium partners started a project to develop indicators specific to urban that are sensitive to the dynamics that impact on food security. A 5 (five) year research project dubbed ‘Indicator Development for Surveillance of Urban Emergencies (IDSUE)’. The project gave way to the current ‘Urban Early Warning Early Action (UEWEA)’ project that is using the indicators developed to initiate a model that can inform early warning and early action for urban dwellers as a way of cushioning them against food and nutrition insecurity.
The project aims to improve urban early action by improving the alignment of local and municipal governance institutions and response agencies towards analyzing early warning information and implementing early response based on these, supported by an increase in the allocation of financial support from key donors towards urban Early Warning Early Action. The ultimate change envisaged by this program is a reduction in the impact of crises on the urban poor including a decrease in the number of people resorting to negative coping mechanisms. A fundamental hypothesis of this project is that investment in Early Warning Early Action (EWEA) in urban settings will create a significant cost saving to the state as has been proven to be the case in the rural context. It must be noted that in Kenya, we are at a critical juncture as the newly devolved County Governments take root. The level of access to Government structures has significantly increased as well as the accountability of these structures to the people they serve and therefore the timing of this intervention is very opportune. By impact evaluations post early action, the project will build an evidence base that demonstrates that Early Warning Early Action (EWEA) is effective and provides value for money from the perspective of the duty bearer.
Below is a summary of the project details: –
Total duration
15th November 2015 to 31st March, 2018
Objectives of the action
Enable the newly devolved County Government of Nairobi, with support from relevant stakeholders, to effectively coordinate an early response among government, non-government, UN and Kenya Red Cross Society (KRCS).
Action research which will generate evidence that urban EWEA makes sense both from a humanitarian and financial stand-point, and will be used to influence financial allocation decisions of key donors.
Consortium Partners
Concern Worldwide – Consortium Lead
Oxfam GB
Kenya Red Cross Society
Project outcomes and outputs
Facilitate the set-up of a coordinated urban early action mechanism within the Nairobi County Government with agreed actions.
Output 1.1 – Set up coordinated multi-stakeholder urban early warning, early action platform.
Strengthen the capacity of 6 Nairobi Sub-Counties and 1 Informal Settlement Community to mitigate and respond quickly to the impacts of slow onset emergencies.
Output 2.1 – On a phased basis, roll out the Integrated Management of Acute Malnutrition (IMAM Surge) Capacity Model[1] (developed and tested by Concern in Marsabit County) to Health facilities in the urban informal settlements of Nairobi.
Output 2.2 – In the event of indications of an escalation of an urban crisis in the Nairobi informal settlements in the duration of this project, and based on early warning information generated by the Indicator Development for Surveillance of Urban Emergencies (IDSUE) project, conduct appropriate early action activities.
Output 2.3 – After any early action activities, conduct evaluation research to understand and document the impact of those actions, including a cost-benefit analysis of early action compared against late or no action.
Output 2.4 – As indicators in the surveillance system deteriorate communities, via established community conversation groups, will be informed for their own preparedness measures.
Routine Surveillance in Urban Informal Settlements in Kenya is fully coordinated and managed by the County Disaster Management Committee (CDMC) by the end of 2017.
Output 3.1 – Routine Surveillance in at least two key informal settlements in Nairobi is continued without direct technical or administrative involvement from Concern post 2016.
Advocacy for Early Action in urban settings.
Output 4.1 – Collate evidence for the human value and cost effectiveness of Early Warning Early Action (EWEA) in urban informal settlements into an advocacy and learning document spanning the concept that urban emergencies are different through the identification of indicators unique to the urban context, negotiating of thresholds and finally their operationalization should an urban emergency occur.
Output 4.2 – Conduct information dissemination workshops with key target groups at both a national and international level to build consensus on the relevance and need for Early Warning Early Action (EWEA) in urban settings.
Output 4.3 – Advocate at key international conferences (World Humanitarian Summit, Habitat III, and Global Platform for Disaster Risk Reduction 2017) based on the learning.
PURPOSE OF THE ASSIGNMENT
The purpose of this assignment is to generate learning on how urban appropriate metrics can inform early warning and early action decisions to save lives. The study shall also identify identify/or formulate lessons and best practices to be shared nationally, regionally and globally. The analysis should also recommend the best format and venue for dissemination of the lessons. The Consultant shall compare the conventional Integrated Food Security Phase Classification (IPC) method and sensitivity of indicators to the Urban Early Warning Early Action (UEWEA) indicators with an aim of documenting how the latter is revolutionizing the understanding and measurement of urban vulnerability. The comparison will also include implications of other incrementally significant approaches to measuring vulnerability. Delayed response to emergencies has often been considered very expensive and could result to loss of lives. Effective early warning mechanism has been associated with reduced costs of preparedness and response. However, there exists challenge of measuring vulnerability in a mixed population where the wealth differentiation and population intensity easily masks the extent of vulnerability for the urban population. As result, addressing food and nutrition insecurity has not been given the weight it deserves in the urban areas.
The ultimate change envisaged by this programme is a reduction in the impact of crises on the urban poor including a decrease in the number of people resorting to negative coping mechanisms. A fundamental hypothesis of this project is that investment in EWEA in urban settings will create a significant cost saving to the state as has been proven to be the case in the rural context. It must be noted that in Kenya, we are at a critical juncture as the devolved County Governments take root. The level of access to Government structures has significantly increased as well as the accountability of these structures to the people they serve and therefore the timing of this intervention is very opportune.
The key objectives of the evaluation will be as follows;
To document the process of generating information appropriate in measuring urban vulnerability and how that is or can be used to inform decision making process at the various level of the government.
Document key lessons learnt so far related to the design and implementation of urban appropriate metrics that inform early warning and action in slow onset urban emergencies.
Drawing and making technical recommendations on how to adopt a best-practice approach for measuring UEWEA mechanisms
EVALUATION SCOPE
The whole program evaluation process will take approximately 30 days to be completed, this will include: time in the field with consortium partners and beneficiaries, and report writing. The expected date of the start of program evaluation is 1st February 2018 and the final evaluation report will be produced on 26th February 2018. However, the team will be flexible to discuss with the successful consultant the breadth of the scope highlighted herein section 4.
The data collection will be conducted both at the county level and informal settlements. The former will include key informant interviews with county government officials such as CDMC officers and latter will focus on target beneficiaries living in Mukuru, Korogocho and Kibera slums. In addition, focus group discussions, interviews will be done with the consortium partners. Moreover, the evaluation will integrate gender, Violence Against Women and Girls (VAWG) and inclusion of vulnerable groups, as a cross-cutting concern.
EVALUATION CRITERIA AND QUESTIONS
KEY EVALUATION QUESTIONS
Relevance and Appropriateness
To what extent does the project address the identified need?
How well does the project align with government and agency priorities?
Was the design appropriate for the geographical areas?
Was the intervention logic coherent and accurate?
Were recommendations from previous projects and evaluations incorporated in the design?
Were the objectives, indicators and targets formulated relevant and realistic?
Effectiveness
How effective was project delivery? What delivery mechanisms worked well and what did not work? What are the key lessons regarding implementation?
To what extent did the project contribute to greater preparedness and response among local organizations, communities and governments?
To what extent and in what ways has the project led to improved knowledge and understanding[2] of best practices relating to disaster and emergency preparedness and response?
In what ways has the project influenced institutional and policy environments?
How has evidence been used and shared by the project?
Efficiency
Was the intervention well-coordinated with and complementary to the work of other major partners including the government, private sector and non-governmental organizations working on early warning systems.
Has the project changed the lives of the beneficiaries in any meaningful way?
Are there any exceptional experiences that should be highlighted e.g. case-studies, best practice?
Were there any capacity gaps within the consortium partners and local implementing partners and how were they dealt with during project implementation?
Were there internal or external factors outside the project implementers’ control that affected the achievement or non-achievement of results? (e.g. government policies, funding levels etc.)
Have programme funds and activities been delivered in a timely manner?
Sustainability
To what extent can the outputs be expected to be sustainable over the longer term?
What characteristics make the outputs sustainable or unsustainable?
To what extent is the county government harnessing information generated and lessons learnt to inform the implementation of activities at county level?
Value for Money
Were the inputs procured cost effective?
How was the outcome/output efficient in comparison to the value of the projects inputs?
How did the project ensure equity of resources to the most vulnerable and marginalized groups among the target beneficiaries?
METHODOLOGY
The evaluation will use a mixed methods approach. The evaluation team will determine the survey methodology and key research questions and draw up a detailed evaluation plan with timelines which will be discussed and agreed with the consortium secretariat team and Consortium partners.
To measure results the evaluation will draw on the existing structure of monitoring and other program data available through the consortium e.g. data on Indicator Development for Surveillance of Urban Emergencies (IDSUE) to be found on this link: https://www.dropbox.com/sh/qha14yoq7jen1hr/AABfUqcJcuCypdhe2au9hBNTa?dl=0 , project reports and Nairobi County Disaster and Emergency Management Act 2015
This data will be complemented and triangulated through primary data collection at National, County and community levels. To complement quantitative information Key Informant Interviews and Focus Group Discussions will be undertaken.
MAIN DELIVERABLES AND WORK PLAN
The evaluation will consist of 5 phases:
Recruitment of evaluation team, preparatory visit to the consortium to discuss TOR, design and agree on methodology and to draw up a detailed work and evaluation plan; initial briefing with consortium secretariat team and the Consortium lead team to ensure that the evaluation team is clear on the principle proposition for this evaluation exercise.
Preparatory desk review: drafting evaluation matrix with evaluation questions, indicators, data requirements and sources; secondary data and literature review 5 days
Main evaluation phase: design of data collection tools, possible pretesting of tools, training of data clerks, planning of field visits and discussions/interviews with consortium partners and implementing partners’ staff (local NGOs) and other stakeholders including Government line departments, other local and international humanitarian actors in the areas of operation to get their feedback to reach conclusions against benchmarks; conduct field visits to collect data through a combination of data collection methods including PRA methodologies. It is expected that the team will use gendered lenses and participatory approaches to seek the view of beneficiaries and, where appropriate, non-beneficiaries. – 15 days
Reporting: Analysis of data (data should be collected and analyzed disaggregated by gender and data sets), production of a draft report and discussion of this report with consortium partners to give opportunities for the team/s to agree on action points, learning and recommendations; submission of final report with at least 2 appropriate and quality case studies or success stories focusing on best practices and lesson learnt as evidence and presentation of findings and recommendations. – 5 days
Follow up: Consortiums’ follow up to evaluation findings and dissemination of final report to donors and partners – 5 days
The expected deliverables from the evaluation exercise are the following:
Complete bibliography of documents/materials/data used during desk review of secondary sources;
Power point presentation of evaluation plan, timelines and activities;
Final data collection tools, data bases and analysis plan;
First draft of evaluation report;
Final evaluation report;
Power point presentation of main findings and conclusions for debriefing purposes; and
All data collection questionnaires, hard copies of filled in questionnaires, clean data set and analysis files.
Evaluation report
The production of the evaluation report will be the liability of the consultant covering all the aspects as outlined in the ToR. Consortium partners’ staff and management will be responsible for coordinating the evaluation exercise. The evaluation report should be:
Produced in English language and should be simple in expression and easy to understand.
Maximum of 25 pages with some short annexes.
The report format and text, should be an A4 paper size and a legible font (e.g. Times New Roman 11 or 12, Arial 10 or 11.
The evaluation team will be liable to submit at least 02 hard copies and 01 electronic copy of the evaluation report by the agreed deadline.
Report Format
An evaluation report should contain the different elements mentioned below. All parts should be clearly distinguished from each other and of sufficient quality.
Cover page
Table of contents
An executive summary that can be used as a document in its own right. It should include the major findings of the evaluation and summarise conclusions and recommendations.
The objectives of the evaluation
The main questions and derived sub-questions.
A justification of the methods and techniques used (including relevant underlying values and assumptions, theories) with a justification of the selections made (of persons interviewed, villages or activity sites visited).
Eventual limitations of the evaluation.
A presentation of the findings and the analysis thereof (including unexpected, relevant findings). All research questions should be addressed, paying attention to gender issues
Conclusions, which will analyse the various research questions. Conclusions must be derived from findings and analysis thereof.
Lesson learned and recommendations should be clearly related to conclusions but presented separately. Recommendations should be practical and if necessary divided up for various actors or stakeholders and include guidelines of how they can be implemented.
Report annexes that include: data collection tools and schedule, sampling procedures and frame and sampled villages, and list of survey team members.
The reporting style should be clear and accessible. References to sources used, such as interviews, literature, reports, must be given.
Confidentiality of information: all documents and data collected will be treated as confidential and used solely to facilitate analysis. Interviewees will not be quoted in the reports without their permission.
Consultant Qualifications
The desired specification and qualities of a consultant shall be:
Post Graduate qualification in relevant fields of study.
Proven background in evaluating projects implemented in consortium and in depth knowledge of project cycle management for disaster and early warning emergency programmes.
Strong interpersonal and analytical skills.
Familiarity with the different cultures in Kenya and ability to speak and understand Kenyan languages preferably Swahili language;
Experience of effective interaction with local and national organizations, government departments, and international humanitarian actors.
Conversant with application of cross cutting themes like gender mainstreaming, and gender competence, M&E, Advocacy and child protection to programming. Good spoken and written communication skills in English.Reporting Lines
The consultant shall work under the supervision of the Director of Humanitarian programs with strong liaison with Oxfam’s Consortium Technical Manager and partner management at field level.
[1] The model aims to: Strengthen the capacity of government health systems to effectively manage increased caseloads e.g. of severe acute malnutrition (and moderate acute malnutrition, where appropriate) during predictable emergencies without undermining on going systems strengthening efforts.
[2] Of project beneficiaries (ex. Local organizations, community members, governments, humanitarian staff)
