4.3 Design M&E Plan

← previous next →
Download this chapter

Summary

Forecast-based Financing (FbF) is a relatively new concept with potential to reduce disaster impacts through increased use of available science to inform decision-making. Since FbF is in its early stages of implementation, monitoring, evaluation, and learning are crucial to measure the effectiveness of the approach and to adjust where necessary.

As building evidence and adjusting the system where needed is so important in these early stages of FbF, each activation of an Early Action Protocol should be used to collect data to that effect and document learnings.

Thus every EAP should include an M&E protocol to 1) assess the impact of the early actions and the extreme event after each activation and 2) ensure that all procedures were carried out as planned and generate evidence that early actions were taken. This M&E system should be harmonized with the existing NS PMER system. If a good and well thought out M&E Plan for the EAP is in place, data collection upon activation will be greatly facilitated (see Chapter Activate, Monitor, Evaluate for steps to be taken following the activation). For EAPs submitted to the FbA by the DREF, it is required that an M&E plan is in place, detailing how activation will be monitored and impact will be assessed.

M&E allows you to measure, manage and track progress against plans and achievement of desired results. It is a prerequisite for FbF project learning, answering, “where and how can we do better?”, as well as integral to demonstrating accountability and results.

Ultimately, M&E enables us to show the communities we work for and the organizations that fund us how FbF is making a difference.

This chapter outlines how to monitor EAP activations and provide evidence whether these goals are being reached. Key questions this chapter addresses:

  • How to set up a M&E plan?
  • What type of evidence we are looking for?
  • How do we best measure impact for FbF?
  • What tools are available?
  • How do we best capture FbF learnings for improved delivery?

Although components of all FbF M&E systems vary from country to country and context, the following steps to guide your development of the M&E plan for your EAP will remain the same.

M&E commences at the onset of project and programme development, employing various tools to measure, monitor and evaluate progress throughout. Components of M&E set-up and Early Actions selection may be found integrated throughout the preceding chapters of this Manual (See chapter Select Early Actions and Make your National Society FbF ready); however, an overview of necessary steps to develop the M&E plan for your EAP may be found below.

Step 1: Define how EAP activations will be monitored

The project or programme that was established in a given country to set up the FbF system, should have its own monitoring plan, in line with the project logframe. Once, the system is set up, a monitoring plan is also needed for the implementation of the EAP. This plan should cover the period starting from the moment the forecast reaches the pre-identified impact level and the EAP is triggered until the end of the activation. Monitoring tracks the performance of the implementing actors vis-a-vis their agreed and planned roles and responsibilities. The collection of monitoring data related to the performance of the EAP implementation will be important to later analyze whether the NS managed to act as early as planned and with the expected efficiency. The objective is to make sure that practices, behaviors and ways of implementing the EAP are analyzed to identify any shortcomings, constraints or bottlenecks that need to be removed and to identify design problems that must be addressed before the next activation.

Develop an EAP monitoring to clearly define which elements of the activation will be monitored and by whom – you can print the form and give it to NS staff and volunteers who are deployed to the field and then return them to the M&E focal point for consolidation.

You can use the EAP monitoring form template or EAP online monitoring form with examples and adapt it to your EAP.

Note:
You need to create a KoBo Toolbox account to be able to access, copy and adapt this form; accounts are free for humanitarian organisations with unlimited use.

Clemens Gros describes the process of developping an M&E plan:

Step 2: Define how to assess the impact of the EAP activation

The ultimate goal is to determine whether the early actions implemented enabled the expected risk reduction (and effective response) goals (i.e. reduced the prioritized impact) that were planned and how those goals were achieved. Did the actions make a difference in the lives and livelihoods of people at risk? To what extent? Do people in FbF-assisted communities experience less adverse impacts on their lives, health or property than people who were not assisted through FbF? Impacts will be measured on key indicators related to health, well-being, physical assets and productive capacity.

Develop the impact assessment section of your EAP M&E plan, as follows:

  • Define indicators (see examples in Table 2).
  • Define your counterfactual approach.
  • Choose data sources, data collection tools Develop/modify data collection tools.
  • Define if the impact evaluation (and/or cost benefit analysis) will be conducted by the NS itself, by the IFRC or jointly, or via an external contract- e.g. academic institution, consultant etc.).

Assess household /community level impacts.

Investigate whether FbF has contributed to improve humanitarian outcomes. How you measure your impacts will vary according to your interventions.

Tip:
Keep in mind gathering impact data on the success of your FbF interventions will depend on the type of intervention. For example, early actions for shelter are immediately observable directly after the disaster event, while early actions such as the distribution of water purification tablets to prevent a cholera outbreak could take weeks to show full effect.

The following materials can support you in planning your impact assessment.

The choice of indicators depends on the type of hazard, the impacts to be prevented or mitigated and the early actions to be taken.

Indicators to measure the impact of early actions are often taken from the following categories:

Health & well-being

  • Mortality
    • “Did less/no people die because of the disaster, as a consequence of FbF assistance?”
  • Morbidity
    • “Did less people fall ill during/after the disaster, thanks to FbF assistance?”
  • Stress / anxiety
    • “Did people feel less stressed and better able to cope with the impacts of the disaster, thanks to FbF assistance?”

Shelter & housing

  • Household housing structures
    • “Did less people experience severe damages to the roofs and walls of their houses, as a result of received early assistance through FbF?”
  • Communal shelters
    • “Did communal cyclone shelters withstand the disaster impacts and protect community members as planned?”

Assets

  • Personal assets
    • “Did less people experience severe damages to their valuable possessions, as a result of received early assistance through FbF?”
  • Productive assets (livestock, orchards, sheds, etc., for example:
    • “Did people experience fewer livestock deaths and injuries because they received forecast-based early assistance?”

Factors impacting health, well-being, livelihoods, and others

  • Food / water supply
    • “Did people who received FbF cash assistance before the disaster suffer from less food insecurity during the disaster?”
  • Labour constraints
    • “Did forecast-based actions help to reduce the time that people were unable to work due to the disaster impacts?”
  • Public infrastructure (roads, clinics, schools, etc., for example:
    • “Were community health centres better able to provide medical care to affected vulnerable people, thanks to FbF assistance?”
  • There are many other possible measures, depending on the programme/project theory of change, logframe and M&E plan.

Counterfactual approach

For example, “did households who were assisted through FbF experience fewer disaster impacts than households who did not receive this type of early help?”

How can we say with certainty that it was FbF assistance that led to the achievement of positive results, such as reduced suffering and fewer disaster impacts, rather than other interventions or external factors?

The use of counterfactuals has become an accepted and widely-used approach to causal inference in social science research.
In the context of FbF, a counterfactual is employed to answer a question such as: “What would have happened if the community hadn’t received assistance through forecast-based actions?”. The impact of FbF is estimated by comparing counterfactual outcomes (what would have happened without FbF) to those observed under the intervention (what happened with FbF assistance).

The challenge is that the counterfactual cannot be observed directly.
They must be approximated with reference to a comparison group that resembles the conditions of the counterfactual as closely as possible.

In practice, FbF teams will usually aim to use one of two types of comparisons (or both) to estimate the counterfactual:

Historical impact data from the same or comparable communities/areas that have been affected by a comparable disaster in the past.

+ Opportunities:

Historical data can be cheaper to obtain because they have been collected by someone else in the past. Since people have lived through the past disaster, historical data also provides a common reference point that may yield additional credibility to the analysis. (Check the risk analysis conducted at the beginning of the EAP development in case information is relevant)

– Challenges:

The comparability of historical data is often problematic on several levels: the past disaster must be comparable to the disaster that triggered FbF actions in magnitude and timing; its impacts on the vulnerable and exposed population must have been similar. The data about the disaster and its impacts must be available for the same units of analysis, and the same level of disaggregation, which are used to assess the current (FbF-triggering) disaster and its impacts, and to analyse the effects of FbF.

Example:

If one of the primary indicators to measure the success of forecast-based actions is the reduction in the proportion of people suffering from diarrheal diseases during/after a disaster, the historical data must contain information on the incidence of diarrhea among the vulnerable and affected population group during/after the past disaster event. It will not suffice to have data only on the disaster itself or the damages to infrastructure and houses. The historical data must be available for the same geographical area in which the EAP implementation took place.

Impact data from comparison communities or households who have been affected by the same disaster (which triggered forecast-based actions) and who are comparable in every other aspect, except that they did not receive assistance through forecast-based actions before the disaster.

+ Opportunities:

It is more likely to achieve data comparability when a random sample is drawn from the population of affected and vulnerable communities. Given the limited amount of funding and therefore coverage of most FbF interventions, it is likely to find comparison communities that were affected by a disaster but were not reached by assistance through forecast-based actions.

– Challenges:

The sampling frame needs to be designed and implemented carefully to avoid introducing bias into the data. Primary data collection is typically more expensive than working with historical, secondary data sets. Also collecting data in comparison communities can lead to expectations by interviewees that they will receive assistance, as following a disaster, assessments by a National Society are usually done to plan response.

Using a counterfactual is not necessary but strongly recommended given the current stage and funding of FbF projects. Otherwise, the analysis cannot show a causal relationship between the intervention and outcomes convincingly.

It is unlikely in the case of FbF, there are situations where non-experimental approaches (without a comparison group) are the only feasible research design. For example, when a programme/project is implemented universally and every exposed and vulnerable person is being reached, there are no more isolated comparison groups. Unfortunately, FbF programmes – with their limited amount of funding – are far from this scenario. Therefore, the use of experimental or quasi-experimental assessment designs with comparison groups to assess the impact of FbF projects/programs  is strongly recommended.

M&E Tasks and Tools

Review the availability of reliable secondary data sources

  • IFRC M&E Guide on assessing the availability of secondary data (section 2.2.2, p. 33)

Identify a comparison group

Step 3: Define responsibilities and timeframes
  • Identify potential collaborators, universities, consultants etc. for the data collection.
  • Develop volunteer/enumerator training materials in advance (e.g. on data collection, conducting of focus group discussions etc.), where possible and relevant. Consider in this step already existing plans under the National Society PMER plans/strategy.

© GRC

Step 4: Summarize in the EAP M&E plan

Based on the prioritized impacts and early actions that were selected, M&E capacity of the National Society and goals of the FbF system, and following the preceding steps, summarize  in an M&E plan what indicators will be used to measure early action impact, how and when the required data will be collected, and who will collect it, analyze it, report and disseminate.

Based on your M&E plan, develop the data collection tools (monitoring forms, questionnaires, checklists, etc.) and set up the processes for data collection.

M&E Tasks and Tools

FbF M&E Plan  

Theory of Change (per action), from EAP (Table 1)

Areas of focus (sector) Risk Action Outputs Short-term outcomes Long-term outcomes Available capacities to implement action
Health X% of the population in the target areas affected by a diarrheal disease outbreak during/after a flood event. Distribute chlorine containers at HH level. Population in flood-prone area equipped with chlorine for water purification, to ensure clean water access during/after flood event. Proportion of population in target area suffering from diarrheal diseases is reduced. A healthy population is able to attend school and work more regularly given the reduced impact of disasters. Local suppliers of water treatment tablets | Volunteer network of more than X volunteers per branch.

The IFRC seven areas of focus are: DRR, Shelter, livelihoods and basic needs, health, WASH, Protection Gender and Inclusion, Migration.

M&E Plan (Table 2)

Here you can download a hypothetical example of a M&E Plan.

Step 5: Consolidate M&E plans from different EAPs in an M&E master plan

If a National Society is developing more than on EAP, once the early actions that will be part of the EAP are selected, it is recommended to consolidate all the early actions and respective indicators for different EAPs in one M&E master plan and review for consistency. This also ensures that synergies can be detected and that there will be no duplication in data collection.

The M&E plan(s) should be updated after each activation to integrate any new learnings, and whenever the early actions are changed.

Step 6: Adapt/review the EAP monitoring form
  • Ensure volunteers and staff know their roles and responsibility for the performance of the monitoring process to be carried out during the EAP implementation.
  • Use the EAP monitoring form for the triggered EAP to monitor implementation.

Tool: EAP monitoring form template