Forecast-based Financing (FbF) is a relatively new concept with potential to reduce disaster impacts through increased use of available science to inform decision-making. Since FbF is in its early stages of implementation, monitoring, evaluation, and learning are crucial to measure the effectiveness of the approach and to adjust where necessary.
As building evidence and adjusting the system where needed is so important in these early stages of FbF, each activation of an Early Action Protocol should be used to collect data to that effect and document learnings.
Thus every EAP should include an M&E protocol to 1) assess the impact of the early actions and the extreme event after each activation and 2) ensure that all procedures were carried out as planned and generate evidence that early actions were taken. This M&E system should be harmonized with the existing NS PMER system. If a good and well thought out M&E Plan for the EAP is in place, data collection upon activation will be greatly facilitated (see Chapter Activate, Monitor, Evaluate for steps to be taken following the activation). For EAPs submitted to the FbA by the DREF, it is required that an M&E plan is in place, detailing how activation will be monitored and impact will be assessed.
M&E allows you to measure, manage and track progress against plans and achievement of desired results. It is a prerequisite for FbF project learning, answering, “where and how can we do better?”, as well as integral to demonstrating accountability and results.
Ultimately, M&E enables us to show the communities we work for and the organizations that fund us how FbF is making a difference.
This chapter outlines how to monitor EAP activations and provide evidence whether these goals are being reached. Key questions this chapter addresses:
- How to set up a M&E plan?
- What type of evidence we are looking for?
- How do we best measure impact for FbF?
- What tools are available?
- How do we best capture FbF learnings for improved delivery?
Although components of all FbF M&E systems vary from country to country and context, the following steps to guide your development of the M&E plan for your EAP will remain the same.
M&E commences at the onset of project and programme development, employing various tools to measure, monitor and evaluate progress throughout. Components of M&E set-up and Early Actions selection may be found integrated throughout the preceding chapters of this Manual (See chapter Select Early Actions and Make your National Society FbF ready); however, an overview of necessary steps to develop the M&E plan for your EAP may be found below.
Step 2: Define how to assess the impact of the EAP activation
The ultimate goal is to determine whether the early actions implemented enabled the expected risk reduction (and effective response) goals (i.e. reduced the prioritized impact) that were planned and how those goals were achieved. Did the actions make a difference in the lives and livelihoods of people at risk? To what extent? Do people in FbF-assisted communities experience less adverse impacts on their lives, health or property than people who were not assisted through FbF? Impacts will be measured on key indicators related to health, well-being, physical assets and productive capacity.
Develop the impact assessment section of your EAP M&E plan, as follows:
- Define indicators (see examples in Table 2).
- Define your counterfactual approach.
- Choose data sources, data collection tools Develop/modify data collection tools.
- Define if the impact evaluation (and/or cost benefit analysis) will be conducted by the NS itself, by the IFRC or jointly, or via an external contract- e.g. academic institution, consultant etc.).
Assess household /community level impacts.
Investigate whether FbF has contributed to improve humanitarian outcomes. How you measure your impacts will vary according to your interventions.
Keep in mind gathering impact data on the success of your FbF interventions will depend on the type of intervention. For example, early actions for shelter are immediately observable directly after the disaster event, while early actions such as the distribution of water purification tablets to prevent a cholera outbreak could take weeks to show full effect.
The following materials can support you in planning your impact assessment.
The choice of indicators depends on the type of hazard, the impacts to be prevented or mitigated and the early actions to be taken.
Indicators to measure the impact of early actions are often taken from the following categories:
Health & well-being
- “Did less/no people die because of the disaster, as a consequence of FbF assistance?”
- “Did less people fall ill during/after the disaster, thanks to FbF assistance?”
- Stress / anxiety
- “Did people feel less stressed and better able to cope with the impacts of the disaster, thanks to FbF assistance?”
Shelter & housing
- Household housing structures
- “Did less people experience severe damages to the roofs and walls of their houses, as a result of received early assistance through FbF?”
- Communal shelters
- “Did communal cyclone shelters withstand the disaster impacts and protect community members as planned?”
- Personal assets
- “Did less people experience severe damages to their valuable possessions, as a result of received early assistance through FbF?”
- Productive assets (livestock, orchards, sheds, etc., for example:
- “Did people experience fewer livestock deaths and injuries because they received forecast-based early assistance?”
Factors impacting health, well-being, livelihoods, and others
- Food / water supply
- “Did people who received FbF cash assistance before the disaster suffer from less food insecurity during the disaster?”
- Labour constraints
- “Did forecast-based actions help to reduce the time that people were unable to work due to the disaster impacts?”
- Public infrastructure (roads, clinics, schools, etc., for example:
- “Were community health centres better able to provide medical care to affected vulnerable people, thanks to FbF assistance?”
- There are many other possible measures, depending on the programme/project theory of change, logframe and M&E plan.
For example, “did households who were assisted through FbF experience fewer disaster impacts than households who did not receive this type of early help?”
How can we say with certainty that it was FbF assistance that led to the achievement of positive results, such as reduced suffering and fewer disaster impacts, rather than other interventions or external factors?
The use of counterfactuals has become an accepted and widely-used approach to causal inference in social science research.
In the context of FbF, a counterfactual is employed to answer a question such as: “What would have happened if the community hadn’t received assistance through forecast-based actions?”. The impact of FbF is estimated by comparing counterfactual outcomes (what would have happened without FbF) to those observed under the intervention (what happened with FbF assistance).
The challenge is that the counterfactual cannot be observed directly.
They must be approximated with reference to a comparison group that resembles the conditions of the counterfactual as closely as possible.
In practice, FbF teams will usually aim to use one of two types of comparisons (or both) to estimate the counterfactual:
Historical impact data from the same or comparable communities/areas that have been affected by a comparable disaster in the past.
Historical data can be cheaper to obtain because they have been collected by someone else in the past. Since people have lived through the past disaster, historical data also provides a common reference point that may yield additional credibility to the analysis. (Check the risk analysis conducted at the beginning of the EAP development in case information is relevant)
The comparability of historical data is often problematic on several levels: the past disaster must be comparable to the disaster that triggered FbF actions in magnitude and timing; its impacts on the vulnerable and exposed population must have been similar. The data about the disaster and its impacts must be available for the same units of analysis, and the same level of disaggregation, which are used to assess the current (FbF-triggering) disaster and its impacts, and to analyse the effects of FbF.
If one of the primary indicators to measure the success of forecast-based actions is the reduction in the proportion of people suffering from diarrheal diseases during/after a disaster, the historical data must contain information on the incidence of diarrhea among the vulnerable and affected population group during/after the past disaster event. It will not suffice to have data only on the disaster itself or the damages to infrastructure and houses. The historical data must be available for the same geographical area in which the EAP implementation took place.
Impact data from comparison communities or households who have been affected by the same disaster (which triggered forecast-based actions) and who are comparable in every other aspect, except that they did not receive assistance through forecast-based actions before the disaster.
It is more likely to achieve data comparability when a random sample is drawn from the population of affected and vulnerable communities. Given the limited amount of funding and therefore coverage of most FbF interventions, it is likely to find comparison communities that were affected by a disaster but were not reached by assistance through forecast-based actions.
The sampling frame needs to be designed and implemented carefully to avoid introducing bias into the data. Primary data collection is typically more expensive than working with historical, secondary data sets. Also collecting data in comparison communities can lead to expectations by interviewees that they will receive assistance, as following a disaster, assessments by a National Society are usually done to plan response.
Using a counterfactual is not necessary but strongly recommended given the current stage and funding of FbF projects. Otherwise, the analysis cannot show a causal relationship between the intervention and outcomes convincingly.
It is unlikely in the case of FbF, there are situations where non-experimental approaches (without a comparison group) are the only feasible research design. For example, when a programme/project is implemented universally and every exposed and vulnerable person is being reached, there are no more isolated comparison groups. Unfortunately, FbF programmes – with their limited amount of funding – are far from this scenario. Therefore, the use of experimental or quasi-experimental assessment designs with comparison groups to assess the impact of FbF projects/programs is strongly recommended.
Step 4: Summarize in the EAP M&E plan
M&E Tasks and Tools
Review the availability of reliable secondary data sources
- IFRC M&E Guide on assessing the availability of secondary data (section 2.2.2, p. 33)
Identify a comparison group
Based on the prioritized impacts and early actions that were selected, M&E capacity of the National Society and goals of the FbF system, and following the preceding steps, summarize in an M&E plan what indicators will be used to measure early action impact, how and when the required data will be collected, and who will collect it, analyze it, report and disseminate.
Based on your M&E plan, develop the data collection tools (monitoring forms, questionnaires, checklists, etc.) and set up the processes for data collection.
M&E Tasks and Tools
FbF M&E Plan
Theory of Change (per action), from EAP (Table 1)
|Areas of focus (sector)
||Available capacities to implement action
||X% of the population in the target areas affected by a diarrheal disease outbreak during/after a flood event.
||Distribute chlorine containers at HH level.
||Population in flood-prone area equipped with chlorine for water purification, to ensure clean water access during/after flood event.
||Proportion of population in target area suffering from diarrheal diseases is reduced.
||A healthy population is able to attend school and work more regularly given the reduced impact of disasters.
||Local suppliers of water treatment tablets | Volunteer network of more than X volunteers per branch.
The IFRC seven areas of focus are: DRR, Shelter, livelihoods and basic needs, health, WASH, Protection Gender and Inclusion, Migration.