C4D Hub: Create a questions-led M&E framework

Participatory

Holistic

Critical

Realistic

Learning-based

Accountable

Complex

Steps:

Step 1. Recommended preparation tasks: a checklist

Step 3: Download a matrix template to fill in as you make decisions:

Step 4: Sort Questions by Type

Step 5: Decide how to answer descriptive questions and compile a matrix

Step 6: Decide how to answer causal questions and compile a matrix

Step 7: Decide how to answer evaluative questions and compile a matrix

Step 8: Decide how to answer Action/ Predictive Questions and compile a matrix

Step 9: Develop a summary evaluation matrix with all planned data collection and analysis, including use of existing data

Step 10. List required tasks, studies, events, processes

Participatory:

Complex:

Examples

Resources

A questions-led M&E Framework starts with thinking about the information needs (questions) of the primary intended users, and builds a plan for answering those questions.

This is a good method for C4D and is consistent with the C4D Evaluation Framework in the following ways:

The potential uses that stakeholders, especially the primary intended users, have are the focus of the M&E. These stakeholders and users should be involved in deciding on the purpose and questions, and selecting options for answering questions.

The key M&E questions drive the direction of the framework. These questions should go beyond 'what happened' and also question the causes, how good programs and results are, and what to do next.

A questions-led M&E Framework encourages mixed methods to build a rich understanding of what is working, and what is not working, for different groups.

A questions-led M&E Framework prioritises efforts around the questions that matter most to users. It does not try to measure everything. If primary intended users want to know about impact of C4D initiatives, that implies certain types of strategies, and should be planned for as part of the M&E Framework. If there are lots of uncertainties about what might work, an M&E Framework can be built to allow for trialling and comparison of different strategies that are investigated through smaller studies and inform an emergent approach.

A questions-led M&E Framework takes learning from RM&E seriously, beyond a list of recommendations at the end. If key users priorities understanding how to make improvements during implementation, this implies certain strategies. Further, learning structures, events and processes (such as committees, annual reviews etc.) can be built into the M&E Framework.

A questions-led M&E Framework supports a true accountability, beyond compliance-oriented reporting against indicators, through building a rigourous, mixed-methods M&E Framework that can be designed to answer questions about effectiveness, impact, relevant and other quality standard criteria.

A questions-led M&E Framework is much easier to design around complicated and complex types of C4D initiatives and problems. Depending on the framing of key questions, a Questions-Led M&E Framework can be designed to support emergent and responsive implementation using methods and strategies suited to understanding uncertainty. The focus on questions means it remain realistic, rather than trying to measure every single thing that might possibly be measured.

The M&E Framework should be informed by several other important decisions and tasks. The C4D Evaluation framework approach would suggest consideration of the following aspects as preparation for undertaking this task:

Have you identified and engaged with stakeholders? Will they be involved in developing the M&E Framework or Evaluation Plan?

Have you reviewed aspects of the C4D initiatives that are simple, complicated and complex, and considered the implications? Have you developed a Program Theory that includes possible intended and unintended changes?

Different types of questions require different types of methods and strategies to get answers. The four main types are:

Descriptive
Causal
Evaluative
Predictive and Action

C4D Matrix Template

DOCX

22 KB

Start by sorting all the smaller questions by their type. This means making a new list of all the descriptive questions, all the causal questions, all the evaluative questions, and all the action/predictive questions (it is helpful to keep the numbers, i.e. 1.1, 1.2 etc. for resorting according to the Key Question later).
Identify any questions that are the same or similar, and if possible adjust the wording of very similar questions slightly to avoid unnecessary duplication, making sure not to lose the essence of any questions.
Paste the list of questions under each of the headings (Descriptive, Causal, Evaluative, Action/Predictive) in Matrix Template document the space provided.

In your matrix template add all the descriptive questions to the first column:

In the second column make a clear statement about what will be described (e.g. types of/number of communication activities undertaken, or levels of knowledge on a specific topic). A theory of change can be very helpful here. (see here for more on Develop a Theory of Change).

In the third column list any existing or accessible data that could be used to answer that question, and assess their quality and relevance (see Determine and secure resources). There are often statistics available that can be used for C4D indicators. Other existing data that might be useful can come from previous research and evaluation studies, official records and publicly available statistics.

Finally, make selections for additional data collection/retrieval, sampling and analysis, and add these to the matrix. More information on options for these is below:

The matrix for answering causal questions is slightly different. Often a matrix to answer causal questions will refer to descriptive data and will use analysis strategies that investigate causal relationships between variables.

Causal relationship	Comments	Strategy 1: Scope for a credible counterfactual?	Strategy 2: Scope for checking consistency of evidence?	Strategy 3: Scope for ruling out other alternative explanations?
Variable 1	Variable 2

First we need to identify the variables. Looking at each of your causal questions try to identify what the variables are. A very simple example might be:

Variable 1: Exposure to communication materials

Variable 2: Level of understanding of a specific topic

In a question about bottleneck and barriers, Variable 1 might be 'the presence of a barrier' and variable 2 the intermediate outcome. Your theory of change can be useful for clarifying variables (see Develop program theory or logic model).

Use the comments column to note any important information e.g. the treatment of groups of variables, or use of answers from descriptive questions.

There are three main strategies for answering questions about the causal relationships between variables.

Review these strategies, note whether or not a credible counterfactual will be feasible; and the list selected strategies for checking the consistency of evidence and for ruling out alternative explanations. It is recommended that you include multiple strategies of different kinds.

National program for child protection communication M&E plan (page 20-22): Matrix for answering causal questions as filled in by Vietnam CO and their counterparts with variables identified

Watch a webinar on Answering causal questions and investigating C4D contributions . Use the password evaluatingC4D

The matrix for answering evaluative questions needs to show the processes you will use to select and apply criteria, standards and weighting. Each evaluative question in your list might need its own processes, or a group of evaluative questions might be answered using the same processes.

What will be evaluated	Criteria	Standards	Synthesis/Weighting	Process for developing agreed standards, criteria and synthesis

Begin by making a statement about what will be evaluated (that is, what will be judged or valued). This might be particular activities, particular C4D approaches, particular sites, or particular outcomes.

To judge and value something we can apply criteria, standards and then we would synthesise and weight those to come to conclusions.

Once you have made your selections, add these to the matrix and describe the processes to be used.

Answering action questions in a credible way often requires a process of identifying and assessing options for action. It is often useful to have a wider group of people involved in this process than simply an external evaluation team.

Action/Predictive Questions (AQ)	Process and participants for answering Action/Predictive Questions
AQ x.x
AQ x.x
AQ x.x

Begin by listing the Action/Predictive questions in the first column.

The material on how to generalise findings and decide on actions indicates some methods for answering action/predictive questions

Once you have made your selections add these to the second column. Note: you may use the same process to answer all questions and in these cases you may simplify the matrix to indicate this.

The next step is to compile a matrix that summarises how you will answer each of the Key Questions and associated smaller questions. This is intended as a summary table; in most cases the more detailed matrixes for answering descriptive, causal, evaluative and action questions will remain in the final document.

KQ	Data source / method / analysis 1	Data source / method / analysis 2	Data source / method / analysis 3	Data source / method / analysis 4
1 [add KQs]
1.1 [add sub questions]
1.2
1.3
1.4
2.
2.1
2.2
3.
3.1
3.2
3.3
4.
4.1
4.2

Add the Key Questions into the shaded rows, the associated sub or smaller questions underneath. You may need to add or remove rows. Add short descriptions of the data source or method in the corresponding boxes. Where possible, make note of timing, (i.e. baseline + every six months; baseline, midline, endline; ad hoc or as triggered etc.)

You can either rename the column headings (e.g. Existing data; Data Collection and Analysis methods; Causal Analysis methods; Stakeholder workshops), which makes it easy to see all the additional data collection in one column; or you could leave the headings as listed and fill in from left to right in the corresponding rows. This makes sense where there are a high number of different methods being used.

Descriptive Question (DQ)

What will be described

Existing data

Additional data collection/ retrieval

Sampling/ disaggregation (equity)

The final step is to extract a list of the tasks, studies, events and processes that are outlined in the matrix, and the associated methods (e.g. baseline studies, bottleneck analysis studies, evaluations, workshops, expert analysis or review processes etc). This list will later be used as the basis of a cost estimate and a workplan.

If who is doing the evaluation had not yet been decided, decide who will conduct the evaluation