Qualitative comparative analysis

Qualitative Comparative Analysis (QCA) is an evaluation approach that supports causal reasoning by examining how different conditions contribute to an outcome. It explores whether an outcome of interest is present across various configurations of conditions.

Overview

QCA is a theory-driven approach, meaning the choice of conditions to examine needs to be driven by an initial theory of change about what aspects of the intervention and context matter. The list of conditions can be revised during the QCA process if the analysis shows certain configurations as being associated with a mixture of outcomes.

When determining whether a condition is present or absent, it’s important to have a clear and precise definition of that condition, including specifying the circumstances when and where this condition can be considered present. This allows for consistent and accurate coding of the presence or absence of the condition in different cases.

Data coding and expression of results

Data for a QCA are collated in a matrix form (sometimes called a ‘QCA matrix’ or a ‘truth table’), where rows represent cases, columns represent conditions, and the rightmost column indicates the presence or absence of the outcome for each case.

For each case, the presence or absence of a condition is recorded numerically. There are different ways of coding this data. The most simple form uses a binary code to indicate the presence or absence of a condition, using (1) or (0), respectively. This is used for clear-cut, yes/no data, such as the presence or absence of a policy or funding. Some forms of QCA may use fuzzy sets to show a condition's partial presence or absence, presenting this as values between 0 and 1, such as, (0.2) or (0.6) to indicate the degree in which a condition exists, such as the level of stakeholder engagement. In cases where the condition can be present in more than one way, this can be represented by values of (0), (1), (2) or more – referred to as "multi-value". This can be used in situations where a condition has multiple distinct states, for example, different fields of study, such as Economics, Arts, Law, etc.

The results of a QCA are often expressed in ordinary language, for example, “A combination of Condition A and condition B or a combination of condition C and condition D will lead to outcome E.” Results can also be expressed using Boolean algebra for brevity – for example, the previous statement can be written in Boolean notation as: “A*B + C*D→E”.

The four phases in a QCA

1. Identify relevant cases and causal conditions
2. Construct the truth table and resolve contradictions
3. Analyse the truth table
4. Evaluate the Results

Software packages such as fsQCA can assist with these kinds of analyses.

QCA begins by documenting different configurations of conditions, or aspects of an intervention and context, that are associated with each case of an observed outcome, incorporating substantive contextual knowledge and hypotheses.

These configurations are then simplified through a minimisation procedure that identifies the simplest set of conditions that account for all the cases of observed outcomes and their absence, helping to determine key contributing factors.

QCA is able to use relatively small and simple data sets. There is no requirement to have enough cases to achieve statistical significance, although ideally there should be enough cases to potentially exhibit all the possible configurations. The latter depends on the number of conditions present. In a survey of QCA uses (Mello, 2012), the median number of cases was 22 and the median number of conditions was 6.

Key characteristics and concepts of QCA

QCA results are able to distinguish various complex forms of causation, including:

• Configurations of causal conditions, not just single cases. Truth tables in QCA are able to show how different causal configurations are made up of different conditions.
• Equifinality (‘same final state’) refers to a situation in which the same outcome can be achieved through different starting conditions and in different ways. In other words, there are different causal pathways that can lead to the same outcome or result. The QCA approach allows exploration of these different pathways through the analysis of different combinations of conditions in which the outcome of interest is present
• Multifinality (‘multiple final states’): Where the same cause can lead to different outcomes in different contexts The QCA approach allows exploration of these different pathways through the analysis of different outcomes arising from a particular starting condition.
• Necessary and/or sufficient causal conditions (or both or neither). A condition is necessary if it must be present for an outcome to occur. A condition is sufficient if its presence ensures the outcome. A condition can be both necessary and sufficient if it is required and ensures the outcome, or it can be neither necessary nor sufficient.
• INUS conditions are Insufficient but Necessary parts of a configuration that is Unnecessary but Sufficient. In other words, the condition must be combined with other conditions to produce the outcome, and there can be other combinations of conditions that also produce the outcome.
• Asymmetric causes are where the causes of failure may not simply be the absence of the cause of success.
• The relative influence of different individual conditions and causal configurations in a set of cases being examined. Configurations can be evaluated in terms of coverage (the percentage of cases they explain) and consistency (the extent to which a configuration is always associated with a given outcome).

Causal Pathway features

How this approach might be used to incorporate features of a causal pathways perspective

A causal pathways perspective on evaluation focuses on understanding how, why, and under what conditions change happens or has happened. It is used to understand the interconnected chains of causal links that lead to a range of outcomes and impacts. These causal pathways are likely to involve multiple actors, contributing factors, events and actions, not only the activities associated with the program, project or policy being evaluated or its stated objectives.

QCA can be used in ways that incorporate the following features of a causal pathways perspective:

• Addressing power and inclusion: QCA can involve a diversity of stakeholders in the calibration phase, which consists of a process of assigning numerical values to conditions across the cases before constructing the truth table. This step encourages evaluation teams and stakeholders to discuss and explicitly define success, degrees of success or lack of success. This step can involve the development of rubrics and may be strongly values-driven.
• Articulating explicit causal pathways: QCA translates qualitative data, including potential causal factors, into a numerical format that allows systematic analysis of causal patterns, including plural pathways to change. Substantive and theoretical knowledge or additional evaluation data are necessary to identify the underlying mechanism(s), including converting combinations of conditions into pathways.
• Paying attention to a range of outcomes and impacts: QCA pays attention to outcomes that did or didn’t occur, including unintended ones.
• Understanding contextual variation: This is a key feature of QCA as the relative influence of different conditions is the focus of analysis.
• Taking a complexity-appropriate approach to evaluation quality and rigour: QCA processes are both transparent and replicable. QCA involves causal reasoning to assess the meaning of empirically observed cross-case regularity. QCA supports transferability by developing mid-range theories of the complex causality of social phenomena.

Background

History of this approach

Although QCA was originally developed by Charles Ragin in the late 1980s and has been used in comparative political science research since then, its use has only become more common among evaluators in the last two decades. Articles on its use have appeared in the journals Evaluation and the American Journal of Evaluation.

Example

For a worked example, see Charles Ragin’s What is Qualitative Comparative Analysis (QCA)?, slides 6 to 15 on The bare-bones basics of crisp-set QCA.

[A  summary of the example is presented here]

In his presentation, Ragin provides data on 65 countries and their reactions to austerity measures imposed by the IMF. This has been condensed into a Truth Table (shown below), which shows all possible configurations of four different conditions that were thought to affect countries’ responses: the presence or absence of severe austerity, prior mobilisation, corrupt government, rapid price rises. For example, the first row refers to a situation where there was no prior mobilisation, no severe austerity, no government corruption and no rapid price rise.  

Next to each configuration are data on the outcome associated with that configuration – the numbers of countries experiencing mass protest or not. In the first row, there were no cases matching that configuration.

There are 16 configurations in all, one per row. The rightmost column describes the consistency of each configuration: whether all cases with that configuration have one type of outcome, or a mixed outcome (i.e. some protests and some no protests). Notice that there are also some configurations with no known cases (where consistency is shown as ??).

Ragin’s next step is to improve the consistency of the configurations with mixed consistency. This is done either by rejecting cases within an inconsistent configuration because they are outliers (with exceptional circumstances unlikely to be repeated elsewhere) or by introducing an additional condition (column) that distinguishes between those configurations which did lead to protest and those which did not. In this example, a new condition was introduced that removed the inconsistency, which was described as “not having a repressive regime”.

The next step involves reducing the number of configurations needed to explain all the outcomes, known as minimisation. Because this is a time-consuming process, this is done by an automated algorithm (aka a computer program) This algorithm takes two configurations at a time and examines if they have the same outcome. If so, and if their configurations are only different in respect to one condition this is deemed to not be an important causal factor and the two configurations are collapsed into one. This process of comparisons is continued, looking at all configurations, including newly collapsed ones, until no further reductions are possible.

[Jumping a few more specific steps] The final result from the minimisation of the above truth table is this configuration:

SA*(PR + PM*GC*NR)

The expression indicates that IMF protest erupts when severe austerity (SA) is combined with either (1) rapid price increases (PR) or (2) the combination of prior mobilization (PM), government corruption (GC), and non-repressive regime (NR).

Advice for choosing this QCA

What types of projects and programs would this approach be appropriate for?

QCA is a useful approach for small and intermediate-size research designs with 5 to 50 cases. In this range, there are often too many cases for researchers to keep all the case knowledge “in their heads” but too few cases for most conventional statistical techniques. However QCA can also be used in evaluations with larger numbers of cases. (Hanckel et al., 2021)

QCA is particularly useful for understanding interventions implemented in complex systems, where there are multiple contributing factors leading to variations in outcomes and impacts. It can also be used to systematically compare the outcomes of different combinations of system components and elements of context (‘conditions’) across a series of cases. (Hanckel et al., 2021)

What types of evaluation is QCA appropriate for?

Barbara Befani (2016) in In Pathways to Change: Evaluating development interventions with Qualitative Comparative Analysis (QCA), discussed the evaluation questions that QCA addresses: The broad, overarching question answered by QCA is “what sets of factors are likely to influence an outcome”?

Different procedures within QCA are used to answer three related evaluation questions:

1. What causal factors are needed for the outcome to occur?
2. What causal factors are most effective (alone or combination) for the outcome?
3. What causal factors make the difference for the outcome, under what circumstances?

QCA can be used as part of a broader evaluation design and additional data collection and analysis to test, refine or develop theories of change, including nested theories of change from empirical case data. QCA has the potential to be used for meta-evaluation, syntheses and systematic reviews.

What level and type of resources are required for this approach?

QCA requires technical expertise in qualitative data analysis and the use of QCA software programs. It is difficult to precisely predict how long the analysis will take, as further iterations may be needed to make sense of the data.

How might this approach be part of an effective overall evaluation design?

Rick Davies explained how evaluators can use QCA with Process Tracing to navigate between the avoidance of generalizations and generalizations that are overly inclusive:

"The two methods [process tracing and QCA] are more valuable when used in conjunction. Examination of individual cases can prompt initial ideas about what conditions to examine across all cases when looking for common causal configurations. When they are found their validity as causal explanations can then be tested by using various process tracing methods (e.g. hoop tests and smoking gun tests), which make use of the same concepts of causal configurations and necessary and/or sufficient causes. When such tests fail further examination of individual cases may be needed to generate new conditions that then need to be included in a re-analysis of causal configurations.

The QCA perspective provides us with a useful middle ground … between overly-inclusive generalisations and the complete avoidance of generalisations. By ensuring that there is an on-going dialogue between within-case analysis and between-case analysis we can also avoid evidence-free generalisations and ensure that within-sample generalisations are as strong as possible." (Davies, 2013)

Advice for using QCA effectively

In An Introduction to Applied Data Analysis with Qualitative Comparative Analysis (QCA) Nicolas Legewie (2013) advised that:

"It is always recommended to use one of the available software packages to conduct a QCA. Among the software packages for fuzzy set QCA, fs/QCA(RAGIN & DAVEY, 2009) is the most widely used option. It is a freeware program that allows both crisp and fuzzy set analyses, provides a function to produce an intermediate solution, as well visualizations such as XY plots. As a further plus, it does not require computation commands, but runs with a graphical user interface."

Challenges and potential pitfalls

In Pathways to Change: Evaluating development interventions with Qualitative Comparative Analysis (QCA), Barbara Befani (2016) drew attention to several pitfalls, challenges and limitations:

"… the need for consistently available data across comparable cases; the need for technical skills in the evaluation team; the relative unpredictability of the number of iterations needed to achieve meaningful findings; and finally the need for sense-making of the synthesis output, which can be accomplished in many ways, including drawing on other evaluation approaches like Contribution Analysis, Realist Evaluation and Process Tracing."

Some evaluators question whether QCA on its own can warrant a leap from purely configurational data (the co-occurrence of different factors) to causal information (some configuration of some of the factors cause another), saying that this is nothing more than the well-known fallacy that one can infer causation from correlation. There is more agreement that QCA can be very useful to refine and test underlying but perhaps incomplete knowledge and hypotheses which are themselves already at least partially causal.