Multiple lines and levels of evidence

Multiple lines and levels of evidence (MLLE) is a systematic approach to causal inference that involves bringing together different types of evidence (lines of evidence) and considering the strength of the evidence in terms of different indicators of a causal relationship (levels of evidence).

It was originally developed in environmental impact studies, drawing on approaches from epidemiology, beginning with a paper on ‘Causal inference in environmental impact studies’ (Beyers, 1998), followed by other publications on monitoring ecological impacts (Downes and others, 2002; Norris and others, 2004).

MLLE is an element of the Collaborative Outcome Reporting approach to impact evaluation. More recently the MLLE method has been labelled ‘causal criteria method’ and developed into an Eco Evidence framework which uses a scored weighting approach to draw conclusions.

MLLE involves more than just triangulation of different types of evidence. It involves:

• Lines of evidence – different types of relevant evidence, whether existing or new data.
• Levels of evidence – strength of their support for a causal relationship, not in terms of a hierarchy of evidence but in terms of different indications of a causal relationship, such as temporality (the hypothesised cause is observed to occur before the event) or specificity of association (where the effect is only found in association with the hypothesized cause, or the only one effect is associated with the cause).
• A systematic process for judging the credibility of the evidence and the causal analysis, often involving an expert cross-disciplinary panel review of the process.

The approach to the analysis of the strength of evidence builds on the Bradford Hill causal criteria adapted for ecological applications. The Bradford Hill causal criteria were presented by Sir Austin Bradford Hill, Professor Emeritus of Medical Statistics, as the President’s Address at the Royal Society of Medicine, drawing on the work of the USA Surgeon-General in assessing the health effects of cigarette smoking in the absence of experimental evidence.

Bradford Hill criteria

The following, extracted verbatim from Table 1 in Norris et al (2012, p7), provides a definition of the Bradford Hill criteria:

"The causal criteria defined by Hill (1965) for use in epidemiology. These were developed from criteria originally defined in the US Surgeon General’s report on the health effects of smoking (USDHEW 1964). Subsequent users have tended to concentrate on smaller subsets of the 9 criteria.

• Strength of association ^abc: Increase in disease incidence associated with exposure to the risk factor. Measured as the likelihood (i.e., increased risk) of developing the disease when exposed to the risk factor.
• Consistency of association ^abc: An association between the hypothesized cause and effect has been observed repeatedly in a variety of places, circumstances, and times.
• Specificity of association ^ac: Refers to the case where either: 1) the effect is found only in association with the hypothesized cause or 2) only one effect is associated with the cause.
• Temporality ^ac: The hypothesized cause is observed to precede effect.
• Biological gradient (dose–response) ^bcd: A conceptually sensible biological explanation exists for the relationship between cause and effect.
• Coherence ^a: Support for the cause–effect relation is provided by known facts including laboratory results or patterns in association between cause and effect between populations.
• Experiment ^c: Results from experimental studies support the cause–effect relationship. This criterion applies only to situations where the hypothesized cause is manipulated experimentally.
• Analogy:  The hypothesized cause–effect relation can be argued on the basis of similarity with an established cause and effect.

^aOriginal criteria defined by USDHEW (1964)

^bCriteria quantitatively identified by Weed and Gorelic (1996) as being the most commonly used in epidemiology

^cPrincipal causal criteria identified by Adams (2005)

^dWeed and Gorelic (1996) and Weed (1997) used the name ‘dose-response’ for this criterion"

Source: Norris et al. (2012)

Criteria to consider when recording lines of evidence

These questions have been adapted in a template developed by the New South Wales (NSW) Government Office of Environment and Heritage.

• "Is the evidence relevant to the issue? Is this evidence expected to be in the study area?
• Is there a reliable explanation for the evidence in the study area?
• Is this evidence restricted to particular spatial or temporal situations? Is there evidence of a relationship (spatial or temporal) between the activity and the response to that activity?
• Does the expected response always occur in the presence of the activity? Is the response always evident after the activity occurs?
• Are there consistent findings across different lines of evidence? Is there a credible linkage between intervention and response? A clear relationship between the activity and the response would be expected where the investment was based on a conceptual model of the system. Consistency of association between the activity and the response is observed where testing the evidence against all the criteria provides positive findings.
• Is there any other evidence that should be considered? Determine whether there is evidence from other investments or management activities in which the expected change occurred after the activity. Determine whether there is any basis for including additional assessment criteria. Consider where appropriate."

Source: New South Wales (NSW) Government Office of Environment and Heritage 'Using the template'

Adapting the criteria for MLLE analysis

The following, extracted from a table in Young et al. (2006) (cited in Department of Environment NSW, n.d), sets out the adaptation of these criteria for an MLLE analysis and provides examples of how they might be used:

"Criteria for assessing relevance of evidence

Criterion - Credible linkage

• Question posed by criterion - Is the evidence relevant to the issue, i.e. is this evidence expected in the study area?

Example of response to criterion from different lines of evidence:

• Literature (references) demonstrates there is a relationship between improved riparian management (activity) and sediment nutrient fluxes (response) of waterways.
• Local catchment studies (referenced) support the findings of the above literature.
• Performance measure monitoring for catchment targets is consistent with investment trends and outputs for riparian management.

Criterion - Presence of a response

• Question - Is there a reliable explanation for the evidence in the study area?

Examples:

• Performance measure monitoring for catchment targets linked to SedNet scenario modelling within a case study provides evidence for the management of sediments and nutrients within the catchments.
• Performance measure monitoring for catchment targets is consistent with investment trends and outputs for riparian management for those estuaries.
• Reported catchment investigations for like systems.

Criterion - Evidence of ‘intervention response’ relationship with activity

• Question - Is this evidence likely only under different spatial or temporal situations?

Examples:

• Performance measure monitoring for catchment targets is consistent with investment trends and outputs for riparian management.
• SedNet scenario modelling highlights how the application of relevant management actions in priority catchments will have the greatest reduction in sediment inputs to estuaries.
• Performance measure monitoring for catchment target linked to SedNet scenario modelling within a case study provides evidence for management of sediments and nutrients within the catchments.

Criterion - Consistency of association

• Questions - Does the expected response always occur in the presence of the activity? Is there a credible linkage between intervention and response?

Examples:

• Performance measure monitoring for catchment target linked to SedNet scenario modelling within a case study provides evidence for management of sediments and nutrients within the catchments.
• Performance measure monitoring for catchment target in particular estuaries is consistent with investment trends and outputs for riparian management.

Criterion - Consistency

• Question - Are there consistent findings across different lines of evidence?

Example:

• All lines of evidence tested against criteria and show consistent findings.

* The examples only show how existing and new sources of information may be used with the criterion."

Source: Young (2006)