Strategies to reduce costs

Reducing costs is something to consider if evaluation costs outweigh the predicted benefits or available resources.

There are a number of strategies for reducing costs by modifying the evaluation design.  Consider these methods carefully because most of them involve threats to the validity of the evaluation findings and recommendations. 

Often the available resources are less than required to implement your ideal evaluation design.  You may need to revise your evaluation to make the best use of resources. Strategies for reducing costs of evaluation planning, data collection, and analysis include:

  1. Focus the evaluation to eliminate nonessential data collection
  2. Reduce the frequency of data collection
  3. Use available data
  4. Reduce the sample size
  5. Use cheaper data collection methods

1. Focus the evaluation to eliminate nonessential data collection

Clarify the information needed and the decisions to which the evaluation n findings will contribute.  In consultation with relevant stakeholders distinguish between “nice to know” and “need to know” information to eliminate nonessential data collection and analysis.  Including only essential data can bring savings in design, data collection, analysis and reporting. This may involve revisiting the purpose of the evaluation and the key evaluation questions

2. Change the research design

For example, “eliminating the collection of data on the project or comparison group before the project begins (pretest) or on the comparison group after the project is implemented (posttest). In the simplest design, when data are collected on only the posttest project group, the data collection budget can be reduced by as much as 80%.” Such designs may compromise the validity of the research. RealWorld Evaluation (archived link, 2012) describes 19 impact evaluation designs and ranks them according to their strengths, and their estimated cost savings (pp. 46-47).

3. Reduce the frequency of data collection

Reducing the number of data collection points in a time series (for example, every 2 months instead of every month) will reduce costs. Make sure you get expert advice to check that the data will still be sufficiently robust to provide valid results and be seen as credible.

4. Use available data

Start with the project’s own monitoring records.  What information is currently available?  How might this information be mined to address evaluation questions?  Consult with staff in the field because sometimes they may be collecting or have access to information that is not requested on reports. 

If the evaluation’s information needs cannot be met through the project’s own monitoring records, look outside the project at other potential sources: 

  1. Census or survey data covering the project and comparison communities.
  2. Public services agencies, such as schools, health centres, agricultural extension centres, etc.
  3. Newspapers and other mass media
  4. Information from community organizations
  5. Dissertations and other academic studies

Assess the information from secondary sources carefully to ensure that it can be used to answer the evaluation questions. Ensure that the data sources match the proposed evaluation sample and that the data be obtained easily and ethically.  

5. Reduce the sample size

Reducing the sample size to work within budget is a matter of balancing information needs with technical considerations.  First consult with key stakeholders to understand the decisions to which the evaluation will contribute, and the degree of precision appropriate to those decisions. After coming to an understanding of the degree of precision required, consider the factors affecting the sample size for:

  • quantitative methods
  • qualitative methods
  • mixed-method designs

6. Use cheaper data collection methods

This strategy is similar to eliminating nonessential data collection but also includes substituting time-consuming or expensive methods with more cost-effective methods.  For example, consider:

  1. focus groups and participatory rural appraisal instead of surveys and interviews
  2. direct observation instead of surveys 
  3. interviews or surveys of a few key informants instead of surveys of a larger sample
  4. self-administered questionnaires instead of evaluator-administered questionnaires 

Advice for choosing this method

Consider each of the five cost reduction strategies carefully because most of them involve threats to the validity of the evaluation findings and recommendations. 

Keep in mind:  “With emergent designs, issues and questions arise as the research progresses, and often, many of what prove to be the critical issues were not even included on the initial list of questions” (Real World Evaluation, 2012, p. 55).  These method substitutions do not in all cases result in cost savings; be sure to work out the details of your evaluation carefully to ensure a good method choice. For example, reducing data collection costs will not necessarily reduce analysis costs. 

Advice for using this method

The validity of findings, and possibly the quantity and type of information, will be affected by these cost reduction strategies.  Once you’ve adjusted the design to reduce the costs of the evaluation, return to the evaluation’s original purpose and ask again:  will the evaluation be worth it?  Will the new evaluation design meet the original purpose? 

Be sure to triangulate methods and sources to ensure accuracy when collecting limited amounts of data.

Resource

Method

Bamberger, M., Rugh, J., & Mabry, L. (2012). Chapter 3:  Not Enough Money:  Addressing Budget Constraints in RealWorld evaluation:  working under budget, time, data and political constraints. In RealWorld Evaluation Working Under Budget,Time, Data and Political Constraints. (2nd ed.).Thousand Oaks, CA: Sage, (pp. 40-59) via http://www.realworldevaluation.org (archived link)

Marynowski S. Denny C. & Colverson P. 2006 Best Practices Guide to Program Evaluation For Aquatic Educators (archived link) Pandion Systems, Inc. Gainesville, Florida (p. 34). www.pandionsystems.com

Various summaries, presentations and other materials from RealWorld Evaluation are available in English, Spanish, French, Russian and Portuguese via https://edge.sagepub.com/bamberger3e

'Strategies to reduce costs' is referenced in: