Evaluation costing

Evaluation expenses are highly situational and there are no magic formulas for calculating costs. 

After developing the budget matrix to identify what needs to be costed, the next step is to calculate the costs of individual expenses and sum them to obtain the total cost of the evaluation.  When determining the resources needed to carry out the evaluation effectively take into account time, money, and expertise.

Depending on budgeting and planning processes in your organization, you may be asked to make a rough estimate of evaluation costs some time before the start of the evaluation planning, and to develop a more detailed budget at a later stage. An accurate version of an evaluation budget can be based on the evaluation scope of work (or Terms of Reference) which outline the evaluation design, methods, sample sizes and other considerations that have a direct bearing on the costs.

Be sure to estimate resources needed to facilitate the participation of those outside the organization commissioning the evaluation, such as program participants, partner organizations, and others. You may want to consult with key stakeholders who will be asked to participate in the evaluation. 

One effective way to break down the budget is by evaluation activities.  Some activities will have repeatable costs.  For example, hosting a focus group may have similar expenses each time, for example, transportation allowances and refreshments for respondents, note-taking and transcription services, and meeting space rental. In the case of data collection activities, determine the cost of each activity and multiply those costs to achieve the desired sample size. Another approach is to tackle large budget items such as personnel and travel.

Calculating personnel time (level of effort) and costs

The basic method for calculating personnel costs is to determine the amount of time required for each personnel member and multiply that by the daily rate to determine the cost for each person.

An accurate estimate of the time required goes hand-in-hand with developing the budget.  “Consider developing an evaluation time budget, as well as a cost/resources budget. … Many projects fail to budget sufficient time for the evaluation, producing results that are not as helpful or useful as initially expected.” (W.K. Kellogg Evaluation Handbook 2004 p57) Don’t forget to include sufficient time for reporting, as well as learning and follow-up on findings.

Some organizations elect to pay for evaluations in their entirety or discrete tasks, using lump-sum contracts rather than contracts specifying billable days.  Nevertheless, an estimate of time and fair market price for personnel costs ensures that lump sum figures are reasonable. 

Complex and changing circumstances - allowing for contingencies & complexity.

When situations are relatively stable, good planning is key to the execution of a successful evaluation. In complex or dynamically changing situations, responsiveness is of equal or greater importance.

There are several ways that unpredictability may affect an evaluation plan and budget including: personnel turn-over, bureaucratic holdups, scheduling delays, developments in the field, shifting organizational priorities, tangled communications, and other unexpected changes arising from its context.

Evaluation choices, such as the type of options used, can also contribute to the unpredictability of an evaluation’s costs. For example, the time and level of expertise needed to prepare a case study or to observe participants can vary widely and be difficult to estimate.  

How do you approach complexity and unpredictability in evaluation budgeting?  Should unpredictability be managed or embraced as central to the evaluation process? The answer depends largely on the kind of evaluation. Develop budgets that estimate high and low boundaries of costs depending on various scenarios.  Set aside some funds for unexpected changes in the design or execution.  Negotiate some flexibility in meeting expenditure or reporting deadlines. 

Budgeting for developmental evaluation

Developmental evaluation, on the other hand, is designed to provide real-time evaluative information to guide an initiative continually changing in response to its environment. If you have selected a developmental evaluation approach in recognition of the complexity of the evaluation and its context, the planning and budgeting process should be in alignment with the adaptive nature of this evaluation approach.  Your budget should be as responsive as you expect your evaluation to be. The interview with Michael Quinn Patton provides specific advice on budgeting for developmental evaluations.

Review and finalise the budget

Once there is an accurate assessment of the resources (time, money, and expertise)  required to carry out the evaluation, it’s time to review and finalise the budget. This involves error-checking, reviewing cost allocations across the evaluation stages, and stepping back to reconsider the evaluation purpose in light of the costs. 

Error-checking can be as simple as reviewing your sums or may include consulting with technical specialists, potential vendors, or field staff, about expense estimates. It’s also helpful to get an overview of how funds (and time) are distributed throughout the process.

One way of assessing the evaluation budget is to step back and review how the resources have been distributed across the various stages of the evaluation.  Calculate the percentage of funds (and time) allocated to each of the major stages. 

There are no hard and fast rules about how resources should be distributed across an evaluation process; the particulars of design and context will have a significant effect on how resources are allotted to each stage and activity. This overview may provide a helpful perspective as you review and finalize your evaluation budget.  Are the resource allocations an accurate representation of the evaluation purpose, context and design? 

“Remember to include time for follow up and application of evaluation results – the most thorough evaluation will not help your program if the results remain in a folder on a shelf!” (Marynowski, 2006, p. 34).

Finally, check that the benefits of conducting the evaluation will exceed the costs and that it is worthwhile to proceed.  Evaluation should be purchased or undertaken only “when the likely usefulness of the new information outweighs the costs of acquiring it” (Wholey, 1983, p. 119).

Advice for choosing this method

This step is essential to completing the task of determining and securing resources for your evaluation.

These resource allocation estimates do not hold true for developmental evaluation.  (See interview with Michael Quinn Patton on Budgeting For Developmental Evaluations)

Advice for using this method

Consider developing a cost-per-activity budget because it facilitates evaluation planning, including finalizing the design, staffing and scheduling. Often changes will have to be made to the design and budget during the planning process.  If your organization requires another format, such as according to cost categories, consider keeping the cost-per-activity version as a working document. 

Don’t underestimate the amount of time required for managing and conducting the evaluation.  Miscalculations are frequently made for planning, reporting, learning and follow-up on findings. 

A common error is to underestimate the resources required for reporting.  It takes time to draft a high-quality report or develop an engaging presentation.  The “Illustrative Allocations of Resources to various Study Stages and Activities” resource estimates that reporting requires 20% of the total resources. Depending on the reporting medium (professional quality presentations or reports, video clip, web pages), special skills or resources may be required.

Another common error is to underestimate the time required for planning and study preparations. 

It can be useful to make a “Three-Point Estimation” using experience, historical information, and expert judgment to define an optimistic estimate, a pessimistic estimate, and a most likely estimate.

Refer to evaluation or project budgets with similar expenses. Consult with those with procurement and/or budget experience, as well as those working in the area where data collection activities will be taking place.

Assess the budget with the stakeholders who have been engaged in planning the evaluation. 

Do not elevate allocations across study stages above other more important considerations in drafting or evaluating an evaluation budget. 

Resources

W.K. Kellogg Foundation (2004), Evaluation Handbook, W.K. Kellogg Foundation, USA. Retrieved from https://www.wkkf.org/~/media/62EF77BD5792454B807085B1AD044FE7.ashx

Marynowski S., Denny C. & Colverson P. (2006). Best Practices Guide to Program Evaluation For Aquatic Educators. Pandion Systems, Inc. Gainesville, Florida. Retrieved from: http://www.rbff.org/uploads/Resources_bestpractices/Best_Practices_Guide_to_Program_Evaluation.pdf (archived link)

The Pell Institute and Pathways to College Institute. (2009). Create a Budget. Evaluation Toolkit. Retrieved June 28, 2012, from http://toolkit.pellinstitute.org/evaluation-guide/plan-budget/develop-a-budget/

RealWorld Evaluation (2012) via http://www.realworldevaluation.org/

Scriven, M. (1976a). Payoffs from evaluation.  In C. C. Abt (Ed.), The evaluation of social programs (pp. 217-224). Beverly Hills, CA: Sage.

Wholey, J. S. (1979). Evaluation: Promise and performance. Washington, DC: Urban Institute.

Wholey, J. S. (1983). Evaluation and effective public management.  Boston: Little, Brown.

Expand to view all resources related to 'Evaluation costing'

'Evaluation costing' is referenced in: