3.01 Evaluation Plan Introduction
The goal of this step is to present and discuss the purpose and components of an evaluation plan. The working group should understand the work that is entailed, and the importance both internally and externally of having a fully articulated and written plan.
A written evaluation plan captures important program and evaluation information and guides the implementation of the evaluation, which is especially important if there are new staff who will be doing some of the work. In addition, having a written evaluation plan can help to establish credibility with stakeholders – especially funders – because it conveys information to them and is a demonstration of the commitment to quality evaluation.
The written evaluation plan developed by users of this guide will likely include section headings that map onto many of the steps in the Protocol. The written plan should be formatted so that sections describing the program and its boundaries appear early in the document. Sections titled “Measures” “Sampling” and “Analysis” (for example) should appear later in the document. In addition to the written sections of the plan, a complete document will include several appendices, including a copy of the logic and pathway models, a stakeholder diagram, and copies of any intended measures.
Several of the outputs from Stage 2 will be used to develop the written evaluation plan. Typically we include the program mission statement and program description, the program logic model and pathway model (perhaps as appendices or attachments), and stakeholder map. Decisions about evaluation scope from the previous stage will inform decisions made within the evaluation plan. The plan will also include new products– the evaluation purpose statement, the evaluation questions, a description of the proposed sample, the proposed measures, the proposed design, a data collection and management plan, the plans for data analysis, plans for evaluation reporting and utilization, and an evaluation timeline. These new components are the classic elements of any good research plan, covering the objectives and methodology to be used.
A key distinction in evaluation planning is differentiating between an evaluation cycle and the evaluation of the program over its entire life. An evaluation “cycle” is the evaluation that you do over a specific period of time. Many organizations use an annual evaluation cycle. The endpoint of an evaluation cycle is often determined by when the organization and program are doing major reports. Again, many organizations find it convenient to do annual reports, which is a tip-off to the notion that they are on an annual evaluation cycle. When the working group developed the evaluation scope in Step 2.08, they were essentially determining the extent of the program evaluation they were going to take on in the next evaluation cycle (e.g., over the next year). An evaluation plan is typically written in terms of the next evaluation cycle, describing, in detail, how the staff will evaluate their program over that time period.
We have already seen that programs evolve through phases over time. Evaluation needs to change and adapt so that it is always appropriate (that is, symbiotically linked) with where the program is in its development. This means that over the life of a program there are likely to be multiple evaluation cycles and the evaluation plan for each cycle will change. We will keep coming back to this idea in the sections below.
Q&A
Q: What is the purpose of having a written evaluation plan?
The written plan serves a number of different purposes. First, the process of developing and writing an evaluation plan helps you be more intentional and thorough in your approach to evaluation. In the absence of a formal planning process, it is easy to focus too much on simply finding or creating a measure. Writing out the plan helps you think through and clearly articulate all components (purpose, questions, design, analysis, etc.) of the evaluation, and helps ensure alignment between those various components and the larger needs that this evaluation should serve. In addition, there are benefits to having it as a written document. The written document increases institutional memory, both about the evaluation and about the program itself. The written plan can provide continuity in how evaluation is conducted. Since evaluation is an iterative, evolving process, the plan provides a history upon which future evaluation cycles can be built. Last but not least, the written plan (like the logic and pathway models) is an excellent communication tool for internal and especially external communication. The plan can be included in reports to funders, in applications for funding, or in presentations to legislatures or other important stakeholders. To these audiences, the written plan exemplifies your commitment to and expertise in evaluation.
Q: How will the pathway diagram be used for evaluation planning?
The pathway model will help guide your evaluation by revealing key outcomes and pathways that may serve as focal points for this evaluation cycle. This focusing is important because it is (usually) not feasible to evaluate the whole program in one year. The visual nature of the pathway model offers a uniquely powerful way to see the key “nodes” in the overall process of change, for example, outcomes that may have many arrows “going in” or “coming out”, or both. (Ironically, although the pathway model is built from the knowledge held by the model-builders, it is sometimes only when they step back and look at the resulting visual display that these patterns can be recognized fully.) These model components become candidates for consideration, along with other factors such as stakeholder priorities, lifecycle considerations, and feasibility, in finalizing the evaluation questions.
A pathway model can also help with what is often a challenge for program staff, namely, that funders or other stakeholders want “evidence” about the program’s impact on long-term outcomes which is simply not feasible for program staff to evaluate. In this case, the pathway model is useful for demonstrating the conceptual linkage between shorter-term outcomes (which can be evaluated) and more distant outcomes. The burden of evaluation is reduced if you can then find published research to support these longer-term conceptual linkages.
Presentations
- Length 7:08