Stage 3: Evaluation Plan Creation
Introduction
Now that the program models have been refined, the working group can move from talking about the program to talking about evaluation. This stage of the Protocol involves several steps that build on the prior stage to aid in the creation of an evaluation plan that will guide the implementation of the evaluation. This stage includes: Introduction to Evaluation Plans, Development of an Evaluation Purpose Statement, Development of Evaluation Questions, Development of Measures and Measurement Strategies, Development of a Sampling Plan, Identification or Selection of an Evaluation Design, Development of an Analysis Plan, Development of an Evaluation Reporting Plan, Development of Evaluation Schedule and Implementation Plan, and the Finalization of the Evaluation Plan.
One of the key aspects of systems evaluation planning that distinguishes it from traditional evaluation planning is the concept of the program lifecycle as described earlier. When you are developing your evaluation plan, it is crucial to consider the lifecycle phase of your program. That is, the evaluation questions, sampling approach, measurement choices, design, analysis, reporting and use will be different for programs at different phases of development. In the sections below we will emphasize this concept and how it is used in developing your evaluation plan.
See Also:
Q&A
Q: How long should each section of the evaluation plan be?
Each section should be complete, yet concise. The length of a particular section can vary widely based on what that section is and based on the nature of the evaluation plan. Some sections, like Program Mission, are supposed to be very brief. Other sections, like Measurement or Analysis will be longer or shorter depending on the number of evaluation questions, the diversity of approaches used, and (perhaps) the lifecycle stage of the evaluation (later stage evaluations may require lengthier text to explain, though not necessarily). Overall, the goal is to make a plan that is thorough yet concise and is readable and understandable by someone external to your program.
Q: What should each section of the evaluation plan include?
Each section should include a complete, concise description of what you plan to do and why you chose that strategy. For the Measurement section, for example, you should include a description of the data collection method(s) you have chosen and which tool(s) you plan to use, along with an explanation for why those choices were made. (Otherwise, skeptical readers might wonder, “Why is an online survey the best choice here?” or “Why will ten interviews be sufficient?”) Without going into too much detail, explain why the approach you have adopted is most likely to generate credible, accurate, useful insights about the program in a way that is feasible. Overall, the plan should hang together well, have internal consistency, and serve as a good guidance and communication tool.
Q: What is the purpose of having a written evaluation plan?
The written plan serves a number of different purposes. First, the process of developing and writing an evaluation plan helps you be more intentional and thorough in your approach to evaluation. In the absence of a formal planning process, it is easy to focus too much on simply finding or creating a measure. Writing out the plan helps you think through and clearly articulate all components (purpose, questions, design, analysis, etc.) of the evaluation, and helps ensure alignment between those various components and the larger needs that this evaluation should serve. In addition, there are benefits to having it as a written document. The written document increases institutional memory, both about the evaluation and about the program itself. The written plan can provide continuity in how evaluation is conducted. Since evaluation is an iterative, evolving process, the plan provides a history upon which future evaluation cycles can be built. Last but not least, the written plan (like the logic and pathway models) is an excellent communication tool for internal and especially external communication. The plan can be included in reports to funders, in applications for funding, or in presentations to legislatures or other important stakeholders. To these audiences, the written plan exemplifies your commitment to and expertise in evaluation.
Q: How will the pathway diagram be used for evaluation planning?
The pathway model will help guide your evaluation by revealing key outcomes and pathways that may serve as focal points for this evaluation cycle. This focusing is important because it is (usually) not feasible to evaluate the whole program in one year. The visual nature of the pathway model offers a uniquely powerful way to see the key “nodes” in the overall process of change, for example, outcomes that may have many arrows “going in” or “coming out”, or both. (Ironically, although the pathway model is built from the knowledge held by the model-builders, it is sometimes only when they step back and look at the resulting visual display that these patterns can be recognized fully.) These model components become candidates for consideration, along with other factors such as stakeholder priorities, lifecycle considerations, and feasibility, in finalizing the evaluation questions.
A pathway model can also help with what is often a challenge for program staff, namely, that funders or other stakeholders want “evidence” about the program’s impact on long-term outcomes which is simply not feasible for program staff to evaluate. In this case, the pathway model is useful for demonstrating the conceptual linkage between shorter-term outcomes (which can be evaluated) and more distant outcomes. The burden of evaluation is reduced if you can then find published research to support these longer-term conceptual linkages.