2.07 Evaluation Scope
Typically evaluation occurs over time as a program evolves. In any given evaluation cycle (e.g., annual evaluation) we usually don’t try to assess every activity and outcome on a logic or pathway model, we make choices about what is most important, relevant and feasible. That is, we determine what the “scope” of the evaluation will be at this point, which elements of the program we will focus on. In most cases, the scope of earlier lifecycle phase evaluations focuses on assessing program activities (e.g., implementation and process), outputs and perhaps a few key short-term outcomes. Over time the scope tends to move to the right on a pathway model, encompassing later short- and medium-term outcomes.
This step in the Protocol represents a change in focus. The Evaluation Champion and the working group will now move from thinking purely about the program, its boundaries, logic and stakeholders move toward thinking about evaluation. This shift should be acknowledged, and the group may want to discuss any learning and other benefits that have come out of their work so far.
One of the Evaluation Champion’s responsibilities is to help the working group determine where evaluation efforts can be most valuable for them. There are many points to consider and discuss during this step. In order to help guide this decision-making process, we have developed a goal statement which may serve as a touchstone for evaluation planners as they move through the rest of the steps of the Protocol.
Discuss Feasibility, Credibility, Accuracy and Usefulness
It is important to be strategic and to determine what is appropriate and feasible in the coming evaluation cycle. Feasibility discussions should take account of program activity, time constraints, funding constraints, and/or reporting constraints. Keep this concept in mind as you continue, and use caution about selecting too large a scope. In addition, evaluation time and resources are limited so it is essential to direct the evaluation efforts toward generating information about the program that is useful and credible to both internal and external stakeholders. This has implications for the evaluation scope (what to focus on) and for the evaluation strategies that will be used. Accuracy refers to reliability and validity of a given evaluation strategy and set of tools. Program staff should plan an evaluation that will provide the most accuracy (which will lead to credibility and usefulness) within their contextual feasibility restraints.
As the group works through the sections of the evaluation plan, the building blocks established in the program modeling phase of the Protocol – stakeholder analysis, lifecycle analysis, program logic and pathway models – will come together to help establish strategies and priorities for achieving the evaluation goal in the coming evaluation cycle.
Examine the Logic and Pathway Models
The pathway model and logic model are the primary tools you will use to assess and decide upon the scope of evaluation for a particular evaluation cycle. The program models represent a broad overview of everything that is done and assumed in a program and all of the expectations for the program’s effect. It can help program staff to think of the pathway model and logic model as a blueprint for their program over the next “X” number of years. Keep in mind that it may not be possible to address every aspect of this model in the next year, or maybe even in the next five years. The “Evaluation Scope” encompasses the components of the program that will be focused on in the upcoming evaluation cycle.
Revisit Stakeholders
An evaluation scope discussion should include a re-examination of the Map of Stakeholders. Are there any outcomes or pathways that multiple stakeholders are interested in? Are there any outcomes or pathways that none of the stakeholders have an interest in? Do certain stakeholders get precedence over others? Once again, visual clues are helpful. Transfer the pathway model onto paper or a drawing board. Create a numbered list of the key stakeholders and place each stakeholder’s number onto the pathway model next to the outcomes and pathways that that stakeholder would be most interested in (see Figure 13).
Figure 13: Stakeholders Located on Program Pathway Model
Revisit Lifecycle
This is an opportune time to reconsider the lifecycle of the program. There may be a desire to focus on a medium-term outcome but if the program is still in its initiation phase it might be more appropriate to focus on demonstrating effects on short-term outcomes first.
Finding the Key Pathway Markers
In any pathway model there are some paths that are more important than others, and some outcomes that are more central to the model. The following questions can help guide thinking about which pathways to focus on in the current evaluation cycle.
Required: Is there a particular activity, outcome, pathway (or set of pathways) on which the program is required to report? If so, then evaluation should definitely include this in its scope.
Easiest: The primary (or most obvious) pathway may be the best focus for the evaluation. Is the program already collecting information that addresses some aspect of the model? Is there an “easy” or obvious point on which an evaluation could focus? We refer to this as the “low-hanging fruit.”
Hub (Grand Central): Like Grand Central Station, there may be a “central point” through which many of the pathways pass. By evaluating this outcome, the program may potentially be able to address many aspects of its model simultaneously.
Once again, visual clues are helpful for many people. On the pathway model circle hubs, highlight key pathways, and so forth.
Determine the Scope for this evaluation cycle
Once the key markers have been identified and stakeholders have been placed on the pathway it is time to determine the scope for the evaluation. What we mean by scope is essentially how far into the pathway model (from left to right) the group intends to evaluate over a certain period of time (this evaluation cycle). For instance, if someone is determining the scope for an evaluation for the upcoming year, and they have determined the key markers as described above, they will want to set the scope for the evaluation to include as many of the key markers as can reasonably be accommodated within the year. It is useful to show the scope of an evaluation graphically. You can do this by drawing a line that encloses the pathways that will be included in your evaluation within it as shown in Figure 14.
Keep in mind that the evaluation scope will generally change from year to year and will generally move from a focus on activities, outputs and short-term outcomes in earlier evaluations to one that includes medium- or long-term outcomes later on.
See also:
Figure 14. Evaluation Scope.
