II.1.01 Reconsider Alignment and Context Appropriateness

There are three types of evaluation plan alignment that should be reviewed:

• Alignment with the program and its context – is the evaluation plan feasible? Does it fit the program setting, duration, the needs and norms of the target audience and sample groups?

• Alignment of purpose – is the evaluation addressing program and stakeholder priorities? Are the evaluation questions focusing on elements that are important to key stakeholders and to the decisions that are currently in play for the organization and program?

• Internal alignment – do the measurement, sampling, design, data management, analysis and other sections fit each other, and do they fit the evaluation questions?

To assess alignment with the program and its context, review the program description and the context section of the logic model, review the descriptions of program participants and the sample groups, and consider the tasks enumerated in the timeline. The following prompt questions are suggestive of the kinds of mismatches that can arise that make it difficult to complete a plan or that can reduce the quality of the data you are likely to obtain. Obviously the list is not exhaustive; use it to think as thoroughly as you can about how well the planned evaluation fits the program:

• Think about the resources and support needed for the evaluation:
• Are there enough staff members available at the necessary times (look out for “crunch times” in people’s calendars!)
• Do they have the necessary skills to develop and refine the measures, assemble the contact lists for the sample group(s), administer the measures, enter the data, clean the data, conduct the analysis, process and integrate the results, prepare reports and presentations in time for program deadlines, and so on?
• Will the evaluators have access to data needed to conduct the evaluation? These might come from registration or attendance records, mailing lists, standardized tests or entrance exams, transcripts, archives of key materials, etc.

• Think about the people that are being drawn upon for data or responses to measures:
• Will they have time to complete the planned measures? (If the program is only an hour long, using 20-minutes of that time to complete a survey would not work!)
• Will the measures work given the program setting? (indoors or outdoors, daytime or nighttime, with or without tables to write on, etc.)
• Will the measure fit the respondents’ circumstances? Will they need to be at home, do they have transportation to get to a focus group meeting, will they have internet access if that’s needed, will they be able to complete the measure without interruption, will they have access to personal records they might need, will they be able to read and understand the questions or prompts?
• Are the evaluation plan components culturally responsive? Are there instructions, questions, or expectations regarding interactions that could be misunderstood or that could make individuals uncomfortable or uncertain? Are there cultural factors that need to be considered in order to interpret responses accurately? Are the selected measures and sampling strategies the most effective for the context, traditions, and norms of the particular groups involved in the evaluation?

 Note any concerns that arise with these prompt questions and return to those plan elements in order to make adjustments that are needed.

To assess alignment of purpose, revisit the Stakeholder Map that was developed early in the evaluation process, and review the priorities of the key stakeholders. Think about what decisions are “coming up” for this program in the near future – are there decisions to be made about program process or re-design? About staffing levels, funding, or program scale? Is the organization that runs the program facing decisions that depend on what is known or learned about this program’s implementation or impact?

Consider the Evaluation Purpose Statement in light of these factors and priorities – is the evaluation focused on obtaining information that will be most useful and strategic at this point in the life of the program? Finally, will the Reporting and Utilization plans make full use of the evaluation results, and will they fulfill the stakeholder priority needs? Now is the time to make changes to the plans to ensure that valuable evaluation resources are put to their best use.

To assess the internal alignment, it can be helpful to review or re-create a “grid” in which each row is devoted to an evaluation question and the columns cover the key elements of measurement, sampling, design, and analysis. Read across each row and check for consistency. Think about the strength of the claim that you would like to be able to make in response to each evaluation question. If it is a claim of effectiveness, are the design and sampling strategy sufficient to rule out other possible causes? If it is a claim of efficacy (that the program is “associated with” a certain change), can the design and measurement strategy capture change? If it is a claim about future participants’ likely satisfaction with the program, is the sampling strategy going to yield a sample that is sufficiently representative of future program participants?

These internal alignment issues tend to be considerations that are more technical. The evaluation elements are inherently inter-dependent, and their alignment is what gives power to the analysis and claims that can be made. In practice, however, the sections of an evaluation plan tend to be written sequentially, and sometimes there are gaps of time between when one section and the next are written. Misalignment can creep in unexpectedly if the thinking drifts as the evaluation plan is developed over time.