Evaluation Planning – 3.03 Evaluation Questions
Evaluation questions are the broad questions asked about the program in general – “Is the program being implemented well?” or “Does our program have an effect on outcomes A, B, or C?”. They will focus the evaluation planning and anchor all subsequent sections of the Plan. Note that evaluation questions in this context are not the same thing as questions that would be included in a questionnaire or interview for participants; those are measurement questions. Evaluation questions are the major questions your evaluation is trying to address in a specific evaluation cycle. Every question will have at least one associated measure in place by the time the evaluation plan is complete.
It is important for the working group to keep in mind the lifecycle phase of the program and its relationship to the evaluation lifecycle. Each stage of the program lifecycle emphasizes different types of evaluation questions. Remember that the goal is to eventually align the program lifecycle and evaluation lifecycle. Depending on where the program is starting from, this may not be feasible in just one evaluation cycle. It may take several evaluation cycles (i.e., years) for the program and evaluation lifecycles to be aligned. When developing evaluation questions, it is important to consider what can be done to move the lifecycle and evaluation phases into better alignment.
In addition to maintaining alignment with the program and evaluation lifecycles, fidelity to the program’s pathway model and purpose statement, language and format have to be carefully considered as the working group continues to brainstorm, revise and edit evaluation questions. Each evaluation question should clearly identify the program element(s) that are being examined (one or more activities, the overall program, specific outcomes, etc.) The phrasing of the evaluation questions matters. Consider the difference between asking whether a program is “associated with a change” as opposed to “causes a change.” The difference could mean a much more (or less) controlled evaluation strategy, and enable a much different claim that could be made at the end of the evaluation cycle.
When writing the evaluation questions, make sure the questions are clearly worded and understandable to someone who is unfamiliar with the program. Be sure to consider (and possibly revisit and revise) the evaluation purpose statement so that the questions make sense given the overall purpose of the evaluation.
The following framework will help to craft evaluation questions for the selected scope of the evaluation. Keep in mind that these are only examples of questions and that there are many “correct” ways to word questions (though some are more appropriate for certain programs than others). To develop an evaluation question, start by identifying a program element (activity or set of activities, outcome, or link) that falls within the evaluation scope in your pathway model. For instance, examples of a Phase 1 evaluation lifecycle question might be:
How well is [ACTIVITY “A”] implemented?
To what extent are participants satisfied with [Activity “B”]?
How useful were the program handouts?
To what extent do participants demonstrate [OUTCOME “X”]?
Phase 2 evaluation questions are typically focused on assessing changes in outcomes associated with participation in the program. Examples of Phase 2 evaluation lifecycle questions might be:
Is participation in the program associated with a change in [OUTCOME “X”]?
How do participants’ levels of [OUTCOME “Y”] compare to non-participants’ levels of [OUTCOME “Y”]?
Phase 3 evaluation questions will typically explore issues of causality or “effectiveness” of the program. These require the use of evaluation designs that are more carefully controlled and/or longitudinal (i.e., follow participants over a longer period of time). Many programs will not reach a Stage 3 evaluation in terms of their program lifecycle phase. Either their purpose is not to demonstrate the effectiveness of the program, or evidence from earlier phase evaluations will cause the organization to abandon the program and try an alternative approach to the problem. For those programs where a Phase 3 evaluation lifecycle question would be appropriate, some examples of questions might be:
To what extent does participation in the program cause [OUTCOME “Y”]?
To what extent does participation in the program cause changes in [OUTCOME “Z”]?
Phase 4 evaluation questions are focused on understanding the generalizability of the program to other contexts, settings, and participants and are typically used when the goal is to broadly disseminate a program that has already clearly demonstrated its effectiveness through rigorous prior evaluations. Examples of Phase 4 questions might be:
Does participation in the program have the same effect on participants of [different ages, different races, living in different settings (e.g., urban, rural, suburban)]?
Can the program be replicated with similar results across multiple settings?
In general, if there are multiple activities that lead to one crucial outcome –or as is more often the case, multiple outcomes that arise from an individual activity– you may certainly use a semi- colon and bullet points to collapse more than one evaluation question into a single sentence, such as:
What is the effect of the program on: [OUTCOME “X”]; [OUTCOME “Y”]; [OUTCOME “Z”]?
Keep in mind that when we are developing evaluation questions we are generally only looking for a few key overarching questions that can guide the evaluation during its next cycle. Imagine we have a program with five activities. We could develop five separate evaluation questions that ask “How well are we implementing Activity 1?”, “How well are we implementing Activity 2?” and so on, or we could shorten this to “How well are we implementing the program?” When stating evaluation questions, use fewer and more general questions if possible.
Typically you want to keep the number of evaluation questions manageable. Three or four evaluation questions will in most cases be sufficient to accomplish a reasonable evaluation in any given evaluation cycle. Use Appendix XXI to help you develop your questions. The remainder of the evaluation plan is constructed to address these questions.
See Also: