Evaluation Planning – 3.03 Evaluation Questions
Evaluation questions are the broad questions asked about the program in general – “Is the program being implemented well?” or “Does our program have an effect on outcomes A, B, or C?”. They will focus the evaluation planning and anchor all subsequent sections of the Plan. Note that evaluation questions in this context are not the same thing as questions that would be included in a questionnaire or interview for participants; those are measurement questions. Evaluation questions are the major questions your evaluation is trying to address in a specific evaluation cycle. Every question will have at least one associated measure in place by the time the evaluation plan is complete.
It is important for the working group to keep in mind the lifecycle phase of the program and its relationship to the evaluation lifecycle. Each stage of the program lifecycle emphasizes different types of evaluation questions. Remember that the goal is to eventually align the program lifecycle and evaluation lifecycle. Depending on where the program is starting from, this may not be feasible in just one evaluation cycle. It may take several evaluation cycles (i.e., years) for the program and evaluation lifecycles to be aligned. When developing evaluation questions, it is important to consider what can be done to move the lifecycle and evaluation phases into better alignment.
In addition to maintaining alignment with the program and evaluation lifecycles, fidelity to the program’s pathway model and purpose statement, language and format have to be carefully considered as the working group continues to brainstorm, revise and edit evaluation questions. Each evaluation question should clearly identify the program element(s) that are being examined (one or more activities, the overall program, specific outcomes, etc.) The phrasing of the evaluation questions matters. Consider the difference between asking whether a program is “associated with a change” as opposed to “causes a change.” The difference could mean a much more (or less) controlled evaluation strategy, and enable a much different claim that could be made at the end of the evaluation cycle.
When writing the evaluation questions, make sure the questions are clearly worded and understandable to someone who is unfamiliar with the program. Be sure to consider (and possibly revisit and revise) the evaluation purpose statement so that the questions make sense given the overall purpose of the evaluation.
The following framework will help to craft evaluation questions for the selected scope of the evaluation. Keep in mind that these are only examples of questions and that there are many “correct” ways to word questions (though some are more appropriate for certain programs than others). To develop an evaluation question, start by identifying a program element (activity or set of activities, outcome, or link) that falls within the evaluation scope in your pathway model. For instance, examples of a Phase 1 evaluation lifecycle question might be:
How well is [ACTIVITY “A”] implemented?
To what extent are participants satisfied with [Activity “B”]?
How useful were the program handouts?
To what extent do participants demonstrate [OUTCOME “X”]?
Phase 2 evaluation questions are typically focused on assessing changes in outcomes associated with participation in the program. Examples of Phase 2 evaluation lifecycle questions might be:
Is participation in the program associated with a change in [OUTCOME “X”]?
How do participants’ levels of [OUTCOME “Y”] compare to non-participants’ levels of [OUTCOME “Y”]?
Phase 3 evaluation questions will typically explore issues of causality or “effectiveness” of the program. These require the use of evaluation designs that are more carefully controlled and/or longitudinal (i.e., follow participants over a longer period of time). Many programs will not reach a Stage 3 evaluation in terms of their program lifecycle phase. Either their purpose is not to demonstrate the effectiveness of the program, or evidence from earlier phase evaluations will cause the organization to abandon the program and try an alternative approach to the problem. For those programs where a Phase 3 evaluation lifecycle question would be appropriate, some examples of questions might be:
To what extent does participation in the program cause [OUTCOME “Y”]?
To what extent does participation in the program cause changes in [OUTCOME “Z”]?
Phase 4 evaluation questions are focused on understanding the generalizability of the program to other contexts, settings, and participants and are typically used when the goal is to broadly disseminate a program that has already clearly demonstrated its effectiveness through rigorous prior evaluations. Examples of Phase 4 questions might be:
Does participation in the program have the same effect on participants of [different ages, different races, living in different settings (e.g., urban, rural, suburban)]?
Can the program be replicated with similar results across multiple settings?
In general, if there are multiple activities that lead to one crucial outcome –or as is more often the case, multiple outcomes that arise from an individual activity– you may certainly use a semi- colon and bullet points to collapse more than one evaluation question into a single sentence, such as:
What is the effect of the program on: [OUTCOME “X”]; [OUTCOME “Y”]; [OUTCOME “Z”]?
Keep in mind that when we are developing evaluation questions we are generally only looking for a few key overarching questions that can guide the evaluation during its next cycle. Imagine we have a program with five activities. We could develop five separate evaluation questions that ask “How well are we implementing Activity 1?”, “How well are we implementing Activity 2?” and so on, or we could shorten this to “How well are we implementing the program?” When stating evaluation questions, use fewer and more general questions if possible.
Typically you want to keep the number of evaluation questions manageable. Three or four evaluation questions will in most cases be sufficient to accomplish a reasonable evaluation in any given evaluation cycle. Use Appendix XXI to help you develop your questions. The remainder of the evaluation plan is constructed to address these questions.
See Also:
Q&A
Q: How will the pathway diagram be used for evaluation planning?
The pathway model will help guide your evaluation by revealing key outcomes and pathways that may serve as focal points for this evaluation cycle. This focusing is important because it is (usually) not feasible to evaluate the whole program in one year. The visual nature of the pathway model offers a uniquely powerful way to see the key “nodes” in the overall process of change, for example, outcomes that may have many arrows “going in” or “coming out”, or both. (Ironically, although the pathway model is built from the knowledge held by the model-builders, it is sometimes only when they step back and look at the resulting visual display that these patterns can be recognized fully.) These model components become candidates for consideration, along with other factors such as stakeholder priorities, lifecycle considerations, and feasibility, in finalizing the evaluation questions.
A pathway model can also help with what is often a challenge for program staff, namely, that funders or other stakeholders want “evidence” about the program’s impact on long-term outcomes which is simply not feasible for program staff to evaluate. In this case, the pathway model is useful for demonstrating the conceptual linkage between shorter-term outcomes (which can be evaluated) and more distant outcomes. The burden of evaluation is reduced if you can then find published research to support these longer-term conceptual linkages.
Q: What if stakeholders are pressuring me to use an evaluation strategy that is out of alignment?
Unfortunately, this is often the case. There are many realities that have to be factored into the decisions in an evaluation plan. However, your understanding of the importance of lifecycle alignment (review that section, if needed) should help you make a case to those stakeholders that their requested evaluation strategy is not appropriate (and is probably an inefficient use of resources!) Sometimes, of course, you will simply have to meet the stakeholder’s needs even if it is not an ideal evaluation strategy. In this case, the benefit of understanding the lifecycle alignment issue is that you will be better able to explain to the stakeholder what the risks or consequences of a misaligned evaluation strategy might be, including the possible misinterpretation of results (whether favorable or unfavorable). You also may be able to do some appropriately aligned evaluation work on a smaller scale, concurrent with the required evaluation.
Q: How are evaluation questions different from survey questions?
Evaluation questions organize and direct your whole evaluation effort. Survey questions are the questions you would ask your participants or sample group to get information that will help you answer an evaluation question. (It may take many survey questions, and perhaps more than just a survey, to properly answer an evaluation question.)
Q: How should evaluation questions be worded?
The more precise your Evaluation Question, the easier your evaluation planning will be. Evaluation Questions should be precise in two ways. First, they need to identify in some detail the program components that are being explored (the activity or activities you are focusing on, the specific kind of knowledge, awareness, behavior change being examined, etc.). Second, the wording of Evaluation Questions must be clear about what kind of causal relationship (if any) between components is being explored. To illustrate the importance of causality, consider the difference between asking “How effective is this program at changing [X]…?” compared to asking “To what extent is this program associated with change in [X]?” The latter means you will be looking for patterns of change in participants; the former means you will also be testing whether your program caused the changes you may observe, which means you will have to have a comparison group or some means of ruling out competing explanations for the change.
Throughout the process of developing Evaluation Questions, keep in mind that the words you choose will determine both the evaluation methods you will need in order to answer the question, as well as the claims you will ultimately be able to make once you have your results.
Presentations
- Length 7:34