Evaluation Planning – 2.02 Program Review
The goal of this step is to gain a firmer understanding of the components and characteristics of the program and its parent organization, including how the program operates and whom it serves. The output will be a fairly comprehensive program description, which will probably be more helpful than the benchmark initial description given in the earlier Evaluation Capacity Surveys. This is an iterative step – any tools that have been used or developed in previous steps in order to describe the program should be discussed and reviewed – such as the surveys and stakeholder map. It is also appropriate to examine any curricula, guidelines, manuals or prior evaluations that might exist in relation to the program.
The following questions may help guide the discussions:
- Why does the program exist?
- What is the program’s mission or vision?
- What are the program’s activities?
- Who participates in these activities?
- When does the program take place?
- Where does the program take place?
Even though these questions are listed from the broadest to the more specific, that is not necessarily the best order to follow when working with a group. Sometimes the best place to start is with what program staff actually do – brainstorm the activities of the program – and then build from there to the more general issues of program mission/vision. This may be best if the program is underway or well-established and the working group includes active program managers. For programs that are being developed, or that are especially mission-driven, it may work best to start with broad themes and work toward discussing specifics. Regardless of the sequence, the discussion should be inclusive and ideally should help build relationships within the working group and with the Evaluation Champion.
This step should be approached knowing that some of these questions can probably never be completely answered to everyone’s satisfaction. There may be some areas of disagreement, or responses for which answers are uncertain, but it is not necessary to have complete agreement at this point. The descriptions outlined should be fairly comprehensive so as to “fill out” a relatively complete picture of the elements of the program. This effort to define the program can be surprisingly difficult. This is an important discussion to have before moving on to the next step, which will be to set some boundaries for the program.
In our experience this step has been an exciting one for many programs. It offers staff the opportunity to step back and reflect on ALL that goes into their program. Sometimes some unexpected connections arise as they recognize the contributions of program components they may previously have undervalued. When you’re doing this step it is probably a good idea to use a white board or black board and draw things out as the group is saying them.
The working group shouldn’t get ahead of themselves —don’t start drawing the logic or pathway models yet. Stay focused on encouraging brainstorming. The next few steps will give shape to what is generated in this step. The result of this step will typically be a tangled mess that shows the level of detail in the program. The complexity of the program may even surprise many in the working group, who previously may have had a rather narrow, organized view of the program.
Q&A
Q: What is the difference between the program mission statement and program description?
A program mission statement is a very short description of the ultimate goals of the program (help prepare students for college, increase energy efficiency of farms through education, etc.), while a program description is a concise yet complete description of the program itself.
Q: What should be included in a program mission statement?
A program mission statement is a statement about the major goals of the program. It should be specific to the program, but at the “big-picture” level. A program mission statement should be only a few sentences.
Q: How long should a program description be?
A program description should be relatively brief, though long enough to cover the details relevant to key stakeholders (often 1-3 paragraphs is about right). Try to imagine describing your program, completely and accurately, to someone riding next to you on an elevator…and you’re not in the Empire State Building.
Q: Why should I include long-term outcomes that I will not be around to see (things that may emerge 20 years from now, for example)?
A logic model is not supposed to just show things you can see or measure. It is meant to convey information and a vision of how the program works. Having a broad and full vision of what your program is about provides valuable information about what motivates the program, who might want to fund it, and what needs it is addressing. The long-term is an important part of that picture. Note also that including long-term, distant outcomes in your model does not imply that you are claiming that your individual program is solely capable of, or responsible for, huge outcomes like community well-being, poverty eradication, scientific breakthroughs, and so on. But you can claim how the program is contributing to broader changes and what you are working toward.