2.08 Program-System Links
No program is an island. That is, it is a rare program that is so unique that it cannot learn or benefit from knowing how others view the program and have addressed similar challenges in the past.
At some point each program should examine what others are doing that might be related to or inform the program and its evaluation, and integrate this information into the program’s pathways. This step involves turning the working groups’ attention to other programs and to the research literature. These sources may suggest measures that could be used for the current evaluation or provide evidence that could help support the logic conveyed by the links in the pathway model.
Ask the working group if there are other programs like this one. These may be in the same organization or in another local organization, or may be physically distant. Are there other programs with similar or shared outcomes, even if the program activities are very different? What evaluation tools are used for comparable programs, and are they available for this program to use? What research supports the logic represented in the pathway model? Are there measures in the research literature that may be useful for this evaluation? To find similar programs, draw upon professional connections as well as links or resources that may be available from the larger systems that this program is part of – national organizations, professional associations, and so on.
Don’t rely only on similar programs. The working group should search the scientific literature for current research that is being conducted in their general program area. This can be a time-consuming task, but it is also an important one. (In our case, we assist in this step by having undergraduate students work on conducting literature searches, and we are making the aggregate results available to many similar programs.) Having an understanding of the evidence that already exists can help the program to focus its evaluation efforts and to identify well-researched and widely used measures to use. Depending on the program’s resources and capacity, as well as the interests of its stakeholders, organizations may choose to devote more or less time to this step.
For example, consider a course that teaches youth how to reason scientifically. Suppose that a short-term outcome for this course is that youth will be able to reason scientifically about everyday problems. An example of evidence for this causal relationship could be a research article that shows that courses in scientific reasoning methods that utilize relevant everyday topics may enhance the transfer of scientific reasoning to other problems/situations (e.g. Williams, et al., (2004). Thinking Like A Scientist About Real-World Problems: The Cornell Institute for Research on Children Science Education Program. Journal of Applied Developmental Psychology, 25:1, 107-126).
Once the working group has surveyed resources from other programs and the literature, it is useful to visually depict this on the pathway model (see Figure 15). Relevant literature (evidence) that relates to the causal relations can be “mapped” onto the pathway model (we call this evidence mapping). Similarly, relevant measures that relate to specific outcomes can also be “mapped” onto the pathway model (we call this measurement mapping). See the Activities: Mining the Model Parts 1 and 2 for an example of how to conduct the mapping exercise. Results from the evidence mapping and measurement mapping may lead the working group to want to revise the scope of the current evaluation; particularly if there is pressure to report on longer-term outcomes. For example, if the working group is able to identify strong research evidence that already links one of their medium-term outcomes (e.g., “increase student excitement about science and engineering topics”) to one of their long-term outcomes (e.g., “increase the number of individuals who choose science and engineering careers”), they may decide to focus the current evaluation on demonstrating that program activities lead to that medium-term outcome. If the evaluation can provide evidence that program participation increases student excitement about science, they can then logically argue (based on the appropriate research evidence) that the long-term outcome should follow.
The working group should keep in mind that they may not be able to find evidence or measures for all of the relations or elements that have been articulated in their pathway model.
See Also:
Figure 15. Literature Mapping
