We use a systems perspective as the framework for developing evaluation capacity, enhancing evaluation quality and ultimately improving programs. Several theoretical systems constructs have guided our work including complexity theory, evolutionary theory and natural selection, general systems theory, ecology, system dynamics, developmental systems theories, and ideas on research-practice integration (including evidence-based practice and translational research).
The field of evaluation can itself be viewed as an evaluation system, and refers to the comprehensive and integrated set of capabilities, resources, activities and support mechanisms for conducting evaluation work. This should not be confused with systems evaluation, which refers to the assessment of the functions, products, outcomes and impacts of a system (set of programs, activities or interventions). Systems evaluation is an approach to conducting program evaluation that considers the complex factors that are inherent within the larger “structure” or “system” within which the program is embedded. Systems evaluation provides both a conceptual framework for thinking about evaluation systems and a set of specific methods and tools that enhances our ability to accomplish high-quality evaluation with integration across organizational levels and structures.
In the Systems Evaluation Protocol (SEP) we have tried to integrate principles associated with systems theories into program evaluations in order to assure that programs that use it will incorporate such principles when developing program pathway models , identifying key pathways and nodes (outputs and outcomes), determining the boundary conditions for program models, assessing program lifecycles, and selecting evaluation designs that are appropriate to program evolution.
It was the examination of some of our beliefs about evaluation that led to our work in systems evaluation. Here are some of our assumptions about evaluation, which were precursors to developing a systems perspective of evaluation:
- Evaluation is a dynamic on-going process that is applied to programs that are (themselves) dynamic.
- Evaluation is a form of feedback that can be used for program or organizational improvement (see sidebar “Driving with your Eyes Open”).
- A formalized and standardized evaluation framework utilizing a systems perspective is needed to develop consistent and high-quality evaluations.
- Program Evaluation is best viewed as a three-phase process, beginning with Evaluation Planning, followed by Evaluation Implementation, and completing the cycle with Evaluation Utilization (which leads back to planning for the next iteration of the program). Figure 1 provides a pictorial representation of the Phases of Evaluation.
- The Evaluation Planning phase is a critical step for systems evaluation. This is where introducing the systems perspective will shape how evaluators and program staff view the program, program boundaries, stakeholders, and its evaluation.
- Because of the need to evaluate multiple programs within an organization, there is value in developing systems for evaluation that encompass multiple programs, rather than conducting isolated evaluations of individual programs.
Figure 1. The Cycle of Evaluation 
A Systems Perspective on Programs and Program Evaluation
Below are some key points that we believe should anchor a systems evaluator’s perspective.
- An organization is a system, and is composed of a collection of parts (see sidebar “Greater Than the Sum”). Systems involve parts, wholes, and their interrelationships.
- Any program necessarily occurs within a complex environment composed of “nested systems”. “Nested systems” refers to the structure where a system is embedded within another system, which is embedded within yet another system. For example, Ms. Smith’s third grade class is a system within the entire third grade, which is part of the elementary school, which is part of the school district, which is part of the state school system, and so on.
- Human systems are dynamic (see sidebar “The Rock and The Bird”). A dynamic system is necessarily composed of evolving relationships and programs.
- Consequently, evaluation needs to be dynamic and should change in order to successfully link with the needs and maturity of the program being evaluated (see sidebar “The Flower and the Bee”).
- Programs have lifecycles, and move through various phases. Different evaluation approaches are appropriate for different program phases. In other words, like programs, evaluations should evolve (see sidebar “In the Course of a Lifetime”).
- Many organizations have multiple programs and many programs are implemented in multiple organizations – perhaps the third grade consists of Ms. Smith’s class, Mrs. Jones’ class, and Mr. Perez’s class, and at the same time there are many schools with numerous other third grade classes. But each school also has multiple grade levels, and you quickly see that systems can rapidly become complicated.
- Nested and dynamic systems create an environment where there are multiple perspectives. Each stakeholder has their own perspective (see sidebar “Eye of the Beholder”).
- Each stakeholder of a system has specific expertise, and brings a distinct perspective and motivation for evaluation. This comprehensive set of stakeholders should be identified and included in the evaluation design and/or process and be viewed as partners in the evaluation. Placing the program being evaluated at the center of a circle, and surrounding it with stakeholders placed closer to or further away from the program of focus – depending upon their relationship to that program – can create a representative map of stakeholders (see Protocol step 2.01 – Stakeholder Analysis). The perspectives of these stakeholders are described in relationship to their association with the program. A stakeholder within the organization would have a local perspective, and one that is placed at the outermost ring would have a more global perspective (see sidebar “The Local and the Global“).
Terminology
Before proceeding further we would like to clarify some of our hierarchical terminology. This is not meant as a standard hierarchy applicable to all systems, but is simply a convenient hierarchy for describing how the steps of this Protocol might be applied when working with an organization. The broadest level is that of the system (rather like a network of organizations). Within that there are assumed to be multiple organizations. Each organization may have one or more programs. Programs in turn are made of up multiple components, including inputs, activities, outputs and outcomes (see Figure 2).
Figure 2. Hierearchy of Systems for this Discussion
