Protocol at a Glance – Steps of the Systems Evaluation Protocol

Richly based in developmental systems, The Systems Evaluation Protocol (SEP) includes three broad Phases – Evaluation Planning, Evaluation Implementation, and Evaluation Utilization.

The Preparation stage is intended to acquaint the participants with this Systems Evaluation Protocol (SEP) and the Evaluation Partnership arrangement, and identify current evaluation resources. The Preparation stage involves the following steps:

Enter the System: Connect with key decision makers of the organization to discuss commitment to evaluation planning and responsibilities.
Memorandum of Understanding: Work with key decision-makers in the organization to create a written document that: describes the roles and responsibilities of participants in the evaluation project; details the expectations for the evaluation consulting team, partner site administrators and staff; and provides a timeline of project activities and completion.
Internal Stakeholders: Identify people in the program/organization who should be involved or consulted in evaluation planning.
Working Group(s): Identify those who will play a key role in developing the evaluation plan.
Assess Evaluation Capacity: Identify the resources available within the organization and within the program – the degree of evaluation training the staff has already received, information technology resources, and evaluation policies.

Stage 2 – Program Model Development

The Modeling stage is intended to enhance participant knowledge of evaluation concepts, and identify how their program “works”. The Modeling stage involves the following steps:

Stakeholder Analysis: The process of identifying, and describing the perspectives of all of the potential people and/or organizations that have a stake in the program and its evaluation
Program Review: Gain a firm understanding of the components and characteristics of the program including how it operates and whom it serves. The program mission statement is a description of the overarching or long-term goals specific to the program being described.
Program Boundary Analysis: Program definition and boundary analysis involve defining the structure and elements of a program; specifically, what is considered to be part of the program as opposed to activities or elements that may be just outside the definition of the program.
Lifecycle Analysis: Lifecycle analysis refers to the process of characterizing the history and current phase of development of a program and its evaluation. The program lifecycle is the individual course a program takes as it evolves, changes, or remains the same over time. Typically, programs earlier in the lifecycle are smaller in scale and more variable in their implementation. Later phase program often, though not always, become more stable and reach a wider audience. Understanding a program’s lifecycle history is a critical part of planning for its evaluation. The evaluation lifecycle exists in parallel with the program lifecycle. Early lifecycle evaluations are characterized by rapid feedback, basic designs, and less of a focus on external validity and reliability. Later lifecycle evaluations are typically more involved, build upon prior knowledge of the program, and allow for broader claims.
Logic Model: A logic model is a representation of the thinking behind a program. Logic models come in many different formats. Some are more graphical than others. They all include the basic components of program activities and outcomes. Many also include program inputs and outputs. Some include program context and assumptions. Overall, logic models represent the theory of change behind a program – the ways in which the program planners imagine the program activities have an effect on the program’s intended outcomes. Generate an initial logic model including the assumptions, context, inputs, activities, outputs, short-, mid-, and long-term outcomes.
Pathway Model: A pathway model is a type of logic model. It is a graphical representation of the relationships between the activities, outputs, and outcomes that make up a program. Pathway models communicate the “story” or “theory of change” of a program. They are the essential foundation for determining the scope and questions that guide the evaluation of the program being modeled. Use the logic model as a basis for articulating clear and direct linkages between program activities and outcomes.
Evaluation Scope: Determine the specific components of the pathway model that will be the focus in the upcoming evaluation cycle.
Program-System Links: Introduce tools and strategies for finding similar programs and shared outcomes, develop research support by drawing on literature and on resources in the systems within which the program exists
Reflection and Synthesis: Finalize the logic and pathway models including reviewing the program logic model, assessing the model from the perspectives of key stakeholders, reviewing the Program Boundary Analysis, reviewing the Program and Evaluation Lifecycle Analyses, and revising the models as needed. This step also involves integrating relevant research literature as it relates to the causal pathways that have been articulated in the Pathway Model.

Stage 3 – Evaluation Plan Creation

The third stage, “Evaluation Plan Creation,” focuses on the creation of an evaluation plan that will guide the implementation of the evaluation. The Evaluation Plan Creation stage involves the following steps:

Evaluation Plan Introduction: Present and discuss the components of an evaluation plan.
Evaluation Purpose: The evaluation purpose statement is an introduction to the evaluation plan document that briefly describes the scope of the evaluation being planned, how it fits in with both prior as well as intended future evaluation work, as well as a brief summary of the methodology and intended use of the evaluation. Think of the evaluation purpose statement as an executive summary of your evaluation plan. This purpose statement will be useful for readers of the plan and as a touchstone document for people implementing the plan to make sure that it stays on track.
Evaluation Questions: Evaluation questions are the broad inquiries about the program that the evaluation will seek to address. The language used in evaluation questions has broad implications for both the methodology of the evaluation as well as the claims that can be made as a result of completing the evaluation. Develop evaluation questions based on the logic and pathway models, lifecycle analysis, stakeholder analysis, and systems insights. The evaluation questions will function as the core determinants of all the evaluation plan components.
Measurement and Measures: Measurement is the process of collecting information systematically, using appropriate methods and/or tools, to address an evaluation question. At this stage in the evaluation planning process, this involves selecting a data collection strategy that fits the evaluation question you’ve identified, and then finding or developing whatever tools (often referred to as “measures”) are needed in order to fulfill that strategy. Identify measures already being used in evaluating the program and assess them for quality and feasibility; identify other existing measures that might fit the program evaluation needs; and/or develop any new measures that are needed.
Sampling Plan: In the context of program evaluation, the sample is the group (of people, objects, etc.) that you will collect data about in order to address the evaluation question(s). For some evaluations, it will be appropriate to collect data about the entire population of interest (for example, all program participants); for other evaluations, the sample may be a sub-group of the population of interest, selected based on the needs of the evaluation and practical considerations (feasibility, resources, etc.).
Evaluation Design: The evaluation design lays out the sequence of program implementation and data collection. In its most basic form, the evaluation design tells when information is collected and from whom. For example, an evaluation design might specify a “pre-post” data collection strategy, a “post-only”, or might include plans for collecting data from a comparison group, etc.
Data Management and Analysis Plan: Data management is the ongoing process of recording, documenting, tracking, securing, and organizing data that is collected during the course of an evaluation project. It’s important to plan this out in advance, in order to ensure that nothing gets lost as data come in, that there is a record of when it came in and through what means, and that data are stored in a way that is retrievable and organized to facilitate the analysis. Analysis is the process of describing and making meaning from data. The appropriate analysis strategy for a given evaluation depends on the evaluation question(s), the type, quantity and quality of the data gathered, as well as the insights and potential claims that emerge from initial review of the data.
Evaluation Reporting and Utilization Plan: “Reporting” refers to the process of communicating results and recommendations to stakeholders. This includes internal (program staff, participants) and external (funders and administrators) stakeholders. “Utilization” refers, more generally, to the process of using evaluation results to make decisions, if any, about program design, support, staffing, and so on.
Implementation Plan and Timeline: The evaluation plan timeline outlines the program and evaluation schedule for the evaluation. To ensure smooth progress, overall feasibility, and availability of staff for various tasks, it is important to include as much detail as possible. For example, a good timeline would include tasks such as measure development, pilot-testing of measures, and training of data collectors and/or data entry staff as needed, in addition to the obvious such as program implementation, data collection and analysis.

Phase II: Evaluation Implementation

Stage 1 – Preparation for Implementation

The Preparation stage of the Implementation Phase includes steps that need to be taken before data can be collected, that are not included in the formal evaluation plan. As in other phases these steps do not necessarily have to be concluded in this order. The steps include:

Reconsider Alignment and Context Appropriateness: Double check that all plan components, methods, tools and strategies are appropriately matched to the evaluation questions and to the program. Revise as necessary.
Address Ethics for Human Participants: Review ethics considerations, set rules for consent and confidentiality, obtain IRB approval if necessary, assess cultural responsiveness.
Set Up for Data Collection: Obtain and prepare online tools and/or other materials as necessary.
Set Up for Data Management: Create/revise plan for when, how, and by whom data will be entered, set up spreadsheet(s) and/or other data organizers, create coding schemes as necessary.
Conduct Pilot Tests: Double check credibility, accuracy, usefulness and feasibility of all aspects of the evaluation plan (sampling, measures, data collection, data management and analysis) as appropriate. Revise as necessary.
Train Data Entry and Analysis Staff: Develop training materials and conduct training to ensure attention to detail in data entry and documentation – including a QA check methodology.
Train Data Collectors: Develop training materials and conduct training in order to ensure quality and consistency in measure administration, data handling, and documentation.

Stage 2 – Data Collection and Management

The second stage of the implementation phase is data collection. This includes accessing, collecting, and entering data as described in the following steps:

Access Data Sources: Recruit, select and assign participants and any comparison groups. Obtain consent and offer incentives as necessary. Summarize how and when data was collected as well as response rates and overall data counts.
Collect, Enter and Secure Data: Administer measures; follow data collection and management plans.
Review and Clean Data: Quantitative data: Identify missing and/or inaccurate data and either correct or remove these entries. Qualitative data: Read, transcribe, edit and label narrative data as needed for planned analysis.
Create Codebook and Code or Categorize Data: Quantitative data: Develop a key for how variables and responses will be labeled/identified, as needed for the software being used. Qualitative data: identify themes or patterns, create categories and record definitions.
Explore and Summarize Data: Quantitative data: Calculate descriptive statistics. Assess whether planned statistical analyses are appropriate (checking sample size, data quality, distributions.) Qualitative data: summarize patterns.
Transform Data: Combine data, treat as different variable type and or calculate “change score” as necessary to allow for appropriate analysis

Stage 3 – Data Analysis

The analysis stage includes tasks necessary to convert raw data into interpreted results. These include:

Conduct Statistical Tests: For quantitative data as needed, calculate correlations, statistical significance of difference or change, cross tabulations, regressions, etc. as indicated in the evaluation plan, and as possible based on results of descriptive statistics.
Synthesize and interpret data: Quantitative data: Use descriptive and inferential statistics results to explain findings and draw conclusions. Qualitative data: synthesize and interpret results. Record process.

Phase III: Evaluation Utilization

Stage 1 –Feedback and Utilization

Internal and External Feedback: Report to program staff (and volunteers, if appropriate) formally and/or informally, Report to program and organization leadership,Report to external stakeholders who have an interest in the results.
Utilization: If feedback and reporting are done well, in meaningful and relevant ways, than it is more likely that the evaluation results will effect some change, such as program improvement, creating new programs, soliciting funding, building support.

Stage 2 – Reflection, Synthesis, and Revision

Review Evaluation Implementation and Decisions :reflect upon the process and decisions made on the fly; document suggested issues or changes
Review and Revise Program Models:revise models to incorporate new insights, consider documenting lifecycle changes.
Revise Scope and Evaluation Plan: document for next stage of evaluation.

Protocol at at Glance