I.2.03 Program Boundary Analysis

Evaluation Planning – 2.03 Program Boundary Analysis

Program names are simply labels given to a set of related activities and goals. Understanding the meaning behind these labels and constructs is the purpose of this exercise. This task builds off the previous one, and is frequently one of the most difficult steps because there are many ways to define, structure, and parse the elements of a program. The goal of this step is to determine what is “in the program” and what is “outside of the program”. While this may seem simple or unnecessary at first glance, the boundaries of a given program are seldom made explicit and frequently vary with the context. Different people within the same organization often draw a program’s boundaries quite differently, and other stakeholders define it still more diversely.

The Program Boundary Analysis exercise is focused on language and terminology, and asks participants to clarify and make precise the statements they make about their program and may take for granted. Boundaries are artificial constructs created by humans, and now is the time to question exactly what those boundaries are. The output should be an improved program description, improved understanding of the program, and the possibly new common ground for those who participated in this conversation.

Keep in mind that staff cannot realistically evaluate every aspect of their program during this evaluation cycle, but they may evaluate different parts at different times. Participants should be encouraged to be broad and to include all the things that matter and go into or come out of the program even though they may not be evaluated in this evaluation cycle. Below are several guidelines for thinking about what the program is, and developing a comprehensive program description.

Ask the program staff exactly what it is that they do, and ask other stakeholders what they think of as the program effort. The chances are that you will get different answers, but all of those things are part of clarifying the boundaries of the program.

What are the “elevator stories” of the program – if someone were to describe the program in two sentences during a short elevator ride, what would they say?
Is there a formal definition of the program?” If there is such a definition this can be the working program definition and boundary. If not, do they have any informal descriptions of it? Did you come up with a better definition during the previous step (2.02 Program Review?) Examples might include program descriptions on their website, or in various types of promotional literature. Do these say the same thing, or are they different? Synthesize a working definition from these descriptions and move on.
Another thought to consider is what statements would they like to make about their program? If they want to say that their program increases community health, then the community should be within the boundaries of their program.
If they were to package up the program and hand it over to someone else, what elements would it consist of? Usually the program staff training is not considered part of the program itself, assuming that the staff come to the program adequately trained to carry out the program (note, this would be one of your assumptions for later on in the Logic Model).
Look at the Map of Stakeholders. How would the various stakeholders define the program? Would strangers, reporters, board members, funders, etc. be able to “get” what the program is doing if they read your description? Would they include or exclude elements that have been included in the description? Have the working group take the perspectives of different stakeholders or stakeholder groups and play the roles of those people. How are they similar to or different from how another stakeholder would describe the program? Perspective taking is absolutely critical to understanding the program. For example, how does a teacher define “school”, as opposed to a student, the parents, or even the government? Why do different stakeholders, including program staff, describe the program differently (i.e. draw the program boundaries in different places)? What different information or values inform these differing descriptions? Can the group reconcile these differing views in a meaningful way?
Do you describe your program differently in internal communications (such as memos or program plans) than in external communications (such as websites or mailings)? Does the program description include the information from both types of sources?
How would the program evaluation be affected if elements were included or excluded from the program description? For example, what would happen if you included or excluded activities that are aimed at different audiences but share similar resources and goals?
Look back at any previous program logic models. Is the information in there (including inputs, participants/audience, activities, outcomes, and assumptions) still in the picture with this new program description? If you have excluded or included elements, are you able to justify that?
Write a new program description with the boundaries clearly established, and in language that would be understandable to someone with no knowledge of the program.

One way to think about the drawing of common boundaries is to consider it from the perspective of a biological scientist. Imagine that the program is a specimen and you are examining it under the microscope. You can zoom all the way in and see all of the intricate details of a portion of the specimen. Alternatively, you can zoom all the way out so that you can see the broader structure of the whole specimen. Or, you might want your level of magnification to be somewhere in the middle so that you maintain a sense of the broader structure while still gaining an understanding of some of the more nuanced details.

It is important to note that there is no one “correct” way to draw program boundaries, but there are ways of drawing boundaries that will be more or less useful for your purposes. To continue the biology example, the boundaries of the system may be the incubator full of petri dishes, a single dish, a single colony, a single cell, or a cell organ. These are all systems that are nested within other systems, and the job of the working group is to define their boundaries for their purposes. They should be guided by the kinds of statements they wish to be able to make at the end of the analysis (i.e. “X% of sample dishes share evidence of…” vs. “cell walls in the treated colonies showed the effect of…”).

This conversation can help guide the efforts toward a tighter boundary with finer detail, or a wider boundary with broader components. The objective is to get the definition of the program to a point where it is not too detailed and not too vague. Here another helpful metaphor from the story of Goldilocks and the Three Bears comes to mind – Goldilocks’ quest to find the perfect bowl of porridge which was neither too hot nor too cold, but just right. Also demonstrated in this example is that we did not focus on the whole story – instead we chose to focus on the porridge (rather than on the chairs or the beds, or on the Bears’ return).

As one last example of different possible boundaries, consider how someone might define “school”? Does school begin when the first class begins, or does it begin when a student walks out of their home or gets on the bus? Is lunch part of school? Does school end after the last class, or when a student gets home. Are after-school activities – such as drama or sports – part of school? The selection among these options is ultimately a practical one driven by what participants want to do with their evaluation results. Under the circumstances, the decision about boundaries tends to get revisited several times as the working group moves through the modeling and evaluation planning steps.

For a program that is designed to “train the trainer” a key boundary question is whether the program includes the activities done with the trainers or only the subsequent activities with their students. Another classic boundary question is whether program planning and preparatory training is considered part of the program, or do program activities only involve what happens after program staff are trained?

The output of this whole step should be a newly revised and even more precise program description upon which the working group has reached consensus. (See Appendix VIII for guidance). At this point they should agree on what activities are considered to be part of the program, and which ones are not. In addition, the working group should develop a mission statement for the program. Often, the organization will have an existing mission statement for each of its programs. The working group should keep a copy of this statement with their evaluation files. If no mission statement exists, or the existing statement needs revision, this should be done as part of this step.

Some groups get very ambitious about defining the program boundaries, and they may pay a price later when trying to evaluate it. If their boundary includes an extended view of the program, their whole logic model becomes much bigger and more complicated. Some groups will try to scale back and define more precisely what they are trying to do and they run the danger of leaving key program components out. This is all an iterative process that will be revisited throughout the other steps. Redefining the boundaries will likely continue as you work through other steps, such as when completing the logic model, when determining the evaluation scope, or prior to the next evaluation cycle.