Evaluation Planning – 3.07 Data Management and Analysis Plan
The information on how to analyze data is vast and careers are based on this skill alone. This should not discourage or cause fear in the average program practitioner because most programs’ needs can be met through relatively simple analysis methods. Also, programs that don’t have the capacity for their analysis needs frequently have resources available to address this. Having a plan (and tools) in place for analyzing evaluation data, in addition to working with an experienced evaluation facilitator or statistician, can help allay concerns that program staff may have. For an introduction to qualitative, quantitative and mixed methods strategies for manipulating and synthesizing data see http://www.socialresearchmethods.net/kb/analysis.php.
The goal of this step is to articulate and put into writing the plan for managing and analyzing the evaluation data once they have been collected. This section can be quite succinct. Before even considering what kind of statistical techniques might be appropriate, it is important to first step back and think about how the data you plan on collecting will be used to answer the evaluation question(s). The analysis plan should explain how each variable (and corresponding measure) will be used (if you aren’t planning on using the information, then why are you collecting it). Consider what kind of data you will have and how the information could be summarized. For example, if you are using surveys, can you calculate individuals’ scores on those surveys? If you are conducting observations, are you using a coding sheet and can numerical scores be calculated? What would a high score indicate? What would a low score indicate? Could you compute average scores on the surveys? If you are comparing groups, could you compare the average scores for one group to the average scores for the other group? If you are doing a pre- and post-test could you compare the average post-test scores to the average pre-test scores? If you are collecting qualitative data through interviews or focus groups, can you look for specific themes that are relevant to the evaluation question(s)? In the analysis plan section, describe how the data you plan on collecting could be used to address the evaluation question(s).
In addition to thinking about how the data will be used to answer the evaluation question(s) it is also important to begin thinking about how the data will be managed. This includes thinking about what software program if any will be used for data storage, how the data will be organized, how the data will be coded, and how any sensitive data will be kept secure. Be sure to include a discussion of data management issues in the analysis plan section of your report.
As with the previous sections, look at the draft analysis section and ask:
- Is there a clear connection between evaluation questions, measures, sample, design and analysis?
- Is this analysis strategy appropriate for this program’s design which is appropriately connected to the stage of development (lifecycle)?
- Will the analysis answer the evaluation questions?
- Is the selected analysis feasible given the program resources and organizational capacity? If not, how will the organization attain either the assistance or professional development necessary?
When writing the analysis plan for the evaluation consider each evaluation question and describe in detail the data analysis strategies that will be used to address the question, making sure that the analysis strategies are appropriate for generating evidence to answer the questions. Also, be sure to describe a plan for data collection (how measures will be administered), how the data will be handled and stored, and how the data will be organized in preparation for analysis.