Evaluation Planning – 3.07 Data Management and Analysis Plan
The information on how to analyze data is vast and careers are based on this skill alone. This should not discourage or cause fear in the average program practitioner because most programs’ needs can be met through relatively simple analysis methods. Also, programs that don’t have the capacity for their analysis needs frequently have resources available to address this. Having a plan (and tools) in place for analyzing evaluation data, in addition to working with an experienced evaluation facilitator or statistician, can help allay concerns that program staff may have. For an introduction to qualitative, quantitative and mixed methods strategies for manipulating and synthesizing data see http://www.socialresearchmethods.net/kb/analysis.php.
The goal of this step is to articulate and put into writing the plan for managing and analyzing the evaluation data once they have been collected. This section can be quite succinct. Before even considering what kind of statistical techniques might be appropriate, it is important to first step back and think about how the data you plan on collecting will be used to answer the evaluation question(s). The analysis plan should explain how each variable (and corresponding measure) will be used (if you aren’t planning on using the information, then why are you collecting it). Consider what kind of data you will have and how the information could be summarized. For example, if you are using surveys, can you calculate individuals’ scores on those surveys? If you are conducting observations, are you using a coding sheet and can numerical scores be calculated? What would a high score indicate? What would a low score indicate? Could you compute average scores on the surveys? If you are comparing groups, could you compare the average scores for one group to the average scores for the other group? If you are doing a pre- and post-test could you compare the average post-test scores to the average pre-test scores? If you are collecting qualitative data through interviews or focus groups, can you look for specific themes that are relevant to the evaluation question(s)? In the analysis plan section, describe how the data you plan on collecting could be used to address the evaluation question(s).
In addition to thinking about how the data will be used to answer the evaluation question(s) it is also important to begin thinking about how the data will be managed. This includes thinking about what software program if any will be used for data storage, how the data will be organized, how the data will be coded, and how any sensitive data will be kept secure. Be sure to include a discussion of data management issues in the analysis plan section of your report.
As with the previous sections, look at the draft analysis section and ask:
- Is there a clear connection between evaluation questions, measures, sample, design and analysis?
- Is this analysis strategy appropriate for this program’s design which is appropriately connected to the stage of development (lifecycle)?
- Will the analysis answer the evaluation questions?
- Is the selected analysis feasible given the program resources and organizational capacity? If not, how will the organization attain either the assistance or professional development necessary?
When writing the analysis plan for the evaluation consider each evaluation question and describe in detail the data analysis strategies that will be used to address the question, making sure that the analysis strategies are appropriate for generating evidence to answer the questions. Also, be sure to describe a plan for data collection (how measures will be administered), how the data will be handled and stored, and how the data will be organized in preparation for analysis.
Q&A
Q: Where can I learn more about how to do analysis?
Quantitative Analysis:
One good, succinct source on analyzing quantitative data is: Taylor-Powell, E. (1996). Analyzing Quantitative Data. Retrieved May 5, 2015, from University of Wisconsin-Extension Cooperative Extension, Program Development and Evaluation Unit Web site: http://learningstore.uwex.edu/Assets/pdfs/G3658-06.pdf
Another useful source is: Research Methods Knowledge Base: Trochim, William M. The Research Methods Knowledge Base, 2nd Edition. Internet WWW page, at URL: http://www.socialresearchmethods.net/kb/analysis.php
Qualitative Analysis:
Although it is rather lengthy and in-depth, a very good, highly readable source on qualitative data analysis (and qualitative evaluation and research in general) is: Patton, M. Q. (2015). Qualitative Research & Evaluation Methods. Thousand Oaks, CA: Sage.
For a more succinct source (12 pages) specifically about qualitative analysis, see: Taylor-Powell, E., & Renner, M. (2003). Analyzing Qualitative Data. Retrieved May 5, 2015, fromUniversity of Wisconsin-Extension Cooperative Extension, at http://learningstore.uwex.edu/Assets/pdfs/G3658-12.pdf
Q: How is evaluation data analysis influenced by program lifecycle?
Evaluation data analysis is not directly influenced by program lifecycle, but it is heavily influenced indirectly, by way of the choices that go into the evaluation purpose, evaluation questions, and, especially, measurement and design sections of your evaluation plan (which are very directly related to your program’s lifecycle stage). In other words, your program lifecycle strongly influences your measurement approach, and your measurement approach, in turn, strongly influences your data analysis plan. As such, analysis for earlier lifecycle programs often involves summarizing, averaging and comparing data in relatively simple ways and can be done internally; analysis for later lifecycle stage programs will tend to be more difficult and complicated, due to the statistical tests required–here, it may be advisable to seek external help from a statistician or research methodologist.
Q: What determines which analysis strategy I should use?
Analysis is essentially about making sense of the data in such a way that you answer or otherwise gain insights in response to your evaluation questions. So it is important that your plan for analysis is closely aligned to your evaluation questions and especially to your chosen measurement strategy. For example, if your evaluation question is about how participants experienced the program, then your measurement would likely include interviews. Then, your analysis strategy can be selected from among the options which exist for analyzing interview data (many of which involve categorizing the data, or text, into codes). If your evaluation question is about whether or not your program caused a desired outcome to occur among participants, your measurement and design would likely have involved a pre-post quantitative survey administered to two randomly assigned, representative samples of participants and non-participants. In this case, your analysis will require specific statistical tests, usually performed using a statistical software package such as SPSS. In such cases, it is common to seek outside assistance from a statistician.
Q: What are the most common considerations when planning for data management?
When planning for data management, consider (1) what your data will be like, and (2) what you will want to do with your data to analyze them. For the first consideration, ask yourself: Will I have qualitative data, quantitative data, or both? How much data will I have? Will there be a need for data entry (e.g., entering the responses to a paper-based survey into a spreadsheet or database). If so, who will do so, following what guidelines? What will I need to do to ensure that data are well organized and safe (especially if anonymity or confidentiality must be maintained)? For the second consideration, ask: Will I want to aggregate data or keep it all separate? What variables or factors will I summarize, and how? What, if any, comparisons among variables will I want to make? Because there are so many different possible answers to the set of questions above, it is difficult to prescribe any particular way of managing data. However, considering these questions (and testing out whatever approach you decide on, to be sure it will work for your data entry and analysis) is really important.
Generally, data are managed with computer software, such as an Excel spreadsheet or a specialized data management and analysis program. If using a spreadsheet program, the spreadsheet should be set up where rows contain responses (e.g., individual respondents’ survey answers) and the columns are variables (e.g., gender, individual survey questions, etc.) However, remember that hard copies of data (paper surveys, audio recordings of interviews) should be retained as a backup or for future review, and protected for confidentiality, even after the data have been entered into a computer program.
Presentations
- Length 5:30
- Length 1:59
- Length 4:56