Evaluation Implementation – 1.03 – Set Up for Data Collection.
As part of planning the evaluation, you will have selected or created a tool or tools for data collection. The design of your instrument should make sense in light of what broader evaluation questions you want answer, and what analyses you want to be able to do. Your data collection tools might include a paper survey, an electronic log, or interview data captured by a recording device. In any of these cases, as well as a myriad of others, there is some work that must go into preparing these tools before they can be used.
Paper documents must be formatted correctly, including titles, demographic information about the respondent, space for respondent identification if applicable, and the use of a font size that is readable and leaves ample space for the participant to write a response. When these documents are ready to print, the working group should consider how they will be organized. Will the measure be used twice, in a pre-post design? If so, the working group might consider printing the pre- and post- copies on different colored paper for ease of identification. Are different tools being used for different respondent groups? If this is a matched pre-post evaluation with anonymity, have you included a unique identifier code so that an individual’s pre- and post- responses can be linked to each other? A similar solution might be useful in this case as well. One critical tool that is often forgotten by survey implementers is the writing utensil.
In the case of an electronic (including online) tool, parallel considerations must be made. Is the tool formatted in a way that makes sense on the screen? How will respondents know that they will need to “scroll down” below the boundary of their monitor to see the full survey? How will they know when they are done? How can the working group ensure that each respondent’s data gets saved accurately? How will anonymity be maintained if that has been promised to participants?
Beyond these two common examples, other data collection methods may involve additional materials. These might include:
- Tape recorders
- Video equipment
- Cameras
- Counting devices
- Art supplies
- Large format paper for discussion notes
- Reusable interview or discussion guides
In setting up for data collection, no matter what the method, the working group should consider each of the following (where applicable):
- How are respondents identified (if they are)?
- How will pre- and post- measures be matched?
- How can paper materials be best organized and identified?
- Are there instructions? Are they adequate?
- Is there written assurance of confidentiality or anonymity?
- Are all necessary materials on hand (pencils, recorders, paper, etc.)?
- Are back-up materials available (memory cards, batteries, etc.)?
- Have online/electronic tools been tested and de-bugged?
Depending on the method in question, there may be other more obvious questions that should be asked. Each evaluation context is unique and will pose its own data collection challenges. This step is meant to help the working group brainstorm and discuss these logistical considerations before data collection begins so that the process will be efficient and chance of error will be reduced.