Evaluation Implementation – 2.02 Collect, Enter, and Secure Data
Although actually collecting data you have been planning on may seem like one of the most important steps in an evaluation, without a well thought out strategy for storing, securing, and backing up that data, there is a high risk of wasted effort.
Before any data is collected, the working group should agree on and set up a system for storing the data as soon as it is collected. Along with the form of the data (hand written, electronic, audio, video, freshly collected or gleaned from previously gathered archives), the working group should consider the intended analysis strategy when choosing a storage strategy. For example, working groups using a pre-post closed-ended survey design will likely need to set up a spreadsheet that will track responses to each of the items (columns) and the responses for each individual (rows). If the analysis plan calls for comparing pre- and post- scores on the level of the individual, the working group will need to decide how to identify and match pre- and post- responses. If the data is in narrative form, descriptive information on participant characteristics may be relevant for answering evaluation questions. If so, qualitative databases should be set up to receive participant descriptors (such as gender, race or age) stored together with their narrative data.
If the data is in a non-written format, such as tape-recorded interviews, the working group will need to decide on a method for storage as well. Typically, interviews will need to be transcribed. Depending on how that transcribed data will be used, the working group may want to identify and install software used for coding and sorting text and/or content analysis.
“Securing” the data refers to protecting participant’s anonymity or confidentiality and ensuring that all data – in any form – is backed up in case of loss. Electronic files should be backed up on an external hard drive or disk and any original paper versions of the data should be kept in a locked file. Data entry and back-up should be done as data is collected so that the raw data are not lost and the working group does not accumulate a larger quantity of data than they are feasibly able to enter efficiently later on. Those involved in data collection should carefully document decisions made along the way, especially any changes to initial data collection protocols.
How do you know when data collection is complete? Whether your sampling strategy is probabilistic or non-probabilistic, where you planned a pre-determined sample size, you will know data collection is complete when you’ve attempted to collect data from everyone in your planned sample, and you have obtained a sufficiently high rate of response. If response rate is too low, this may affect your ability to make certain claims from your results. In the case of an alternate sampling strategy (for example, one that calls for data collection until you are at thematic “saturation”) your data collection is complete when fresh themes stop appearing, and each successive data collection merely yields repeats of themes from earlier collections.
At the end of one complete run through data collection, you may want to review data collection notes for any needed changes to the evaluation plan and/or logic model.