Determine Collection Method
Below is a summary of the most commonly used methods in evaluation:
- Document analysis
- Focus groups
- Case studies
Carefully consider each option before deciding which method(s) to use.
In the figure below we provide an abbreviated overview of each method. A more detailed description and explanation of each method along with its unique benefits and challenges is located below the figures.
Overview of Evaluation Data Collection Methods
|Method||Primary Data Type||Examples of Data Sources||Advantages||Challenges|
|Document analysis||Primarily quantitative but can also collect qualitative data in the form of documented narratives.||Program applications, finances, memos, minutes, etc.||Provides comprehensive and historical informationDoesn’t interrupt program or participants’ routineInformation already existsInexpensive||Can be time-consumingInformation may be incomplete or unreliableData is restricted to what already exists; not flexible|
|Surveys||Primarily quantitative but can also collect qualitative data through open-ended or free response questions.||Participant questionnaires, surveys, and checklists||Easy to compare and analyzeAdminister to any size sampleCan collect a lot of data at onceParticipant anonymitySample surveys/ questionnaires already existInexpensive||Possible response bias; wording can bias participants’ responsesPossible sampling biasSometimes difficult to get a high response rate|
|Interviews||Primarily qualitative but can also collect quantitative data by numerically coding interview responses and/or observations.||Documented questions and answers with program participants||Get full range and depth of informationPersonalDevelops relationship with participantAllows for participant flexibility||Need a good and trained interviewerCan be time-consumingCan be hard to analyze and compareInterviewer can bias participant’s responseData reflects participants’ biasesCan be expensive|
|Focus Groups||Primarily qualitative but can also collect quantitative data by numerically coding participant responses and/or
questions and answers with multiple program participants interviewed as a group
|Quickly and reliably collect common impressionsEfficient way to get range and depth of information in a short timeCapture key participants’ perspectives about programs||Need a good and trained facilitatorCan be difficult to analyze responsesDifficult to schedule a group of people togetherOther participants and/or the facilitator may bias responsesCan be expensive|
|Observations||Primarily qualitative but can also collect quantitative data by numerically coding observations.||Trained observers’ field notes||View operations of a program as they are actually occurringCan adapt to events as they occur||Need a good and trained observerCan be difficult to interpret observationsCan be difficult to categorize observationsObserver’s presence can influence behaviors of program participantsCan be expensive|
|Case Studies||Primarily qualitative but can also collect quantitative data by coding observations, using surveys and document analysis.||In-depth interviews and longitudinal observations of a particular case, group, event, or program||Fully depicts participants’ experience in programPowerful way to portray program to outsiders||Very time-consuming to collect, organize and describeRepresents depth of information, rather than breadthVery expensive|
Source: The table was adapted from the Basic Guide to Program Evaluation, http://www.managementhelp.org/evaluatn/fnl_eval.htm#anchor1585345
Document analysis is the most common form of data collection because it involves the gathering of existing documents and records.
Begin this process by collecting your own program’s records and taking inventory of the collected data.
- You can supplement your own data with documents from partners and local, regional, and national agencies (United Way of America, 1996).
- Using existing information is typically cheap and sometimes free.
- Disruption to daily program functions and participants’ routines is limited
- Information is limited to what already exists.
- Data is often incomplete, old, or unreliable.
Surveys are probably the most recognized and popular form of data collection because they provide an easy way to collect a lot of information at once in a systematic and standardized way.
- Surveys ask the same questions in the same way for every study participant.
- Surveys are effective tools for collecting data for various sample sizes, small to large.
- Surveys can be conducted at a single point in time or can be given over time as a pre- and post-assessment or continuously as a formative assessment.
Refer to Resources & References at the end of this section for a list of websites that provide sample survey instruments, scales, and items that you can download and modify.
- Quick, easy, and rather inexpensive
- Easy to collect data in a safe, non-threatening or obtrusive way
- Easy to maintain participant confidentiality or anonymity
- Can be administered in person, mail, telephone, or electronically
- Existing surveys exist and can be adapted to fit specific needs (McNamara, 1997)
- Response bias – When participants respond in a socially desirable manner
- This often occurs when questions are leading or loaded
- A leading or loaded question: “All successful people have gone to college. Do you want to go to college?” This question is equating college-going to success. By including this value judgment at the beginning of the question, the participant is being led to provide an affirmative response because the question implies that people who do not go to college are unsuccessful.
- This often occurs when questions are leading or loaded
- Sampling bias - When survey respondents do not comprise a representative sample
- Sampling bias is generally caused by faulty collection practices
- An example of a practice that would lead to a sampling bias is administering an online survey to low-income families about their awareness of and perceptions about the new FAFSA electronic filing system. By administering the survey only in an online electronic format you are automatically eliminating families who do not have access to the internet from your survey response pool; these non-respondents are probably the least aware and the most affected by the new online filing system and their valuable perspectives would be lost from your study.
- Sampling bias is generally caused by faulty collection practices
For more information on surveys, visit the following link: http://learningstore.uwex.edu/Assets/pdfs/G3658-10.pdf.
Interviews are commonly used to collect qualitative data through oral questioning and response.
Interviews are helpful in collecting participants’:
- Opinions about a topic
The types of approaches to conducting interviews are varied, and you should determine which is best for your project needs and most suited to your interviewer’s style. Below, we describe four approaches to interviewing (Patton, 1990).
1) Conversational approach – informal and does not have a pre-determined set of topics or questions
- The conversation is allowed to evolve between the interviewer and interviewee.
- The interviewer should be well-trained and have strong interpersonal skills that will help him/her to connect personally with the interviewee.
- Data can be difficult to analyze since this type of interview is not standardized.
2) Interview guide approach – structured and uses an outline to guide the interviewer on the topics or issues he/she must cover
- Uses a conversational format but is more structured to ensure that interviewees discuss the same set of topics
- Increased structure makes the data more systematic across interviews, so it is easier to analyze than an informal conversational approach
- Analysis may still be difficult since the questions may be slightly different
3) Standardized, open-ended approach – very structured and provides interviewers with a script so that he/she asks the same exact set of questions, worded in the same way, for every interviewee with no flexibility
- This approach standardizes the questions, ensuring data quality if you have more than one interviewer, or less experienced interviewer(s).
- Data is standardized across all participants so it is easier to analyze and identify patterns in the data for specific topics.
- Interviewer has no leeway to respond to specific concerns or topics of value that the participant may bring up during the interview.
4) Fixed, closed-response approach – very structured format where the interviewer asks a standardized set of questions, each with a fixed set of response choices (e.g., strongly agree, agree, disagree, and strongly disagree) to all interviewees.
- This approach is specifically used to collect quantitative data.
- This approach is used when administering a survey in an interview format.
- This interview approach shares the same benefits (i.e. useful for inexperienced interviewers) and disadvantages (i.e. may be too structured) as the standardized, open-ended approach, and produces data that is even more standardized and easy to analyze.
If neither approach works, you can combine the approaches listed above. [e.g., you can use a fixed, closed-response approach in the beginning to collect basic demographic information, then you can move into an interview guide approach to cover the required topics, and you can close with an informal conversational approach to allow the interviewer to address any questions or concerns the participant(s) may have.]
Interviews do not need to be conducted in a physical face-to-face format. Interviews may be done by phone or through live video streaming.
A Note on Response Bias
- It is possible that 1) the wording of the questions or 2) certain characteristics of the interviewer will produce a response bias
- Some interviewer characteristics that may produce this type of bias include:
- Education level
- Physical appearance
- Voice accent
- Perceived sexual orientation
- The interviewer should be aware of biases the participant may hold, so that he/she can use this knowledge to contextualize the participant’s responses.
Visit the following link for more information about interviewing: http://www.gao.gov/special.pubs/pe1015.pdf
Focus groups are group discussions led by a facilitator, where participants are asked to talk about or react to a topic and/or question.
- Like interviews, focus groups generate qualitative data.
- In focus groups, participants are free to talk with one another and can support or disagree with each others’ points.
- The interview approaches discussed previously can be used in focus group facilitation (McNamara, 1997).
To conduct a focus group:
- Select and interview approach and develop questions.
- Contact a small sample of participants that is representative of the larger population you are trying to understand.
- If you are interested in learning more about participants’ satisfaction with and suggestions for improvement for an adult college access walk-in service you will want to include individuals who represent the racial/ethnic diversity your service participants, as well as gender and economic diversity.
- Coordinate with all focus group participants to find a time in common when they can all meet together at a central physical or virtual (phone or online) location.
- Find a trained facilitator who will guide the group discussion without influencing participants’ responses.
- Low cost
- Allows data collection from multiple participants at once
- Affords instant ability to hear similarities and differences in participants’ perspectives
- Difficult to schedule a time that is good for all participants
- Group dynamics can cause response bias
- Participants may exhibit biases towards other members of the focus group as well. This is why it is important to carefully select and group your focus group participants into discussion groups that will be productive. Some evaluators choose group participants by characteristics they have in common. For example, you can conduct two focus groups, one with all adolescent participants, and one with all parents. It is easy to understand how adolescents might censor their reactions and responses if placed in the same group with parents.
Observations are useful in capturing data through recorded direct observations of people, places, or things.
- When observing people, be sure to take note of their characteristics, interactions with others, speech, and non-verbal behaviors.
- Before observing, it is important to define what you are looking for, if anything.
Observations are a useful collection method when:
- You want to view direct information
- Physical and observable evidence exists that you want to collect data examine
- Written or self-reported data are inappropriate
- You are seeking to understand an ongoing process or behavior
Things you can observe:
- Program participants
- Program staff
- Physical surroundings
- Program artifacts and products
- This method provides you with a wealth of data.
- You may identify interesting things about the program that you hadn’t thought of before.
- Data from unstructured observations are not standardized and can be difficult to analyze across multiple observation times and observers.
- Challenges to this approach mimic those associated with informal conversational interviews.
The quality of observation data is a reflection of the observer’s knowledge of the object being studied and their interpretation of what they are observing. Therefore, it is incredibly important that the observer(s) are well trained and have a good understanding of the program. The observer should be able to identify the behaviors or elements they are supposed to be looking for in the case of a structured observation, or be able to identify things that might be interesting or useful to the program evaluation, as they emerge, in the case of an unstructured observation.
Types of Observations
Structured versus Unstructured (Taylor-Powell & Steele, 1996)
1) Structured observations are used when you are looking for something specific
- Used to track and record specific things about a program and the observer’s efforts are purposeful and targeted
- Directed by a checklist or guide which details exactly what the observer should be looking for and tracking
- The guide or checklist is accompanied by a rating scale allowing the observer to:
- Document whether something occurs or not, and
- Rate the quality with which the program element being observed was carried out
- Produce more standardized data across observation times and different observers; this will make your data easier to analyze and gives you the ability to quantify the data (i.e. report the frequency that a behavior was observed across multiple observations).
2) Unstructured observations are used when you are observing something as a whole and are expecting themes to emerge
- Enable the researcher to observe all facets of the phenomenon being monitored
- Useful for observing things as they naturally occur
- Provides the observer with the flexibility to focus in on themes or interesting elements as they emerge
- Observer is encouraged to take on the perspective of the participant(s) and experience the program as the participant(s) would
Low-inference versus High-inference (Mcmillan, 2004)
Low-inference observations – recorded exactly as they are observed with minimal inferences or interpretations made by the observer.
- Useful for recording quantitative data
High-inference observations – derived from the observer’s interpretation and inference about the meaning of what they see.
- Useful in assessing program quality (i.e. asking the observer to rate the effectiveness of a program element after observing it across several time periods) but require the observer to be very skilled and well-trained
A Note on Bias
- Because the observer is the reporter of the data, there is a potential for the observer’s personal biases to be reflected in the data they report.
- Study design can also create a bias. For example, if the observer observes an extremely high-quality program session they may view subsequent sessions they observe as less positive and rate them more harshly.
- Another potential for bias in observation data is a performance bias which is when participants behave differently because they know they are being observed.
Visit the following link for more information on observations: http://learningstore.uwex.edu/assets/pdfs/G3658-5.PDF
Case studies (Yin, 1994):
- Provide an in-depth and longitudinal examination of a specific individual, group, event, situation, behavior, program, or service
- Use a combination of quantitative and qualitative data collection methods (see those described earlier: document analysis, surveys, interviews, focus groups, and observations to fully explore and/or explain a particular thing)
- Are becoming increasingly common in social science research and very useful in conducting education and program evaluation
Case studies are best when:
- Answering very broad or complex research questions
- The nature or cause of what is being studied is uncertain and umbiguous
There are two important things to consider when conducting a case study.
- Define your case.
- The case is your unit of analysis (the thing you will be studying).
- Your case can be as small as one individual or as large as an internationally implemented program. Regardless of its size, your case should be clear and identifiable.
- Your case should not be average or typical of the group it is representing (this kind will provide the least useful information). Select a case that is atypical, an extreme example (either positive or negative), a prototype, or a strategically selected example of what you are trying to understand.
- Decide which methods you will use to collect your data about the case.
- You must determine which combination of methods will give you the most comprehensive view of the case you are examining.
- Gather an extremely in-depth understanding from the data collected
- Data collected is case specific and may not be generalizable
- Method takes a large amount of resources (e.g., money, time, staff)
- Requires staff who are trained and skilled qualitative and/or quantitative researchers