Surveys are probably the most recognized and popular form of data collection because they provide an easy way to collect a lot of information at once in a systematic and standardized way.

• Surveys ask the same questions in the same way for every study participant.
• Surveys are effective tools for collecting data for various sample sizes, small to large.
• Surveys can be conducted at a single point in time or can be given over time as a pre- and post-assessment or continuously as a formative assessment.

Refer to Resources & References at the end of this section for a list of websites that provide sample survey instruments, scales, and items that you can download and modify.

Benefits

• Quick, easy, and rather inexpensive
• Easy to collect data in a safe, non-threatening or obtrusive way
• Easy to maintain participant confidentiality or anonymity
• Can be administered in person, mail, telephone, or electronically
• Existing surveys exist and can be adapted to fit specific needs (McNamara, 1997)

Challenges

• Response bias – When participants respond in a socially desirable manner
  • This often occurs when questions are leading or loaded
    • A leading or loaded question: “All successful people have gone to college. Do you want to go to college?” This question is equating college-going to success. By including this value judgment at the beginning of the question, the participant is being led to provide an affirmative response because the question implies that people who do not go to college are unsuccessful.
• Sampling bias - When survey respondents do not comprise a representative sample
  • Sampling bias is generally caused by faulty collection practices
    • An example of a practice that would lead to a sampling bias is administering an online survey to low-income families about their awareness of and perceptions about the new FAFSA electronic filing system. By administering the survey only in an online electronic format you are automatically eliminating families who do not have access to the internet from your survey response pool; these non-respondents are probably the least aware and the most affected by the new online filing system and their valuable perspectives would be lost from your study.

For more information on surveys, visit the following link: http://learningstore.uwex.edu/Assets/pdfs/G3658-10.pdf.

Interviews are commonly used to collect qualitative data through oral questioning and response.

Interviews are helpful in collecting participants’:

• Feelings
• Thoughts
• Beliefs
• Opinions about a topic

The types of approaches to conducting interviews are varied, and you should determine which is best for your project needs and most suited to your interviewer’s style. Below, we describe four approaches to interviewing (Patton, 1990).

1) Conversational approach – informal and does not have a pre-determined set of topics or questions

• The conversation is allowed to evolve between the interviewer and interviewee.
• The interviewer should be well-trained and have strong interpersonal skills that will help him/her to connect personally with the interviewee.
• Data can be difficult to analyze since this type of interview is not standardized.

2) Interview guide approach – structured and uses an outline to guide the interviewer on the topics or issues he/she must cover

• Uses a conversational format but is more structured to ensure that interviewees discuss the same set of topics
• Increased structure makes the data more systematic across interviews, so it is easier to analyze than an informal conversational approach
• Analysis may still be difficult since the questions may be slightly different

3) Standardized, open-ended approach – very structured and provides interviewers with a script so that he/she asks the same exact set of questions, worded in the same way, for every interviewee with no flexibility

• This approach standardizes the questions, ensuring data quality if you have more than one interviewer, or less experienced interviewer(s).
• Data is standardized across all participants so it is easier to analyze and identify patterns in the data for specific topics.
• Interviewer has no leeway to respond to specific concerns or topics of value that the participant may bring up during the interview.

4) Fixed, closed-response approach – very structured format where the interviewer asks a standardized set of questions, each with a fixed set of response choices (e.g., strongly agree, agree, disagree, and strongly disagree) to all interviewees.

• This approach is specifically used to collect quantitative data.
• This approach is used when administering a survey in an interview format.
• This interview approach shares the same benefits (i.e. useful for inexperienced interviewers) and disadvantages (i.e. may be too structured) as the standardized, open-ended approach, and produces data that is even more standardized and easy to analyze.

If neither approach works, you can combine the approaches listed above. [e.g., you can use a fixed, closed-response approach in the beginning to collect basic demographic information, then you can move into an interview guide approach to cover the required topics, and you can close with an informal conversational approach to allow the interviewer to address any questions or concerns the participant(s) may have.]

Interviews do not need to be conducted in a physical face-to-face format. Interviews may be done by phone or through live video streaming.

A Note on Response Bias

• It is possible that 1) the wording of the questions or 2) certain characteristics of the interviewer will produce a response bias
• Some interviewer characteristics that may produce this type of bias include:
  • Gender
  • Race/ethnicity
  • Age,
  • Education level
  • Physical appearance
  • Voice accent
  • Perceived sexual orientation
• The interviewer should be aware of biases the participant may hold, so that he/she can use this knowledge to contextualize the participant’s responses.

Visit the following link for more information about interviewing: http://www.gao.gov/special.pubs/pe1015.pdf

Focus groups are group discussions led by a facilitator, where participants are asked to talk about or react to a topic and/or question.

• Like interviews, focus groups generate qualitative data.
• In focus groups, participants are free to talk with one another and can support or disagree with each others’ points.
• The interview approaches discussed previously can be used in focus group facilitation (McNamara, 1997).

To conduct a focus group:

• Select and interview approach and develop questions.
• Contact a small sample of participants that is representative of the larger population you are trying to understand.
  • If you are interested in learning more about participants’ satisfaction with and suggestions for improvement for an adult college access walk-in service you will want to include individuals who represent the racial/ethnic diversity your service participants, as well as gender and economic diversity.
• Coordinate with all focus group participants to find a time in common when they can all meet together at a central physical or virtual (phone or online) location.
• Find a trained facilitator who will guide the group discussion without influencing participants’ responses.

Benefits

• Low cost
• Allows data collection from multiple participants at once
• Affords instant ability to hear similarities and differences in participants’ perspectives

Challenges

• Difficult to schedule a time that is good for all participants
• Group dynamics can cause response bias
  • Participants may exhibit biases towards other members of the focus group as well. This is why it is important to carefully select and group your focus group participants into discussion groups that will be productive. Some evaluators choose group participants by characteristics they have in common. For example, you can conduct two focus groups, one with all adolescent participants, and one with all parents. It is easy to understand how adolescents might censor their reactions and responses if placed in the same group with parents.

Observations are useful in capturing data through recorded direct observations of people, places, or things.

• When observing people, be sure to take note of their characteristics, interactions with others, speech, and non-verbal behaviors.
• Before observing, it is important to define what you are looking for, if anything.

Observations are a useful collection method when:

• You want to view direct information
• Physical and observable evidence exists that you want to collect data examine
• Written or self-reported data are inappropriate
• You are seeking to understand an ongoing process or behavior

Things you can observe:

• Program participants
• Program staff
• Physical surroundings
• Program artifacts and products

Benefits

• This method provides you with a wealth of data.
• You may identify interesting things about the program that you hadn’t thought of before.

Challenges

• Data from unstructured observations are not standardized and can be difficult to analyze across multiple observation times and observers.
• Challenges to this approach mimic those associated with informal conversational interviews.

The quality of observation data is a reflection of the observer’s knowledge of the object being studied and their interpretation of what they are observing. Therefore, it is incredibly important that the observer(s) are well trained and have a good understanding of the program. The observer should be able to identify the behaviors or elements they are supposed to be looking for in the case of a structured observation, or be able to identify things that might be interesting or useful to the program evaluation, as they emerge, in the case of an unstructured observation.

Types of Observations

Structured versus Unstructured (Taylor-Powell & Steele, 1996)

1) Structured observations are used when you are looking for something specific

• Used to track and record specific things about a program and the observer’s efforts are purposeful and targeted
• Directed by a checklist or guide which details exactly what the observer should be looking for and tracking
• The guide or checklist is accompanied by a rating scale allowing the observer to:
  • Document whether something occurs or not, and
  • Rate the quality with which the program element being observed was carried out
• Produce more standardized data across observation times and different observers; this will make your data easier to analyze and gives you the ability to quantify the data (i.e. report the frequency that a behavior was observed across multiple observations).

2) Unstructured observations are used when you are observing something as a whole and are expecting themes to emerge

• Enable the researcher to observe all facets of the phenomenon being monitored
• Useful for observing things as they naturally occur
• Provides the observer with the flexibility to focus in on themes or interesting elements as they emerge
• Observer is encouraged to take on the perspective of the participant(s) and experience the program as the participant(s) would

Low-inference versus High-inference (Mcmillan, 2004)

Low-inference observations – recorded exactly as they are observed with minimal inferences or interpretations made by the observer.

• Useful for recording quantitative data

High-inference observations – derived from the observer’s interpretation and inference about the meaning of what they see.

• Useful in assessing program quality (i.e. asking the observer to rate the effectiveness of a program element after observing it across several time periods) but require the observer to be very skilled and well-trained

A Note on Bias

• Because the observer is the reporter of the data, there is a potential for the observer’s personal biases to be reflected in the data they report.
• Study design can also create a bias. For example, if the observer observes an extremely high-quality program session they may view subsequent sessions they observe as less positive and rate them more harshly.
• Another potential for bias in observation data is a performance bias which is when participants behave differently because they know they are being observed.

Visit the following link for more information on observations: http://learningstore.uwex.edu/assets/pdfs/G3658-5.PDF

Case studies (Yin, 1994):

• Provide an in-depth and longitudinal examination of a specific individual, group, event, situation, behavior, program, or service
• Use a combination of quantitative and qualitative data collection methods (see those described earlier: document analysis, surveys, interviews, focus groups, and observations to fully explore and/or explain a particular thing)
• Are becoming increasingly common in social science research and very useful in conducting education and program evaluation

Case studies are best when:

• Answering very broad or complex research questions
• The nature or cause of what is being studied is uncertain and umbiguous

There are two important things to consider when conducting a case study.

1. Define your case.
   • The case is your unit of analysis (the thing you will be studying).
   • Your case can be as small as one individual or as large as an internationally implemented program. Regardless of its size, your case should be clear and identifiable.
   • Your case should not be average or typical of the group it is representing (this kind will provide the least useful information). Select a case that is atypical, an extreme example (either positive or negative), a prototype, or a strategically selected example of what you are trying to understand.
2. Decide which methods you will use to collect your data about the case.
   • You must determine which combination of methods will give you the most comprehensive view of the case you are examining.

Benefits

• Gather an extremely in-depth understanding from the data collected

Challenges

• Data collected is case specific and may not be generalizable
• Method takes a large amount of resources (e.g., money, time, staff)
• Requires staff who are trained and skilled qualitative and/or quantitative researchers