Understanding Data Collection

Data collection captures evidence about indicators and enables analysis to measure change. Systematic data collection is essential to establish credible findings.

Types of Data Collection

Quantitative data collection methods typically use standardized response categories. Surveys are the most common example. Respondents are asked to choose among responses that best characterize their perceptions, attitudes, knowledge, or opinions.   The advantage of quantitative data is that it efficiently measures the reactions of many people which facilitates statistical aggregation of the data, including making comparisons by subgroups. Using sound sampling procedures to represent the population and obtaining adequate response rates are critical. Provided your sample size is large enough, and your methods and analysis are sound, this method of data collection provides a broad, generalizable set of findings. This means that they can be used to learn about the entire population that you are studying.

By contrast, qualitative data collection methods typically produce detailed data about a much smaller number of people. Qualitative data can provide rich information through direct quotation and careful description of programs, events, people, interactions, and observed behaviors. The advantage and disadvantage of such descriptions, quotations, and case studies is that they are collected as open-ended narratives. Observations are not fit to categories so rigorous and systematic analysis of content can be tedious and time-consuming.

One of the most common qualitative data collection techniques is the interview which may be with individuals or a group. In a group interview, or focus group, a moderator conducts a discussion among five to ten people in order to learn their opinions, attitudes, and thought processes about a given topic. The group dynamic encourages a deeper level of discussion and allows the moderator to probe for topics that are important. Note that the term focus group is often misused to refer to any meeting of any group of people about a given topic. In actuality, focus groups as well as individual interviews, are systemically structured and discussion is carefully guided to allow for drawing conclusions and making comparisons.  Qualitative data can also be collected from written sources such as journals, open-ended survey questions, and reaction sheets completed by observers or participants.

An ethnographic approach to evaluation collects qualitative data. Maribel Alvarez describes in her case study, “Two-Way Mirror: Ethnography as a Way to Assess Civic Impact of Arts-based Engagement in Tucson, AZ,” that ethnographic evaluation emphasizes listening carefully and observing real-life actions to understand how people make sense of their lives. An ethnographic evaluation produces “data collection” of a distinct kind—subjective accounts of how people actually interact with systems, programs, and policies. This data is collected through experiences of the evaluator in the field, side by side with participants.

Systematic Collection of Qualitative Data

In order to be credible, qualitative data needs to be gathered objectively and systematically. Systems can be set up to code open-ended comments from interviews, surveys, or themes from meetings and public forums according to their meaning and then group those with similar meanings together for analysis. This two-part analysis provides quantifiable statistics by counting number of like responses (prevalence of responses) as well as the deeper meaning behind them (themes that emerge from the coding process).

In "Two-Way Mirror," Maribel Alvarez provides insight on how ethnographers are methodical in their field work, for example, they:

• Gather a variety of information from different sources.
• Understand both who is present and absent from the experience; understand and locate the group, community in its broader context (geographic, cultural, ethnic, occupational, etc).
• Keep a journal, diary, or notebook of field notes; are descriptive in their note-taking. Are disciplined and attentive to details and things unspoken. Include notes about their own feelings, experiences as an observer.
• Record interviews and extended conversations; transcribe key quotations to represent program participants in their own terms.

Sampling

Sampling is the process of selecting participants for your study or evaluation. In cases where the entire group of people that you are studying is small, it is desirable to sample all of them. This would be the case, for example, with a group of 10 artists who are core to the arts project, or 20 community organizers who are linking arts-based efforts to activism efforts. When the population is so large that you cannot include all participants in your study, for example a whole city’s population, or all residents of a neighborhood, the commonly accepted practice is to select a sample based on agreed-upon criteria.

In general, when the population is large, the best and most commonly accepted practice is to randomly select respondents. Random sampling helps to ensure that the findings from your sample are likely to reflect the characteristics of the entire population that you are studying. In instances where participants hesitate to be interviewed, offering incentives (such as cash, free tickets or other items) may help.

However, just as important (depending on the type of research being conducted), is purposeful sampling, in which people are intentionally selected because they have a distinct set of characteristics that are of interest to the researcher. Examples of purposeful samples might be audience members who attend a series of public dialogues that integrate performance prompts, people self-described as opposed to an issue, students who volunteer in a community revitalization project, or artists who participate in civic engagement skill-building workshops. Often helpful in small studies, the reasons why a purposeful sample would be used are:

• To get adequate representation, where random sampling would be impossible, such as to evaluate an arts program that serves homeless people in a particular jurisdiction.
• To deliberately select typical cases, such as artists in a particular jurisdiction who volunteered in the presidential election.
• To examine critical cases, such as a teenage mothers who had found ways to complete long-term programs that an organization had hosted.
• To compare differences between cases, such as the differences between artists who work in partnership with community arts organizers versus those who conduct similar work on their own.

Featured Resources on Data Collection

Four Ways to Collect Information