Evaluation Terms: A Glossary
The terms here will help increase your general understanding of evaluation as well as search and navigate this site. The terms selected include the types and approaches to evaluation and standard research concepts.
| A-C | D-F | G-K | L-0 | P-R | S-Z |
A-C
Accountability — the responsibility of leaders to provide evidence of project or initiative effectiveness to partners, stakeholders, and funders and that the project conforms to standards with its coverage, service, legal, and fiscal requirements. Adapted from Lipsey: Rossi, Peter H., Mark W. Lipsey, and Howard E. Freeman. 2004. Evaluation: A Systemic Approach. Thousand Oaks, CA: Sage Publications.
Baseline — the starting point for data collection, in the form of initial measurement, before a program commences. Later this measurement is repeated at key points, or at the end of the program, in order to compare the change.
Benchmark — An actual measurement of group performance against an established standard at defined points along the path toward the standard. Subsequent measurements of group performance use the benchmarks to measure progress toward achievement.
Case — each one of the things that you are evaluating, such as audience members, artists, participants in your program, volunteers, voters in your area, community leaders, members of the media, cultural and civic groups, schools, or theaters.
Coding — the process whereby evaluators identify and categorize similar responses to qualitative questions, in order to consolidate findings.
Complexity evaluation — views programs and the environment in which they operate as complex systems, whose dynamics have a high degree of connectivity and interdependence. The ramifications of changes are not readily apparent and can be difficult to understand. Diverse elements interact to produce unpredictable, emergent results. Solutions are tried. Problems are then re-examined in light of what was learned, and additional stakeholders can be consulted or included. This approach may resonate with innovative situations and is relevant in confronting stubborn social issues, like poverty, or policy making. (From The Broker’s Three Approaches to Evaluation: Evaluation Evolution? and Developmental Evaluation Primer)
Content analysis — a systematic process of reviewing qualitative data for recurrent issues and themes that emerge which are of interest to stakeholders, usually for the purpose of writing reports.
Context — the circumstances and setting in which a program takes place. For most programs, it includes events, location, cultural norms, beliefs, resources, timing/history, people, and other circumstances. Context for arts and civic engagement work might include the size, nature, and demographics of the community; cohesion or polarization in the community; history of civic engagement or community arts initiatives; and the history or position of the sponsoring organization. (From Patton’s Qualitative Research and Evaluation Methods, factors from Suzanne Callahan’s training materials and Dwyer’s “Arts and Civic Engagement: Briefing Paper for Working Group of the Arts and Civic Engagement Impact Initiative”)
D-F
Developmental evaluation — is used when goals and outcomes are not pre-set but rather evolve as learning occurs. It supports continuous progress and rapid response to complex situations with multiple variables. The evaluator is often an integral member of the program design team. It is best suited for initiatives that are at an initial stage of development or undergoing significant change, and can benefit from careful tracking of the process. Organizations and programs focused on innovation and social change are especially appropriate for developmental evaluation. (From “J. W. McConnell Family Foundation: What We Are Learning,” the section on Patton’s Developmental Evaluation, Sustaining Social Innovation)
Documentation — the process of recording what happened, or creating a record of a project, usually with little or no judgment attached. Documentation may include meeting notes, letters and memos, oral history records, journals, voting results, census data, audio and video recordings, media coverage of events, etc. In many instances, documentation materials are sources used for evaluation.
Evidence-based evaluation — aims to find measurable changes that can be directly attributed to specific policies. It uses experimental or quasi-experimental research methods. It is evaluation by testing, just as the effect of a medical treatment is assessed in laboratories by administering it to some members of a test group and not to others. By using a large sample group, one can determine the effects of a program or project objectively. (From The Broker’s Three Approaches to Evaluation: Evaluation Evolution?)
Focus group — a common method of gathering qualitative research, where a moderator conducts a group discussion among five to ten people in order to learn their opinions, attitudes, and thought processes about a given topic. The group dynamic encourages a deeper level of discussion and allows the moderator to probe for responses to important topics. In addition to being employed for evaluation purposes, focus groups can be useful in the design and planning stage of an arts and civic engagement program as a means of getting input from stakeholders.
G-K
Generalizability — the extent to which findings from a sample reflect the population studied. Generalizability depends on factors such as the design of the evaluation or study, sample size, and selection process.
Goal — the desirable long-term result(s) of a program, project, or effort. Arts-based civic engagement goals may involve policy change, reduction or elimination of certain conditions (such as violence in a community), or economic equity.
Impact — generally used to connect outcomes to the broader mission of an organization or partnership for social change.
Indicators — observable, measurable evidence of change by which the effects of a program can be assessed. They answer the question, “How will you know a stated outcome has been achieved?” Indicators indicate the degree to which the outcomes stated have been attained. Distinctions are made between process indicators, which point to what happens during a program, and outcome/impact indicators, or the change that results during or after a program has been launched.
Input — time and/or resources invested or the activities undertaken to make change
Intermediate effects — important changes that occur as a result of a program or initiative that may be considered prerequisites of longer-term impact or appropriate ends themselves. They are the types of outcomes often achieved by arts-based civic engagement initiatives and fall in the realm of social and civic capacity building for individuals, institutions, networks, and communities. Intermediate outcomes may be the explicit focus or desired effects of a project or program. Examples of intermediate capacity-building effects are: heightened awareness; increased numbers or diversity of individuals who are engaged; and new relationships built among organizations. (From Chris Dwyer’s “Arts and Civic Engagement: Briefing Paper for Working Group of the Arts and Civic Engagement Impact Initiative”)
Key informants — people whose personal or professional position gives them a knowledgeable perspective on the nature and scope of a social problem or a target population and whose views are obtained during an assessment or evaluation. For arts-based civic engagement work, key informants may include community members, organizers, elected officials, or cultural group leaders. Adapted from Lipsey: Rossi, Peter H., Mark W. Lipsey, and Howard E. Freeman. 2004. Evaluation: A Systemic Approach. Thousand Oaks, CA: Sage Publications.
L-O
Logic model — a diagram that explicitly shows relationships between a project’s goals, inputs (or resources), activities, and outcomes (or results). Logic models reflect a project’s theory of change, that is, how the program works (or logic behind) in order to effect the desired change. The goals state desired change over the life of the project or program. Inputs are the resources involved in your program, including human resources, space, materials, equipment, and technology. Activities are the action steps that will be taken to implement and reach the goal and are usually summarized into major categories, such as marketing, production, presenting, community organizing, etc. Outputs are the distinct products that the project produces and which can be counted, such as number of art work produced, pamphlets distributed, people who attended, or voters registered. Outcomes are the intended results of the program. (From Suzanne Callahan’s Singing Our Praises)
Metric — a unit of measure like a standard score or a rating, essentially how an instrument calibrates degrees of change.
Monitoring — the regular reporting of program results in ways that stakeholders can use to understand and judge those results.
Non-response bias — bias that results when information is absent in your study from those who chose not to respond.
Outcome — the intended results from a program, usually indicated by the kinds and degree of change that occur. Examples of arts-based civic engagement outcome goals could be heightened public awareness of a civic issue; media coverage of an issue is more balanced; or a policy is enacted or changed.
Output — the quantifiable, measurable, and distinct end product(s) of a program. Examples are: artworks (performance, mural, spoken word event), collaborations created, dialogues forums or engagement activities (e.g. rallies, river clean-up, neighborhood planning meetings), votes, news stories generated, information vehicles (web sites, brochures, creative posters).
P-R
Participatory evaluation — a process that involves key participants in planning and implementing evaluation, including: setting goals, developing research questions, interpreting data, making decisions, and using the information. The participatory approach is designed to increase participation in and ownership of collective inquiry on the part of stakeholders, as well as the usefulness of the information gathered.
Population — the entire group of people you seek to study and learn about in your evaluation, such as targeted community segments, audiences, youth voters, community leaders, artists participating in your civic program, a cultural group in the community, members, etc.
Process evaluation (same as formative and implementation evaluation) — focuses on the way a program is working in order to understand the strengths and weaknesses of its design and implementation. It asks and answers the question, “What is happening and why?” It looks at how a product is produced rather than the product itself. It focuses on indicators that are process or activity related, such as how many activities were undertaken, the effectiveness of choices made in program design, such as choice of a particular art form as a catalyst for dialogue. Examples might include evaluation of: strategies used to recruit participation of a desired group; workings of a partnership; the reach of a particular informative campaign. (From Suzanne Callahan’s Singing Our Praises)
Program evaluation — efforts to systematically assess the quality of a program’s performance and the outcomes and impacts of programs.
Purposeful sample — a sample in which people are intentionally selected because they have a distinct set of characteristics that are of interest to the researcher.
Qualitative design — the approach to collecting information that captures experience, meaning, and stories that illuminate impact. Typical methods used to conduct qualitative research include, for example: in-depth, open-ended interviews and focus groups, direct observation, open-ended responses, journals, letters from program participants, notes, and program records. It is important to note that qualitative design can employ either qualitative or quantitative data.
Quantitative design — the approach to collecting information that expresses change in numerical terms. Typical methods include, for example: close-ended questionnaires, attendance information, and census/population data. It is important to note that quantitative design can employ either qualitative or quantitative data.
Random sampling — a process that gives each person in the population an equal chance of being included in the sample. This is the most common and statistically valid method of selecting participants, especially for quantitative studies.
Representative sample — a sample that is representative of the key characteristics of the population from which it is drawn. Such characteristics vary according to the study and may include demographics such as age, race, and gender, or other areas such as artistic discipline or degree of civic involvement.
Research — systematically and empirically gathering information about a topic of interest.
Response rate — the percentage of those who provide information on a survey, which is calculated by dividing those who did respond by the total number of surveys distributed. The response rate is one factor that determines the confidence of drawing conclusions from data.
S-Z
Sampling — the process of selecting individuals or entities to participate in an evaluation, usually with the goal of being able to use the findings from these people to make inferences about the population you are serving and studying.
Social desirability bias — also known as social response bias, the skew that is present in a study when participants feel compelled to respond in socially desirable ways, in order to make the researcher (or arts organization or funder) feel good about the evaluation or subject of the study. For example, a person being interviewed by a volunteer at a voter registration drive might feel more compelled to say that s/he intends to vote in the next local election than s/he otherwise would.
Sphere of influence — in evaluation, the area, as defined by geography; number of people, artists, citizens; or other characteristics that can be influenced by a program, given its scope and other limitations.
Stakeholders — all individuals who hold a stake, or value, in the success of your program. These can include: program participants, partner agencies, community leaders/organizers, advocates, funders, as well as artists, dialogue facilitators/mediators, staff, board, instructors, counselors, evaluators.
Statistical significance — also known as statistical power, procedures that test the likelihood that differences in data are due to true difference, rather than chance or random error, and that they can thus be inferred from the sample studied to the total population.
Strategy — a step taken with the express purpose of meeting a goal.
Target audience — the people, group, and/or community you intend to reach with your project, program, or initiative. This includes those you intend to benefit from or be influenced by your program and the group to which you intend to present your findings and recommendations.
(From Cultural Mapping Toolkit)
Theory of change — articulates the relationship between the social problem, issue, or change you are addressing and the strategies you’re using to get the work done. A theory of change identifies preconditions that will enable or possibly inhibit change, determines the activities that will produce positive conditions, and offers logic about why those activities are likely to work. It helps you to articulate exactly what propositions and assumptions you are testing and, therefore, what you should be assessing in your evaluation plan.
Unit of analysis — the major entity that you are analyzing in your study. For instance, a unit of analysis in a study could be: individuals, groups, a geographical unit (neighborhood, town, region). A unit of analysis could also be social interactions (arrests, youth seeking help). It is the analysis you do in your study that determines what the unit is (i.e., in one case it might be studying change in individual children and in another case the collective immigrant youth in a city).
