A successful and effective evaluation depends on having good information. Identifying what information (data) you will need and how you will gather it (method) are key parts of the evaluation plan. Ideally, you will think about these things before your program starts because it is easier to collect information as you go. That said, it is possible to gather data after the fact.
As discussed in “Planning an Evaluation,” there are different kinds of information that you can gather: quantitative and qualitative. Quantitative data is information that can be measured or represented using numbers. Qualitative data is information that is expressed using words. Both types of information can help you better understand the value of a program or how you might improve it, so it’s good practice to gather both quantitative and qualitative data.
There are lots of ways to gather information for the purpose of evaluation. In this section, we consider the benefits and challenges of different methods. Collecting data is one of the most time-consuming tasks of evaluation work, so you will want to leave enough time in your evaluation plan to gather information and do a thorough job of analyzing it.
Remember to use your evaluation questions to guide what information you will collect and how you will collect it (see “Planning an Evaluation” for tips on developing your evaluation questions).
Your first source of information should be documents and files that have been produced by or about your program. These may include funding proposals, program reports, budgets, publications about the program, or policies and guidelines that have been produced by the program staff. Before accessing files that might contain information about program participants, be aware of any rules about accessing private information.
Program files are likely to be the best source of quantitative data about a program. They might tell you, for example, how much money was spent, how many people took part in the program, or how many chainsaws were purchased.
External Information Sources
Other organizations may also have information that can help you to better understand your program, such as Statistics Canada, NWT Bureau of Statistics, or other government departments like Health and Social Service or Environment and Natural Resources. For example, an internal program report might show that 12 youth participated in your summer fish camp. A visit to the NWT Bureau of Statistics website can tell you that that represents 37% of the youth in your community.
Sometimes actions speak louder than words. In other words, we may learn more about people’s behaviour, knowledge, and actions from watching them than from having them answer questions. If you are working with an external evaluator, it is critical to have them visit the program site and observe the program in action.
Observation for the purpose of evaluation can be structured or unstructured. You may have a list of things that you are “looking for,” such as how many participants are speaking Dene Zhatie, or participants’ progress with a skill such as filleting fish. Unstructured observation is more general and also more concerned with context. For example, you may observe the nature and quality of interactions between youth and Elders in your program. Sometimes observing what doesn’t happen is as valuable as what does.
Make note of your observations when they happen. If you are using a structured approach, you can use checklists or observation guides. Alternatively, you can take more general field notes. You may want to supplement your observations with photos or videos.
People have a tendency to observe the people at the ends of the spectrum, those who are incredibly successful or those who are really struggling. Don’t forget to keep an eye on those in the middle. You want to make sure that your observations are balanced and reflect a wide range of experiences.
Sharing circles are a good opportunity to understand the thoughts, feelings, and opinions of program participants. They can allow for deeper discussion of complex issues.
Sharing circles provide more detailed information than questionnaires and are more accessible than surveys for people with low literacy. They are also more efficient than one-on-one interviews. However, they tend to favour more vocal group members. They may also be alienating for people who are uncomfortable sharing in a larger group. Sharing circles are not an effective evaluation method for discussing sensitive or highly personal issues.
If you are using a sharing circle for the purpose of evaluation, it is a good idea to guide the discussion. In other words, identify someone who can facilitate the circle and ask participants a series of questions that address your evaluation goals. Ideally, you want a facilitator who is a good listener, but who can also re-direct the discussion if need be. One of the challenges of sharing circles is ensuring that one or two people don’t dominate the discussion or influence the opinions of others. It is also a good idea to have someone take notes so that you have a record of the discussion.
Art making can provide a fun, accessible, and engaging way to gather information about your program. Because art-based methods tap into our creative brain, they can provide different ways of seeing and understanding your program. Arts-based methods can be particularly useful for gathering information that might otherwise be hard to gather, such as thoughts and feelings. You can use art with participants of all ages, but it can be especially valuable when you're working with children and youth.
There are lots of different arts-based approaches that you can use in evaluating your program, including drawing, sculpture, collage making, dance, drama, and photography.
We often use photographs and videos to share information about our programs with potential participants and funders. However, photographs and videos can also be used in evaluation. In particular, they can help program participants to make sense of and communicate their experiences in the program.
As with other methods of data collection, there are different ways to approach taking videos or photos. For example, you can provide participants with cameras and see what they choose to capture on their own. Alternatively, you can provide them with some direction. For example, you may ask participants to take five to ten photographs that illustrate their experiences as a participant in the program and to include in those photographs images that capture the strengths of the program, as well as areas for improvement. Or you may ask them to make a short video or a digital story showing what they learned during your program.
While all photos and videos tell stories, it is a good idea to have the participants explain their choices. They can do this orally or, if they feel more comfortable, they can write captions for the images or a script for a video explaining their choices.
Journals provide a record of where we have been and what we have done. They also help us to make sense of our experiences. For evaluators, journals provide a different perspective on the experiences of participants and program staff, one that is close to the event.
Assuming that participants take the time each day to document their experiences, journals can provide a window into less obvious strengths, weaknesses, and important moments than evaluation tools used only at the program’s end. Some people will be more comfortable writing down their thoughts and experiences than sharing them verbally.
Diaries can be free form or participants can be provided with guiding questions. Not everyone will want to write. Some may feel more comfortable sketching or drawing. If literacy is an issue, participants can create audio or video diaries.
During an interview, participants are asked a series of questions orally. Interviews are good for collecting in-depth information on opinions, thoughts, experiences, and feelings. They allow the interviewer to ask follow-up questions and clarify responses.
Interviews can be structured, semi-structured, or unstructured. A structured interview has a list of standard questions, each with a set of possible responses. A semi-structured interview also has a set of pre-determined questions, but the interviewee responds in their own words and the interviewer can provide clarification and ask follow-up questions. In an unstructured interview, the interviewer has no specific guidelines, restrictions, predetermined questions, or list of options. The interviewer asks a few broad questions to engage the respondent in an open and informal spontaneous discussion.
Regardless of the approach, successful interviews start with good planning. Begin by identifying your focus. What is it that you want to learn from the interviewee?
During the interview, it is important the interviewee feels comfortable. Show empathy, listen actively, and maintain eye contact. You may want to start with general questions before asking more sensitive questions. Most people will not want to say things they think you might disagree with so it is important that you not show strong reactions or emotions.
It can be difficult to balance interviewing with note taking, so you may want to record the interview. A recording not only allows you to capture what is being shared for the sake of the program, but stories from the interviews can be shared with others (with the correct permissions).
Questionnaires can be used to measure opinions, attitudes, behaviours, and perceptions. Questionnaires are commonly used because they are easy to conduct, inexpensive, efficient, and can easily be made anonymous. That said, it is difficult to produce a survey that is easy to complete and analyze. A good questionnaire requires careful planning.
Questionnaires may use a mix of closed-ended and open-ended questions. Closed-ended questions ask participants to select an answer from a list. They can be yes/no answers, a list of possible responses, or rating scales. Open-ended questions allow participants to respond in their own words. They typically begin with “what” or “why.” Closed-ended questions are easier to analyze. Open-ended questions can be useful if you do not know the possible answers to questions or for gathering insightful or unexpected information.
Questions should be easy to read and understand, they should only ask for one piece of information at a time, and they should be organized logically. Typically, questionnaires open with questions that are easy to answer and not particularly sensitive. It’s also important to avoid questions that are leading. For example, instead of asking, “Is this the best camp you have ever been to?,” ask “What did you like about this camp?.”
Questionnaires can be administered in several different ways. You can provide participants with a paper version, use an online survey tool like Survey Monkey, load the questionnaire onto a tablet, conduct the questionnaire by phone, or have someone sit down with the participant and ask the questions. The last two options are good ones for participants who may not have strong literacy skills. Be sure to choose a method that will work well in your community and for your participants. For example, you won’t want to use an online survey if your participants don’t have easy access to computers.
An important part of designing an interview guide, questionnaire, or survey is testing it out with a few people before you finalize it. A question that might seem clear to you may be confusing for an interviewee. Or, you might go to analyze the responses and realize you didn’t get the information you were looking for. Don’t expect to get it right the first time, and don’t be discouraged if it takes two or three versions before you have a set of questions that you think will work.
A successful and effective evaluation depends on having good information.
TIP! Use your evaluation questions to guide what information you will collect and how you will collect it.
TIP! If you are working with an external evaluator, it is critical to have them visit the program site and observe the program in action.
TIP! If you are using a sharing circle for the purpose of evaluation, it is a good idea to guide the discussion.
TIP! Successful interviews have a focus and a plan.
TIP! Questions should be easy to read and understand, they should only ask for one piece of information at a time, and they should be organized logically.