Your choice of evaluation methodology should be based upon:
- What you are trying to find out
- From whom
- Where and when you conduct your evaluation
- The money, staff and volunteer time you can allocate to the evaluation
Before you choose your evaluation methods you must have clear answers to all these questions. For more this see how to plan your evaluation.
There are four broad approaches to collecting data.
- You can talk to people – face-to-face or via video-link / phone – asking them questions and recording their answers
- You can ask people to record their thoughts, experiences and opinions – usually on some form of self-completion questionnaire, either on paper or an electronic form
- You can watch people while they take part in an activity, while they use a website, while they move through an exhibition
- You can look at data people leave behind e.g. bookings data, website usage statistics; social media feeds
In the tables below we look at the main ways of doing each of these and their pros and cons. Every method has weaknesses as well as strengths. Where time and resource allow, you should use at least two methods so that the strengths of one, compensate for the weaknesses of the other; and vice versa. For example, if you were using self-completion questionnaires to evaluate a school workshop, it would be a good idea to additionally observing a couple of the sessions and conduct some interviews with the teachers.
Interviews
|
Method |
Cost / staff needed |
Quant or Qual |
Good for evaluating |
Advantages |
Disadvantages |
|
Interviews Talking to individuals or pairs of people; in person, by phone, Zoom, Teams etc An interviewer asks the questions & records the interviewee’s answers by hand or with a digital recorder Can be large sample – lots of closed question; or small sample – lots of open-ended questions
|
Medium to High Depends on number of questions; number of open-ended questions; & /or size of the sample Interviewers need training Data analysis can be time consuming if lots of open-ended questions are used
|
Both
|
How audiences feel about an exhibition, live event, online resource etc; what they learnt; the problems they encountered; how they feel it could be improved
|
Easier to get a representative sample of the audience than with self-completion questionnaires. You select who takes part Can be quick – 3 or 4 closed questions – or longer & more in depth with open-ended questions More likely to get thoughtful responses to open-ended questions than when using self-completion questionnaires Better quality answers as the interviewer can clarify what a question means & probe for more detail Can collect verbatim quotes which make results more compelling |
Need to recruit people to take part. Interviewers need training in how to do this. Interviews of more than 15 minutes may require incentives. Need to conduct interviews at a time & place that is convenient & comfortable for interviewees. Interviewees will be reluctant to give answers that reflect badly upon themselves or the people who developed the activity, resource, exhibition etc. Difficult to capture verbatim answers to open-ended questions when taking written notes Digital recording of interviews requires extra reassurance about how the data will be used
|
Self-completion questionnaires
|
Method |
Cost / staff needed |
Quant or Qual |
Good for evaluating |
Advantages |
Disadvantages |
|
Self-completion questionnaires Paper, online, via social media, pop-ups on websites Mostly closed questions but can include 2 or 3 open-ended questions |
Low But you still need to allocate staff / volunteer time or pay someone to analyse the data |
Quantitative |
People’s feelings about a live event, online resource, an exhibition etc.; the benefits they have gained; satisfaction ratings Profile of audience at a live event |
Inexpensive compared to interviews or focus groups Anonymous – potentially yielding more honest responses than a one-to-one interview Large sample sizes possible. But that doesn’t mean it is representative of your audience Quick & easy for people to provide answers since you are mostly using closed questions Can send out questionnaires via email, text message or QR code This is often the only viable option for gathering feedback from the audience at live events or users of online resources |
Often get a very low response rate A self-selecting sample that may be a poor match to your actual audience. People who are very happy or very angry tend to be massively over-represented. No way to clarify questions if people struggle to understand them Often get very superficial answers to open-ended questions – just a single word. No opportunity to probe for further details Respondents often skip questions Difficult (but not impossible) to use with children below the age of 12 |
Observations
|
Method |
Cost / staff needed |
Quant or Qual |
Good for evaluating |
Advantages |
Disadvantages |
|
Observation Recording people’s behaviours & sometimes conversations |
Low to Medium Depends on the number of people you observe & amount of data you collect about each person |
Both
|
How people are responding during live events such as workshops, drama performances, lectures & tours How people navigate a website, use an interactive exhibit, move through an exhibition, play an online game etc
|
Provides a more reliable assessment of what people do, as opposed to what they remember or say they did. Often worth doing a bit of observation in addition to questionnaires or interviews as it helps you understand the answers you are getting. Can be quick & easy e.g. recording 2 or 3 basic measures of success / failure such as the percentage of the audience who stay until the end of a guided tour. Can provide very detailed data e.g. exact descriptions of how people behave during a workshop or while they use an interactive exhibit |
People change their behaviour when they know they are being watched - expending more time and effort than if you weren’t there. Very difficult to simultaneously observe & make written notes. Waiting till the end of the observation is risky; you may not remember everything. Handwritten notes only capture a small fraction of what happens. Video & audio recordings capture vast amounts of data but have significant privacy issues, and analysis is extremely time-consuming |
Focus Groups
|
Method |
Cost / staff resource needed |
Quant or Qual |
Good for evaluating |
Advantages |
Disadvantages |
|
Focus groups Group interview with 6 to 10 people with 2+ characteristics in common e.g. parents with children aged …; none have been to a science festival. 1 to 2 hours in duration Discussion chaired by a moderator, assisted by a silent observer The discussion is digitally recorded for later analysis Can be run in-person or via Zoom / Teams |
High A lot of time required to plan & organise the session especially recruiting participants You will need to pay participants an incentive to recompense them for their time & travel costs Data analysis is very time consuming
|
Qualitative |
People’s feelings about an event they have taken part in, an exhibition or website they have visited; a resource they have used. Why they feel that way; particularly liked or disliked something; how it compares to their expectations & past experiences; the benefits they gained People’s feelings about proposals for a new event, resource, website, exhibition etc |
Very detailed information about people’s feelings, experiences & ideas Group setting encourages respondents to be much more honest & thoughtful Opportunity to explore different scenarios in detail using storyboards, mock-ups, models etc Can use a variety of approaches to gathering data – as well as asking people to respond verbally. For example: filling in speech & thought bubbles on cartoons; drawing or selecting pictures that represent their ideas or feelings; placing markers on a scale e.g. ‘the best schools’ workshop I’ve ever attended’ to ‘the worst schools’ workshop I’ve ever attended’ |
Moderation requires training & practice: keeping the discussion on track; ensuring nobody dominates it; encouraging quieter members to contribute; keeping to time. Crucial to ensure the people involved really represent the audience you want to research. This may require professional recruiters who add extra expense. Recruiting yourself will be very time-consuming. Analysis of the discussion is very time-consuming Much more difficult to get a free-flowing discussion when running focus groups via video link. Also need to reduce the size of the group to 4-6 participants |