This seemingly simple question is a tricky to answer. It depends on what you are trying to find out, from what types of people and how much staff time or money you can devote to your evaluation. Evaluation is always ‘the art of the possible’ so aim for a sample that is good enough, rather than perfect.
Some things to consider:
- Are you trying to understand or to measure your findings? Understanding can often benefit from fewer, more in-depth conversations (i.e. qualitative research) whilst measurement, e.g. of how much or how often, benefits from larger sample sizes (i.e. quantitative research). For more on this see The difference between qualitative and quantitative research
- Is it OK to treat the audience as a single group? Or do you need to understand the experiences of different sub-groups within your audience? The answer will affect how big a sample you will need. Breaking the audience into sub-groups will mean you need to hear from more people.
- What is the expected size of your audience? If you are working with five engineers then your sample of engineers is … five. If your project is going reach 10,000 people you can only feasibly gather data from a sample of this audience. Nonetheless you will want to hear from enough of them to get a reliable picture of what happened.
- How much time, money and staff / volunteers do you have? If you have 60 hours of volunteer time, and each interview takes 5 minutes, it is likely they will complete 200 to 240 interviews - an average of 4 interviews per hour plus time for recruitment, regular breaks.
Reflecting on the constraints you are working within will help you set realistic targets for your sampling. A smaller, better thought through and more representative sample is almost always better than a larger sample that does not reflective your audience.
Quantitative research samples
Quantitative research is where you are seeking to measure how much, how often etc. So samples need to be much larger than those for qualitative research i.e. you need to gather data from more people. The number of people you need to include depends on how precise you want your findings to be.
To compare results from different audience sub-groups, you need at least 30 people in each one - ideally more. For instance, if you wanted to compare first-time and repeat visitors, you would need to recruit a sample of 60 to 100 people, assuming an even split between first-time and repeat visitors.
Qualitative research samples
Qualitative research should be conducted as rigorously as quantitative research. It is not a collection of random anecdotes from a self-selecting sample. Instead you need to carefully plan who you are going to include, where you are going to find them and how you are going to persuade them to take part.
There are no hard and fast rules about qualitative sample sizes. It often comes down to the time, money, and people you can allocate to the evaluation.
- Focus groups should each consist of 6 to 10 people (never more) so it is important to ensure these people are selected to represent the audience you are researching. For more on this see How to plan and run focus groups.
- In-depth interviews typically involve 10 - 30 interviewees. However, aim to undertake at least 5 interviews for each audience sub-group you want to research.
A cautionary tale: size isn’t everything
A common mistake is to assume that a large sample must be a good sample. This is not true. The quality of the sample - how closely it mirrors the composition of the total audience - is equally important. If you do not believe me, ask Alf Landon.
The largest survey ever undertaken was by the Literary Digest during the run-up to the 1936 US Presidential election. They achieved an astonishingly large sample of over 2 million voters. By comparison most pre-election polls today reach just 1,000 to 5,000 voters.
Not only was this the largest pre-election survey ever undertaken, it is also famous for getting the result more wrong than any other before, or since. The Literary Digest’s survey predicted a landslide victory for the Republican Alf Landon. In fact it was the Democrat incumbent Franklin D Roosevelt who won by a huge margin. The survey got the result wrong by a whopping 19%!
What went wrong? It was an enormous sample, but it was massively skewed towards people who voted Republican. The Literary Digest recruited voters from automobile club membership lists, telephone directories and their own readership. In 1936, in the midst of the Great Depression, if you were poor you almost certainly did not own a car, or a telephone, or subscribe to the Literary Digest. But you were much more likely to vote Democrat. The sample gathered by the Literary Digest was heavily skewed towards people who voted Republican. So the data they collected was not an accurate reflection of the whole electorate.
Just because it is a big sample, does not mean it is a good sample.