Q1. What do I really need to find out?
This is the most important question and where you need to start your planning. The Academy does not want you to “prove your project is a success” nor is it expecting you to demonstrate long-term impact on public attitudes to engineering. Instead the Academy wants you to learn useful lessons from your project about how to:
- Build audiences’ interest and understanding of engineering
- Reach under-served audiences through public engagement
- Increase engineers’ interest, understanding and skills in public engagement
- Co-create public engagement activities and resources with engineers
Start by reviewing the learning outcomes you have defined for your engineers and the public audiences and ask yourself – what would engineers / audience members think, feel, say and do if we were achieving each learning outcome? And what would they think, feel, say and do if we weren’t achieving our learning outcomes?
In addition to this evaluation, the Academy also needs you to collect some specific information from your public audiences.
- A basic assessment of how they feel about the activities you provided
- (And for family and adult audiences) an assessment of their socio-economic profile
For more about these see: Using the standardised questions in your evaluation.
Q2. Which audience groups will you need to research? Where and when will you find them?
Who is your project’s target audience - those for whom it needs to be appealing, accessible and impactful, for it to be deemed a success? You need to collect data from a representative sample of these audience groups.
Think about where and when will be the best time to recruit members of the public to take part in your evaluation. Where will it be comfortable and convenient for them? When will they have the time to answer your questions? What might distract them or dissuade them from taking part? What could you do to alleviate those problems
The other key group for your evaluation is the engineers that have worked on your project. Likewise you need to consider when and where it will be convenient for them to take part in your evaluation.
Q3. What challenges might you encounter persuading them to take part in your evaluation?
For your public audiences you need to consider what might make them reluctant to take part in your evaluation. For example they may be looking after young children who will quickly become bored or English might be their second language.
If you are asking for more than 15 minutes of their time you should consider offering an incentive. This will certainly be the case if you are asking people to take part in an hour long focus groups. But you may want to offer a small token of thanks even for those taking part in a 5 minute interview or completing a short questionnaire. For more on this see When to provide incentives for people taking part in evaluation
You also need to consider what might make people reluctant to be entirely honest about their feelings, knowledge and experiences? What can you do to reassure them, or to provide ways in which they can respond to your questions without risking their self-esteem? For more about this see How to get thoughtful, honest answers to your questions
Q4. How much money and staff / volunteer time can you allocate?
This question will help you identify which methods you can use. Some methods are a lot more time-consuming and costly than others – focus groups in particular. Others require specialist skills and training. For more about this see Choosing the best evaluation methods for your evaluation
You will need to allocate staff or volunteer time for designing the evaluation tools – interview questions, observation sheets etc - and analysing the data, as well as for data collection. As a rule of thumb it will take you at least as long to analyse the data as it took you to collect it. Have a look at How to analyse data from open-ended questions to get a sense of what will be required.
Q5. What methods should I use?
It is tempting to start with this question, but you that is a bad idea. Your choice of methods needs to match the aims of your evaluation, the audiences you need to study, and the resources you can allocate to data collection and data analysis.
Every method has cons as well as pros so you need to carefully select which ones best serve your needs and those of the audiences you are researching. For more about this see Choosing the best evaluation methods for your evaluation
Q6. How am we going to make use of the results?
It is worth reflecting on this question at the start of your project, as it will influence how you plan and conduct the evaluation. Evaluating Ingenious projects is not about ‘proving success’ or ticking a box. It should be an opportunity for you – and the wider STEM engagement community – to learn useful lessons about working with engineers to broaden the public’s appreciation, interest and understanding of engineering.
You should aim to design an evaluation study that will help you:
- Improve future public engagement activities through better understanding of the needs and wants of audiences; and the barriers to engaging with them
- Recruit engineers for future projects – through a better understanding of what motivates them to take part in public engagement; the training and support they need; and the benefits they can gain
- Raise funding for future public engagement work – by providing evidence that you understand your audiences; the challenges involved in engaging them
- Develop new approaches to public engagement – by testing innovative activities and resources with audiences and identifying how they can be improved