Closed questions
Questions where people are provided with a set of answer options to choose from, rather than using their own words. Closed questions can offer single or multiple-choice options or some form of rating scale.
Closed questions are used in both quantitative and qualitative research; in interviews, questionnaires and to a limited extent in focus groups.
Closed questions are quick and easy for people to complete. The data is already sorted into categories so it is easy to analyse. However, closed questions provide only a limited amount of information compared to open-ended questions.
Data-mining / Data analytics
Analyses of data that is automatically collected e.g. bookings data, website usage statistics, likes on social media; Government statistics such as Indices of Multiple Deprivation, Pupil Premium funding.
Evaluation
The collection of quantitative and qualitative data using interviews, questionnaires, observational techniques and/or focus groups.
The two main goals of evaluating your Ingenious project are: a) to provide useful lessons for your future public engagement projects; and b) provide feedback about the extent to which you achieved your Reach, Quality and Impact objectives.
Focus group
A group discussion lasting 1 to 2 hours, involving 6 -10 specially invited people who share various characteristics in common. Focus groups are one of the main techniques used in qualitative research.
The discussion is chaired by a moderator who guides the discussion using a set of pre-designed questions and stimulus materials – storyboards, picture sorting activities, bubble cartoons to complete etc. The discussion is digitally recorded for later analysis.
Focus groups provide immensely rich data about your audience but are expensive and time-consuming to run, and to analyse the data. They can only provide qualitative data.
GDPR (aka UK General Data Protection Regulation)
UK legislation – based on EU law - that defines how people’s personal data should be collected, stored and used. In terms of evaluating your Ingenious project the most important principles of GDPR are: that members of the public must give permission for their personal data to be collected; you must be clear and honest about how this data will be used; this data should only be collected if there is a valid reason for doing so; it must be stored securely and deleted at the earliest opportunity after it has been used.
GDPR applies to data that would allow someone to identify an individual (personal data). Such data includes names, postal addresses, and email addresses. It also applies to data which, in combination, could identify an individual. For example if you collected: job title, income, age, gender, ethnicity, post-code and whether they have children under 16 living at home it is becoming increasingly easily to identify one particular person.
Indices of multiple deprivation (IMD)
An official Government measure of the relative deprivation within an area. It is a combined measure of: income, employment, education and training, health and disability, crime levels, quality of environment; provision of housing and services. Areas are ranked using IMD from 1 (most deprived area) to 32,844 (least deprived area).
Similar indices of deprivation have been developed for Northern Ireland, Scotland and Wales (MDM, SIMD, WIMD).
It is possible to identify the IMD score of individual postcodes using free to use ‘look-up’ tables. This provides a quick, easy way of assessing the socio-economic status of members of your family and adult audiences. See also Pupil Premium.
Interviews
An evaluation method where a researcher asks questions in person, by phone, or via video link. The researcher records the interviewee’s answers digitally or by hand.
Interviews usually include a mixture of open-ended questions and closed questions. They can be used for either quantitative or qualitative research.
Interviews provide much richer and more reliable data than would be obtained using self-completion questionnaires. However they require more time and effort to conduct.
Learning outcomes
These are descriptions of the benefits that engineers and public audiences could, or do, gain from an Ingenious project. The outcomes are divided into categories:
- Increased knowledge and understanding
- Impact on beliefs, attitudes and values
- Development of skills – practical, thinking, social, and creative skills
- Increased motivation and self-confidence to learn more, take part in future activities
- Increase sense of self-worth, belonging, identity
Learning outcomes are used to define the aims and objectives of a project; and to plan evaluation.
Long-term tracking questionnaire
An online, self-completion questionnaire that the Academy sends to all the engineers who took part in Ingenious projects. The survey is sent out 12 months after a round of Ingenious projects has been completed. The survey provides a top-level assessment of the long-term impact on engineers’ public engagement knowledge, skills, motivation and practice.
See also: Pre-project questionnaire.
Open-ended questions
These are questions where someone must answer using their own words instead of selecting an answer from a menu of options. Open-ended questions are used in both quantitative and qualitative research; in interviews, questionnaires and focus groups.
Open-ended questions provide much richer data than closed questions but they take more time and effort to complete, and more time to analyse the data.
Observational techniques
A set of evaluation methods that involve watching members of your audience and recording their behaviours.
Data can be recorded using hand-written notes or video-recording. Filming people requires their informed consent and the resulting data must be analysed with care to protect participants privacy and dignity.
Observational techniques reveal what people actually do, as opposed to what they recall, or are willing to admit, doing.
Pre-project questionnaire
An online, self-completion questionnaire, that your engineers need to complete at the start of your Ingenious project. It gathers basic data about the engineers you have recruited – age, gender, ethnicity, employment status, prior experience of public engagement work.
This questionnaire is also used to gather email addresses from the engineers so that they can be sent the Long-term tracking questionnaire, 12 months after the end of your project.
Project planning template
A document you need to complete at the start of your Ingenious project. In it you set out your project objectives and outputs; define your Target audience, and the Learning outcomes for the engineers and the public audiences you engage with; and how you will evaluate your project.
Pupil Premium
Funding provided to improve education outcomes for disadvantaged pupils in primary and secondary schools in England.
The Government publishes a spreadsheet every year showing the percentage of pupils in each school who are receiving this funding. This data provides a quick and easy way to assess the socio-economic profile of your school audience.
See also Indices of multiple deprivation.
Qualitative research
Explores in detail what people think, feel, know and do, as well as the underlying reasons for these opinions and behaviours. Qualitative research involves collecting a lot of data from a small number of people (10 to 100).
Qualitative research methods include in-depth interviews, focus groups and various observational techniques. Merely asking an open-ended question on a questionnaire does not count as qualitative research.
See also: Quantitative research
Quantitative research
Assesses the frequency or amount of outcomes: how much, how often, how many, who, when and where. Quantitative research involves collecting a relatively small amount of data from a lot of people (100s to 10,000s, sometimes even larger samples).
Quantitative research involves the use of surveys and self-completion questionnaires as well as data-mining and observational techniques. Quantitative research makes extensive use of closed questions, however open-ended questions are also often used.
See also: Qualitative research
Questionnaires (self-completion questionnaires)
A set of questions presented on a printed or electronic form. Members of the public or engineers answer the questions themselves – ticking boxes, writing or typing answers –, rather than speaking their answers out loud for someone else to record. A questionnaire usually consists mostly of closed questions plus one or two open-ended questions.
Compared to Interviews and Focus groups, questionnaires are much quicker and cheaper to use. However the quality of the data is much lower.
Sample
The individuals with whom we undertake research. Ideally, they are selected at random from a defined population. The selected sample of people should reflect the target population as closely as possible.
Target audience
The segments of the general public for whom your project outputs must be appealing, informative, thought-provoking and accessible for the project to be deemed a success. These are the people with whom you will be conducting your evaluation.
Target audiences can be defined by various characteristics e.g. age, type of group, whether they have a specific interest or specialist knowledge, whether they have previously taken part in public engagement and so on.