Download
1 / 58
580 likes | 589 Views
Evaluation & measuring impact. Why zoo education is important. “Long-term conservation success will be linked to how zoos and aquariums engage with their visitors and change behaviour.” Translation …. http://www.waza.org/en/site/conservation/conservation-strategies. Why evaluate?.
E N D
Why zoo education is important “Long-term conservation success will be linked to how zoos and aquariums engage with their visitors and change behaviour.” Translation… http://www.waza.org/en/site/conservation/conservation-strategies
Why do we evaluate? • To find out what is working? Translation… • To find out what can be improved? Translation… • To ensure you are meeting your learning aims? Translation… • To ensure your participants’ needs are met? Translation… • To measure if you’re meeting your targets Translation… • Because its ‘best practice’? Translation… • To create evidence? If so what sort? Translation… • To tell people how good the programme is? Translation… • To monitor how things are going? Translation…
Who is it for? Why are they interested? • Funders Translation… • Yourselves Translation… • Senior management Translation… • Visitor audiences Translation… • Scientific publication Translation… • Conservation organisations Translation…
Who is it for? Why are they interested? What kind of reporting is needed? How much proof do you need? What sort of data will you need to collect? Translation…
What to measure • Messages or objectives are what you want to say –you can measure whether these are delivered and/or whether they are received • Translation… • Outcomes are what our learner goes away with – you can only measure this by interacting or observing them directly • Translation…
What to measure • Numbers of sessions delivered or people participating, what level of detail – age, gender? • Translation… • Quality of presentations, presenters, activities, resources? • Translation… • Return on investment – what is the cost of each activity, is it value for money? • Translation…
What are your constraints? • Your time • Planning & design • Collecting the data • Analysing the data • Reporting on the evaluation • Participants time • Access to participants – ease, sample size • Timing – when can people take part in evaluation activity • Budget for materials, staff time or software • Motivation of participants • Availability of resource • Translation…
Types of evidence • Photographic Translation… • Video Translation… • Sound Translation… • Questionnaires Translation… • Observations – structured or semi-structured Translation… • Peer reviews Translation… • Q&As Translation… • Drawings Translation… • Notes from activities Translation… • Numerical data Translation… • Written work Translation… • Interviews Translation…
Social Research Quantitative • Seeks to measures the prevalence of something • E.g. How many people have a certain view Translation.. • Translation…
Social Research Qualitative • Seeks to understand why • E.g. ‘I decided to do that because…’ • Doesn’t matter how many people think a certain thing, doesn’t value one reason or opinion more than another • Why do people think differently about the same thing? Translation… • Translation..
How to gather evidence • Basically there are 2 main approaches: • Observing people (in a structured way) • Talking to people or asking them to give their opinion (in a structured or unstructured way)
Use observation when… • You want to understand physical behaviour in a geographic space (asking people is not valid, you need to observe). • Translation… • You want to understand real-time reactions to stimuli (it is often hard to articulate how you thought and felt about something after the fact). • Translation… • You want to look at the things that might predict visitors’ behaviour. • Translation…
Observation • Benefits • Easy, quick and repeatable method • Quantitative data – good for analysis • Allows you to see what visitors are actually doing, rather than what they tell you! • Minimal bias • Translation…
Observation • Problems • Does not tell you what somebody is thinking! (unless you are ‘observing’ someone’s conversation) • Translation…
Examples: Structured Observation: This can be any type of measure of behaviour T.. What do people stop at? T… How long for? T… What do they do when they stop? T… What are they saying? T… Who too? T… Essentially this is animal behaviour research. Almost always quantitative. T….
Talk to people when… • You want to find out what visitors think and feel about things • Translation… • You want to understand change in some kind of cognitive variable (knowledge, attitudes etc.) • Translation… • You want in-depth (probably qualitative) data • Translation…
Talking to people • Benefits • Is really the only way to measure what people know, think or feel • Data collection can be quick depending on method • Quantitative data (rating scales for example) can be easy to input and analyse. • Translation…
Talking to people • Problems • Poorly designed surveys are worse than no survey at all! You need to know what you are doing. • Qualitative data is probably the most difficult and time consuming to analyse – it is not the easy option! • Translation…
Examples: Talking to people (or people talking to you): Many methods T… Surveys (could be online) T… Interviews T… Unstructured feedback (comments board) T… Focus groups T… Drawings T… Data can be quantitative and qualitative.
Example from Chester Zoo
Analysis of drawings (related to lesson content) Significant difference (p<0.001) Pre-test Post-test
Pupil questionnaires results * * * *
How would you evaluate these things … • Driving skills • T… • Knowledge of how to make an omelette • T… • How much a teacher inspires their pupils • T… • Singing ability • T… • Number of people who watch soap operas • T…
Tool trial Your activity? T…
Tool rules – being robust • Use your Theory of Change to plan what you will measure • Translation… • Collect information before and after • Translation… • Account for what else made the difference • Translation… • Be careful with language, order and nature of questions • Translation… • Representative samples • Translation…
Validity (internal) Are you measuring what you think you are measuring? And are you using the right tool to do the measuring? Translation…
Reliability Are you measuring the same thing each time – consistency Translation…
Sampling and generalisability Random (probability) sampling Translation… Each member of the population has an equal chance of being selected for the sample Translation…
Understanding Bias • Response Bias: • Where a survey question is constructed in such a way that one answer is much more likely than another. • E.g. ‘Don’t you agree that zoo animals are amazing?’ • Translation…
Common types of Response Bias • Demand characteristics: • In surveys, this is where the question itself ‘cues’ a participant in to what they think the study is about or what the researcher wants to hear. • E.g. ‘Do you think that the zoo is an important conservation organisation?’ • Translation…
The problem of self-report • All surveys are essentially self-report (people are responding about themselves) but some questions are not really suitable: • T… • Any question that relies on people giving their intention to do something e.g. “as a result of your visit, will you now recycle at home?” (note, other biases too) • T… • Any question that relies on the person understanding a difficult concept to answer accurately: e.g. “do you think you have learnt a lot today?” (what does the person understand by ‘learnt’ and how do they judge what ‘a lot’ is – this might be different for everyone) • T…
Being careful with using proxy measures • For example: • Asking a teacher to comment on the learning outcomes of their students. • T… • Asking a animal professional to comment on the behaviour of an animal. • T… • What is actually being measured ? • T…
