Consider the Evidence

1. Consider the Evidence Evidence-driven decision making for secondary schools A resource to assist schools to review their use of data and other evidence 6 Getting to Information

2. Evidence-driven decision making This module is part of a resource about how we use data and other evidence to improve teaching, learning and student achievement Today we are looking at the second stage of this process – collecting and analysing evidence, and getting to information we can use

3. Evidence Any facts, circumstances or perceptions that can be used as an input for an analysis or decision • how classes are compiled, how classes are allocated to teachers, test results, teachers’ observations, attendance data, portfolios of work, student opinions… Data are one form of evidence

4. Data Known facts or measurements, probably expressed in some systematic or symbolic way (eg as numbers) • assessment results, gender, attendance, ethnicity … Data are one form of evidence

5. What evidence does a school have? • Demographics • Student achievement • Perceptions • School processes • Other practice

6. Speculate A teacher has a hunch about a problem or a possible action Trigger Data indicate a possible issue that could impact on student achievement Explore Check data and evidence to explore the issue Reflect on what has been learned, how practice will change Question Clarify the issue and ask a question Evaluate the impact on the intervention Assemble Decide what data and evidence might be useful Act Carry out the intervention Intervene Plan an action aimed at improving student achievement Interpret Insights that answer your question Analyse data and evidence The evidence-driven decision making cycle

7. A teacher has a hunch - poor writers might spend little time on homework Explore data Survey of students shows that this is only partially true Reflect How will we teach writing in the future? Trigger Significant numbers not achieving well in writing Question What are the characteristics of students who are poor at writing? Evaluate Has writing improved? Analyse NQF/NCEA results by standard Intervene Create multiple opportunities for writing; include topics that can use sport as context; connect speaking and writing. PD for staff. Assemble more data & other evidence: asTTle reading, homework, extracurric, Attendance, etc Interpret information Poor writers likely to play sport, speak well, read less, do little HW Analyse non NQF/NCEA data and evidence The evidence-driven decision making cycle .

8. Evidence-driven decision making Getting to information Trigger Clues found in data, hunches Explore Is there really an issue? Question What do you want to know? Assemble Get all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

9. The evidence-driven decision making cycle Trigger Clues found in data, hunches Explore Is there really an issue? Question What do you want to know? > Assemble Get all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

10. Assembling the evidence • We want to know if our senior students are doing better in one area of NCEA Biology than another. So … we need NCEA results for our cohort. • It could be that all Biology students do better in this area than others. So … we also need data about national differences across the two areas.

11. Are our data any good? A school found that a set of asTTle scores indicated that almost all students were achieving at lower levels than earlier in the year. Then they discovered that the first test had been conducted in the morning, but the later test was in the afternoon and soon after the students had sat a two-hour exam.

12. Think critically about data • Was the assessment that created this data assessing exactly what we are looking for? • Was the assessment set at an appropriate level for this group of students? • Was the assessment properly administered? • Are we comparing data for matched groups?

13. Cautionary tale 1 You want to look at changes in a cohort’s asTTle writing levels over 12 months. Was the assessment conducted at the same time both years? Was it administered under the same conditions? Has there been high turnover in the cohort? If so, will it be valid to compare results?

14. Cautionary tale 2 You have data that show two classes have comparable mathematics ability. But end-of-year assessments show one class achieved far better than the other. What could have caused this? Was the original data flawed? How did teaching methods differ? Was the timetable a factor? Did you survey student views? Are the classes comparable in terms of attendance, etc?

15. The evidence-driven decision making cycle Trigger Clues found in data, hunches Explore Is there really an issue? Question What do you want to know? Assemble Get all useful evidence together > Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

16. Analysing data and other evidence • Schools need some staff members who are responsible for leading data analysis • Schools have access to electronic tools to process data into graphs and tables • All teachers do data analysis • Data is not an end in itself - it’s one of the many stages along the way to evidence-driven decision making

17. Types of analysis We can compare achievement data by subject or across subjects for • an individual student • groups of students • whole cohorts The type of analysis we use depends on the question we want to answer

18. Inter-subject analysis • Have my students not achieved a particular history standard because they have poor formal writing skills, rather than poor history knowledge?

19. Intra-subject analysis • What are the areas of strength and weakness in my own teaching of this class?

20. Longitudinal analysis • Are we producing better results over time in year 11 biology?

21. Basic analysis

22. Basic analysis • Divide the class into three groups on the basis of overall achievement • Identify students who are doing so well at level 2 that they could be working at a higher level • Find trends for males and females, those who are absent often, or have many detentions • Compare this group’s external assessment success rate with the national cohort.

23. Reading levels – terms 1 and 4 .

24. Making sense of the results Think about significance and confidence How significant are any apparent trends? How much confidence can we have in the information?

25. Making sense of the results This table shows that reading levels overall were higher in term 4 than in term 1. Scores improved for most students. 20% of students moved into Level 5. But the median score is still 4A. Is this information? Can we act on it?

26. The evidence-driven decision making cycle Trigger Clues found in data, hunches Explore Is there really an issue? Question What do you want to know? Assemble Get all useful evidence together Analyse Process data and other evidence > Interpret What information do we have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

27. Making sense of information Data becomes information when it is categorised, analysed, summarised and placed in context. Information therefore is data endowed with relevance and purpose. Information is developed into knowledge when it is used to make comparisons, assess consequences, establish connections and engage in dialogue. Knowledge … can be seen as information that comes laden with experience, judgment, intuition and values. Empson (1999) cited in Mason (2003)

28. Information Knowledge gained from analysing data and making meaning from evidence Information is knowledge (or understanding) that can inform your decisions. How certain you will be about this knowledge depends on a number of factors: where your data came from, how reliable it was, how rigorous your analysis was. So the information you get from analysing data could be a conclusion, a trend, a possibility.

29. Information Summative information is useful for reporting against targets and as general feedback to teachers. Formativeinformation is information we can act on – it informs decision-making that can improve learning.

30. Reading levels – terms 1 and 4

31. Questions to elicit information • Did the more able students make significant progress, but not the lower quartile? • How have the scores of individual students changed? • How many remain on the same level? • How much have our teaching approaches contributed to this result? • How much of this shift in scores is due to students’ predictable progress? Is there any data that will enable us to compare our students with a national cohort? • How does this shift compare with previous Year 9 cohorts?

32. Words, words, words … Information can … establish, indicate, confirm, reinforce, back up, stress, highlight, state, imply, suggest, hint at, cast doubt on, refute … • Does this confirm that …? • What does this suggest? • What are the implications of …? • How confident are we about this conclusion?

33. Interrogate the information • Is this the sort of result we envisaged? If not, why? • How does this information compare with the results of other research or the experiences of other schools? • Are there other variables that could account for this result? • Should we set this information alongside other data or evidence to give us richer information? • What new questions arise from this information?

34. Interrogate the information • Does this relate to student achievement - or does it actually tell us something about our teaching practices? • Does this information suggest that the school’s strategic goals and targets are realistic and achievable? If not, how should they change, or should we change? • Does the information suggest we need to modify programmes or design different programmes? • Does the information suggest changes need to be made to school systems?

35. Interrogate the information What effect is the new 6-day x 50-min period structure having on student engagement levels?

36. Interrogate the information What effect is the new 6-day x 50-min period structure having on student engagement levels? Do student views align with staff views? Do positive effects outweigh negative effects? Is there justification for reviewing the policy? Does the information imply changes need to be made to teaching practices or techniques? Does the information offer any hint about what sort of changes might work?

37. The evidence-driven decision making cycle Trigger Clues found in data, hunches Explore Is there really an issue? Question What do you want to know? Assemble Get all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?