Inter-Subject Comparability

1. Inter-Subject Comparability • A brief history of Inter-Subject Comparability (ISC) • Ofqual consultation (Dec ‘15) • Ofqual position (April ‘17) • Opportunity for A/L Bi, Ch, Ph and A/L Fr, Gn, Sp • Action needed for GCSE Fr, Gn, Sp • Evidence Helen Myers and David Blow 18 aug 17

2. A Brief History of ISC - 1 Official position for a long time was that all subjects were of equal difficulty and that a grade C (say) in one subject was the same standard as a grade C in another However, long-standing concern that this was not the case in practice - research by Nuttall in 1974; Cockcroft Report on Maths in 1982 highlighted that 25% of cohort passed O-level Maths whereas 33% passed O-level English ALIS (Carol Fitzgibbon) in 1992 showed clearly large variations in grading at A/L

3. A Brief History of ISC - 2 2004: Peter Tymms and Robert Coe from CEM in Durham coined term “severe grading” particularly with reference to A/L In 2008, QCA / Ofqual seminar on Inter-subject Comparability (see ISC Paper on official views at the time) – Helen Myers and David Blow presented on behalf of ML community (together with STEM and English reps) Robert Coe presents to ASCL Conference Then pause until Ofqual launch formal consultation on ISC in Feb 2016 Helen Myers (Past President, ALL & Asst Head, The Ashcombe School) website: www.all-london.org.uk

4. Ofqual consultation (Dec ‘15) “New research published today (18 December) examines whether different GCSE and A level subjects are comparable to each other and whether a better alignment could be achieved. The working papers identify the policy options that Ofqual could adopt in relation to the comparability of different subjects.” https://www.gov.uk/government/news/can-different-gcse-and-a-level-subjects-be-compared-accurately

5. Ofqual consultation (Dec ‘15) 6 working papers, Comparability of Different GCSE and A level Subjects in England: An introduction. Inter-Subject Comparability: A Review of the Technical Literature. Inter-Subject Comparability of Examination Standards in GCSE and GCE. Inter-Subject Comparability: An International Review. A Recent History of Regulatory Perspectives on Inter-Subject Comparability in England. Exploring Implications of Policy Options Concerning Inter-Subject Comparability. conference in Feb ’16 with presentations incl from Charles Tracy (Institute of Physics) and Nick Mair (Independent Schools Modern Language Association)

6. Ofqual: Question + 4 options  ?  Q. Should Ofqual have an approach to ISC? “No action should be taken” – this would perpetuate all the existing problems “Action to achieve inter-subject comparability” – probably too far for ALL subjects; need bands/groupings “Post hoc action to achieve inter-subject comparability” – too remote and does not address the reality in the classroom “Action to achieve an alternative to inter-subject comparability” – same norm-referencing for all subjects does not take account of cohort ability.  Need a pragmatic approach to tackling the anomalies. Vital to take action, and so need to be flexible 

7. Letter from science organisations “In April 2016, leaders of a number of high-profile science organisations wrote to Ofqual expressing disagreement with the finding in some of the working papers Ofqual had produced on inter-subject comparability. Ofqual then undertook research based on the letter to look at the progression of students from GCSE to A level.”Ofqual website https://www.gov.uk/government/publications/inter-subject-comparability-2015-to-2016

8. Ofqual policy position Written by Dennis Opposs, and presented to Ofqual Board Nov 2016; published April 2017 Ofqual moved quickly at point of publication with purdah imminent and combined with announcement on A/L ML native speaker research https://www.gov.uk/government/news/setting-grade-standards-in-a-level-modern-foreign-languages https://www.gov.uk/government/publications/inter-subject-comparability-2015-to-2016

9. Ofqual policy position - 1 “Ofqual has decided that the challenges apparent in conceptualising and measuring how subjects may compare means that it will not try to take coordinated action to align grade standards across the full range of GCSE or A level subjects. This decision follows more than a year of public discussion and debate about a number of potential policy options.” “However, the regulator will consider making one-off adjustments to standards where there is a compelling case. Ofqual has already begun conversations with subject communities in relation to A levels in physics, chemistry and biology, as well as MFL.”

10. Ofqual policy position - 2 “Sally Collier, Chief Regulator, Ofqual, said: ‘Everybody expects that grade standards between exam boards in any particular subject should be comparable. However, inter-subject comparability is more difficult to consider. We are extremely grateful to the stakeholders who have offered their opinions and the many researchers who have helped us reach our policy decision. It, combined with the actions we are taking this summer specifically in relation to A level languages, will mean we are better placed than ever to ensure standards in GCSEs, AS and A levels are appropriate.’”

11. Reflection Suggest active support for science organisations and ML organisations in making “compelling case” for one-off adjustments to A/L Bi, Ch, Ph and A/L Fr, Gn, Sp to take up the specific invitation offered by Ofqual Urge Ofqual to also allow “compelling case” for one-off adjustments to GCSE Fr, Gn, Sp What is the evidence?

12. Severe Grading in MFL- brief history Peter Downes (former Chief ML examiner, former Head, former President ALL & SHA now ASCL) . Peter Downes raised concerns over time (from O-level), leading to recommendation in the Nuffield Report Series of presentations from 2005 onwards by Helen Myers and David Blow using official data (DfES/DCSF/DfE, Ofsted, QCA/Ofqual etc) Continuing concern in ML community, esp from 2004 when Languages no longer compulsory at KS4 and GCSE numbers go into freefall Series of meetings by ALL, ISMLA, ASCL etc with ministers, QCA, exam boards ISMLA, ALL, ASCL and HMC have worked closely together on severe grading in MFL for over 10 years now – state & indept; teachers & Heads

13. Severe Grading in MFL- brief history (2) Initially, 2005 – 2008 during Dearing Review etc, phrase was “perception” 2008 Ofqual seminar on Inter-subject Comparability (see ISC Paper on official views at the time) – Helen Myers and David Blow presented on behalf of ML community (together with STEM and English reps) – see later slides So focus then was on convincing about the reality; that has now been accepted and evidenced in the Ofqual ISC papers

14. Dearing Review National outcry at drop in numbers GCSE Aug 06 - Alan Johnson sets up Dearing Review “Mr Johnson wants to see what more can be done to encourage14-16 year olds to study GCSE or other language courses leading to a recognised qualification.” DfES Press Notice 2006/0144 Oct 06 Meeting Nov 06 with Lord Dearing Idea of broad comparability with Maths grading Awareness of public and political sensitivity

15. Dearing Review Led to recommendation in Consultation Report (Jan 07) confirmed in Final Report (Mar 07) to have definitive study published on “perception” of severe grading (with implicit corollary that there should be action...) - the consultation had “found strong confirmation of the view that award of grades is more demanding than for most other subjects”

16. Dearing Review – press - 2007 Dr Coe, of Durham's curriculum, evaluation and management centre, believed such trends were repeated in other years, but insisted "the question of difficulty is not about content of the subject ... It is purely about the examination and grading process." David Willetts, Conservative education spokesman, said: "If there is evidence modern languages is tougher than other GCSEs, then that is something that has to be corrected. They should be the same level of challenge as traditional academic GCSEs."http://education.guardian.co.uk/gcses/story/0,,2031769,00.html James Meikle, education correspondent Monday March 12, 2007

17. Grade standards in GCSE modern foreign languages February 2008 QCA/08/3570 12. To test its practical application we asked AQA, Edexcel and OCR to calculate what changes there would be to their 2007 GCSE French results if the grades were based only on the relationship between candidates’ key stage 3 test scores and their results in GCSE mathematics. The analyses indicated that the changes would be marked at the higher grades. About half the candidates presently awarded a grade B would gain a grade A as the threshold mark or performance standard for a grade A would have to move down by about half a grade width. There would be a similar effect at grade C. QCA Report – Feb 2008 17

18. Joint ASCL, ALL, ISMLA statement re QCA report 2008 in response to Dearing Dr John Dunford, General Secretary of ASCL said: “The main problem is that modern language GCSEs are graded more severely than other subjects and it is extremely disappointing that the QCA, while recognising the problem, does not intend to bring the grading of GCSE languages into line with mathematics, English and other similar subjects.” “This is not an issue of ‘dumbing-down’ language GCSEs, it is levelling the playing field with other similar subjects .....not to maintain a grading standard that was unfair to begin with. 20 Feb 2008

19. Citations in Ofqual docs "Inter-Subject Comparability: A Review of the Technical Literature- ISC Working Paper 2“ Page 10: "In more recent years, the debate on inter-subject comparability has been driven largely by subject groups. Those with an interest in languages (Myers, 2006; Dearing and King, 2007) and mathematics and sciences (Coe et al., 2008; Royal Society, 2008) have been particularly vocal.“ Page 45: Myers, H. (2006) The ‘severe grading’ of MFL grades at GCSE and A level. London, Association for Language Learning. Inter-subject comparability: research documents www.all-london.org.uk/severe_grading.htm

20. Official Reports JCQ Report into M.L. (July ‘14) – GCSE and A/L Paper 1: The decline in uptake of A-level Modern Foreign Languages: literature review (Dr Debra Malpass) Paper 2: Attitudes towards Modern Foreign Languages (Ipsos MORI) Paper 3: A-level Modern Foreign Languages: why so few A*s? (Ben Jones, Beth Black & Christine Wong) Paper 4: A-level Modern Foreign Languages: assessment instruments and script analysis (Beth Black) Ofqual Report (Sep’14) – A/L esp A* JCQ report and Ofqual report into ML looked in detail at all the factors from content through to assessment (questions) to marking to grading that contribute to variability (some typical, others undesirable) http://www.jcq.org.uk/media-centre/news-releases/mfl-review-press-notice Ofqual Report: http://ofqual.gov.uk/news/improvements-made-level-foreign-languages/

21. A* in A/L ML Subtle distinction NOT looking at the judgemental boundary A/B BUT looking at the proportion of A* relative to A + A* We are comfortable that the overall % of A+A* will vary from one subject to another because cohort profiles vary, but argue that the RELATIVE proportion of A* should be comparable esp amongst “facilitating subjects”

22. A* in A/L ML The highly technical 2014 Ofqual report showed that the “difficulty” of the exam was out of kilter with the ability profile of the cohort, as the latter had changed significantly in last 10 years becoming almost all very high prior attainment Because of the way A* was calculated, increasing the proportion of challenging questions to bring the two back into line would affect the proportion of A* to A + A* without affecting the A/B judgemental boundary – highly technical reasons Put into practice in Summer 2016 succcessfully And will continue into reformed with “comparable”

23. A/L ML native speaker “The decision stems from new research, published by the regulator today, which suggests that awarding should take into account the fact that native language speakers take these subjects. The adjustment to grade standards will be decided in early summer. “ “If the ability of the cohorts is similar to previous years we would anticipate small increases in the proportion of students getting top grades in each subject this August.” Ofqual https://www.gov.uk/government/news/setting-grade-standards-in-a-level-modern-foreign-languages

24. A/L native speaker - Fr 320 = A Note relatively high proportion of native to non-native at lower grades 200 = D

25. A/L ML native speaker Outcome in Aug 17: Ofqual Key Points: 3. In A level French, German and Spanish, outcomes at grades A* and A are up following an agreement with exam boards to make an adjustment to take account of native speakers in these languages. So the percentage of candidates gaining grades A and A* has risen from 2016 to 2017 in all 3 of those languages: French 37.3% to 39.0%; German 39.6% to 41.4% and Spanish from 34.4% to 36.9% The proportion of A* grades relative to the number of A and A* grades in total is now around 26% meaning that ML candidates are being treated comparably to other "facilitating" arts subjects such as History. https://www.gov.uk/government/news/guide-to-as-and-a-level-results-for-england-2017

26. Severe grading at A/L Sadly this has become almost “horse has bolted” as much of the damage has been done. In 2004 clearly, severe grading was a major issue especially at AS level. Look at the stark drop in candidates in the last 10 years at E grade; no. of E grades in 2015 was 29% of the number in 2002

27. A/L nos Which has led to a major mismatch at A/L between the “difficulty” of the exam and the cohort profile – Ofqual report And since 2010 we have had the A* issue as a final nail in the coffin A&A* stable until 2012, when a drop of 1,000 candidates to 2015

28. GCSE MFL grading • Our outline principle is that typical students studying a mainstream GCSE ML should have a reasonable expectation that they will get similar grades across EBacc subjects, without any systematic variation. • So, looking at the many different ways of analysing cohort performance, ML should be brought broadly into line with the other EBacc subjects e.g. Maths

29. Issues for pupils • Perceive that they are performing worse in languages in other subjects • Reduces the take-up at KS4 and A-level • In each case, pupils are doing relative comparisons on grades (note key issue at AS in French as compared with other subjects - at A2 Univs can make subject by subject adjustments)

30. Issues for staff & managers • Heads and Governors may think that languages are “under-performing” so issues re Performance Management etc • External people e.g. Ofsted may think same or that the school as a whole is “under-performing” against current performance measures

31. Language Trends survey - 2016 It says: "The exam system is seen as one of the principal barriers to the successful development of language teaching. "The comparative difficulty of exams in languages in relation to other subjects, and widely reported harsh and inconsistent marking, are deeply de-motivating for both pupils and teachers." The report says the EBacc, where pupils have to study English, a language, maths, science and history or geography to GCSE, "appears to be having very little impact on the numbers of pupils taking languages post-16". Uptake after GCSE is found to be a particular concern, with some state schools suggesting the small numbers of students opting to take languages at A-level means the subject is becoming "financially unviable".

32. Impact on nos. at GCSE ML Slight rise in 2013 after announcement of EBacc in 2011 Under half compared with 2004 Drop in 2016 as consequence of Progress 8 EBacc3 definition

33. Impact on nos. at A/L Qual. Reform and funding cuts hitting ML at A/L Hitting indept sector as well http://educationdatalab.org.uk/2017/05/young-people-are-taking-fewer-a-levels-as-qualification-reforms-kick-in-and-per-student-spending-falls/

34. Way forward - 2008 data from CEM SCORE 2008 paper We believe it is unrealistic to move to comparable grading for all subjects Currently bands ML - His, Geo, Maths, Sci - English - Art, Drama, PE, We are proposing simply that ML moves into the 3rd band - tiny changes in grade boundaries involved, similar to those which have taken place - “one-off step change” Slide from Ofqual Oct ‘08 ISC seminar

35. data from Ofqual ISC paper 3 Way forward - 2017 We believe it is unrealistic to move to comparable grading for all subjects We are proposing simply that ML moves up to Hi / Ge (tiny changes in grade boundaries involved), similar to those which have taken place - “one-off step change”

36. Original Ofqual graph in ISC Paper 3: 2010 - 13 Ma + Sci En Ge Sp, Gn, Fr . Top row: grade A Middle row: grade C Bottom row: grade E

37. Media Daily Mail Jan 2016 article about the Ofqual ISC papers "At GCSE the hardest subjects – modern foreign languages, Latin and statistics, are about half a grade harder than the average." http://www.dailymail.co.uk/news/article-3398347/Official-Exams-soft-subjects-really-easier-Board-admits-pupils-unfairly-marked-courses-maths-discuss-overhaul-grading-system.html

38. Importance of language Terminology is important. Words like “hard”, “standards”, “demand” carry multiple connotations. "An earlier version of this article used the expression 'marking down'. Ofqual would like us to clarify that the issue not about the quality of the marking but rather about how likely a student is to get a good grade depending on the exam subject. We have amended the article to avoid confusion.“ note at end of Daily Mail article on ISC (Jan ‘16) thanks to Peter Tymms and Robert Coe for the phrase “severe grading” from Sept ‘04. The Times: “Student offers “should recognise easy A-levels”

39. Distractors Need to avoid invalid comparisons / deductions re severe grading, numbers etc STEM v ML etcOur line on numbers has been: “Making changes here [grading] will not in itself solve the overall problem [pupil nos], but, to use a metaphor, will remove having a hand tied behind one's back.” ASCL / ISMLA / ALL to Dearing Similarly with pedagogy, time, etc etc Need to look at each subject, Govt incentives, etc

40. Sensitivity We do not underestimate the sensitivity of this topic Through all of our involvement, have sought to make politically (and media) realistic proposals - e.g Lord Dearing in 2006 liking the idea of comparability with Maths: “no-one will say that Maths is easy” So he would have been pleased with headline in Jan 2016 Daily Mail article on ISC !

41. Latest situation June 17: ASCL adopt Position Statement “ASCL supports Ofqual in tackling severe grading in GCSE MFL so that students learning mainstream GCSE MFL should have a reasonable expectation that they will get similar grades across EBacc subjects, without any systematic variation” https://www.ascl.org.uk/policy/position-statements/curriculum-and-assessment.html

42. Joint letter to Ofqual from ALL, ASLC, HMC, ISMLA We cautiously welcome the action they [Ofqual] propose to take this year regarding grading at AL ML this year linked to the issue of native speakers, as we did to the action taken effective in June 2016 to address the anomalies in the proportion of A* grades in A/L ML We welcome the offer of further work on A level ML We are deeply disappointed that Ofqual's announcement about inter-subject comparability (ISC) does not address the issue of severe grading at GCSE.

43. Joint letter - 1

44. Joint letter - 2

45. Joint letter - 3

46. Next steps Representatives of the 4 organisations are meeting with Ofqual in September to discuss the points raised