1 / 45

Questions of evaluation

This article discusses the evaluation of English for Academic Purposes (EAP) programs and its impact on the success of international students in university. It explores various evaluation techniques and findings, as well as suggestions for improvement.

mlindo
Download Presentation

Questions of evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Questions of evaluation Tony Lynch

  2. Somebasicquestions What? Who? How? When? / Howoften? Why?

  3. 1984

  4. Terminology of the time • overseasstudents • Personnel • problems • in thefuture • teachers • forks

  5. EAP in theearly 1980s • relatively new • English forAcademicStudy(1975) British Council • English forAcademicPurposes(1977) BAAL / SELMOUS

  6. EAP at Edinburgh • InstituteforAppliedLanguageStudies (1979-2010) • Pre-sessionalsummercourse • Matriculationtesting (ELBA, ELTS, TEAM) • In-sessioncourses (English LanguageTesting and Tuition – ELTT)

  7. ELTT in-sessioncourses 1984 AutumnTerm • Listening and Notetaking • Reading • Speaking • Grammar Winter vacation: Intensivecourse (56 hours) Spring Term • Writingexamanswers • Thesiswriting

  8. Theawkwardquestion DrCarpenter (Agriculture): Wouldmystudents´ vacation time be betterspent in ELTT classes, orstudyingfortheirMSccourse?

  9. Thebasicquestions What? Who? How? When? / Howoften? Why?

  10. Example: Howthequestionsinteract • The impact of English for Academic Purposes (EAP) programs on international students' success in university = • The subjective perceptions, gathered by interview, of the • impact of one EAP pre-sessional programme • on eight international students' success • in coping with the demands of the first term • oftheir university degree course

  11. What to evaluate? • Programmes • Courses • Tasks • Materials • Techniques

  12. Techniques • Evaluating alternative forms of feedback on EAP students´ spoken English

  13. Implicit feedback Performance Monitoring

  14. Transcript-based feedback study • Performance • Feedback • Revision • Second round of performance

  15. Explicit feedback (via transcripts) Performance Revision

  16. Comparing two techniques Student-initiated feedback • taskrehearsal • recorded performance • transcript 1 (verbatim) • transcript 2 (revised) • transcript 3 (revisedby T) • discussionwith T • recorded performance 2 • final performance (4 wkslater) Teacher-initiatedfeedback • taskrehearsal • recorded performance • T listens to recordings • transcribes problematicextracts • Ss revise extracts • discussionwith T • recorded performance 2 • final performance (4 wkslater)

  17. Findings • Student-initiated feedback group achieved higher accuracy in final performance of L2 items highlighted in transcripts • Student-initiated feedback group also expressed greater satisfaction with their feedback technique

  18. Whodoestheevaluation? • Teachers • Students • EAP course directors • University departments • External bodies

  19. Students • Pre-sessional EAP programme 1995 • Evaluation: Suggestionsforimprovement • ”More opportunities to speak English” • ”With a normal person, not a teacher” • Course Assistant 1996 onwards • Assumption confirmed by research • S/CA talk (in class) differed in topic and structurefrom S/T talk

  20. External bodies • 1992 BASC accreditation visit • Language learning strategies → independent learning • 1993 Learning to Learn English (Ellis & Sinclair) • 1993 evaluation of LLE use (Stratford MScdissertation) • 1994 PROFILE – Principles, Resources and OptionsfortheIndependentLearner of English (Lynch & Anderson) classroomversion • 1996 and 2007: PROFILE self-studyversion • 2012: online version, Effective English Learning • http://www.ed.ac.uk/files/imports/fileManager/UNIT_8_Pronunciation.pdf

  21. How to evaluate? • Formal (tests, questionnaires, coursecommittee meetings) • Informal (tutorials, conversations) • Reflection

  22. Reflectionon pre-sessional EAP • “EAP classes are long and very similar. In Writingwetalk, in Speakingwetalk, in Listeningwetalk”. • “Thesummerwasmucheasier. Nowwehave to hurryallthe time”. • EAP teacher: 90-minute norm • AL lecturer/tutor/supervisor: 50 minutes • Lack of fitbetween EAP and subjectstudy • Pre-sessionalbuildingunrealisticexpectations?

  23. Restructuring of 2012 pre-sessional Format • 50 minutes • Lectures; lab-based practicals; and tutored classes Timing • All sessions in morning (9:00-13:10). Afternoon 1:1 tutorials Content • Preparation → lecture → practical → feedback

  24. Anon (senior colleague) • “Thisismuch more likethelivesthey´lllivefortherest of theyear”

  25. When to evaluate (and howoften)? • End-of-course: annualcycle • Mid-course: opportunityfor a response • Post-course: +experience of degreework

  26. Winter Vacationwritingcourse (1996) • Intensive: 20 hoursoverfourconsecutivedays • Ssproduced complete draftessay (commontopic) • 12 study units (e.g. Day 3;Criticaldistance / Caution / Coverage ) • Dailyevaluationsheetonclass´swork / materials / tuition • Days 2, 3 and 4: Opportunity to discuss / adjust / negotiate • End of course: Overallevaluation of course • Followingterm: Evaluation of usefulness of materials

  27. Whyevaluate? • Narrowly, to improveEAP support • throughrevision of materials, courses, etc. • throughteachereducation • More broadly, to enhancestudents´readinessfordegreework • But… languageonlyoneelement

  28. “Do foreignstudentshaveproblems?” (Davies 1977) Edinburgh University (students) Edinburgh FE college (tutors) language academicstudies social contacts accommodation finance health • accommodation • social contacts • climate • finance • academicstudies • language • health

  29. Students’ evaluation of gainsfrom a course • Writing Up QualitativeResearch(piloted 2014) • confidence to ask supervisor Qs • interactionwithotherstudents • awareness of PhD studentnetwork • betterwritten English

  30. Perspectivesonevaluation • Process • Product • Outcome (academic)

  31. Roomforexpansion? In today´sabstracts: • perceive/perception21 occurrences • experience22 • reflect/ion 21 = total 64 • success/outcome7 (2 related to successonanacademicassignment)

  32. MSc outcome and matriculation scores: Edinburgh • ELTS / outcome = 0.35 • TEAM 1 / outcome = 0.32 • IELTS study, 1994 [n=24]: • IELTS / outcome = 0.39 • TEAM 1 /outcome = 0.49 • (PBT-TOEFL / outcome 0.34)

  33. Back to DrCarpenter…

  34. ELTT in-sessioncourses 1984 AutumnTerm • Listening and Notetaking • Reading • Speaking • Grammar Winter vacation: Intensivecourse (56 hours) Spring Term • Writingexamanswers • Thesiswriting

  35. DevelopmentssinceDrCarpenter´squestion… In 1984 • Face-to-faceclassesonly By 2014 wehadadded: • Online (email) courses, with EAP tutor feedback • Blendedcourses - f2f and email • Independentstudyversions – adaptedmaterialswithstudy notes • School-basedwritingcourses – withfeedbackfrom EAP tutor and final feedback/assessmentfromsubjectspecialist

  36. MyquestionsforDrCarpenter • Whenwould be thebest time foryourstudents to getadditionalwritingtuition? • Wouldyoulikeus to run a dedicatedwritingcourseforyourMScgroup?

  37. Summary Focus • What? Who? • How? When? / Howoften? • Why? Perspective • Product • Process • Outcome

  38. Summary Focus • What? Who? • How? When? / Howoften? • Why? Perspective • Product • Process • Outcome (= To whateffect?)

  39. !

  40. A.J.Lynch@ed.ac.uk

  41. Publishedworkfeatured in the talk Davies A. 1977. Do foreignstudentshaveproblems? In JB Heaton & AP Cowie (eds) English forAcademicPurposes: Papersonthelanguageproblems of overseasstudents in HigherEducation in the UK. Reading:BAAL/SELMOUS. 34-36. Lynch T. 1994. TheUniversity of Edinburgh Test of English at Matriculation: Validationreport. Edinburgh WorkingPapers in AppliedLinguistics5: 66-67. Lynch T. 2007. Learningfromthetranscripts of an oral communicationtask. ELT Journal61/4: 311-320. Lynch T. 2014. Writing Up Qualitative Research. Independent study version. University of Edinburgh. http://www.ed.ac.uk/files/atoms/files/writing_up_your_phd_qualitative_research.pdf Lynch T. & K. Anderson. 2003. Learner/non-teacherinteractions: Thecontribution of a courseassistant to EAP speakingclasses. In J Burton & C Clennell (eds) Interaction and Language Learning.Alexandria, VA: TESOL. 7-22. Lynch T. & K. Anderson K. 2012. Effective English Learning. University of Edinburgh. http://www.ed.ac.uk/files/imports/fileManager/UNIT_1_Preparation.pdf

  42. (Extra slides)

  43. Matriculation tests at Edinburgh • 1967-82 Edinburgh Language Battery (ELBA) • 1982-86 English Language Testing System (ELTS) • 1987-1994 TEAM version 1 • 1994-2000 TEAM version 2 • 2001- ? TEAM version 3

  44. A pottedhistory of IALS, ELTC and ELE • Board of Extra-Mural Studies ↓ • Department of Adult and ContinuingEducation ↓ • Office of LifelongLearning ←← ↓ • Centre for Open Learning • (Dept of AppliedLinguistics) begat • InstituteforAppliedLanguageStudies (1979) ↓ English LanguageTeaching Centre (2010) • now English LanguageEducation

More Related