Acknowledgements

1. Enriching understandings of the contribution of language background and time-on-task to school achievement in Asian languages Presenters Angela Scarino, Catherine Elder, Michelle Kohler, Noriko Iwashita, Sun Hee Ok Kim, Andrew Scrimgeour SAALE

2. Acknowledgements • Other members of project team • Dr Anne-Marie Morgan –Project officer • Dr Ute Knoch and Hyejeong Kim – Research data managers • Ms Jamie Mazraeh- Systems developer • Ms Anna Dinys – Administrative officer • Mr Ari Bickley – Web developer • Dr Robyn-Spence-Brown – Consultant • Languages and Asian Studies section, National Curriculum Branch, DEEWR • Amanda Day – Director • Josephine Holzner – Project officer • Project Advisory Group • Ms Connie Andreana, • Ms Gillian Cordy • Ms Kylie Farmer • Dr Antonio Mercurio SAALE

3. The need for contextualised descriptions of learner achievement in Asian languages Angela Scarino SAALE

4. Context - 1 Variability in: • languages • learners (learner background in the TL) • program conditions (time-on-task), intensity, pathways) • orientation of programs (program-type, view of language and culture) • teaching, learning and assessment practices (local and state) SAALE

5. Context -2 1. Competing discourses • Assessment of learning through generic curriculum and assessment frameworks  improvement through standardisation • Assessment for learning  improvement through sensitivity to variability  Role of teachers and their understandings of (1) assessment and learning to assess (2) learner achievements – all of which are viewed through existing professional and cultural ways of knowing and doing. 2. … and with the development of the Australian Curriculum we have entered another phase in curriculum and assessment framework development. SAALE

6. Different conceptions of ‘achievement’, K-12 - (1) • Relates to the nature of the construct (i.e. communicative language ability/communicative language use) • Influences the dimensions of language use/performance that are considered: • language • language use in context/diverse contexts • language use in context for diverse purposes • language and literacy • language and content • language and culture • language and learning SAALE

7. Different conceptions of ‘achievement’, K-12 -(2) • A ‘proficiency’ view (e.g. Common European Framework of Reference) • a focus on what students can do in the target language irrespective of the context in which it is acquired • foregrounds the scaling process • An ‘achievement’ view • a focus on what students can do in the target language based on the context of the program: conditions, curricula • sensitivity to language and learning • foregrounds sensitivity to context: especially time-on-task and learner background as major variables that structure teaching, learning and assessment SAALE

8. The SAALE Project Two studies: • Conceptualising, gathering and analysing data to provide the basis for establishing an initial baseline through descriptions of achievement. • Investigating the impact of two particular variables on student achievement: learner background and time-on-task. SAALE

9. Process of research and development • Establishing a network of contributors • Consultations and reviews: • consultation workshop • literature review on - time-on-task and learner background - research available in specific languages (Chinese, Japanese, Indonesian, Korean) - national and international assessment systems - issues in language assessment especially young learners • Analysis of state and territory systems for grouping learners (at senior secondary level) SAALE

10. Considerations • Assessment at 3 assessment points (Year 6/7 / 10 / 12) • 4 specific languages: Chinese, Japanese, Indonesian, Korean • Curriculum and assessment based descriptions • Research based • Catering for the span of achievement • Dimensions of achievement • Documenting time-on-task and language background (self report) • Quantitative and qualitative data analysis SAALE

11. Assessment instruments: development and implementation • Learner background and program profile questionnaire • Test specifications and assessment instruments: reading, writing and oral • Sampling • Marking procedures • Analyses – of reliability, of program profile data, of test score data according to language background and time-on-task variables • Development of descriptions of student achievement and exemplars. • for each group identified as meaningful for each language at the three assessment points SAALE

12. Contextualised descriptions of achievement • Student groupings provide the structure of the descriptions at a macro level – derived from statistical analysis • Curriculum-based descriptions – derived from qualitative analyses of student responses and teacher expertise • Exemplars SAALE

13. Quantitative analyses and key findings Catherine Elder SAALE

14. Aim of quantitative analysis • To determine any differences in level of achievement according to language background and time-on-task variables • To aid decisions on groupings and selection of “high” and “average” samples for profiling purposes SAALE

15. Data set 1 • Program profiles • years of instruction • years of compulsory study • other languages offered at the school • availability of a study abroad program • availability of transition arrangements with other schools • practices for streaming or grouping students • specific information about the program at 3 year levels: number of students, learner background, number of lessons per week, duration of lessons, and at Year 12 the actual course offered (Beginners, Continuers/Second Language, and Background Speakers/First Language). SAALE

16. Data set 2 • Background information from participating learners • country of birth • parents’ country of birth • time spent in country where target language is spoken • year of arrival in Australia, if born overseas, e age on arrival, actual years of study completed overseas, language of instruction of the overseas education, study of the target language overseas • years of studying target language in Australia • target language learning at community school • learning school subjects through the target language in Australia • learning of other languages at school, in addition to the target language • languages used at home and domains of use. SAALE

17. Criteria for grouping learners • First language learner • Born in a country where target language (or a variety of it) is official medium AND arrived in Australia at age 8 or more AND • The target language was the first language used before starting school AND/OR uses the target language at home. SAALE

18. Background language learner • Born in Australia but with one or more parents born in country where target language is official medium AND target language was first language used before starting school AND/OR uses the target language at home OR • Born in a country where target language (or a variety of it) is official medium BUT arrived in Australia before the age 8 (and therefore has limited experience of target language medium instruction). SAALE

19. Second language learner • Born in a country where the target language (or a variety of it) is NOT the official medium AND • No more than one parent born in the country where the target language is official medium AND • First language before school AND/OR language used at home is NOT the target language or a variety of it. SAALE

20. Data set 3 • Scores on custom-designed assessment tasks or school examinations at relevant year levels • Years 6/7 and Year 10 • Reading • Writing • Oral (for a subset of Yr 10 learners only) • Year 12 • Writing • Oral (for a subset only) SAALE

21. Assessment criteria • Writing • Content • Vocabulary • Forms/structures • Discourse • Scripts and/or characters (except for Indonesian) • Speaking • Content • Vocabulary • Forms/structures • Fluency • Intelligibility • Comprehension • Discourse SAALE

22. Procedure • Divide into language background groups for each language • Compare mean scores for Reading, Writing and Oral for each language background group (t-test/ANOVA) • Calculate effect of other variables where relevant • L1 (English vs. other) • Study of another language (apart from TL) • Participation in community schooling • Years of instruction (years) • Intensity of instruction (minutes per week) • In-country/ study abroad experience SAALE

23. Some key findings: language background • Statistical analysis supports language background groupings used for descriptions of achievement • First language learners consistently achieve at higher levels than second language learners • Highly variable levels of performance among background language learners, not always statistically distinct from first and second language learner groups • Some L1 groups do better than others (e.g. learners with Chinese & Korean as L1 achieve better in Japanese than those with English as L1) • Studying multiple languages may aid achievement SAALE

24. Some key findings: time-on-task • Difficult to disentangle time variables from language background • Studying for longer at primary school yields high levels of achievement at Year 6/7 • Early starters do not necessarily do better at Year 10 (Indonesian and Japanese) • Intensity of instruction (via bilingual education or study abroad) yields higher levels of achievement (in Japanese and Chinese respectively) SAALE

25. Issues in describing achievement in primary (Indonesian) language programs Michelle Kohler SAALE

26. Capturing achievement in primary language programs Designing and eliciting: a suitable protocol • Reading and Writing - pen and paper ‘test’ • Oral - recording of interaction/lesson AND focus group ‘interview’ A protocol that: • recognises achievement is program, task & text dependent • captures evidence of learning with and without support • enables a range of performance across learner sample Challenge of capturing achievement that is context dependent in a way that can be generalised across the learner sample Culture of assessment in primary – characterised by short term, single tasks, high level of support SAALE

27. Time on task matters- primary Indonesian Mean Indonesian Writing & Reading by years of instruction *First year of junior secondary (i.e. Year 7 in some states) Sustained language study at primary results in higher levels of achievement than what is attained in a single year of study at secondary SAALE

28. Descriptions: Year 6/7, L2, Writing-high Forms and Structures Students create simple sentences using the subject-verb-object construction. Students use the structure of the adjective following the noun at the clause level when modelled (e.g. Tempat favorit saya rumah teman). They use a possessive pronoun following the object if modelled (e.g. Nama teman baik saya Alice) and the possessive form –nya (e.g. Teman saya namanya James, rupanya tinggi). Students use the simple verb suka to express their preferences (e.g. Saya suka…). They negate verbs by using tidak (e.g. Saya tidak suka…) and use compound verbs (e.g. Saya tidak suka membaca buka, Saya suka main tenis). Activities are typically described using ber- verbs (e.g. belajar, berbelanja, berjalan)and a small range of formulaic me- verbs (e.g. membeli, menonton, mendengarkan). Verbs are used at times with auxiliaries and adjectives extending the idea (e.g. Saya suka sekali… Saya mau ke sekolah naik bis). Students use prepositions of place di, dari, and ke to refer to where events are located (e.g. Saya mau ke sekolah naik bis, Saya tinggal di …Saya berasal dari Australia). Students use cardinal numbers following the noun (i.e. as an adjective) to refer to quantities, including when describing people (e.g. Kakak saya satu. Adik saya dua,Saya ada dua kakak). Occasionally preferences or opinions are attempted using comparatives and superlatives, for example, Saya suka sekali futbal, … lebih kecil (daripada) kamarsaya. Opinions are occasionally expressed using colloquial terms that reflect a judgment (e.g. asyik). SAALE

29. Examples: Year 6, L2, Writing-high (Task: Self-introduction) name name name Learner Background and Program Context This student is a second language learner at school. Her first language is English. She and her parents were born in Australia. The student has studied Indonesian at school for 6 years. The Indonesian program has two 50-minute lessons per week. place name name SAALE

30. Features of achievement in this example The student provides a range of information, including personal details such as name and age, family, pets, likes and dislikes, and future career. The response is sequenced logically, moving from personal details to family, hobbies, preferences, and friends. The student refers to quantity using the numbers and the nouns for animals (anjing, kucing)to describe her pets. She uses several adjectives to describe her own character (i.e. jujur, ramah, rajin). She uses the personal pronoun saya and its possessive form accurately (e.g. keluarga saya, teman saya). She uses a range of simple verbs such as suka, makan, minum, tinggal as well as formulaic ber- and me- words (e.g. berenang, membaca, mempunyai, menonton). She uses the formulaic phrase saya mau menjadi to state her preferred occupation. Spelling is accurate throughout including less familiar words such as dokter hewan.   The student creates cohesion using the simple conjunction dan. The response is structured using a series of sentences starting with saya, with occasional variation of subject (e.g. teman saya, keluarga saya). SAALE

31. What is important for Indonesian? • Evidence of the cohort - majority L2 learners (some Malay, varied degrees of ‘background’) • Achievement is context dependent e.g. more time on task = higher achievement in primary • The data reflects current orientation in programs e.g. language and cultural awareness, communicative competence, continuous/supported assessment • Challenge of defining achievement for (primary) school language learning • A suitable construct e.g. macro skills, communication strategies, higher order thinking, use of ICTs, intercultural understanding (NB student survey data yet to be analysed) • Ways of attending to context (learners, time, and…) SAALE

32. Variation in performance on a communicative writing task (Japanese) Noriko Iwashita, University of Queensland SAALE

33. Aim • To present a snapshot of variation in Year 12 writing task performance focusing on discourse feature SAALE

34. Japanese learners in Australia • Mainly second language (L2) learners • A substantial number of students are native speakers of a Chinese or Korean language. • In-country experience at senior level • Some background speakers • Some first language (L1) speakers SAALE

35. Assessment Criteria - Writing • Content – amount/quality of information to fulfill the task requirement • Vocabulary – choice of words • Forms/structures – choice and accuracy • Discourse - cohesion, coherence, genre, awareness of the reader, register, structure - e.g., use of pronouns, connectives, sentence final particles, plain form • Scripts and/or characters – accurate choice of scripts and a variety of characters SAALE

36. Why discourse? • Rich content • Correct use and choice of forms and words • Accurate use of scripts and use of characters DOES NOT guarantee the text has a communicative value • Analysisbeyond sentence level analysis is required. SAALE

37. Analysis • Statistical analysis → L1 > L2 • Qualitative analysis of discourse features • Compare L1 vs. L2 speakers • Compare Average vs. High L2 learners • Descriptors SAALE

38. Excerpt from Descriptors – L1 • Cohesive devices are used less often than is the case with L2 learners. Students refer to relevant information in context using あの (that) while L2 users tend to use その(that)この(this) more often than あの (e.g. あのオリンピックがアジア最大の都市東 京で再び開かれるなんてすばらしいではないか. ) • Appropriate use of register for the text type and audience (e.g. students select correct vocabulary items to differentiate slight nuances in meaning 経験する、体験する，学ぶ、身につける and use words of Chinese origin in expository writing増える 減少する instead of 減る 悪化する instead of 悪くなる). SAALE

39. (Cont’d) • Engage with the reader involving them in the composition (e.g. みなさんこんにちは。きっと皆さんもすぐにご存知でしょうが、今、 東京は2016年のオリンピック開催地の一つとして選ばれています。そこで、本当に東京でオリンピックをするべきなのかということについて、私なりの意見を皆さんに聞いていただくて、 今日はお手紙を書きます。) but also remove themselves by using indirect expressions to express their own opinions (e.g. 非常にこん難を極めるのではないだろうか。). • Deliberate use of tense shifts to make an impact in their writing (e.g. 残念ながら北京オリンピックに私は行くことができなかった。 北京に行くだけの時間も経済力もない。今日はとにかく体力を付けるために走ることにしていた。外は大雨で５メートル先も見えないような天気だ。だけど、そんなことで休むことなんて許されない。朝ゴハンを食べて全身ぼう水のカッパを着て準備ができた。). SAALE

40. Excerpt from Descriptors - L2 High • Structure of the text conforms to the chosen text type (e.g. Introduction, body, conclusion as per example in Content). According to text type, students use appropriate opening and closing (e.g. opening – みなさん、私はオーストラリアから来た です。 closing – (1) では、トレーニングがんばって下さい。よろしくおねがいします。 (2) 僕は日本にいた時を永遠にわすれません。この一年間本当にありがとうございました。) and show engagement with audience explicitly and implicitly (e.g. (1) みなさんは、僕がこの高校をはじめて来たときを覚えていますか。 (2) えみさんの聞く事は私と同じ事でした。だから、今その事について教えてあげます。). • An extensive range of cohesive devices (e.g. この、その referring to what has come before in the text; discourse markers — conjunctions 例えば、だから、はじめに、このように、 けれども、そして、なぜなら、それに、それから, つまり). SAALE

41. (Cont’d) • Use of appropriate register for the text type and audience (e.g. use of plain form for personal essay, email, すうじつかんeメールを見ていなかったから、知らなかった。 元気？NTはどう？), use of polite form when addressing the general audience (e.g. 私は東京マラソンを走りはじめて前にたくさんのことを考えなければなりませんでした。). • Awareness of different words/phrases/endings according to gender, but tend to mix them up (e.g. (1) ぼくが4月に日本に行くとき、しんが、むかえに来てくれたらうれしいけれど、 むりはしないでね。ぼくははやくしんと日本の高校でべんきょうしたい。 (2) へん事をするこはおくれてごめなさいね。ぼくのコンピューターはウィルスをもらった。 わるいね。). SAALE

42. Excerpt from Descriptors – L2 average • The structure of the text conforms to the chosen text type (e.g. introduction, body, conclusion). • There is appropriate use of cohesive devices (e.g.そして、なぜなら、でも、じゃ、たとえば、 いっぽうで、最後に) and use of references for a report (e.g. 新聞によると). • Cohesive devices この、その , referring to what has come before in the text, are observed. • Occasionally students use register inappropriately for the text type and audience (e.g. じゃ、去年オーストラリアに行った時、メルボルンだけおとずれましたね。 お会いできる (honorifics), use of あなた, きみにアドバイスを上げるよ。 They use plain form in writing an email to a friend 日本に行きたい。しんは会いたいよ。). SAALE

43. Sample - L2 High (1)

44. Sample - L2 High (2)

45. Sample - L2 Average SAALE

46. Summary • Effective use of cohesive device (e.g., あの）appropriate choice of words, forms etc. for the text are important as well as control of linguistics items (e.g., forms/structures and vocabulary) • Teaching implications • Exposure to everyday language usein different types of texts and contexts e.g., In-country experience and use of multimedia and online materials • Strategic competence SAALE

47. Korean learners: Similarities and differences in achievement across different learner groups Sun Hee Ok Kim SAALE

48. Korean as a LOTE in Australia: Current context • Small enrolment overall • All three types of learners present across year levels • Primary • Both L1 and L2 backgrounds • Junior secondary • Mostly L2 beginners in a few programs • Learners from a L1 background tend not to study Korean. • Senior secondary • The majority of the few students from both backgrounds who were enrolled at the junior secondary level discontinue studying. • The majority: Recent arrivals from Korea • Not all jurisdictions have learners in all three year levels. (Shin, 2010) SAALE

49. 3 groups of Year 10 learners of Korean • L1 learners (5 students): • Age of arrival: 9-15 years of age • Korean-medium education: 2-6 years • Background language learners (3 students) • 2 born in Australia; 1 arrived in Australia at 5 years of age • First language(s) before school: Korean (1); both Korean and English (2) • L2 learners (12 students) • All were studying Korean since Year 9. • None had experience of living in Korea. • Chinese speaking background (4); L1s not related to Korean (8) SAALE

50. Overview of achievement: Writing Figure 1: Comparison of overall achievement among learner groups SAALE