Miriam Corris,* Christopher Manning, † Susan Poetsch,* and Jane Simpson*

1. Bilingual Dictionaries for Australian Languages: User studies on the place of paper and electronic dictionaries Miriam Corris,* Christopher Manning,† Susan Poetsch,* and Jane Simpson* *Linguistics, University of Sydney †CS and Linguistics, Stanford University http://www.sultry.arts.usyd.edu.au/SULTRY/dict.html

2. Background • Dictionaries have long been seen as an essential contribution by linguists to work on endangered languages – more useful than grammars, papers • Many field linguists produce dictionaries – there has been an explosion of work in this area in recent decades in Australia • But what is the actual use and usability of dictionaries of endangered languages? • This has been little explored • Most have been produced without much consultation with speakers about their structure

3. Background (2) • We were interested in innovative computer interfaces for electronic dictionaries within this domain • But this required us to investigate the actual and potential uses of paper or electronic dictionaries for indigenous languages by various user groups • Who might use an electronic interface? • How might they use it? – For what purpose? • Would they be able to use it?

4. The promise of e-dictionaries • Why electronic dictionaries? • There are many needs for EL dictionaries, but little money/human labour available • Can make multiple versions at low cost • Can provide learner supports, active learning • Computers are engaging and exciting • Why not computers: • Cost (of computers) • Need electricity • Can’t use it in camp, under a tree, in the bath

5. Existing dictionary use studies • Almost entirely for world languages • mainly paper dictionaries, though increasing work on electronic dictionary usability • Almost entirely highly-educated people (usually university students or dictionary society members) • Studies appropriate to our domain were required • Though see (Lindstrom 1985, Hansford 1991)

6. (Australian) endangered language (EL) dictionaries • Usually alphabetical (some thesaurus/semantic) • All bilingual or trilingual (makers usually not speakers) • Mostly EL–LWC organization; only sometimes have LWC-EL finderlist • Sometimes organized via semantic domains • A few have vernacular definitions (incl. Warlpiri) • Bigger ones have examples, structured entries • Some are little more than wordlists • Structure has often been taken for granted

7. Endangered language dictionaries Reasons for EL-LWC organization: • Useful for speakers of LWC, usually including lexicographer, in decoding EL • Idea of documentation of EL in its own terms • Symbolic value Doesn’t always fit with user needs: • Commonly more need for encoding the EL Competing goals: • documentation vs. maintenance/learning • symbolic vs. practically useful organization

8. Testing • 76 speakers, semi-speakers, and learners of Australian Aboriginal languages (from primary school age through adult literacy workers) • Alawa: Feb 1999 at Minyerri – dictionary workshops and task-based activities • Warlpiri: Feb 1999 – demonstrations and observation; May 1999 – demonstrations and task-based activities; Feb 2000 – demonstration; June 2000 – task-based activities, educational courses, demonstration, and observation • Warumungu: May 1999 and June 2000 – courses, task-based activities, and observation

9. Alawa Warlpiri Warumungu Orientation • Warlpiri: active bilingual programs in three schools. Children learn Warlpiri as their first language. Young people can write Warlpiri. • Alawa: children and young adults do not speak as first language; people are most likely to be literate in English, but mainly speak Kriol. LOTE program at Minyerri. • Warumungu: children and young adults do not speak as a first language; some can write it through Batchelor College courses. No LOTE program.

10. The dictionaries • Warlpiri dictionary (unpublished, but various xeroxed versions – over 2000 pages of material): 10,000 headwords incl. subentries, lengthy definitions, exemplification. Also shorter beginner’s dictionary (100 pages) • Electronic interface: Kirrkirr • Alawa-Kriol-English dictionary: draft, A-K-E main dictionary, semantic domain list, K-A-E and E-A-K finderlists, about 250 pp A4. • Warumungu: draft Warumungu-English dictionary, currently lacks E-W finderlist

11. Current dictionary use Observed natural uses of dictionaries: • Finding meanings for old Warlpiri words for translation and art documentation • Making materials for school language programs • Browsing • Checking spelling [ + symbolic function] Factors inhibiting use: • Low levels of vernacular literacy • Lack of availability of dictionaries [not in classrooms] • They are not standardly perceived or chosen as a source of information about language

12. Sources of information “One day I was sitting around the table with three young women. We had done two two-hour sessions on different activities with the dictionary, and then I proposed that we should find all the Alawa words for different kinds of kangaroo. They said, ‘Oh, there’s the plains kangaroo, …’. They wrote down all the words in Kriol, and then they said, ‘We’ll go home and ask the old people.’ Even though the dictionary was sitting there.” – Susan Poetsch, Alawa, Feb 1999

13. Method Qualitative/ethnographic study • We demonstrated and then observed people using paper and electronic dictionaries, and got their feedback on them • We gave people dictionary-use tasks, and observed their performance • Tasks often completed with assistance • We have attempted to directly compare the usability of paper vs. e-dictionaries on tasks • but confounding factors are too great

14. Rationale • Language skills, literacy, educational skills, life experiences of subjects varied greatly • In general, very limited literacy, education, and dictionary skills, particularly for older people • Subjects required a great deal of time to complete tasks (e.g., commonly 45-60 mins for a 12 lookup crossword puzzle) • Subjects often required assistance/training • No established approaches to indigenous dictionary usability: pilot studies were needed

15. Task-based testing • Designed 21 task-based activities • Constructed assuming low levels of literacy • In the end many were never used • Order words alphabetically • Lookup for filling in crosswords • Recognizing parts of entries • Translation: understanding long entries • Sorting by semantic domain; using cross-references • Checking spelling

16. Example task: Part of Task 4a 3. How do you write bark (of a tree) in Alawa? 7. How do you write bull ant in Alawa?

17. Results • We concentrate here on practical considerations • Learning to use a dictionary: people generally lacked the necessary literacy and dictionary-specific skills to make adequate use of dictionaries • Ongoing opportunities for training are required • Keeping to conventional dictionary structure allows crossover of skills from LWC (English)

18. Alphabetical order • People know words are ordered, but not how • A separate skill from literacy; often not mastered • A big obstacle to effective lookup • The flick randomly strategy • The proceed from the beginning strategy • Linguists often regard digraphs like ng, ny as single letters in alphabetization • this retards skills transfer from English • stick to alphabetization of LWC [community choice] • Effective typography helps [cut index? – expensive]

19. Alphabetical order • E-dictionary avoids reliance on alphabetical order by providing multiple routes to dictionary information • typing, fuzzy spelling, lists, links [sight words] • The ability to type in words was quickly adopted • can just type a few letters • In an e-dictionary, it is easy to provide multiple alphabetizations, but there is unlikely to be value in doing this [standardization]

20. Comprehensiveness • Linguists, lexicographers, and sometimes older (commonly illiterate) speakers want to include as much information as is known • For most potential users, with emerging literacy skills, short entries with illustrations are essential: • information overload of large dictionaries is overwhelming • even a shorter dictionary will be a challenge to use effectively

21. Comprehensiveness • Competing needs • Teachers felt that comprehensive printed dictionaries were too long and detailed for school use • But they remarked with frustration that the beginner’s dictionary didn’t have enough words to be useful • Suggests a lot of words, but short entries?

22. Macro-structure • EL-LWC effective for proficient speakers decoding hard/old words • EL-EL dictionary possible or preferred where EL literacy is strong(er) – but this is rare • For others, LWC-EL order is more useful, and helps improve their English literacy • This commonly meant that they were using the finder list as a dictionary • Usage didn’t match dictionary design: a LWC-EL dictionary would have been far superior

23. Citation forms • Some found the concept hard to grasp • Would have liked inflected forms in dictionary • E-dictionaries easily can include all inflected forms as headwords • Being able to do morphological analysis is a form of learner support

24. Pronunciation • Lack of confidence in pronouncing words was a problem in Alawa, and discouraged use of dictionary (what does it sound like?, where is the stress?) • E-dictionaries can solve this through use of audio • some concerns on use of individual’s voices • synthesis may be best, though questionable for language learning

25. Extracting relevant info • Long entries were hard/overwhelming to process • People want simple entries, even if lexicologically questionable • one word correspondences easiest for users • even school teachers queasy on notion of word senses • Either need sufficient user training or • Several versions of dictionaries designed for different users • need spacious clear entries for beginners

26. Microstructure details • Sub-entries were confusing • converting them to a crossreference better • Definitions often overly technical • People were confused by conventions such as reversal in word lists (kangaroo, plains) • Part of speech abbreviations puzzling – read as part of definition (minimally put at end) • Cross-reference abbreviations (syn., ant.) confusing • Small font size a problem (literacy level, eyesight)

27. E-dictionary use • E-dictionaries have novelty, engaging interactiveness • Children willing to spend time with it, even when they didn’t understand everything • Some teachers have thought our interface too advanced for primary school children, but in our experience, they underestimate the facility of today’s young (indigenous) children with computers – they manipulate them with ease • E.g., one (bright) 10 yo returned after a demo and played with interface for 2 1/2 hours

28. Teachers • Generally enthusiastic • Would encourage kids to learn to read Warlpiri • Would help teachers to prepare lessons – they found the interface more efficient to use for lesson preparation than a domain-specific dictionary extract. • Felt could use as a basis for classroom activities (better with some further development: games and puzzles)

29. Primary school kids • One major benefit is that it was on computer. • It maintained interest • “They were enthusiastic about the computer side of things and negotiated the interface’s various windows easily” • e.g., wanted “back” button • Sometimes, working on sense relations and definitions was of less interest than moving things around • Word list – and English search – was found helpful (can compensate for poor spelling)

30. Older children • More thoughtful; had dictionary experience • Still really liked word list • Interested in word relationships • Liked finding pictures and audio • Students used it voluntarily during lunch breaks (confounding factor: air-conditioning)

31. Adult literacy workers • Less interested in graphical interface • Mainly interested in looking at definitions • Started discussing and disagreeing with them immediately • although they had and used paper printout of dictionary, first real chance to see what was there? • They wanted to, and were able to, annotate the definitions with notes

32. E-dictionary links • Promoting subentries to entries appeared very effective: People enjoyed exploring and explaining relation of derived terms to main word (even if sometimes folk etymologies?) • Synonyms, and alternate forms used and liked • Concept of antonym seemed confusing • The semantically uninterpreted cf link was confusing

33. User-initiative, computer support “One of the introductory Warlpiri literacy students, who had not been very interested in the literacy class, spent nearly 3/4 hour looking at Kirrkirr apparently in absorbed concentration. She wasn’t especially interested in the sound and picture possibilities. She moved between words, scrolling along the list, typing in the search, clicking on the words in the network pane. She wasn’t even put off when the dictionary definitions stopped appearing – looking at the networks of words instead. … After the Kirrkirr demo she asked if she could have a printed dictionary to take away with her to use in camp to learn the words. I interpret this as a desire to learn words in her own time and place.” – Jane Simpson Lajamanu May 1999

34. Conclusions • Regular dictionary users (especially, compilers!) grossly underestimate the time they have spent becoming familiar with dictionary structure • If a dictionary is going to be made for a speech community, then the people in that community need to feel confident in using it. • Most of our users need a different type of dictionary from the current documentation dictionaries, (implicitly?) prepared to meet the needs of professionals – My First Alawa • Would probably be used by adults as well as kids

35. Conclusions • E-dictionary showed great promise, though still room for improvement: • Make more interactive (more for kids to do) • Add in more views of dictionary (e.g., search by colour) • But it did show how an electronic dictionary can effectively offer multiple pathways and user supports in mediating between well-structured dictionary data and users’ needs for searching/browsing and presentation