1 / 52

Searching Speech: A Research Agenda

Searching Speech: A Research Agenda. Douglas W. Oard College of Information Studies and Institute for Advanced Computer Studies University of Maryland, College Park. Some Grid Use at Maryland. Global Land Cover Facility 13 TB of raw and derived data from 5 satellites Digital archives

gilead
Download Presentation

Searching Speech: A Research Agenda

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Searching Speech:A Research Agenda Douglas W. Oard College of Information Studies and Institute for Advanced Computer Studies University of Maryland, College Park National E-Science Centre

  2. Some Grid Use at Maryland • Global Land Cover Facility • 13 TB of raw and derived data from 5 satellites • Digital archives • Preserving the meaning of metadata structure • Access grid • No-operator information studies classroom

  3. Expanding the Search Space Scanned Docs Identity: Harriet “… Later, I learned that John had not heard …”

  4. Indexable Speech • What if we could collect “everything”? • 1 billion users of speech-enabled devices • Each producing >10K words per day • Much of it not worth finding • Comparison case: Web search • Google indexes ~10 billion Web pages • Perhaps averaging ~1K words each • Much of it not worth finding

  5. A Web of Speech?

  6. The Need for Scalable Solutions

  7. Some Spoken Word Collections • Broadcast programming • News, interview, talk radio, sports, entertainment • Storytelling • Books on tape, oral history, folklore • Incidental recording • Speeches, courtrooms, meetings, phone calls

  8. Indexing Options • Transcript-based (e.g., NASA) • Manual transcription, editing by interviewee • Thesaurus-based (e.g., Shoah Foundation) • Manually assign descriptors to points in an interview • Catalog-based (e.g., British Library) • Catalog record created from interviewer’s notes • Speech-based (MALACH) • Create access points with speech processing

  9. Search System Query Formulation Query Search Ranked List Selection Query Reformulation and Relevance Feedback Recording Examination Recording Source Reselection Delivery Supporting “Intellectual Access” • Speech Processing • Computational Linguistics • Information Retrieval • Information Seeking • Human-Computer Interaction • Digital Libraries Source Selection

  10. Some Technical Challenges • “Fast” ASR systems are way too slow • 6 orders or magnitude slower than tokenization • Situational sublanguage induces variability • Impedes interactive vocabulary acquisition • Knee in the WER/MAP curve comes early • 30-40% for broadcast news • Somewhere below 30% for conversations • Skimmable summaries from imperfect ASR • Particularly important for linear media • Classic IR measures focus on “documents” • Conversationalboundaries are ambiguous

  11. Start Time Error Cost

  12. Shoah Foundation Collection • Substantial scale • 116,000 hours; 52,000 interviews; 32 languages • Spontaneous conversational speech • Accents, elderly, emotional, … • Accessible • $100 million collection and digitization investment • Manually indexed (10,000 hours) • Segmented, thesaurus terms, people, summaries • Users • A department working full time on dissemination

  13. Interview Excerpt • Audio characteristics • Accented (this one is unusually clear) • Separate channels for interviewer / interviewee • Dialog structure • Interviewers have different styles • Content characteristics • Domain-specific terms • Named entity mentions and relationships

  14. MALACH Languages Testimonies (average 2.25 hours each) As of January 31, 2004

  15. 8 independent searchers Holocaust studies (2) German Studies History/Political Science Ethnography Sociology Documentary producer High school teacher 8 teamed searchers All high school teachers Thesaurus-based search Rich data collection Intermediary interaction Semi-structured interviews Observational notes Think-aloud Screen capture Qualitative analysis Theory-guided coding Abductive reasoning Observational Studies

  16. Relevance Criteria 6 Scholars, 1 teacher, 1 film producer, working individually

  17. Topicality Total mentions 6 Scholars, 1 teacher, 1 movie producer, working individually

  18. Test Collection Design Query Formulation Speech Recognition Automatic Search Boundary Detection Content Tagging Interactive Selection

  19. Interviews Topic Statements Training: 38 existing Evaluation: 25 new Automatic: 35% interview-tuned 40% domain-tuned Manual: Topic boundaries Automatic: Topic boundaries Ranked Lists Relevance Judgments Evaluation Manual: ~5 Thesaurus labels 3-sentence summaries Automatic: Thesaurus labels Mean Average Precision Test Collection Design Query Formulation Speech Recognition Automatic Search Boundary Detection Content Tagging

  20. CLEF-2005 CL-SR Track • Test collection distributed by ELDA • ~7,800 segments from ~300 English interviews • Hand segmented / known boundaries • 63 topics (title/description/narrative) • 38 for training, 25 for blind evaluation • 5 languages (EN, SP, CZ, DE, FR) • Relevance judgments • Search-guided + post-hoc judgment pools • 5 participating teams • DCU, Maryland, Pitt, Toronto/Waterloo, UNED • One required cross-site baseline run • ASR segments / English TD topics

  21. Additional Resources • Thesaurus • ~3,000 core concepts • Plus alternate vocabulary + standard combinations • ~30,000 location-time pairs, with lat/long • Both “is-a” and “part-whole” relationships • In-domain expansion collection • 186,000 3-sentence summaries • Indexer’s scratchpad notes • Digitized speech • .mp2 or .mp3

  22. English ASR ASR2004A ASR2003A Training: 200 hours from 800 speakers

  23. <DOCNO>VHF00017-062567.005</DOCNO> <KEYWORD> Warsaw (Poland), Poland 1935 (May 13) - 1939 (August 31), awareness of political or military events, schools </KEYWORD> <PERSON> Sophie P[…], Henry H[…] </PERSON> <SUMMARY> AH talks about the college she attended before the war. She mentions meeting her husband. She discusses young peoples' awareness of the political events that preceded the outbreak of war. </SUMMARY> <SCRATCHPAD>graduated HS, went to college 1 year, professional college hotel management; met future husband, knew that they'd end up together; sister also in college, nice social life, lots of company, not too serious; already got news from Czechoslovakia, Sudeten, knew that Poland would be next but what could they do about it, very passive; just heard info from radio and press </SCRATCHPAD> <ASRTEXT> no no no they did no not not uh i know there was no place to go we didn't have family in a in other countries so we were not financially at the at extremely went so that was never at plano of my family it is so and so that was the atmosphere in the in the country prior to the to the war i graduate take the high school i had one year of college which was a profession and that because that was already did the practical trends f so that was a study for whatever management that eh eh education and this i i had only one that here all that at that time i met my future husband and that to me about any we knew it that way we were in and out together so and i was quite county there was so whatever i did that and this so that was the person that lived my sister was it here is first year of of colleagues and and also she had a very strongly this antisemitic trend and our parents there was a nice social life young students that we had open house always pleasant we had a lot of that company here and and we were not too serious about that she we got there we were getting the they already did knew he knew so from czechoslovakia from they saw that from other part and we knew the in that that he is uhhuh the hitler spicy we go into this year this direction that eh poland will be the next country but there was nothing that we would do it at that time so he was a very very he says belong to any any organizations especially that the so we just take information from the radio and from the dress </ASRTEXT>

  24. Segment duration (s) 44.5% Min. 1st Qu. Median Mean 3rd Qu. Max. NA's -2044.00 54.01 224.90 391.70 326.00 287400.00 75031.00 ??

  25. Keywords vs. Segment duration

  26. Nodes descending from parents of leaves

  27. Years spoken in ASR

  28. Spoken dates in release ASR Min. : 0.0000 1st Qu.: 0.0000 Median : 0.0000 Mean : 0.6575 3rd Qu.: 1.0000 Max. : 13.0000

  29. Current classifier performance: 46,601 (1,175) 3,610 ( 169) 1,437 (168) 613 ( 47) MAP: .2374, even post-mixing of scratchpad/summary from 20NN, remixed with time-label densities estimated w/ Gaussian kernel at 5x def. bandwidth

  30. An Example English Topic Number: 1148 Title: Jewish resistance in Europe Description: Provide testimonies or describe actions of Jewish resistance in Europe before and during the war. Narrative: The relevant material should describe actions of only- or mostly Jewish resistance in Europe. Both individual and group-based actions are relevant. Type of actions may include survival (fleeing, hiding, saving children), testifying (alerting the outside world, writing, hiding testimonies), fighting (partisans, uprising, political security) Information about undifferentiated resistance groups is not relevant.

  31. Binary qrels 5-level Relevance Judgments • “Classic” relevance (to “food in Auschwitz”) • Direct Knew food was sometimes withheld • Indirect Saw undernourished people • Additional relevance types • Context Intensity of manual labor • Comparison Food situation in a different camp • Pointer Mention of a study on the subject

  32. Comparing Index Terms +Persons Title queries, adjudicated judgments

  33. Searching Manual Transcripts jewish kapo(s) fort ontario refugee camp Title queries, adjudicated judgments

  34. 3,199 Training segments test segments Spoken Words (hand transcribed) Spoken Words (ASR transcript) kNN Categorization Thesaurus Terms Thesaurus Terms F=0.19 (microaveraged) Index Thesaurus Terms ASR Words Title queries, linear score combination, adjudicated judgments Category Expansion

  35. Average of 3.4 relevant segments in top 20 +27% ASR-Based Search Mean Average Precision Title queries, adjudicated judgments

  36. Rethinking the Problem • Segment-then-label models planned speech well • Producers assemble stories to create programs • Stories typically have a dominant theme • The structure of natural speech is different • Creation: digressions, asides, clarification, … • Use: intended use may affect desired granularity • Documentary film: brief snippet to illustrate a point • Classroom teacher: longer self-contextualizing story

  37. Labels Activation Matrix Time

  38. Training Data: 196,000 Segments Location-Time Subject Person Berlin-1939 Employment Josef Stein Berlin-1939 Family life Gretchen Stein Anna Stein interview time Dresden-1939 Relocation Transportation-rail Dresden-1939 Schooling Gunter Wendt Maria + Segment summaries + Indexer’s notes

  39. Preprocessing Training Data • Normalize labeled categories? • Food in hiding -> food AND hiding • Develop class models • Existing hierarchy, types of personal relationships • Determine the extent for each label and class • Merge the extent of repeated labels

  40. Characteristics of the Problem • Clear dependencies • Correlated assignment of applications • Living in Dresden negates living in Berlin • Heuristic basis for class models • Persons, based on type of relationship • Date/Time, based on part-whole relationship • Topics, based on a defined hierarchy • Heuristic basis for guessing without training • Text similarity between labels and spoken words • Heuristic basis for smoothing • Sub-sentence retrieval granularity is unlikely

  41. Modeling Location Berlin Dresden Germany • Presence in a new location negates presence in the prior location • Location granularity varies (inclusion relationships are known)

  42. A Class Model for People father mother sister father mother nobody sister friend • Several people may be discussed simultaneously • Small inventory of relationship types • Relationship type is known for most people that are mentioned

  43. Search • Compute a score at each time based on: • How likely is each descriptor? (~TF) • How selective is each descriptor? (~IDF) • What related descriptors are active? (~expansion) • Determine passage start time based on: • Score trajectory (sequence of scores) • Additional heuristics (e.g., pause, speaker turn) • Rank passages based on score trajectory • e.g., by peak score within the passage

  44. Timelines for the whole interview text

  45. Some Open Issues • Is the expressive power of a lattice needed? • An activation matrix is an unrolled lattice • What states do we need to represent? • Balance fidelity, accuracy, and complexity • How to integrate manual onset marks? • How much training data do we need? • Annotating new data costs ~$100/hour • How will people use the system we build?

  46. Non-English ASR Systems WER [%] 30 40 50 60 70 34.49% 35.51% 38.57% + stand.+LMTr+TC + adapt. 40.69% 41.15% 100h + LMTr + LMTr+TC + standard. 45.75% 45.91% + stand.+LMTr+TC 84h + LMTr 50.82% 100h + LMTr 57.92% 45h + LMTr 66.07% 20h + LMTr Polish Czech Slovak Hungarian Russian 10/01 4/02 10/02 4/03 10/03 4/04 10/04 4/05 10/05 4/06 10/06

  47. Planning for the Future • Tentative CLEF-2006 CL-SR Plans: • Adding a Czech collection • Larger English collection (~900 hours) • Adding word lattice as standard data • No-boundary evaluation design • ASR training data (by special arrangement) • Transcripts, pronunciation lexicon, language model • Possible CLEF-2007 CL-SR Options: • Add a Russian or Slovak collection? • Much larger English collection (~5,000 hours)?

  48. Shoah Foundation Sam Gustman IBM TJ Watson Bhuvana Ramabhadran Martin Franz U. Maryland Doug Oard Dagobert Soergel Johns Hopkins Zak Schefrin U. Cambridge (UK) Bill Byrne Charles University (CZ) Jan Hajic Pavel Pecina U. West Bohemia (CZ) Josef Psutka Pavel Ircing UNED (ES) Fernando López-Ostenero The CLEF CL-SR Team USA Europe

  49. More Things to Think About • Privacy protection • Working with real data has real consequences • Are fixed segments the right retrieval unit? • Or is it good enough to know where to start? • What will it cost to tailor an ASR system? • $100K to $1 million per application? • Do we need to change what we collect? • Speaker enrollment, metadata standards, …

  50. Final Thoughts • The moving hand, having writ, moves on • Ephemeral webcasting • Forgone acquisition opportunities

More Related