1 / 16

Lucene in action

This document provides an in-depth overview of Lucene, a powerful open-source library for full-text search and indexing. Written in 100% Java with no external dependencies, it covers the fundamental components of indexing and searching documents, how to structure them, and execute queries. The guide includes practical code snippets, information on built-in analyzers and token filters, and best practices for optimizing performance. Suitable for developers and researchers, it highlights Lucene's applications in various commercial products and famous websites.

dusty
Download Presentation

Lucene in action

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Lucene in action Information Retrieval A.A. 2010-11 – P. Ferragina, U. Scaiella – – Dipartimento di Informatica – Università di Pisa –

  2. What is Lucene • Full-text search library • Indexing + Searching components • 100% Java, no dependencies, no config files • No crawler, document parsing nor search UI • see Apache Nutch, Apache Solr, Apache Tika • Probably, the most widely used SE • Applications: a lot of (famous) websites, many commercial products

  3. Basic Application • Parse docs • Write Index • Make query • Display results IndexWriter IndexSearcher Index

  4. Indexing //create the index IndexWriterwriter = newIndexWriter(directory,analyzer); //create the documentstructure Document doc = newDocument(); Fieldid = newNumericField(“id”,Store.YES); Fieldtitle = newField(“title”,null,Store.YES, Index.ANALYZED); Field body = newField(“body”,null,Store.NO,Index.ANALYZED); doc.add(id); doc.add(title); doc.add(body); //scroll alldocuments, fillfields and indexthem! for(alldocument){ Article a = parse(document); id.setIntValue(a.id); title.setValue(a.title); body.setValue(a.body); writer.addDocument(doc); //docis just a container } //IMPORTANT! close the Writertocommitalloperations! writer.close();

  5. How to represent text?

  6. Analyzer Text processing pipeline String TokenStream Docs Tokenizer TokenStream TokenFilter1 TokenStream TokenFilter2 TokenFilter3 Analyzer Indexingtokens Index

  7. Analyzer Analyzer Searching tokens Query String Index Results

  8. Analyzer • Built-inTokenizers • WhitespaceTokenizer, LetterTokenizer,… • StandardTokenizergoodformostEuropean-languagedocs • Built-inTokenFilters • LowerCase, Stemming, Stopwords, AccentFilter, manyothers in contribpackages (language-specific) • Built-inAnalyzers • Keyword, Simple, Standard… • PerFieldwrapper

  9. TEXT QUERY the LexCorp BFG-900 is a printer Lexcorp bfg900 printers WhitespaceTokenizer the LexCorp BFG-900 is a printer Lex corp bfg900 printers WordDelimiterFilter the Lex Corp BFG 900 is a printer Lex corp bfg 900 printers LexCorp LowerCaseFilter the lex corp bfg 900 is a printer lex corp bfg 900 printers lexcorp StopwordFilter lex corp bfg 900 printer lex corp bfg 900 printers lexcorp StemmerFilter lex corp bfg 900 print- lex corp bfg 900 print- lexcorp MATCH!

  10. Field Options • Field.Stored • YES, NO • Field.Index • ANALYZED, NOT_ANALYZED, NO • Field.TermVector • NO, YES (POSITION and/or OFFSETS)

  11. Analysis tips • Use PerFieldAnalyzerWrapper • Don’t analyze keyword fields • Store only needed data • Use NumberUtils for numbers • Add same field more than once, analyze it differently • Boost exact case/stem matches

  12. Searching //Best practice: reusable singleton! IndexSearcher s = newIndexSearcher(directory); //Build the queryfrom the input string QueryParserqParser =newQueryParser(“body”, analyzer); Query q = qParser.parse(“title:Jaguar”); //Do search TopDocshits = s.search(q, maxResults); System.out.println(“Results: ”+hits.totalHits); //Scroll allretrieveddocs for(ScoreDoc hit : hits.scoreDocs){ Document doc = s.doc(hit.doc); System.out.println( doc.get(“id”) + “ – ” + doc.get(“title”) + “ Relevance=” + hit.score); } s.close();

  13. Building the Query • Built-in QueryParser • does text analysis and builds the Query object • good for human input, debugging • not all query types supported • specific syntax: see JavaDoc for QueryParser • Programmatic query building e.g.: Query q = new TermQuery( new Term(“title”, “jaguar”)); • many types: Boolean, Term, Phrase, SpanNear, … • no text analysis!

  14. Scoring • Lucene = BooleanModel + VectorSpaceModel • Similarity= Cosine Similarity • TermFrequency • Inverse DocumentFrequency • Otherstuff • LengthNormalization • Coord. factor (matchingterms in OR queries) • Boosts (per query or per doc) • Tobuildyourown: implementSimilarity and callSearcher.setSimilarity • Fordebugging: Searcher.explain(Query q, int doc)

  15. Performance • Indexing • Batch indexing • Raise mergeFactor • Raise maxBufferedDocs • Searching • Reuse IndexSearcher • Optimize: IndexWriter.optimize() • Use cached filters: QueryFilter Segment_3 Indexstructure

  16. Esercizi http://acube.di.unipi.it/esercizi-lucene-corso-ir/

More Related