1 / 22

Evaluating Hiera r chical Clustering of Search Results

Evaluating Hiera r chical Clustering of Search Results. SPIRE 2005, Buenos Aires. Departamento de Lenguajes y Sistemas Informáticos UNED, Spain Juan Cigarrán Anselmo Peñas Julio Gonzalo Felisa Verdejo nlp.uned.es. Overview. Scenario Assumptions

tirza
Download Presentation

Evaluating Hiera r chical Clustering of Search Results

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluating Hierarchical Clustering of Search Results SPIRE 2005, Buenos Aires Departamento de Lenguajes y Sistemas Informáticos UNED, Spain Juan Cigarrán Anselmo Peñas Julio Gonzalo Felisa Verdejo nlp.uned.es

  2. Overview • Scenario • Assumptions • Features of a Good Hierarchical Clustering • Evaluation Measures • Minimal Browsing Area (MBA) • Distillation Factor (DF) • Hierarchy Quality (HQ) • Conclusion

  3. Scenario • Complex information needs • Compile information from different sources • Inspect the whole list of documents • More than 100 documents • Help to • Find the relevant topics • Discriminate from unrrelevant documents • Approach • Hierarchical Clustering – Formal Concept Analysis

  4. Problem • How to define and measure the quality of a hierarchical clustering? • How to compare different clustering approaches?

  5. Physics Physics d1 d1, d2, d3, d4 Astrophysics d4 Astrophysics d4 Nuclear physics Nuclear physics d2, d3 d2, d3 Previous assumptions • Each cluster contains only those documents fully described by its descriptors

  6. Physics Jokes d1 d2 Jokes about physics d3 Jokes about physics d3 Physics Jokes d1 d2 Jokes about physics d3 Previous assumptions • ‘Open world’ perspective

  7. + + + + + - + + - - - + - - + + + + - - Good Hierarchical Clustering • The content of the clusters. • Clusters should not mix relevant with non relevant information

  8. + + + + - - - + + + + - - - - - - + + + + + + - - - Good Hierarchical Clustering • The hierarchical arrangement of the clusters • Relevant information should be in the same path

  9. Good Hierarchical Clustering • The number of clusters • Number of clusters substantially lower than the number of documents • How clusters are described • Cognitive load of reading a cluster description • Ability to predict the relevance of the information that it contains (not addressed here)

  10. Evaluation Measures • Criterion • Minimize the browsing effort for finding ALL relevant information • Baseline • The original document list returned by a search engine

  11. Evaluation Measures • Consider • Content of clusters • Hierarchical arrangement of clusters • Size of the hierarchy • Cognitive load of reading a document (in the baseline): Kd • Cognitive load of reading a node descriptor (in the hierarchy): Kn • Requirement • Relevance assessments are available

  12. + - + - + - + Minimal Browsing Area (MBA) • The minimal set of nodes the user has to traverse to find ALL the relevant documents minimising the number of irrelevant ones

  13. Distillation Factor (DF) • Ability to isolate relevant information compared with the original document list (Gain Factor, DF>1) • Equivalent to: • Considers only the cognitive load of reading documents

  14. + Document List Doc 1 + Doc 2 - Doc 3 + Doc 4 + Doc 5 - Doc 6 - Doc 7 + - + - + - + Distillation Factor (DF) • Example Precision = 4/7 Precision MBA = 4/5 DF(L) = 7/5 = 1.4

  15. Precision = 4/8 Precision MBA = 4/4 DF = 8/4 = 2 - + + + + - - - Distillation Factor (DF) • Counterexample: • Bad clustering with good DF • Extend the DF measure considering the cognitive cost of taking browsing decisions  HQ

  16. - + - - - + +- - + MBA |Nview|=8 Hierarchy Quality (HQ) • Assumption: • When a node (in the MBA) is explored, all its lower neighbours have to be considered: some will be in turn explored, some will be discarded • Nview : subset of lower neighbours of each node belonging to the MBA

  17. Hierarchy Quality (HQ) • Kn and Kdare directly related with the retrieval scenario in which the experiments take place • The researcher must tune K=Kn/Kd before conducting the experiment • HQ > 1 indicates an improvement of the clustering versus the original list

  18. - + - - - + +- - + - + + + + - - - Hierarchy Quality (HQ) • Example

  19. Conclusions and Future Work • Framework for comparing different clustering approaches taking into account: • Content of clusters • Hierarchical arrangement of clusters • Cognitive load to read document and node descriptions • Adaptable to the retrieval scenario in which experiments take place • Future work • Conduct user studies to compare their results with the automatic evaluation • Results will reflect the quality of the descriptors • Will be used to fine-tune the kd and kn parameters

  20. Thank you!

More Related