Relevance in ISR

1. Relevance in ISR Peter Ingwersen Department of Information Studies Royal School of Library and Information Science, Denmark pi@db.dk

2. Agenda • Brief history • Types of relevance • Higher order of relevance • Experimental relevance measures • Socio-cognitive relevance • Concluding discussion Peter Ingwersen, 2007

3. Up until 1990.. • Cranfield: • Topicality - Utility (usefulness): Cleverdon • Saracevic 1975 – Swanson: Objective vs. Subjective relevance • Schamber 1990 / 1994 (JASIS): • Situational relevance vs. topicality • Harter 1992: • Psychological relevance (Sperber/Wilson) • Ørom & Cosijn/Ingwersen 2000: • Socio-cognitive relevance Peter Ingwersen, 2007

4. Higher Order Relevances • Relevance is a multi-dimensional, multi-layered, non-binary, subjective, and dynamic phenomenon • E.g. topical, pertinence, situational, and socio-cognitive relevance. • Incorporation of higher order relevance • improves the realism of evaluation • brings human users and interaction into evaluation • Are cognitive/emotional perceptions and interpretations of document features Peter Ingwersen, 2007

5. Levels of Relevance Types -1Saracevic, 1996 • System or Algorithmic relevance: • query-object (objectivity) • Topical relevance: aboutness relationof • query-object (interpretation) • Pertinence(cognitive relevance): perceived correspondence of information need-objects Peter Ingwersen, 2007

6. Levels of Relevance Types -2Saracevic, 1996; rev. Cosijn/Ingwersen, 2000 • Situational relevance: relation as perceived between work task or interest situation or problem and objects • Affective relevance: intentionality-objects • Saracevic´ idea revised by Cosijn/Ingwersen • Socio-Cognitive relevance: objectsperceived as meaningful by users in social interaction Peter Ingwersen, 2007

7. Algorithmic relevance • The ranked output of information objects - ranked by relevance scores • Commonly judged against expert assessor´s binary relevance assessments of the pooled documents (all scale variances normalized) • Assessor´s judgement seen as topicality & objective- is of courseintellectual Peter Ingwersen, 2007

8. Topical relevance • Contains interpretations • Issues of nature of aboutness & meaning • Inconsistency among several assessors (yet: see Sigir 98 paper by Vorhees) • Used for relative performance indications • Why not apply the mean of the algorithmic output - normalized across sites (TREC) ? Peter Ingwersen, 2007

9. Pertinence • Requires knowledge of intrinsic information need for an observer - difficult to obtain • May not be achieved in case of ill-defined needs • Involves other featuresof objects than simply topical ones(novelty - authorship - cognitive authority of carriers) Peter Ingwersen, 2007

10. Situational relevance • Relates to the WORK TASK (interest) SITUATION- perceivedusefulness of objects • Individual relevance assessment in IS&R • as document assessment; peer review • As document features: references or outlinks • Is observable by users (Borlund, 2000) - also when work tasks are simulated Peter Ingwersen, 2007

11. Experimental issues on relevance in IS&R Peter Ingwersen, 2007

12. Socio-Cognitive Relevance • Proposed by Ørom (JoD, Jan. 2000; Hjørland, 1997) as associated with the socio-cultural / epistemological / domain context. • Discussed by Cosijn/Ingwersen (IPM, May 2000) as related to organisational or social strategies, group situations & perceptions (individuals in social interaction) Peter Ingwersen, 2007

13. Socio-Cognitive Relevance - 2 • In the perspective of information use: • creation of scientific references and Web outlinksis commonly situational(usefulness of cited work) • CITATIONS or INLINKS over time are manifestations of socio-cognitive relevance in domain • Relation to informetrics & scientometrics and knowledge representation Peter Ingwersen, 2007

14. Socio-Cognitive Relevance - 3 • Strength of - e.g. - author (journal …) co-citation (co-linking) shows (socio)-cognitive authority • In the perspective of information creation, selection, processing or presentation: • Multi-authorships • Peer reviews used in editorial decisions • Programme Committee decisions (conf.) Peter Ingwersen, 2007

15. Information objects Org. Program Committee Members Social Context IT Structure Interface Cultural • Cognitive transformation (selection & structure of objects) • Social interaction over time (scientific domain) • Interactions with submissions/reviews/preogram structure • Cognitive transformations & influence over time (submissions) Simplistic model of a Selector groupin IS&R as cognitive agent Peter Ingwersen, 2007

16. Socio-cognitive assessments: • Create features of isness of information objects, e.g. • Journal name – conference… - acceptance • Publication year / date – publisher decision • Database name(s) - inclusion • Corporate source & Geo-location – employment • Such features are represented and accessible • Dependent on Media - Domains - Time Peter Ingwersen, 2007

17. Concluding remarks • Topicality - Pertinence - Situational relevances are • Increasingly subjective • Typically ad hoc manifestations • Situational Rel. difficult to apply in system due to lack of proper features (usefulness?) • Yet: references and outlinks – document structure • Socio-cognitive relevance turns into • objectivity or confined subjectivity • and manifestations over time • lead to perceptions of cognitive authority Peter Ingwersen, 2007