INTRODUCTION TO ARTIFICIAL INTELLIGENCE

1. INTRODUCTION TO ARTIFICIAL INTELLIGENCE Massimo PoesioLECTURE 16: Unsupervised methods, IR, and lexical acquisition

2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FEATURE-DEFINED DATA SPACE .

3. UNSUPERVISED MACHINE LEARNING • In many cases, what we want to learn is not a target function from examples to classes, but what the classes are • I.e., learn without being told

4. Arch. Graphics Theory NLP AI EXAMPLE: TEXT CLASSIFICATION • Consider clustering a large set of computer science documents

5. CLUSTERING • Partition unlabeled examples into disjoint subsets of clusters, such that: • Examples within a cluster are very similar • Examples in different clusters are very different • Discover new categories in an unsupervised manner (no sample category labels provided).

6. Arch. Graphics Theory NLP AI = AI Deciding what a new doc is about • Check which region the new doc falls into • can output “softer” decisions as well.

7. animal vertebrate invertebrate fish reptile amphib. mammal worm insect crustacean Hierarchical Clustering • Build a tree-based hierarchical taxonomy (dendrogram) from a set of unlabeled examples. • Recursive application of a standard clustering algorithm can produce a hierarchical clustering.

8. Agglomerative vs. Divisive Clustering • Agglomerative (bottom-up) methods start with each example in its own cluster and iteratively combine them to form larger and larger clusters. • Divisive (partitional, top-down) separate all examples immediately into clusters.

9. Direct Clustering Method • Direct clustering methods require a specification of the number of clusters, k, desired. • A clustering evaluationfunctionassigns a real-value quality measure to a clustering. • The number of clusters can be determined automatically by explicitly generating clusterings for multiple values of k and choosing the best result according to a clustering evaluation function.

10. Hierarchical Agglomerative Clustering (HAC) • Assumes a similarity function for determining the similarity of two instances. • Starts with all instances in a separate cluster and then repeatedly joins the two clusters that are most similar until there is only one cluster. • The history of merging forms a binary tree or hierarchy.

11. Cluster Similarity • Assume a similarity function that determines the similarity of two instances: sim(x,y). • Cosine similarity of document vectors. • How to compute similarity of two clusters each possibly containing multiple instances? • Single Link: Similarity of two most similar members. • Complete Link: Similarity of two least similar members. • Group Average: Average similarity between members.

12. Non-Hierarchical Clustering • Typically must provide the number of desired clusters, k. • Randomly choose k instances as seeds, one per cluster. • Form initial clusters based on these seeds. • Iterate, repeatedly reallocating instances to different clusters to improve the overall clustering. • Stop when clustering converges or after a fixed number of iterations.

13. CLUSTERING METHODS IN NLP • Unsupervised techniques are heavily used in : • Text classification • Information retrieval • Lexical acquisition

14. CLUSTERING METHODS IN NLP • Unsupervised techniques are heavily used in : • Text classification • Information retrieval • Lexical acquisition

15. Feature-based lexical semantics • Very old idea in lexical semantics: the meaning of a word can be specified in terms of the values of certain `features’ (`DECOMPOSITIONAL SEMANTICS’) • dog : ANIMATE= +, EAT=MEAT, SOCIAL=+ • horse : ANIMATE= +, EAT=GRASS, SOCIAL=+ • cat : ANIMATE= +, EAT=MEAT, SOCIAL=- ANLE

16. FEATURE-BASED REPRESENTATIONS IN PSYCHOLOGY • Feature-based concept representations assumed by many cognitive psychology theories (Smith and Medin, 1981, McRae et al, 1997) • Underpin development of prototype theory (Rosch et al) • Used, e.g., to account for semantic priming (McRae et al, 1997; Plaut, 1995) • Underlie much work on category-specific defects (Warrington and Shallice, 1984; Caramazza and Shelton, 1998; Tyler et al, 2000; Vinson and Vigliocco, 2004) Cog/Comp Neuroscience

17. SPEAKER-GENERATED FEATURES (VINSON AND VIGLIOCCO) Cog/Comp Neuroscience

18. Vector-based lexical semantics • If we think of the features as DIMENSIONS we can view these meanings as VECTORS in a FEATURE SPACE • (An idea introduced by Salton in Information Retrieval, see below) ANLE

19. Vector-based lexical semantics CAT DOG HORSE ANLE

20. General characterization of vector-based semantics (from Charniak) • Vectors as models of concepts • The CLUSTERING approach to lexical semantics: • Define properties one cares about, and give values to each property (generally, numerical) • Create a vector of length n for each item to be classified • Viewing the n-dimensional vector as a point in n-space, cluster points that are near one another • What changes between models: • The properties used in the vector • The distance metric used to decide if two points are `close’ • The algorithm used to cluster ANLE

21. Using words as features in a vector-based semantics • The old decompositional semantics approach requires • Specifying the features • Characterizing the value of these features for each lexeme • Simpler approach: use as features the WORDS that occur in the proximity of that word / lexical entry • Intuition: “You can tell a word’s meaning from the company it keeps” • More specifically, you can use as `values’ of these features • The FREQUENCIES with which these words occur near the words whose meaning we are defining • Or perhaps the PROBABILITIES that these words occur next to each other • Alternative: use the DOCUMENTS in which these words occur (e.g., LSA) • Some psychological results support this view. Lund, Burgess, et al (1995, 1997): lexical associations learned this way correlate very well with priming experiments. Landauer et al: good correlation on a variety of topics, including human categorization & vocabulary tests. ANLE

22. Using neighboring words to specify lexical meanings

23. Learning the meaning of DOG from text

24. Learning the meaning of DOG from text

25. Learning the meaning of DOG from text

26. Learning the meaning of DOG from text

27. Learning the meaning of DOG from text

28. The lexicon we acquire

29. Meanings in word space

30. Acquiring lexical vectors from a corpus(Schuetze, 1991; Burgess and Lund, 1997) • To construct vectors C(w) for each word w: • Scan a text • Whenever a word w is encountered, increment all cells of C(w) corresponding to the words v that occur in the vicinity of w, typically within a window of fixed size • Differences among methods: • Size of window • Weighted or not • Whether every word in the vocabulary counts as a dimension (including function words such as the or and) or whether instead only some specially chosen words are used (typically, the m most common content words in the corpus; or perhaps modifiers only). The words chosen as dimensions are often called CONTEXT WORDS • Whether dimensionality reduction methods are applied ANLE

31. Variant: using probabilities (e.g., Dagan et al, 1997) • E.g., for house • Context vector (using probabilities) • 0.001394 0.016212 0.003169 0.000734 0.001460 0.002901 0.004725 0.000598 0 0 0.008993 0.008322 0.000164 0.010771 0.012098 0.002799 0.002064 0.007697 0 0 0.001693 0.000624 0.001624 0.000458 0.002449 0.002732 0 0.008483 0.007929 0 0.001101 0.001806 0 0.005537 0.000726 0.011563 0.010487 0 0.001809 0.010601 0.000348 0.000759 0.000807 0.000302 0.002331 0.002715 0.020845 0.000860 0.000497 0.002317 0.003938 0.001505 0.035262 0.002090 0.004811 0.001248 0.000920 0.001164 0.003577 0.001337 0.000259 0.002470 0.001793 0.003582 0.005228 0.008356 0.005771 0.001810 0 0.001127 0.001225 0 0.008904 0.001544 0.003223 0 ANLE

32. Variant: using modifiers to specify the meaning of words • …. The Soviet cosmonaut …. The American astronaut …. The red American car …. The old red truck … the spacewalking cosmonaut … the full Moon … ANLE

33. Another variant: word / document matrices ANLE

34. Measures of semantic similarity • Euclidean distance: • Cosine: • Manhattan Metric: ANLE

35. dj θ qk SIMILARITY IN VECTOR SPACE MODELS: THE COSINE MEASURE

36. EVALUATION • Synonymy identification • Text coherence • Semantic priming

37. SYNONYMY: THE TOEFL TEST

38. TOEFL TEST: RESULTS

39. Some psychological evidence for vector-space representations • Burgess and Lund (1996, 1997): the clusters found with HAL correlate well with those observed using semantic priming experiments. • Landauer, Foltz, and Laham (1997): scores overlap with those of humans on standard vocabulary and topic tests; mimic human scores on category judgments; etc. • Evidence about `prototype theory’ (Rosch et al, 1976) • Posner and Keel, 1968 • subjects presented with patterns of dots that had been obtained by variations from single pattern (`prototype’) • Later, they recalled prototypes better than samples they had actually seen • Rosch et al, 1976: `basic level’ categories (apple, orange, potato, carrot) have higher `cue validity’ than elements higher in the hierarchy (fruit, vegetable) or lower (red delicious, cox) ANLE

40. The HAL model (Burgess and Lund, 1995, 1996, 1997) • A 160 million words corpus of articles extracted from all newsgroups containing English dialogue • Context words: the 70,000 most frequently occurring symbols within the corpus • Window size: 10 words to the left and the right of the word • Measure of similarity: cosine ANLE

41. HAL AND SEMANTIC PRIMING

42. INFORMATION RETRIEVAL • GOAL: Find the documents most relevant to a certain QUERY • Latest development: WEB SEARCH • Use the Web as the collection of documents • Related: • QUESTION-ANSWERING • DOCUMENT CLASSIFICATION

43. DOCUMENTS AS BAGS OF WORDS INDEX DOCUMENT broadmay rallyrallied signal stockstocks techtechnology traderstraders trend broad tech stock rally may signal trend - traders. technology stocks rallied on tuesday, with gains scored broadly across many sectors, amid what some traders called a recovery from recent doldrums.

44. THE VECTOR SPACE MODEL • Query and documents represented as vectors of index terms, assigned non-binary WEIGHTS • Similarity calculated using vector algebra: COSINE (cfr. lexical similarity models) • RANKED similarity • Most popular of all models (cfr. Salton and Lesk’s SMART)

45. TERM WEIGHTING IN VECTOR SPACE MODELS: THE TF.IDF MEASURE FREQUENCY of term i in document k Number of documents with term i

46. VECTOR-SPACE MODELS WITH SYNTACTIC INFORMATION • Pereira and Tishby, 1992: two words are similar if they occur as objects of the same verbs • John ate POPCORN • John ate BANANAS • C(w) is the distribution of verbs for which w served as direct object. • First approximation: just counts • In fact: probabilities • Similarity: RELATIVE ENTROPY ANLE

47. (SYNTACTIC) RELATION-BASED VECTOR MODELS attacked subj obj fox dog det mod det mod the red the lazy E.g., Grefenstette, 1994; Lin, 1998; Curran and Moens, 2002 Cog/Comp Neuroscience

48. SEXTANT (Grefenstette, 1992) It was concluded that the carcinoembryonic antigens represent cellular constituents which are repressed during the course of differentiation the normal digestive system epithelium and reappear in the corresponding malignant cells by a process of derepressive dedifferentiation antigen carcinoembryonic-ADJantigen repress-DOBJantigen represent-SUBJconstituent cellular-ADJconstituent represent-DOBJcourse repress-IOBJ…….. ANLE

49. DOG CAT dog pet-DOBJdog eat-SUBJ dog shaggy-ADJdog brown-ADJdog leash-NN cat pet-DOBJcat pet-DOBJcat hairy-ADJcat leash-NN Jaccard: SEXTANT: Similarity measure ANLE

50. MULTIDIMENSIONAL SCALING • Many models (included HAL) apply techniques for REDUCING the number of dimensions • Intuition: many features express a similar property / topic