1 / 47

Python for NLP and the Natural Language Toolkit

Python for NLP and the Natural Language Toolkit. CS1573: AI Application Development, Spring 2003 (modified from Edward Loper’s notes). Outline.

marva
Download Presentation

Python for NLP and the Natural Language Toolkit

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Python for NLP and the Natural Language Toolkit CS1573: AI Application Development, Spring 2003 (modified from Edward Loper’s notes)

  2. Outline Review: Introduction to NLP (knowledge of language, ambiguity, representations and algorithms, applications) HW 2 discussion Tutorials: Basics, Probability

  3. Python and Natural Language Processing • Python is a great language for NLP: • Simple • Easy to debug: • Exceptions • Interpreted language • Easy to structure • Modules • Object oriented programming • Powerful string manipulation

  4. Modules and Packages Python modules “package program code and data for reuse.” (Lutz) Similar to library in C, package in Java. Python packages are hierarchical modules (i.e., modules that contain other modules). Three commands for accessing modules: import from…import reload

  5. Modules and Packages: import • The importcommand loads a module: # Load the regular expression module >>> import re • To access the contents of a module, use dotted names: # Use the search method from the re module >>> re.search(‘\w+’, str) • To list the contents of a module, use dir: >>> dir(re) [‘DOTALL’, ‘I’, ‘IGNORECASE’,…]

  6. Modules and Packagesfrom…import • The from…import command loads individual functions and objects from a module: # Load the search function from the re module >>> from re import search • Once an individual function or object is loaded with from…import,it can be used directly: # Use the search method from the re module >>> search (‘\w+’, str)

  7. Import Keeps module functions separate from user functions. Requires the use of dotted names. Works with reload. from…import Puts module functions and user functions together. More convenient names. Does not work with reload. Import vs. from…import

  8. Modules and Packages: reload • If you edit a module, you must use the reload command before the changes become visible in Python: >>> import mymodule ... >>> reload (mymodule) • The reload command only affects modules that have been loaded with import; it does not update individual functions and objects loaded with from...import.

  9. Introduction to NLTK • The Natural Language Toolkit (NLTK) provides: • Basic classes for representing data relevant to natural language processing. • Standard interfaces for performing tasks, such as tokenization, tagging, and parsing. • Standard implementations of each task, which can be combined to solve complex problems.

  10. NLTK: Example Modules • nltk.token: processing individual elements of text, such as words or sentences. • nltk.probability: modeling frequency distributions and probabilistic systems. • nltk.tagger: tagging tokens with supplemental information, such as parts of speech or wordnet sense tags. • nltk.parser: high-level interface for parsing texts. • nltk.chartparser: a chart-based implementation of the parser interface. • nltk.chunkparser: a regular-expression based surface parser.

  11. NLTK: Top-Level Organization • NLTK is organized as a flat hierarchy of packages and modules. • Each module provides the tools necessary to address a specific task • Modules contain two types of classes: • Data-oriented classes are used to represent information relevant to natural language processing. • Task-oriented classes encapsulate the resources and methods needed to perform a specific task.

  12. To the First Tutorials • Tokens and Tokenization • Frequency Distributions

  13. The Token Module • It is often useful to think of a text in terms of smaller elements, such as words or sentences. • The nltk.token module defines classes for representing and processing these smaller elements. • What might be other useful smaller elements?

  14. Tokens and Types • The term word can be used in two different ways: • To refer to an individual occurrence of a word • To refer to an abstract vocabulary item • For example, the sentence “my dog likes his dog” contains five occurrences of words, but four vocabulary items. • To avoid confusion use more precise terminology: • Word token: an occurrence of a word • Word Type: a vocabulary item

  15. Tokens and Types (continued) • In NLTK, tokens are constructed from their types using the Token constructor: >>> from nltk.token import * >>> my_word_type = 'dog' 'dog' >>> my_word_token =Token(my_word_type) ‘dog'@[?] • Token member functions include type and loc

  16. Text Locations • A text location @ [s:e] specifies a region of a text: • s is the start index • e is the end index • The text location @ [s:e]specifies the text beginning at s, and including everything up to (but not including) the text at e. • This definition is consistent with Python slice. • Think of indices as appearing between elements: I saw a man 0 1 2 3 4 • Shorthand notation when location width = 1.

  17. Text Locations (continued) • Indices can be based on different units: • character • word • sentence • Locations can be tagged with sources (files, other text locations – e.g., the first word of the first sentence in the file) • Location member functions: • start • end • unit • source

  18. Tokenization • The simplest way to represent a text is with a single string. • Difficult to process text in this format. • Often, it is more convenient to work with a list of tokens. • The task of converting a text from a single string to a list of tokens is known as tokenization.

  19. Tokenization (continued) • Tokenization is harder that it seems I’ll see you in New York. The aluminum-export ban. • The simplest approach is to use “graphic words” (i.e., separate words using whitespace) • Another approach is to use regular expressions to specify which substrings are valid words. • NLTK provides a generic tokenization interface: TokenizerI

  20. TokenizerI • Defines a single method, tokenize, which takes a string and returns a list of tokens • Tokenize is independent of the level of tokenization and the implementation algorithm

  21. Example • from nltk.token import WSTokenizer from nltk.draw.plot import Plot #Extract a list of words from the corpus corpus = open('corpus.txt').read() tokens = WSTokenizer().tokenize(corpus) # Count up how many times each word length occurs wordlen_count_list = [] for token in tokens: wordlen = len(token.type()) # Add zeros until wordlen_count_list is long enough while wordlen >= len(wordlen_count_list): wordlen_count_list.append(0) # Increment the count for this word length wordlen_count_list[wordlen] += 1 Plot(wordlen_count_list)

  22. Next Tutorial: Probability • An experiment is any process which leads to a well-defined outcome • A sample is any possible outcome of a given experiment • Rolling a die?

  23. Outline Review Basics Probability Experiments and Samples Frequency Distributions Conditional Frequency Distributions

  24. Review: NLTK Goals Classes for NLP data Interfaces for NLP tasks Implementations, easily combined (what is an example?)

  25. Accessing NLTK What is the relation to Python?

  26. Words Types and Tokens Text Locations Member Functions

  27. Tokenization TokenizerI Implementations >>> tokenizer = WSTokenizer() >>> tokenizer.tokenize(text_str) ['Hello'@[0w], 'world.'@[1w], 'This'@[2w], 'is'@[3w], 'a'@[4w], 'test'@[5w], 'file.'@[6w]]

  28. Word Length Freq. Distribution Example from nltk.token import WSTokenizer from nltk.probability import SimpleFreqDist # Extract a list of words from the corpus corpus = open('corpus.txt').read() tokens = WSTokenizer().tokenize(corpus) # Construct a frequency distribution of word lengths wordlen_freqs = SimpleFreqDist() for token in tokens: wordlen_freqs.inc(len(token.type())) # Extract the set of word lengths found in the corpus wordlens = wordlen_freqs.samples()

  29. Frequency Distributions • A frequency distribution records the number of times each outcome of an experiment has occurred • >>> freq_dist = FreqDist() >>> for token in document: ... freq_dist.inc(token.type()) • Constructor, then initialization by storing experimental outcomes

  30. Methods • The freq method returns the frequencey of a given sample. • We can find the number of times a given sample occured with the count method • We can find the total number of sample outcomes recorded by a frequency distribution with the N method • The samples method returns a list of all samples that have been recorded as outcomes by a frequency distribution • We can find the sample with the greatest number of outcomes with the max method

  31. Examples of Methods • >>> freq_dist.count('the') 6 • >>> freq_dist.freq('the') 0.012 • >>> freq_dist.N() 500 • >>> freq_dist.max() ‘the’

  32. Simple Word Length Example • >>> from nltk.token import WSTokenizer >>> from nltk.probability import FreqDist >>> corpus = open('corpus.txt').read() >>> tokens = WSTokenizer().tokenize(corpus) # What is the distribution of word lengths in a corpus? >>> freq_dist = FreqDist() >>> for token in tokens: ... freq_dist.inc(len(token.type())) • What is the "outcome" for our experiment?

  33. Simple Word Length Example • >>> from nltk.token import WSTokenizer >>> from nltk.probability import FreqDist >>> corpus = open('corpus.txt').read() >>> tokens = WSTokenizer().tokenize(corpus) # What is the distribution of word lengths in a corpus? >>> freq_dist = FreqDist() >>> for token in tokens: ... freq_dist.inc(len(token.type())) • This length is the "outcome" for our experiment, so we use inc() to increment its count in a frequency distribution.

  34. Complex Word Length Example • # define vowels as "a", "e", "i", "o", and "u" >>> VOWELS = ('a', 'e', 'i', 'o', 'u') # distribution for words ending in vowels? >>> freq_dist = FreqDist() >>> for token in tokens: ... if token.type()[-1].lower() in VOWELS: ... freq_dist.inc(len(token.type())) • What is the condition?

  35. More Complex Example • # What is the distribution of word lengths for # words following words that end in vowels? >>> ended_in_vowel = 0#Did last word end in vowel? >>> freq_dist = FreqDist() >>> for token in tokens: ... if ended_in_vowel: ... Freq_dist.inc(len(token.type())) ... ended_in_vowel=token.type()[-1].lower() in VOWELS

  36. Conditional Frequency Distributions • A condition specifies the context in which an experiment is performed • A conditional frequency distribution is a collection of frequency distribtuions for the same experiment, run under different conditions • The individual frequency distributions are indexed by the condition. • NLTK ConditionalFreqDist class • >>> cfdist = ConditionalFreqDist() <ConditionalFreqDist with 0 conditions>

  37. Conditional Frequency Distributions (continued) • To access the frequency distribution for a condition, use the indexing operator : >>> cfdist['a'] <FreqDist with 0 outcomes> • # Record lengths of some words starting with 'a' >>> for word in 'apple and arm'.split(): ... cfdist['a'].inc(len(word)) • # How many are 3 characters long? >>> cfdist['a'].freq(3) 0.66667 • To list accessed conditions, use the conditions method: >>> cfdist.conditions() ['a']

  38. Example: Conditioning on a Word’s Initial Letter • >>> from nltk.token import WSTokenizer >>> from nltk.probability import ConditionalFreqDist >>> from nltk.draw.plot import Plot # >>> corpus = open('corpus.txt').read() >>> tokens = WSTokenizer().tokenize(corpus) >>> cfdist = ConditionalFreqDist()

  39. Example (continued) • # How does initial letter affect word length? >>> for token in tokens: ... outcome = len(token.type()) ... condition = token.type()[0].lower() ... cfdist[condition].inc(outcome) • What are the condition and the outcome?

  40. Example (continued) • # How does initial letter affect word length? >>> for token in tokens: ... outcome = len(token.type()) ... condition = token.type()[0].lower() ... cfdist[condition].inc(outcome) • What are the condition and the outcome? • Condition = the initial letter of the token • Outcome = its word length

  41. Prediction • Prediction is the problem of deciding a likely outcome for a given run of an experiment. • To predict the outcome, we first examine a training corpus. • Training corpus • The context and outcome for each run are known • Given a new run, we choose the outcome that occurred most frequently for the context • Conditional frequency distribution finds the most frequent occurrrence

  42. Prediction Example: Outline Record each outcome in the training corpus, using the context that the experiment was under as the condition Access the frequency distribution for a given context with the indexing operator Use the max() method to find the most likely outcome

  43. Example: Predicting Words • Predict word's type, based on preceding word type • >>> from nltk.token import WSTokenizer >>> from nltk.probability import ConditionalFreqDist >>> corpus = open('corpus.txt').read() >>> tokens = WSTokenizer().tokenize(corpus) >>> cfdist = ConditionalFreqDist() #empty

  44. Example (continued) • >>> context = None # The type of the preceding word >>> for token in tokens: ... outcome = token.type() ... cfdist[context].inc(outcome) ... context = token.type()

  45. Example (continued) • >>> cfdist['prediction'].max() 'problems' >>> cfdist['problems'].max() 'in' >>> cfdist['in'].max() 'the‘ • What are we predicting here?

  46. Example (continued) We predict the most likely word for any context Generation application: >>> word = 'prediction' >>> for i in range(15): ... print word, ... word = cfdist[word].max() prediction problems in the frequency distribution of the frequency distribution of the frequency distribution of

  47. For Next Time HW3 To run NLTK from unixs.cis.pitt.edu, you should add /afs/cs.pitt.edu/projects/nltk/bin to your search path Regular Expressions (J&M handout, NLTK tutorial)

More Related