Lab 14: Text & Corpus Processing with NLTK. Ling 1330/2330: Computational Linguistics Na-Rae Han

Transcription

1 Lab 14: Text & Corpus Processing with NLTK Ling 1330/2330: Computational Linguistics Na-Rae Han

2 Getting started with NLTK book NLTK Book, with navigation panel: NLTK Book, without: Chapter 1. Language Processing and Python Chapter 2. Accessing Text Corpora and Language Resources 10/18/2018 2

3 Install NLTK and NLTK data NLTK (Mac): NLTK (Win): NLTK data: Test to confirm everything works: 10/18/2018 3

4 textstats and pipeline 'Rose is a rose is a rose is a rose.' gettokens ['rose', 'is', 'a', 'rose', 'is', 'a', 'rose', 'is', 'a', 'rose', '.'] gettypes getfreq getword2grams ['.', 'a', 'is', 'rose'] x = 2 getxlengthwords {'a': 3, 'is': 3, '.': 1, 'rose': 4} x = 3 getxfreqwords [('rose', 'is'), ('is', 'a'), ('a', 'rose'), ('rose', 'is'), ('is', 'a'), ] ['is', 'rose'] ['a', 'is', 'rose'] 10/18/2018 4

5 List comprehension can do them all 'Rose is a rose is a rose is a rose.' gettokens ['rose', 'is', 'a', 'rose', 'is', 'a', 'rose', 'is', 'a', 'rose', '.'] gettypes getfreq getword2grams ['.', 'a', 'is', 'rose'] x = 2 getxlengthwords ['is', 'rose'] {'a': 3, 'is': 3, '.': 1, 'rose': 4} x = 3 getxfreqwords ['a', 'is', 'rose'] [('rose', 'is'), ('is', 'a'), ('a', 'rose'), ('rose', 'is'), ('is', 'a'), ] No longer needed. Can be achieved via list comprehension! 10/18/2018 5

6 Multiple ways to build 'Rose is a rose is a rose is a rose.' gettokens ['rose', 'is', 'a', 'rose', 'is', 'a', 'rose', 'is', 'a', 'rose', '.'] sorted(set()) gettypes getfreq ['.', 'a', 'is', 'rose'] {'a': 3, 'is': 3, '.': 1, 'rose': 4} sorted() 10/18/2018 6

7 Multiple ways to build 'Rose is a rose is a rose is a rose.' gettokens ['rose', 'is', 'a', 'rose', 'is', 'a', 'rose', 'is', 'a', 'rose', '.'] sorted(set()) gettypes Goodbye getfreq textstats, ['.', 'a', 'is', 'rose'] Hello, NLTK! {'a': 3, 'is': 3, '.': 1, 'rose': 4} sorted() 10/18/2018 7

8 NLTK's functions 'Rose is a rose is a rose is a rose.' nltk.word_tokenize() ['Rose', 'is', 'a', 'rose', 'is', 'a', 'rose', 'is', 'a', 'rose', '.'] FreqDist({'rose': 3, 'a': 3, 'is': 3, 'Rose': 1, '.': 1}) ['.', 'Rose', 'a', 'is', 'rose'] nltk.freqdist() sorted() nltk.ngrams() sorted(set()) [('Rose', 'is'), ('is', 'a'), ('a', 'rose'), ('rose', 'is'), ('is', 'a'), ('a', 'rose'), ('rose', '.')] 10/18/2018 8

9 NLTK's tokenizer >>> import nltk >>> nltk.word_tokenize('hello, world!') ['Hello', ',', 'world', '!'] >>> nltk.word_tokenize("i haven't seen Star Wars.") ['I', 'have', "n't", 'seen', 'Star', 'Wars', '.'] >>> nltk.word_tokenize("it's 5 o'clock. Call Ted...!") ['It', "'s", '5', "o'clock", '.', 'Call', 'Ted', '...', '!'] >>> rose = 'Rose is a rose is a rose is a rose.' >>> nltk.word_tokenize(rose) ['Rose', 'is', 'a', 'rose', 'is', 'a', 'rose', 'is', 'a', 'rose', '.'] >>> rtoks = nltk.word_tokenize(rose) >>> rtoks ['Rose', 'is', 'a', 'rose', 'is', 'a', 'rose', 'is', 'a', 'rose', '.'] >>> type(rtoks) <class 'list'> nltk.word_tokenize() Note differences! No lowercasing, n't, o'clock a word Good-old list type. 10/18/2018 9

10 NLTK and frequency counts >>> rfreq = nltk.freqdist(rtoks) >>> rfreq FreqDist({'rose': 3, 'a': 3, 'is': 3, 'Rose': 1, '.': 1}) >>> rfreq['is'] 3 >>> rfreq.keys() dict_keys(['a', 'is', 'Rose', 'rose']) >>> rfreq.values() dict_values([3, 3, 1, 3]) >>> rfreq.items() dict_items([('a', 3), ('is', 3), ('Rose', 1), ('rose', 3)]) >>> sorted(rfreq) ['Rose', 'a', 'is', 'rose'] >>> type(rfreq) <class 'nltk.probability.freqdist'> word types nltk.freqdist() FreqDist works very much like our frequency count dictionary but it's NLTK's own custom data type! 10/18/

11 FreqDist can do much more >>> dir(rfreq) ['B', 'N', 'Nr', ' add ', ' and ', ' class ', 'clear', 'copy', 'elements', 'freq', 'fromkeys', 'get', 'hapaxes', 'items', 'keys', 'max', 'most_common', 'pformat', 'plot', 'pop', 'popitem', 'pprint', 'r_nr', 'setdefault', 'subtract', 'tabulate', 'unicode_repr', 'update', 'values'] >>> rfreq.hapaxes() ['Rose', '.'] >>> rfreq.tabulate() rose a is Rose >>> rfreq.most_common(2) [('a', 3), ('is', 3)] >>> rfreq['platypus'] 0 >>> rfreq.plot() >>> rfreq.max() 'a' Graph window pops up nltk.freqdist comes with additional handy methods! No "key not found" error! Defaults to 0. 10/18/

12 NLTK and n-grams >>> rtoks ['Rose', 'is', 'a', 'rose', 'is', 'a', 'rose', 'is', 'a', 'rose', '.'] >>> nltk.ngrams(rtoks, 2) <generator object ngrams at 0x0A18B0C0> >>> for gram in nltk.ngrams(rtoks, 2): print(gram) nltk.ngrams() returns a generator object. Cannot see it as a whole, but works in for loop. ('Rose', 'is') ('is', 'a') ('a', 'rose') ('rose', 'is') ('is', 'a') ('a', 'rose') ('rose', 'is') ('is', 'a') ('a', 'rose') ('rose', '.') Feed to nltk.freqdist() to obtain bigram frequency distribution. >>> r2grams = nltk.ngrams(rtoks, 2) >>> nltk.freqdist(r2grams) FreqDist({('is', 'a'): 3, ('a', 'rose'): 3, ('rose', 'is'): 2, ('rose', '.'): 1, ('Rose', 'is'): 1}) 10/18/

13 New trick: sentence tokenization! >>> foo = 'Hello, earthlings! I come in peace. Take me to your leader.' >>> nltk.sent_tokenize(foo) ['Hello, earthlings!', 'I come in peace.', 'Take me to your leader.'] >>> foosents = nltk.sent_tokenize(foo) >>> foosents[0] 'Hello, earthlings!' >>> foosents[1] 'I come in peace.' >>> foosents[-1] 'Take me to your leader.' >>> len(foosents) 3 Total number of sentences nltk.sent_tokenize() takes a text string, returns a list of sentences as strings. 10/18/

14 Try it out 5 minutes In IDLE shell, copy and paste the "Moby Dick" opening paragraph from Text samples, as string called motxt. Then, use NLTK's functions to produce: word tokens nltk.word_tokenize(txt) word frequency distribution nltk.freqdist(tokens) bigram list nltk.ngrams(tokens, 2) bigram frequency distribution nltk.freqdist(bigrams) sentence list nltk.sent_tokenize(txt) How many words? How many sentences? "I" occurs how many times? 20 most common words? 20 most common bigrams? 10/18/

15 NLTK's corpus methods NLTK provides methods for reading in a corpus as a "corpus object". Many different methods available for different types of corpora (plain text? POS-tagged? XML format?) PlaintextCorpusReader suits our corpora just fine. Once a corpus is "loaded" and a corpus object created, we can use pre-defined methods on the corpus. 10/18/

16 Loading your own Corpus >>> from nltk.corpus import PlaintextCorpusReader >>> corpus_root = 'D:/Lab/mycorpus/ICLE2' >>> jacor = PlaintextCorpusReader(corpus_root, 'JP.*txt') >>> type(jacor) <class 'nltk.corpus.reader.plaintext.plaintextcorpusreader'> >>> jacor.fileids() ['JPSW1001.txt', 'JPSW1002.txt', 'JPSW1003.txt', 'JPSW1004.txt', 'JPSW1005.txt', 'JPSW1006.txt', 'JPSW1007.txt', 'JPSW1008.txt', 'JPSW1009.txt', 'JPSW1010.txt', 'JPSW1011.txt', 'JPSW1012.txt', 'JPSW1013.txt', 'JPSW1014.txt', 'JPSW1015.txt', 'JPSW1016.txt', 'JPSW1017.txt', 'JPSW1018.txt', 'JPSW1019.txt', 'JPSW1020.txt', 'JPSW1021.txt', 'JPSW1022.txt', 'JPSW1023.txt', 'JPSW1024.txt', 'JPSW1025.txt', 'JPSW1026.txt', 'JPSW1027.txt', 'JPSW1028.txt', 'JPSW1029.txt', 'JPSW1030.txt'] >>> len(jacor.fileids()) 30 >>> jacor.fileids()[0] 'JPSW1001.txt' File names without path work as the file ID.* means any character, in any number 10/18/

17 .words() is a list of word tokens >>> jacor.words() ['I', 'agree', 'greatly', 'this', 'topic', 'mainly',...] >>> len(jacor.words()) >>> jacor.words()[:50] ['I', 'agree', 'greatly', 'this', 'topic', 'mainly', 'because', 'I', 'think', 'that', 'English', 'becomes', 'an', 'official', 'language', 'in', 'the', 'not', 'too', 'distant', '.', 'Now', ',', 'many', 'people', 'can', 'speak', 'English', 'or', 'study', 'it', 'all', 'over', 'the', 'world', ',', 'and', 'so', 'more', 'people', 'will', 'be', 'able', 'to', 'speak', 'English', '.', 'Before', 'the', 'Japanese'] >>> jacor.words('jpsw1001.txt') ['I', 'agree', 'greatly', 'this', 'topic', 'mainly',...] >>> len(jacor.words('jpsw1001.txt')) 475 >>> len(jacor.words('jpsw1002.txt')) 559 All tokenized words in this corpus Word tokens in 1 st file only Whoa! Spill-proof! 10/18/

18 .sents() is a list of sentence tokens >>> jacor.sents() [['I', 'agree', 'greatly', 'this', 'topic', 'mainly', 'because', 'I', 'think', 'that', 'English', 'becomes', 'an', 'official', 'language', 'in', 'the', 'not', 'too', 'distant', '.'], ['Now', ',', 'many', 'people', 'can', 'speak', 'English', 'or', 'study', 'it', 'all', 'over', 'the', 'world', ',', 'and', 'so', 'more', 'people', 'will', 'be', 'able', 'to', 'speak', 'English', '.'], Again, spill-proof....] >>> jacor.sents()[0] ['I', 'agree', 'greatly', 'this', 'topic', 'mainly', 'because', 'I', 'think', 'that', 'English', 'becomes', 'an', 'official', 'language', 'in', 'the', 'not', 'too', 'distant', '.'] >>> jacor.sents()[-1] ['Let', "'", 's', 'study', 'English', ',', 'and', 'aommunicato', 'with', 'all', 'over', 'the', 'world', '!"'] >>> len(jacor.sents()) sentences in this corpus! 1 st sentence Last sentence Unlike nltk.sent_tokenize() output, each sentence is a list of word tokens! 10/18/

19 .sents() is a list of sentence tokens Sentences in 1 >>> jacor.sents('jpsw1001.txt') st file only [['I', 'agree', 'greatly', 'this', 'topic', 'mainly', 'because', 'I', 'think', 'that', 'English', 'becomes', 'an', 'official', 'language', 'in', 'the', 'not', 'too', 'distant', '.'], ['Now', ',', 'many', 'people', 'can', 'speak', 'English', 'or', 'study', 'it', 'all', 'over', 'the', 'world', ',', 'and', 'so', 'more', 'people', 'will', 'be', 'able', 'to', 'speak', 'English', '.'],...] >>> len(jacor.sents('jpsw1001.txt')) 22 >>> for f in jacor.fileids(): print(f, 'has', len(jacor.sents(f)), 'sentences') JPSW1001.txt has 22 sentences JPSW1002.txt has 43 sentences JPSW1003.txt has 40 sentences JPSW1004.txt has 28 sentences For-loop through file IDs, print out sentence count of each essay 10/18/

20 .raw() is a raw text string >>> jacor.raw()[:200] 'I agree greatly this topic mainly because I think that English becomes an official language in the not too distant. Now, many people can speak English or study it all over the world, and so more peopl' >>> jacor.raw()[-200:] 's. I like to study English than before. English is the common language. If we want to success the work, you need high linguistic ability. Let\'s study English, and aommunicato with all over the world!"' >>> len(jacor.raw()) >>> jacor.raw('jpsw1001.txt') Character count of entire corpus.raw() is not spill-proof, so we must be careful. Raw text of 1 st file only, small enough to flash in full "I agree greatly this topic mainly because I think that English becomes an official language in the not too distant. Now, many people can speak English or study it all over the world, and so more people will be able to speak English. Before the Japanese fall behind other people, we should be able to speak English, therefore, we must study English not only junior high school students or over 10/18/

21 Try it out 5 minutes Download the ICLE2 "Bulgarian vs. Japanese EFL Writing" corpus from CourseWeb (on HW5B). Then, load the corpus through NLTK: >>> from nltk.corpus import PlaintextCorpusReader >>> corpus_root = 'D:/Lab/mycorpus/ICLE2' >>> jacor = PlaintextCorpusReader(corpus_root, 'JP.*txt') >>> dir(jacor) ['CorpusView', ' class ', ' delattr ', ' dict ',... 'abspath', 'abspaths', 'citation', 'encoding', 'ensure_loaded', 'fileids', 'license', 'open', 'paras', 'raw', 'readme', 'root', 'sents', 'unicode_repr', 'words'] Try: jacor.fileids() jacor.words() jacor.sents() jacor.raw()[:200] jacor.words('jpsw1001.txt') jacor.sents('jpsw1001.txt') jacor.raw('jpsw1001.txt') 10/18/

22 Further corpus processing A corpus object (PlaintextCorpusReader) already has tokenized words, tokenized sentences, and raw text strings. Accessible as a whole corpus, or by-file basis Other data objects should then be built from the tokens: Word frequency distribution nltk.freqdist(tokens) Word type list sorted(set(tokens)) sorted(freqdist) N-gram list nltk.ngrams(tokens, n) N-gram frequency distribution nltk.freqdist(ngrams) 10/18/

23 Word frequency through FreqDist() >>> jafreq = nltk.freqdist(jacor.words()) >>> jafreq['english'] 709 >>> jafreq['the'] 352 >>> jafreq.most_common(20) nltk.freqdist() builds a frequency distribution object [('.', 997), (',', 709), ('English', 709), ('to', 571), ('I', 452), ('is', 413), ('the', 352), ('and', 272), ('we', 240), ('a', 236), ('in', 228), ('of', 197), ('that', 195), ('Japanese', 181), ('language', 180), ('master', 167), ('think', 165), ("'", 163), ('can', 151), ('people', 146)] >>> jafreq.freq('english') >>> jafreq.freq('the') >>> len(jafreq) 1613 >>> sorted(jafreq)[-20:] ['worth', 'would', 'wouldn', 'write', 'writing', 'written', 'wrong', 'year', 'years', 'yen', 'yes', 'yet', 'yhings', 'you', 'young', 'your', 'yourself', 'youth', '\x81', '\x87'] 10/18/

24 Word frequency distribution >>> jafreq.plot(50, cumulative=true) 10/18/

25 Building n-grams >>> ja2grams = nltk.ngrams(jacor.words(), 2) >>> ja2grams[:20] Traceback (most recent call last): File "<pyshell#64>", line 1, in <module> ja2grams[:20] TypeError: 'generator' object is not subscriptable >>> list(ja2grams)[:20] [('I', 'agree'), ('agree', 'greatly'), ('greatly', 'this'), ('this', 'topic'), ('topic', 'mainly'), ('mainly', 'because'), ('because', 'I'), ('I', 'think'), ('think', 'that'), ('that', 'English'), ('English', 'becomes'), ('becomes', 'an'), ('an', 'official'), ('official', 'language'), ('language', 'in'), ('in', 'the'), ('the', 'not'), ('not', 'too'), ('too', 'distant'), ('distant', '.')] >>> list(ja2grams)[:20] [] nltk.ngrams() builds an n-gram list Oops, a generator object -- Cannot slice. list-ify, and then slice. After a full round of "generating" (list-ifying above, a for-loop, etc. ), the generator object is now empty! next page 10/18/

26 n-gram frequency distribution So, in order to use >>> ja2grams = nltk.ngrams(jacor.words(), 2) ja2grams again to build a >>> ja2gramfd = nltk.freqdist(ja2grams) frequency distribution, we >>> len(ja2gramfd) 7154 should re-build ja2grams. >>> ja2gramfd.most_common(20) [(('.', 'I'), 152), (('I', 'think'), 137), (('master', 'English'), 127), (('English', 'is'), 125), (('English', '.'), 124), (("'", 't'), 119), (('to', 'master'), 118), ((',', 'I'), 90), (('speak', 'English'), 87), (('English', ','), 82), (('second', 'language'), 80), (('English', 'as'), 78), ((',', 'we'), 78), (('as', 'a'), 76), (('need', 'to'), 74), (('the', 'world'), 73), (('language', '.'), 70), (('a', 'second'), 68), (('it', 'is'), 68), (('in', 'the'), 65)] >>> for (gram, count) in ja2gramfd.most_common(20): print(gram, count) ('.', 'I') 152 ('I', 'think') 137 ('master', 'English') 127 ('English', 'is') 125 ('English', '.') 124 Whoa. 'master English' is #3 most frequent bigram. 26

27 Try it out 5 minutes Further process the ICLE2 "Bulgarian vs. Japanese EFL Writing" corpus. Word frequency distribution nltk.freqdist(jacor.words()) Word type list sorted(set(jacor.words())) # Don't flash to SHELL! sorted(freqdist) N-gram list nltk.ngrams(jacor.words(), n) N-gram frequency distribution nltk.freqdist(ngrams) Most frequent words? How many times is 'master' mentioned? How many word types? Most frequent bigrams? 10/18/

28 Summary Process a SINGLE TEXT Process a CORPUS You start with -> A single text string A corpus object (1) Text string ^^ Built-in.raw() (2) Word tokens Build using nltk.word_tokenize() Built-in.words() (3) Sentence tokens Build using nltk.sent_tokenize() **Note: each sentence is a string (4) Word frequency distribution Built-in.sents() **Note: each sentence is a tokenized word list Build from (2), using nltk.freqdist() (5) Word types Build from (2) or (4), using sorted(), set() (6) N-grams Build from (2), using nltk.ngrams() (7) N-gram frequency distrib. (8) nltk.text object Build from (6), using nltk.freqdist() Build from (2), using nltk.text() Accessible corpus-wide or per-file Next class 10/18/

29 Wrapping up Homework 6 due next Tuesday Friday recitation Final exam: Monday (Dec 10) 8am 10am is the assigned time We decided to have it later at 4pm-6pm, same day, at LMC (G17). 10/18/