Lab 10: List Comprehension, Handling Large Text. Ling 1330/2330: Computational Linguistics Na-Rae Han

Size: px

Start display at page:

Download "Lab 10: List Comprehension, Handling Large Text. Ling 1330/2330: Computational Linguistics Na-Rae Han"

Shanon Morris
5 years ago
Views:

1 Lab 10: List Comprehension, Handling Large Text Ling 1330/2330: Computational Linguistics Na-Rae Han

2 Objectives HW3, Exercise 6 review List comprehension Filtering Transforming Beyond a single, small text file Going big: how to operate in IDLE shell (without breaking it) 9/27/2018 2

3 HW3, Exercise 6 textstats.py as a module Functions should be placed on top, main() function should be towards the bottom, and finally the part that calls main: gettypes() SLOW FAST Edit your textstats 9/27/2018 3

4 Filtering a list, the old way >>> mary = 'Mary had a little lamb, whose fleece was white as snow.'.split() >>> mary ['Mary', 'had', 'a', 'little', 'lamb,', 'whose', 'fleece', 'was', 'white', 'as', 'snow.'] How to make a list of words that have 'a'? >>> alist = [] >>> for w in mary: if 'a' in w: alist.append(w) >>> alist ['Mary', 'had', 'a', 'lamb,', 'was', 'as'] You need to make a new empty list, and then iterate through mary to find items to put in 9/27/2018 4

5 Filtering with list comprehension >>> mary = 'Mary had a little lamb, whose fleece was white as snow.'.split() >>> mary ['Mary', 'had', 'a', 'little', 'lamb,', 'whose', 'fleece', 'was', 'white', 'as', 'snow.'] How to make a list of words that have 'a'? >>> [w for w in mary if 'a' in w] ['Mary', 'had', 'a', 'lamb,', 'was', 'as'] >>> The power of LIST COMPREHENSION Creating a new list where elements meet a certain condition: [x for x in list if... ] 9/27/2018 5

6 Try it out 2 minutes >>> mary = 'Mary had a little lamb, whose fleece was white as snow.'.split() >>> mary ['Mary', 'had', 'a', 'little', 'lamb,', 'whose', 'fleece', 'was', 'white', 'as', 'snow.'] Syntax: [x for x in list if... ] Words that have 'a' Words that are 5 chars or longer >>> [w for w in mary if 'a' in w] ['Mary', 'had', 'a', 'lamb,', 'was', 'as'] >>> [w for w in mary ifuse len(w) len() >=5] ['little', 'lamb,', 'whose', 'fleece', 'white', 'snow.'] Words that are 5 chars or longer and without symbols >>> [w for w in mary if len(w) use >=5.isalnum() and w.isalnum()] ['little', 'whose', 'fleece', 'white'] 9/27/2018 6

7 Try it out 2 minutes >>> mary = 'Mary had a little lamb, whose fleece was white as snow.'.split() >>> mary ['Mary', 'had', 'a', 'little', 'lamb,', 'whose', 'fleece', 'was', 'white', 'as', 'snow.'] Syntax: [x for x in list if... ] Words that have 'a' Words that are 5 chars or longer >>> [w for w in mary if 'a' in w] ['Mary', 'had', 'a', 'lamb,', 'was', 'as'] >>> [w for w in mary if len(w) >=5] ['little', 'lamb,', 'whose', 'fleece', 'white', 'snow.'] Words that are 5 chars or longer and without symbols >>> [w for w in mary if len(w) >=5 and w.isalnum()] ['little', 'whose', 'fleece', 'white'] 9/27/2018 7

8 A list of English words 2 minutes In Python shell, load up the ENABLE word list we used in last class. If you saved a pickle file 'words.pkl', unpickle it. If you don't have a pickled list, build it from scratch. Download the ENABLE word list, posted on Norvig's site: Open the file and make a list object: >>> f = open('enable1.txt') >>> txt = f.read() >>> f.close() >>> wlist = txt.split() >>> print(wlist[:100]) ['aa', 'aah', 'aahed', 'aahing', 'aahs', 'abaka', 'abakas', 'abalone', 'abalones', enable1.txt abaka abakas abalone abalones 9/27/2018 8

9 ENABLE word list: what's in >>> import pickle >>> f = open('words.pkl', 'rb') >>> wlist = pickle.load(f) >>> f.close() >>> len(wlist) >>> wlist[:10] ['aa', 'aah', 'aahed', 'aahing', 'aahs', 'aal', 'aalii', 'aaliis', 'aals', 'aardvark'] >>> wlist[-10:] ['zymology', 'zymosan', 'zymosans', 'zymoses', 'zymosis', 'zymotic', 'zymurgies', 'zymurgy', 'zyzzyva', 'zyzzyvas'] >>> 'platypus' in wlist True >>> 'syntactician' in wlist False >>> 'a' in wlist False WHAA? >>> 9

10 Try it out 2 minutes Syntax: [x for x in list if... ] >>> [x for x in wlist if 'wkw' in x] ['awkward', 'awkwarder', 'awkwardest', 'awkwardly',?? 'awkwardness', 'awkwardnesses', 'hawkweed', 'hawkweeds'] Words that have 'wkw' >>> [x for x in wlist if len(x)?? >=25] ['electroencephalographically', 'ethylenediaminetetraacetate', 'ethylenediaminetetraacetates', 'immunoelectrophoretically', 'phosphatidylethanolamines'] >>> [x for x in wlist if len(x) >=15 and?? x.startswith('x')] ['xerographically', 'xeroradiographies', 'xeroradiography'] Too easy for you? Get creative! Show us what you could find. Words that are 25+ chars Words that are 15+ chars and start with 'x' 9/27/

11 Try it out 2 minutes Syntax: [x for x in list if... ] >>> [x for x in wlist if 'wkw' in x] ['awkward', 'awkwarder', 'awkwardest', 'awkwardly', 'awkwardness', 'awkwardnesses', 'hawkweed', 'hawkweeds'] Words that have 'wkw' >>> [x for x in wlist if len(x) >=25] ['electroencephalographically', 'ethylenediaminetetraacetate', 'ethylenediaminetetraacetates', 'immunoelectrophoretically', 'phosphatidylethanolamines'] >>> [x for x in wlist if len(x) >=15 and x.startswith('x')] ['xerographically', 'xeroradiographies', 'xeroradiography'] Words that are 25+ chars Words that are 15+ chars and start with 'x' 9/27/

12 Try it out 2 minutes Syntax: [x for x in list if... ] >>> [w for w in wlist if w.startswith('lingui')] ['linguine', 'linguines', 'linguini', 'linguinis', 'linguist', 'linguistic', 'linguistical', 'linguistically', 'linguistician', 'linguisticians', 'linguistics', 'linguists'] >>> [w for w in wlist if len(w) >=7 and 'a' not in w and 'e' not in w and 'i' not in w and 'o' not in w and 'u' not in w] ['glycyls', 'rhythms', 'tsktsks'] Words starting with 'lingui' Words that are 7+ characters and do not have a 'vowel' >>> [w for w in wlist if sorted(w) == sorted('cried')] ['cider', 'cried', 'dicer', 'riced'] Anagrams of cried 9/27/

13 Try it out 2 minutes Syntax: [x for x in list if... ] >>> [w for w in wlist if w.startswith('un') and w.endswith('ed')] Think before you press ENTER Words that start with 'un' and end with 'ed' >>> foo = [w for w in wlist if w.startswith('un') and w.endswith('ed')] >>> len(foo) 1076 >>> foo[:10] ['unabashed', 'unabated', 'unabraded', 'unabridged', 'unabsorbed', 'unabused', 'unaccented', 'unaccepted', 'unacclimated', 'unacclimatized'] >>> 1076 items. This is not a small list. 9/27/

14 Careful with list comprehension How many are 8 chars or longer? >>> [w for w in wlist if len(w) >=8] This is going to return a long list! Unless you're reasonably sure your list is short, assign the list to a new variable first >>> foo = [w for w in wlist if len(w) >=8] >>> len(foo) >>> foo[:10] ['aardvark', 'aardvarks', 'aardwolf', 'aardwolves', 'aasvogel', 'aasvogels', 'abacterial', 'abacuses', 'abalones', 'abampere'] and then look at snippets using slice indexing 9/27/

15 Transforming items in list, the old way >>> mary ['Mary', 'had', 'a', 'little', 'lamb,', 'whose', 'fleece', 'was', 'white', 'as', 'snow.'] How to make a new list with uppercase words? >>> mary.upper()... AttributeError: 'list' object has no attribute 'upper' >>> mup = [] >>> for w in mary: mup.append(w.upper()) >>> mup ['MARY', 'HAD', 'A', 'LITTLE', 'LAMB,', 'WHOSE', 'FLEECE', 'WAS', 'WHITE', 'AS', 'SNOW.'] Cannot uppercase a list You have had to create an empty new list and then put in uppercased words 9/27/

16 Transforming items in list >>> mary ['Mary', 'had', 'a', 'little', 'lamb,', 'whose', 'fleece', 'was', 'white', 'as', 'snow.'] Uppercased list, using list comprehension >>> [w.upper() for w in mary] ['MARY', 'HAD', 'A', 'LITTLE', 'LAMB,', 'WHOSE', 'FLEECE', 'WAS', 'WHITE', 'AS', 'SNOW.'] >>> Creating a new list where each element is transformed: [f(x) for x in list] 9/27/

17 Try it out 2 minutes >>> mary ['Mary', 'had', 'a', 'little', 'lamb,', 'whose', 'fleece', 'was', 'white', 'as', 'snow.'] Syntax: [f(x) for x in list] List of first characters List of word lengths List of True/False for having 'a' as substring >>> [w.upper() for w in mary] ['MARY', 'HAD', 'A', 'LITTLE', 'LAMB,', 'WHOSE', 'FLEECE', 'WAS', 'WHITE', 'AS', 'SNOW.'] >>> [w[0]? for w in mary] ['M', 'h', 'a', 'l', 'l', 'w', 'f', 'w', 'w', 'a', 's'] >>> [len(w)? for w in mary] [4, 3, 1, 6, 5, 5, 6, 3, 5, 2, 5] >>> ['a'? in w for w in mary] [True, True, True, False, True, False, False, True, False, True, False] 9/27/

18 Try it out 2 minutes >>> mary ['Mary', 'had', 'a', 'little', 'lamb,', 'whose', 'fleece', 'was', 'white', 'as', 'snow.'] Syntax: [f(x) for x in list] List of first characters List of word lengths List of True/False for having 'a' as substring >>> [w.upper() for w in mary] ['MARY', 'HAD', 'A', 'LITTLE', 'LAMB,', 'WHOSE', 'FLEECE', 'WAS', 'WHITE', 'AS', 'SNOW.'] >>> [w[0] for w in mary] ['M', 'h', 'a', 'l', 'l', 'w', 'f', 'w', 'w', 'a', 's'] >>> [len(w) for w in mary] [4, 3, 1, 6, 5, 5, 6, 3, 5, 2, 5] >>> ['a' in w for w in mary] [True, True, True, False, True, False, False, True, False, True, False] 9/27/

19 Try it out 2 minutes >>> mary ['Mary', 'had', 'a', 'little', 'lamb,', 'whose', 'fleece', 'was', 'white', 'as', 'snow.'] Words that are 6 chars or longer, in upper case >>> [w.upper() for w in?? mary if len(w) >= 6] ['LITTLE', 'FLEECE'] Calculate the average word length in one line! >>> [len(w) for w in mary] [4, 3, 1, 6, 5, 5, 6, 3, 5, 2, 5] >>> sum([len(w) for w in mary]) 45 >>> sum([len(w) Use for sum() w in mary]) and len() / len(mary) /27/

20 Try it out 2 minutes >>> mary ['Mary', 'had', 'a', 'little', 'lamb,', 'whose', 'fleece', 'was', 'white', 'as', 'snow.'] Words that are 6 chars or longer, in upper case >>> [w.upper() for w in mary if len(w) >= 6] ['LITTLE', 'FLEECE'] Calculate the average word length in one line! >>> [len(w) for w in mary] [4, 3, 1, 6, 5, 5, 6, 3, 5, 2, 5] >>> sum([len(w) for w in mary]) 45 >>> sum([len(w) for w in mary]) / len(mary) /27/

21 Back to English words 2 minutes Syntax: [f(x) for x in list] "Most words are 9 characters or longer." True or False? >>> TorF = [len(x) >=9 for x in wlist] >>> TorF[:20] [False, False, False, False, False, False, False, False, False, False, True, False, True, False, False, False, False, False, True, False] >>> TorF.count(True) >>> TorF.count(False) >>> TorF is a list of True/False on word x being at least 9 characters long 9/27/

22 Filtering + transforming 2 minutes Syntax: [f(x) for x in list if...] >>> [x for x in wlist if len(x) >=23] ['carboxymethylcelluloses', 'deinstitutionalizations', 'dichlorodifluoromethane', 'dichlorodifluoromethanes', 'reinstitutionalizations'] filter tuplify >>> [(len(x), x) for x in wlist if len(x) >=23] [(23, 'carboxymethylcelluloses'), (23, 'deinstitutionalizations'), (23, 'dichlorodifluoromethane'), (24, 'dichlorodifluoromethanes'), (23, 'reinstitutionalizations')] >>> sorted([(len(x), x) for x in wlist if len(x) >=23], reverse=true) [(28, 'ethylenediaminetetraacetates'), (27, 'ethylenediaminetetraacetate'), (27, 'electroencephalographically'), (25, 'phosphatidylethanolamines'), (23, 'carboxymethylcelluloses')] and sort! 9/27/

23 so-initial bigrams made easy Many of our past tasks can be accomplished through list comprehension. so-initial bigrams from last exercise: >>> import pickle >>> pf = open('bigramf_austen.pkl', 'rb') >>> bigramf = pickle.load(pf) >>> pf.close() >>> bigramf[('so', 'much')] 207 >>> bigramf[('so', 'will')] 1 >>> sograms = [x for x in list bigramf comprehension if x[0] == 'so'] >>> sograms[:10] [('so', 'to'), ('so', 'indeed'), ('so', 'affectionate'), ('so', 'prudent'), ('so', 'nervous'), ('so', 'mistake'), ('so', 'sought'), ('so', 'cheap'), ('so', 'i'), ('so', 'friendly')] >>> 9/27/

24 so-initial bigrams made easy Many of our past tasks can be accomplished through list comprehension. so-initial bigrams from last exercise: >>> import pickle >>> pf = open('bigramf_austen.pkl', 'rb') >>> bigramf = pickle.load(pf) >>> pf.close() >>> bigramf[('so', 'much')] 207 >>> bigramf[('so', 'will')] 1 so much >>> sograms = [x for x in bigramf if x[0]=='so'] simpler! >>> sograms[:10] [('so', 'to'), ('so', 'indeed'), ('so', 'affectionate'), ('so', 'prudent'), ('so', 'nervous'), ('so', 'mistake'), ('so', 'sought'), ('so', 'cheap'), ('so', 'i'), ('so', 'friendly')] >>> Also: [(w1, w2) for (w1, w2) in bigramf if w1=='so'] 9/27/

25 List comprehension: summary Syntax: [f(x) for x in list if...] >>> mary = 'Mary had a little lamb'.split() >>> mary ['Mary', 'had', 'a', 'little', 'lamb'] >>> [w for w in mary] ['Mary', 'had', 'a', 'little', 'lamb'] >>> [w for w in mary if len(w) >3] ['Mary', 'little', 'lamb'] >>> [w for w in mary if 'a' in w] ['Mary', 'had', 'a', 'lamb'] >>> [w.upper() for w in mary] ['MARY', 'HAD', 'A', 'LITTLE', 'LAMB'] >>> [len(w) for w in mary] [4, 3, 1, 6, 4] Same as mary Filter in only those elements that meet a condition Transform each element in list 9/27/

26 Beyond a single, short text So far, we have been handling relatively short texts, one at a time. Going multiple Find out what's involved in processing a text archive of multiple text files (aka corpus) Going big Find out what's involved in processing HUMONGUOUS text files Let's try this today 9/27/

27 Going big: considerations HOW big? Google Books Ngram files 1+ GB unzipped * 1000s Google 1T 5-gram data 24GB of compressed data COCA n-gram files MB unzipped Norvig 1,2-gram files 4-5MB King James Bible 4 MB Moby Dick 1.2 MB Alice in Wonderland 150 KB The Gift of the Magi 11KB The Gettysburg Address 1.4 KB Not for mere mortals. Might require some caution, but doable. Python on your personal computer should handle these with relative ease. 9/27/

28 IDLE shell is fragile Problems arise, however, with Python's IDLE shell. With Gettysburg Address or Fox in Sox, you can flash whatever objects (string, dictionary, 2-grams ) onto IDLE shell window. Most data objects are a handful of lines at best. With a larger text, an attempt to look up a data object could/will crash your IDLE shell window. >>> f = open('bible-kjv.txt') >>> bibletxt = f.read() >>> f.close() >>> print(bibletxt) DO NOT ATTEMPT 9/27/

29 Too big for your eyes This means you no longer have the luxury of being able to quickly examine your data objects in their entirety even as they get bigger and harder to rein in! We need to develop good strategies and habits when dealing with large texts/data. 9/27/

Our text: Carroll's Alice's Adventures Download the novel from the Project Gutenberg web site: http://www.gutenberg.org/ebooks/11 Download and save the "Plain Text UTF-8" file. Name it "alice.

30 Our text: Carroll's Alice's Adventures Download the novel from the Project Gutenberg web site: Download and save the "Plain Text UTF-8" file. Name it "alice.txt" Clean up the text file Open it up using a text editor. You will find: The file begins with a preamble The file ends with many (over 300!) lines of Project Gutenberg Legalese Remove both parts and save. 9/27/

31 Read in the text >>> f = open('alice.txt') >>> altxt = f.read() >>> f.close() >>> altxt Might need: encoding='utf-8' STOP!!! This sprays the entire text onto screen. It will likely freeze your IDLE shell. >>> altxt[:500] >>> altxt[10000:10500] Slice indexing is your friend. Look at first 500 chars, 500 chars in the middle, and last 500 characters. >>> print(altxt[-500:]) would feel with all their simple sorrows, and find a pleasure in all their simple joys, remembering her own child-life, and the happy summer days. THE END 9/27/

32 Tokenize the text >>> import textstats >>> altoks = textstats.gettokens(altxt) >>> >>> altoks DON'T press ENTER!! This is a LONG list. >>> altoks[100:150] ['well', 'as', 'she', 'could', ',', 'for', 'the', 'hot', 'day', 'made', 'her', 'feel', 'very', 'sleepy', 'and', 'stupid', ')', ',', 'whether', 'the', 'pleasure', 'of', 'making', 'a', 'daisy', '-', 'chain', 'would', 'be', 'worth', 'the', 'trouble', 'of', 'getting', 'up', 'and', 'picking', 'the', 'daisies', ',', 'when', 'suddenly', 'a', 'white', 'rabbit', 'with', 'pink', 'eyes', 'ran', 'close'] >>> Again, use slice indexing to look at a portion at a time. 9/27/

33 Explore the tokens How long is the text? How many times does 'rabbit' occur? Does 'curiouser' occur in the text? >>> len(altoks) >>> altoks.count('rabbit') 51 >>> 'curiouser' in altoks True 9/27/

34 Unique word types >>> altypes = textstats.gettypes(altxt) >>> >>> altypes Nope! >>> altypes[:30] ['!', '"', "'", '(', ')', '*', ',', '-', '--', '.', '0', '3', ':', ';', '?', '[', ']', '_', 'a', 'abide', 'able', 'about', 'above', 'absence', 'absurd', 'acceptance', 'accident', 'accidentally', 'account', 'accounting'] >>> Yep! Also: Ctrl+c to the rescue. 9/27/

35 Explore the types How many types? What's the typetoken ratio? >>> len(altypes) 2591 >>> len(altypes) / len(altoks) How many are 13+ characters long? >>> textstats.getxlengthwords(altypes, 13) ['affectionately', 'circumstances', 'contemptuously', 'conversations', 'disappointment', 'extraordinary', 'inquisitively', 'multiplication', 'straightening', 'uncomfortable', 'uncomfortably'] >>> [(w, len(w)) for w in altypes if len(w) >=13] [('affectionately', 14), ('circumstances', 13), ('contemptuously', 14), ('conversations', 13), ('disappointment', 14), ('extraordinary', 13), ('inquisitively', 13), ('multiplication', 14), ('straightening', 13), ('uncomfortable', 13), ('uncomfortably', 13)] List comprehension! coming right up 9/27/

36 Careful with long lists How many are 8 chars or longer? >>> textstats.getxlengthwords(altypes, 8) This is going to return a long list! Unless you're reasonably sure your list is short, assign the list to a new variable first >>> foo = textstats.getxlengthwords(altypes, 8) >>> len(foo) 627 >>> foo[:10] ['acceptance', 'accident', 'accidentally', 'accounting', 'accounts', 'accusation', 'accustomed', 'actually', 'addressed', 'addressing'] and then look at snippets using slice indexing 9/27/

37 Word frequencies >>> alfreq = textstats.gettypefreq(altxt) >>> alfreq['rabbit'] 51 >>> alfreq['the'] 1644 >>> alfreq['curiouser'] 2 >>> sorted(alfreq.keys())[:30] Does the same thing: returns sorted list of dictionary keys ['!', '"', "'", '(', ')', '*', ',', '-', '.', '0', '3', ':', ';', '?', '[', ']', '_', 'a', 'abide', 'able', 'about', 'above', 'absence', 'absurd', 'acceptance', 'accident', 'accidentally', 'account', 'accounting', 'accounts'] >>> sorted(alfreq)[:30] ['!', '"', "'", '(', ')', '*', ',', '-', '.', '0', '3', ':', ';', '?', '[', ']', '_', 'a', 'abide', 'able', 'about', 'above', 'absence', 'absurd', 'acceptance', 'accident', 'accidentally', 'account', 'accounting', 'accounts'] >>> alfreq['absurd'] 2 >>> 37

38 Explore the frequencies >>> for w in sorted(alfreq, key=alfreq.get, reverse=true)[:10]: print(w, alfreq[w]) ' 2871, 2418 the and 872 to a 632 it 595 she 553 >>> What are the top 10 most frequent words? 9/27/

39 Summary: handling large texts When reading in file: proceed by line or in chunks Mostly unnecessary for us, but good to know Careful with IDLE shell Do not flash a large object onto screen. Use len() and slicing [:20], [-20:], [1000:1020], etc. to look at parts of data. Utilize IDLE shell to understand your data As you build up your complex data objects through pipeline, poke around them at each stage to verify they are what you think they are. 9/27/

40 Public announcement Mac users: is this you? You installed Python wrong. Your IDLE will crash a lot. You should uninstall your Python, and then re-install Python that uses the correct version of Tcl/Tk. Follow instructions here: ll.html 9/27/

41 Wrapping up Next class: How to process external resource as formatted data files Download count_1w.txt and words.js from Norvig's site Homework #4 Bigram Speak List comprehension START EARLY. Midterm exam 10/11 (Thursday) At LMC's PC lab (CL G17) More room! 9/27/

Lab 14: Text & Corpus Processing with NLTK. Ling 1330/2330: Computational Linguistics Na-Rae Han

Lab 14: Text & Corpus Processing with NLTK Ling 1330/2330: Computational Linguistics Na-Rae Han Getting started with NLTK book NLTK Book, with navigation panel: http://www.pitt.edu/~naraehan/ling1330/nltk_book.html