In the preceding chapter, we had the example of the urn which contains N word types, N being the size of the lexicon. We drew M word tokens from the urn, and replaced the word type we drew each time. M is the length of our text. The two values N and M were enough to characterize a simple 1-gram language model.<\/p>\r\n[h5p id=\"31\"]\r\n
Obviously, these assumptions render the model very unrealistic. In the last chapter we saw in the discussion of Zipf's law that different words have extremely different probabilities of being used in language production. Now let us think about the other assumption. To improve our language model, we have to get away from the assumption that draws are independent. This is necessary because we know that word order matters. One might think of a far-fetched example, indeed one from a galaxy far, far away. Yoda's speech is instantly recognizable as his on account of it's unusual subject-object-verb structure. English sentences are most frequently structured along subject-verb-object. This means that if we know the first word of a sentence, we can assign a higher probability to having a verb in the second place than having an object. To account for this in our language model, we can use conditional probabilities, as they are known in statistics. Let\u2019s look at the simplest of the conditional probability models, the bigram language model.<\/p>\r\n
The bigram language model assumes a dependency structure where the probability of a word occurring depends on the previous word. Formally, we can express this as:<\/p>\r\n
[latex]\\text{Conditional probability in a bigram model}=p(word | previous word)[\/latex]<\/p>\r\n[Exercise to come]\r\n\r\n \r\n
The improved monkey at the typewriter<\/strong><\/p>\r\n (Or, Shakespeare in love with random text generation)<\/p>\r\n How much of a difference there is between the 1-gram and more advanced n-gram models becomes apparent when looking at some examples. Trained on a healthy diet of Shakespeare, you can see how well the different models fare.<\/p>\r\n The 1-gram model, with independent draws and equal word probabilities [latex]p(word)[\/latex] looks like this:<\/p>\r\n\r\n The bigram model, with two-word sequences [latex]p(word | previous word)[\/latex], looks only vaguely better:<\/p>\r\n\r\n In contrast, the trigram model, with three-word sequences [latex]p(word | word-1, word-2)[\/latex], is no longer as bad:<\/p>\r\n\r\n But by no means does the trigram model hold the torch to the quadrigram model. Although we probably still wouldn't pay money to sit through a play generated like this, the four-word sequences of [latex]p(word | word-1, word-2, word-3)[\/latex] begins to approximate meaningful language:<\/p>\r\n\r\n These examples of n-gram models with varying degrees of complexity are often cited creations taken from Jurafsky and Martin (2009).<\/p>\r\n In order to understand the progression of these models in more detail, we need a bit of linguistic theory. Chomsky says:<\/p>\r\n Conversely, a descriptively adequate grammar is not quite equivalent to nondistinctness in the sense of distinctive feature theory. It appears that the speaker-hearer's linguistic intuition raises serious doubts about an important distinction in language use. To characterize a linguistic level L, the notion of level of grammaticalness is, apparently, determined by a parasitic gap construction. A consequence of the approach just outlined is that the theory of syntactic features developed earlier is to be regarded as irrelevant intervening contexts in selectional rules. In the discussion of resumptive pronouns following (81), the earlier discussion of deviance is rather different from the requirement that branching is not tolerated within the dominance scope of a complex symbol.<\/em><\/p>\r\n In case this does not answer your questions fully, we recommend you read the original source: http:\/\/www.rubberducky.org\/cgi-bin\/chomsky.pl<\/a><\/p>\r\n Does this leave you quite speechless? Well, it certainly left Chomsky speechless. Or, at least, he didn't say these words. In a sense, though, they are of his creation: the Chomskybot is a phrase-based language model trained on (some of the linguistic) works of Noam Chomsky.<\/p>\r\n If you prefer your prose less dense, we can also recommend the automatically generated Harry Potter and the Portrait of What Looked Like a Large Pile of Ash<\/em>, which you can find here: https:\/\/botnik.org\/content\/harry-potter.html<\/a><\/p>\r\n You can see that different language models produce outputs with varying degrees of realism. While these language generators are good fun, there are many other areas of language processing which also make use of n-gram models:<\/p>\r\n\r\n Collocations<\/strong><\/p>\r\n Take words like strong <\/em>and powerful<\/em>. They are near synonyms, and yet strong tea <\/em>is more likely than powerful tea<\/em>, and powerful car<\/em> is more likely than strong car<\/em>. We will informally use the term collocation<\/em> to refer to pairs of words which have a tendency to \u201cstick together\u201d and discuss how we can use statistical methods to answer these questions:<\/p>\r\n\r\n But first, let\u2019s take a look at the linguistic background of collocations. Choueka (1988) defines collocations as \u201ca sequence of two or more consecutive words, that has characteristics of a syntactic and semantic unit, and whose exact and unambiguous meaning or connotation cannot be derived directly from the meaning or connotation of its components\u201d. Some criteria that distinguish collocations from other bigrams are:<\/p>\r\n\r\n However, these criteria should be taken with a grain of salt. For one, the criteria are all quite vague, and there is no clear agreement as to what can be termed a collocation. Is the defining feature the grammatical relation (adjective + noun or verb + subject) or is it the proximity of the two words? Moreover, there is no sharp line demarcating collocations from phrasal verbs, idioms, named entities and technical terminology composed of multiple words. And what are we to make of the co-occurrence of thematically related words (e.g. doctor and nurse, or plane and airport)? Or the co-occurrence of words in different languages across parallel corpora? Obviously, there is a cline going from free combination through collocations to idioms. Between these different elements of language there is an area of gradience which makes collocations particularly interesting.<\/p>\r\n \r\n Statistical collocation measures<\/strong><\/p>\r\n With raw frequencies, z-scores, t-scores and chi-square tests we have at our disposal an arsenal of statistical methods we could use to measure collocation strength. In this section we will discuss how to use raw frequencies and chi-square tests to this end, and additionally we will introduce three new measures: Mutual Information (MI), Observed\/Expected (O\/E) and log-likelihood. In general, these measures are used to rank pair types as candidates for collocations. The different measures have different strengths and weaknesses, which means that they can yield a variety of results. Moreover, the association scores computed by different measures cannot be compared directly.<\/p>\r\n Frequencies<\/strong><\/p>\r\n Frequencies can be used to find collocations by counting the number of occurrences. For this, we typically use word bigrams. Usually, these searches result in a lot of function word pairs that need to be filtered out. Take a look at these collocations from an example corpus:<\/p>\r\n\r\n\r\n[caption id=\"attachment_446\" align=\"aligncenter\" width=\"225\"] Except for New York<\/em>, all the bigrams are pairs of function words. This is neither particularly surprising, nor particularly useful. If you have access to a POS-tagged corpus, or know how to apply POS-tags to a corpus, you can improve the results a bit by only searching for bigrams with certain POS-tag sequences (e.g. noun-noun, adj-noun).<\/p>\r\n Despite it being an imperfect measure, let\u2019s look at how we can identify raw frequencies for bigrams in R. For this, download the ICE GB written corpus in a bigram version:<\/p>\r\n [file: ice gb written bigram]<\/p>\r\n Import it into R using your preferred command. We assigned the corpus to a variable called icegbbi<\/em>, and so we open the first ten entries of the vector:<\/p>\r\n\r\n We can see that each position contains two words separated by a tab, which is what the \\t<\/em> stands for. Transform the vector into a table, sort it in decreasing order and take a look at the highest ranking entries:<\/p>\r\n\r\n The results in the ICE GB written are very similar to the results in the table above. In fact, among the 16 most frequent bigrams, there are only pairs of function words.<\/p>\r\n [H5P: Search the frequency table some more. What is the most frequent bigram which you could classify as a collocation?]<\/p>\r\n \r\n Mutual Information and Observed\/Expected<\/strong><\/p>\r\n Fortunately, there are better collocation measures than raw frequencies. One of these is the Mutual Information (MI) score. We can easily calculate the probability that a collocation composed of the words x and y occurs due to chance. To do so, we simply need to assume that they are independent events. Under this assumption, we can calculate the individual probabilities from the raw frequencies f(x) and f(y) and the corpus size N:<\/p>\r\n [latex]p(x)=\\frac{f(x)}{N}[\/latex] and [latex]p(y)=\\frac{f(y)}{N}[\/latex]<\/p>\r\n Of course, we can also calculate the observed probability of finding x and y together. We do this joint probability by dividing number of times we find x and y in sequence, f(x,y), by the corpus size N:<\/p>\r\n [latex]p(x,y)=\\frac{f(x,y)}{N}[\/latex]<\/p>\r\n If the collocation of x and y is due to chance, which is to say if x and y are independent, the observed probability will be roughly equal to the expected probability:<\/p>\r\n [latex]p(x,y){\\approx}p(x)*p(y)[\/latex]<\/p>\r\n If the collocation is not due to chance, we have one of two cases. Either the observed probability is substantially higher than the expected probability, in which we have a strong correlation:<\/p>\r\n [latex]p(x,y)>>p(x)*p(y)[\/latex]<\/p>\r\n Or the inverse is true, and the observed probability is much lower than the expected one:<\/p>\r\n [latex]p(x,y)<<p(x)*p(y)[\/latex]<\/p>\r\n The latter scenario can be referred to as \u2018negative\u2019 collocation, which simply means that the two words hardly occur together.<\/p>\r\n The comparison of joint and independent probabilities is known as Mutual Information (MI) and originates in Information Theory => surprise in bits. What we outlined above is a basic version of MI which we call O\/E, and which can be simply calculated by dividing Observation by Expectation:<\/p>\r\n [latex]O\/E=\\frac{p(x,y)}{p(x)*p(y)}[\/latex]<\/p>\r\n After this introduction to the concept, let\u2019s take a look at how to implement the O\/E in R.<\/p>\r\n [R exercise with available corpus?]<\/p>\r\n Okay, let's look at one more set of examples, this time with data from the British National Corpus (BNC). The BNC contains 97,626,093 words in total. The table below shows the frequencies for the words 'powerful', 'strong', 'car' and 'tea', as well as all of the bigrams composed of these four word:<\/p>\r\n\r\n\r\n \t
\r\n \t
\r\n \t
\r\n \t
\r\n \t
\r\n \t
\r\n \t
\r\n \t
\r\n \t
\r\n \t
\r\n \t
\r\n \t
![\"\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/uploads\/sites\/23\/2019\/08\/1.-frequencies.jpg\")
Figure 8.1: Raw collocation frequencies[\/caption]\r\nicegbbi[1:10]\r\n[1] \"Tom\\tRodwell\"\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"Rodwell\\tWith\"\r\n[3] \"With\\tthese\"\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"these\\tprofound\"\r\n[5] \"profound\\twords\"\u00a0\u00a0\u00a0\u00a0\u00a0 \"words\\tJ.\"\r\n[7] \"J.\\tN.\"\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"N.\\tL.\"\r\n[9] \"L.\\tMyres\"\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"Myres\\tbegins\"\r\n<\/code><\/pre>\r\n> ticegbbi = table(icegbbi)\r\n> sortticegbbi = as.table(sort(ticegbbi,decreasing=T))\r\n> sortticegbbi[1:16]\r\nicegbbi\r\n\u00a0 of\\tthe\u00a0\u00a0\u00a0 in\\tthe\u00a0\u00a0\u00a0 to\\tthe\u00a0\u00a0\u00a0\u00a0 to\\tbe\u00a0\u00a0 and\\tthe\u00a0\u00a0\u00a0 on\\tthe\u00a0\u00a0 for\\tthe\u00a0 from\\tthe\r\n\u00a0\u00a0\u00a0\u00a0 3623\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 2126\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1341\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 908\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 881\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 872\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 696\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 627\r\n\u00a0 by\\tthe\u00a0\u00a0\u00a0 at\\tthe\u00a0 with\\tthe\u00a0\u00a0\u00a0\u00a0\u00a0 of\\ta\u00a0 that\\tthe\u00a0\u00a0\u00a0\u00a0 it\\tis\u00a0\u00a0 will\\tbe\u00a0\u00a0\u00a0\u00a0\u00a0 is\\ta\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 602\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 597\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 576\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 571\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 559\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 542\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 442\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 420\r\n<\/code><\/pre>\r\n
![\"\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/uploads\/sites\/23\/2019\/08\/2.-Powerful-tea.jpg\")
Figure 8.2: Powerful tea in the British National Corpus[\/caption]\r\n![\"\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/uploads\/sites\/23\/2019\/08\/3.-New-companies.jpg\")
Figure 8.3: 'New' and 'companies' in all possible and sensible bigram combinations[\/caption]\r\n