At its simplest, language is just words. Imagine language as an urn filled with slips of paper. On each paper there is one word, and together these words constitute the lexicon of the language. Say the urn contains N words. We draw M word tokens from the urn, replacing them every time. At the end of this exercise, we have a text which is M words long. If this is our language model, what are the assumptions behind it?<\/p>\r\n
Basically, there are two assumptions in this model. The first of these is that every draw is independent<\/em>. This means that the word you have just drawn has a probability of [latex]\\frac{1}{N}[\/latex] of occurring, regardless of what precedes it. Of course, this is not a realistic assumption. In actual language, the first word of a sentence narrows the number of possibilities for the second one, which in turn reduces the number of choices for the third word, and so on and so forth. We will discuss this assumption at greater length in the next chapter.<\/p>\r\n The second assumption is that every word type is equally likely to be drawn<\/em>. Since each word has a [latex]\\frac{1}{N}[\/latex] probability of being drawn, it is equally likely that you will draw \u201cthe\u201d and \u201cTheremin\u201d. Again, you are quite right in thinking that, actually, not all words occur with the exact same frequency. What might be less obvious than the observation that different words are used with different frequencies is how frequent different words actually are. How likely, which is to say frequent, are frequent words? How frequent are rare words? How much more frequent are frequent words than rare words? One may also take a sociolinguistic perspective, and ask: who uses how many words? Actually, how many word types are there?<\/p>\r\n In what follows, you will see how the statistical methods from the last chapters can help to answer some of these questions.<\/p>\r\n \r\n\r\nTypes, tokens, rare and frequent words<\/strong>\r\n In order to explore the distribution of words in R, we need to load in a large corpus. For this chapter we use the ICE Great Britain. The file is structured as a long list with one word per line. In R terminology, we have a long vector of strings. Download the raw text file here:<\/p>\r\n[Dataset to come]\r\n\r\nSave it so you can find it again and load it into R:\r\n Now, we can restructure the corpus into a frequency table, using the If you look at the first few entries of the table, it doesn't look too informative yet. However, we can sort the table by descending frequency to see what the most frequent words are:<\/p>\r\n\r\n Now, of course, we can begin to visualize the results, to make them more easily interpretable. For instance, we can create a set of histograms:<\/p>\r\n\r\n [H5P slideshow: histograms]<\/p>\r\n [H5P: What type of distribution is this?]<\/p>\r\n We can also check for the number of different types in the ICE GB, using this sequence of commands:<\/p>\r\n\r\n [H5P: How would you find out the number of tokens in R?]<\/p>\r\n One obvious question that arises when looking at word frequencies is who uses how many words, or rather who uses how many different types? The measure which is used to evaluate vocabulary richness is known as the type per token ratio (TTR). The TTR is a helpful measure of lexical variety, and can for instance be used to monitor changes in children with vocabulary difficulties. It is calculated by dividing the number of types by the number of tokens in a passage of interest:<\/p>\r\n [latex]TTR=\\frac{types}{tokens}[\/latex]<\/p>\r\n Using this formula, we can calculate the TTR of the ICE GB:<\/p>\r\n\r\n Usually the TTR is very small, which is why it is often handier to invert the ratio and calculate the tokens per type:<\/p>\r\n [latex]TTR=\\frac{tokens}{types}[\/latex]<\/p>\r\n In this formulation, the TTR tells us how often, on average, each type is used. When we refer to the TTR henceforth, we talk about the token per type ratio.<\/p>\r\n For the ICE GB, we would then calculate:<\/p>\r\n\r\n This value indicates that, on average, each type occurs 12.6 times in the ICE GB .<\/p>\r\n As soon as we start to think more about this metric, we can see that it depends fundamentally on the sample size. The paragraph in which we introduce the TTR above \u2013 starting with \u201cOne obvious\u201d and ending with \u201cpassage of interest\u201d \u2013 consists of 82 tokens and 59 types. Accordingly, its TTR is [latex]\\frac{82}{59}=1.39[\/latex]. In this paragraph, each type occurs 1.39 times on average. Now, let us look at a very short sentence. For instance the preceding one. The TTR for \u201cnow let us look at a very short sentence\u201d is 1, because every token is of a unique type. This means that when we compare different samples with each other, as we do below, we want to take samples of a comparable size.<\/p>\r\n \r\n\r\nExercise: TTR<\/strong>\r\n You can try this yourself now. You may wonder whether written language has a richer vocabulary than spoken language, as measured by TTR.<\/p>\r\n [H5P: What is the null hypothesis to formulate here?]<\/p>\r\n [H5P: Calculate the TTR for the first 10\u2019000 words of the ICE GB written, which we used above.]<\/p>\r\n Now, download the ICE GB spoken from here:<\/p>\r\n[icegb_spoken.words]\r\n Then, load it into R.<\/p>\r\n [H5P: Calculate the TTR for the first 10\u2019000 words of the ICE GB spoken]<\/p>\r\n You can see that there is a pronounced difference between the spoken and written parts of the ICE GB corpus.<\/p>\r\n [H5P: Which significance test would you use to check whether the difference is statistically significant?]<\/p>\r\n With this basic principle, you have a solid basis for comparing the vocabulary richness of different genres, dialects or speakers. However, when it comes to producing fully fledged research, simply taking the first 10'000 words of two corpora is generally not advisable since the sequence of documents in a corpus can potentially skew the results. Instead, we recommend forming segments of equal length, for instance 2,000 words, calculating the TTR for each segment and then comparing the mean.<\/p>\r\n \r\n Zipf\u2019s law<\/strong><\/p>\r\n Another interesting aspect of word frequencies in corpora is known as Zipf\u2019s law. Named after the Harvard linguistics professor George Kingsley Zipf (1902-1950), it explains the type of distribution that we get when counting the word tokens in a large corpus. Zipf\u2019s law states that in a sorted frequency list of words the measure [latex]rank*frequency[\/latex] is nearly constant. In other words, there is an inversely proportional relationship between the rank of the type and the number of tokens this type contributes to the corpus.<\/p>\r\n We already sorted the written ICE GB corpus according to frequency above, so let\u2019s investigate how well Zipf\u2019s law holds up here. Remember that to view the contents of a table, we can use square brackets:<\/p>\r\n\r\n To only retrieve the frequency in a table like this, we can use two square brackets:<\/p>\r\n\r\n For the first ranked entry, this is of course already the value we need to test Zipf\u2019s law (because the frequency multiplied by 1 is equal to the frequency). For the remaining ranks, we can formulate a sequence of commands which calculates the rank*frequency:<\/p>\r\n\r\n There is a lot of variance here, but five values are clearly not enough for us to draw any conclusions yet. Say we want to see whether Zipf\u2019s law holds up over the 30, or 100, most frequent words. Instead of assigning values from 1 to 30 (or 100) manually, we can tell R to go through each of them, using a loop. Remember that loops work exactly like the indexes in mathematical sums (if not, you might want to take a quick look at the Introduction to R<\/a> again, where we discuss this for the first time).<\/p>\r\n In the loop, we first give the value 1 to the index variable i, then 2, and so on until we reach the last value of interest, 30 in this case. We save the results in the list variable titled zipfconstant, which we again index with i:<\/p>\r\n\r\n Now we have a vector, zipfconstant, which contains the Zipf contstant for the thirty most frequent words in the ICE GB. Even though these are only thirty values, it is rather hard to intuit anything about how constant they are, especially because the values are all very large. It might be easier to see patterns if we plot the values of a larger sample:<\/p>\r\n\r\n There are large jumps in the frequencies, which cause a lot of variation in the Zipf constant. However, this is only true for the first thirty or so values. After those, we see a strong regularity, with the Zipf constant ascending steeply. This indicates that, for the highest ranking values, the increase in rank outweighs the decrease in frequency. Then, the constant begins to earn it's name.<\/p>\r\n Let's see what happens when we extend the analysis further:<\/p>\r\n\r\n When we look at the Zipf constant for the 1,000 most frequent words, we see a similar pattern. We retain the high variance in the highest ranking words, and then see a more stable pattern, a mild upwards trend, starting at around 180 on the index axis. However, the regular pattern we saw starting at around 30 in graph 7.3 looks a lot wilder here. What's going on here? Well, we compress the y-axis by a factor of ten, i.e. we fit ten times more observations on the y-axis, but the x-axis stays the same. So the variance, the vertical fluctuation, stays within the same range but is compressed horizontally. Accordingly, it looks as though the fluctuations are more intense when we plot more observations in a graph of the same size.<\/p>\r\nLet's see what happens when we plot the Zipf constant for the entire ICE GB corpus. To do this, we can let our loop run through the length of the corpus:\r\n If we create a histogram of the Zipf constant, we get a slightly different idea of how regularly the Zipf constant is. We set the parameters for the histogram so that the y-axis shows the Zipf constants between 10,000 and 60,000 and that there are 100 breaks in the histogram:<\/p>\r\n\r\n The histogram shows us that most of the Zipf constants in the ICE GB are concentrated on the left side of the peak, which is at around 33,000. On the right side, we have fewer values but a somewhat longer tail. While neither the histogram nor the plot show the Zipf constant to be as consistent as the name implies, both the plot and the histogram clearly show is that there are certain regularities regarding the relation of rank and frequency. The regularities of word frequencies can help us to think about how language is structured.<\/p>\r\n \r\n Exercise: Test your intuition<\/strong><\/p>\r\n Remember that we can access a table column in R by its position:<\/p>\r\n\r\n We can also access it by the name of the row:<\/p>\r\n\r\n Now, compare for a few words if they are more frequent in written or in spoken language. You should already have the ICE GB spoken loaded into R from the TTR exercise above.<\/p>\r\n Process it into a sorted frequency table like we did with the ICE GB written earlier in the chapter. Do not forget to correct (at least roughly) for the fact that the spoken part of the ICE GB is bigger than the written part. For the correction, you can either cut the corpora into segments of equal lengths, taking for example the first 100\u2019000 words, or you can extrapolate from the frequencies you have in the written corpus and multiple them by 1.2, which is roughly equivalent to the difference in size between the two corpora.<\/p>\r\n Once you have both corpora available in this form, you can begin comparing word frequencies in spoken and written language. You can do so either by looking at the raw numbers, or more elegantly by writing a sequence of commands which results in a ratio:<\/p>\r\n\r\n Applications\r\n<\/strong><\/p>\r\n Although word frequencies are clearly a simple way of using statistics in linguistics, they can lead to insightful (and entertaining) analyses. For instance, Rayson et al. (1997) use word frequencies and chi-square tests in combination with sociolinguistic characteristics to investigate social differentiation in the use of English. The paper is more intent on delivering a proof of concept than on making any grand statements about how different speaker groups use language, but it is still interesting to note, for instance, that younger speakers do not only show \"a marked tendency in favour of certain taboo words\" but also a \"stronger tendency to use the polite words\" (Rayson et al. 1997: 140). We can recommend reading Rayson et al. (1997) for an impression of how word frequencies can be used in original research.<\/p>\r\nAfter this discussion of 1-grams and frequencies, we turn to more complex and elaborate language models in the next chapter, N-grams<\/a>.\r\n\r\n \r\n\r\nReference:<\/strong>\r\n Rayson, Paul and Leech, Geoffrey and Hodges, Mary. 1997. Social differentiation in the use of English vocabulary: some analyses of the conversational component of the British National Corpus. International Journal of Corpus Linguistics<\/em>, 2.1, 133\u2013152.<\/p>","rendered":" In this part, Language Models, we discuss how statistics can inform the way we model language. To put it very simplistically, language consists of words and of the interaction between words. In this chapter we focus on language at the level of words, and show how we can use the methods discussed in the preceding section can help us to evaluate word frequencies.<\/p>\n <\/p>\n The simplest language model: 1-gram<\/strong><\/p>\n At its simplest, language is just words. Imagine language as an urn filled with slips of paper. On each paper there is one word, and together these words constitute the lexicon of the language. Say the urn contains N words. We draw M word tokens from the urn, replacing them every time. At the end of this exercise, we have a text which is M words long. If this is our language model, what are the assumptions behind it?<\/p>\n Basically, there are two assumptions in this model. The first of these is that every draw is independent<\/em>. This means that the word you have just drawn has a probability of The second assumption is that every word type is equally likely to be drawn<\/em>. Since each word has a In what follows, you will see how the statistical methods from the last chapters can help to answer some of these questions.<\/p>\n <\/p>\n Types, tokens, rare and frequent words<\/strong><\/p>\n In order to explore the distribution of words in R, we need to load in a large corpus. For this chapter we use the ICE Great Britain. The file is structured as a long list with one word per line. In R terminology, we have a long vector of strings. Download the raw text file here:<\/p>\n [Dataset to come]<\/p>\n Save it so you can find it again and load it into R:<\/p>\n We can take a first look at our vector:<\/p>\n Now, we can restructure the corpus into a frequency table, using the If you look at the first few entries of the table, it doesn’t look too informative yet. However, we can sort the table by descending frequency to see what the most frequent words are:<\/p>\n And then take a look at the 100 most frequent words:<\/p>\n Now, of course, we can begin to visualize the results, to make them more easily interpretable. For instance, we can create a set of histograms:<\/p>\n [H5P slideshow: histograms]<\/p>\n [H5P: What type of distribution is this?]<\/p>\n We can also check for the number of different types in the ICE GB, using this sequence of commands:<\/p>\n [H5P: How would you find out the number of tokens in R?]<\/p>\n One obvious question that arises when looking at word frequencies is who uses how many words, or rather who uses how many different types? The measure which is used to evaluate vocabulary richness is known as the type per token ratio (TTR). The TTR is a helpful measure of lexical variety, and can for instance be used to monitor changes in children with vocabulary difficulties. It is calculated by dividing the number of types by the number of tokens in a passage of interest:<\/p>\n Using this formula, we can calculate the TTR of the ICE GB:<\/p>\n Usually the TTR is very small, which is why it is often handier to invert the ratio and calculate the tokens per type:<\/p>\n In this formulation, the TTR tells us how often, on average, each type is used. When we refer to the TTR henceforth, we talk about the token per type ratio.<\/p>\n For the ICE GB, we would then calculate:<\/p>\n This value indicates that, on average, each type occurs 12.6 times in the ICE GB .<\/p>\n As soon as we start to think more about this metric, we can see that it depends fundamentally on the sample size. The paragraph in which we introduce the TTR above \u2013 starting with \u201cOne obvious\u201d and ending with \u201cpassage of interest\u201d \u2013 consists of 82 tokens and 59 types. Accordingly, its TTR is <\/p>\n Exercise: TTR<\/strong><\/p>\n You can try this yourself now. You may wonder whether written language has a richer vocabulary than spoken language, as measured by TTR.<\/p>\n [H5P: What is the null hypothesis to formulate here?]<\/p>\n [H5P: Calculate the TTR for the first 10\u2019000 words of the ICE GB written, which we used above.]<\/p>\n Now, download the ICE GB spoken from here:<\/p>\n [icegb_spoken.words]<\/p>\n Then, load it into R.<\/p>\n [H5P: Calculate the TTR for the first 10\u2019000 words of the ICE GB spoken]<\/p>\n You can see that there is a pronounced difference between the spoken and written parts of the ICE GB corpus.<\/p>\n [H5P: Which significance test would you use to check whether the difference is statistically significant?]<\/p>\n With this basic principle, you have a solid basis for comparing the vocabulary richness of different genres, dialects or speakers. However, when it comes to producing fully fledged research, simply taking the first 10’000 words of two corpora is generally not advisable since the sequence of documents in a corpus can potentially skew the results. Instead, we recommend forming segments of equal length, for instance 2,000 words, calculating the TTR for each segment and then comparing the mean.<\/p>\n <\/p>\n Zipf\u2019s law<\/strong><\/p>\n Another interesting aspect of word frequencies in corpora is known as Zipf\u2019s law. Named after the Harvard linguistics professor George Kingsley Zipf (1902-1950), it explains the type of distribution that we get when counting the word tokens in a large corpus. Zipf\u2019s law states that in a sorted frequency list of words the measure We already sorted the written ICE GB corpus according to frequency above, so let\u2019s investigate how well Zipf\u2019s law holds up here. Remember that to view the contents of a table, we can use square brackets:<\/p>\n To only retrieve the frequency in a table like this, we can use two square brackets:<\/p>\n For the first ranked entry, this is of course already the value we need to test Zipf\u2019s law (because the frequency multiplied by 1 is equal to the frequency). For the remaining ranks, we can formulate a sequence of commands which calculates the rank*frequency:<\/p>\n By scrolling back up in the history in the R console, we can quickly generate a small sample of these values:<\/p>\n There is a lot of variance here, but five values are clearly not enough for us to draw any conclusions yet. Say we want to see whether Zipf\u2019s law holds up over the 30, or 100, most frequent words. Instead of assigning values from 1 to 30 (or 100) manually, we can tell R to go through each of them, using a loop. Remember that loops work exactly like the indexes in mathematical sums (if not, you might want to take a quick look at the Introduction to R<\/a> again, where we discuss this for the first time).<\/p>\n In the loop, we first give the value 1 to the index variable i, then 2, and so on until we reach the last value of interest, 30 in this case. We save the results in the list variable titled zipfconstant, which we again index with i:<\/p>\n Now we have a vector, zipfconstant, which contains the Zipf contstant for the thirty most frequent words in the ICE GB. Even though these are only thirty values, it is rather hard to intuit anything about how constant they are, especially because the values are all very large. It might be easier to see patterns if we plot the values of a larger sample:<\/p>\n How can we interpret this? The plot shows us that there is a lot of variance in the Zipf constant of the highest-ranking words. This makes a lot sense, when we consider that the difference between these frequencies vary strongly. Let us calculate the Zipf constant for the five most frequent words:<\/p>\n There are large jumps in the frequencies, which cause a lot of variation in the Zipf constant. However, this is only true for the first thirty or so values. After those, we see a strong regularity, with the Zipf constant ascending steeply. This indicates that, for the highest ranking values, the increase in rank outweighs the decrease in frequency. Then, the constant begins to earn it’s name.<\/p>\n Let’s see what happens when we extend the analysis further:<\/p>\n When we look at the Zipf constant for the 1,000 most frequent words, we see a similar pattern. We retain the high variance in the highest ranking words, and then see a more stable pattern, a mild upwards trend, starting at around 180 on the index axis. However, the regular pattern we saw starting at around 30 in graph 7.3 looks a lot wilder here. What’s going on here? Well, we compress the y-axis by a factor of ten, i.e. we fit ten times more observations on the y-axis, but the x-axis stays the same. So the variance, the vertical fluctuation, stays within the same range but is compressed horizontally. Accordingly, it looks as though the fluctuations are more intense when we plot more observations in a graph of the same size.<\/p>\n Let’s see what happens when we plot the Zipf constant for the entire ICE GB corpus. To do this, we can let our loop run through the length of the corpus:<\/p>\n> icegb <- scan(file.choose(),sep=\"\\n\", what=\"raw\")<\/code><\/pre>\r\nWe can take a first look at our vector:\r\n> icegb[1:20]<\/code><\/pre>\r\n[caption id=\"attachment_428\" align=\"aligncenter\" width=\"451\"]![\"\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/uploads\/sites\/23\/2019\/08\/1.-Head-ICE-GB.jpg\")
Figure 7.1: The first twenty words in ICE GB[\/caption]\r\ntable()<\/code> command we introduced in earlier:<\/p>\r\n\r\n> ticegb = table(icegb)<\/code><\/pre>\r\n> sortticegb = sort(ticegb,decreasing=T)<\/code><\/pre>\r\nAnd then take a look at the 100 most frequent words:\r\n> sortticegb[1:100]<\/code><\/pre>\r\n[caption id=\"attachment_429\" align=\"aligncenter\" width=\"976\"]![\"\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/uploads\/sites\/23\/2019\/08\/2.-Most-frequent-words.jpg\")
Figure 7.2: The 100 most frequent woords in ICE GB[\/caption]\r\n> hist(as.vector(sortticegb[1:100]),breaks=100)\r\n> hist(as.vector(sortticegb[1:1000]),breaks=1000)\r\n> hist(as.vector(sortticegb[1:10000]),breaks=10000)<\/code><\/pre>\r\n> types = length(as.vector(sortticegb)) ; types\r\n[1] 33816<\/code><\/pre>\r\n> types\/tokens\r\n[1] 0.07929615<\/code><\/pre>\r\n> TTR = tokens \/ types ; TTR\r\n[1] 12.61095<\/code><\/pre>\r\n> sortticegb[1]\r\nthe\r\n24614<\/code><\/pre>\r\n> sortticegb[[1]]\r\n[1] 24614<\/code><\/pre>\r\n> myrow = 3; freq = sortticegb[[myrow]] ; zipfconstant = freq * myrow ; zipfconstant\r\n[1] 35463<\/code><\/pre>\r\nBy scrolling back up in the history in the R console, we can quickly generate a small sample of these values:\r\n> myrow = 5; freq = sortticegb[[myrow]] ; zipfconstant = freq * myrow ; zipfconstant\r\n[1] 43055\r\n> myrow = 10; freq = sortticegb[[myrow]] ; zipfconstant = freq * myrow ; zipfconstant\r\n[1] 35060\r\n> myrow = 20; freq = sortticegb[[myrow]] ; zipfconstant = freq * myrow ; zipfconstant\r\n[1] 45740<\/code><\/pre>\r\n> for (i in 1:30) { freq = sortticegb[[i]] ; zipfconstant[i] = freq * i}\r\n> zipfconstant\r\n\u00a0[1] 24614 27778 35463 42460 43055 46080 37436 32904 35496 35060 34738\u00a0\r\n\u00a0\u00a0\u00a0\u00a0 35808 \u00a0\u00a037791 40474 42270 44576 44727 46674 45315 45740 47649 47014\r\n[23] 45356 47256 48675 50622 51435 52276 53505 49980<\/code><\/pre>\r\n> for (i in 1:100) { freq = sortticegb[[i]] ; zipfconstant[i] = freq * i}\r\n> plot(zipfconstant)<\/code><\/pre>\r\n[caption id=\"attachment_430\" align=\"aligncenter\" width=\"677\"]![\"\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/uploads\/sites\/23\/2019\/08\/4.-Plot-Zipf.jpeg\")
Figure 7.3: The Zipf constant for the 100 most frequent words in ICE GB[\/caption]\r\n\r\nHow can we interpret this? The plot shows us that there is a lot of variance in the Zipf constant of the highest-ranking words. This makes a lot sense, when we consider that the difference between these frequencies vary strongly. Let us calculate the Zipf constant for the five most frequent words:\r\n\r\n \t
> for (i in 1:1000) { freq = sortticegb[[i]] ; zipfconstant[i] = freq * i}\r\n> plot(zipfconstant)<\/code><\/pre>\r\n[caption id=\"attachment_431\" align=\"aligncenter\" width=\"677\"]![\"\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/uploads\/sites\/23\/2019\/08\/5.-Zipfplot1000.jpeg\")
Figure 7.4: The Zipf constant for the 1000 most frequent words[\/caption]\r\n> for (i in 1:length(sortticegb)) { freq = sortticegb[[i]] ; zipfconstant[i] = freq * i}\r\n> plot(zipfconstant)<\/code><\/pre>\r\n[caption id=\"attachment_432\" align=\"aligncenter\" width=\"677\"]![\"\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/uploads\/sites\/23\/2019\/08\/6.-Zipfplot-complete.jpeg\")
Figure 7.5: The Zipf constant for the complete ICE GB corpus[\/caption]\r\n\r\n[h5p id=\"30\"]\r\n> hist(zipfconstant,xlim=c(20000,60000),breaks=100)<\/code><\/pre>\r\n[caption id=\"attachment_433\" align=\"aligncenter\" width=\"677\"]![\"\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/uploads\/sites\/23\/2019\/08\/7.-Zipfstogram.jpeg\")
Figure 7.6: Histogram of the Zipf constant for the complete ICE GB corpus[\/caption]\r\n\r\n[h5p id=\"29\"]\r\n> sortticegb[2]\r\n\u00a0\u00a0 of\r\n13889<\/code><\/pre>\r\n> sortticegb['of']\r\n\u00a0\u00a0 of\r\n13889<\/code><\/pre>\r\n> myrow = 'daresay' ; factor[myrow] = ((sortticegb[[myrow]] * 1.2)\/ sortticegbspoken[[myrow]]) ; factor[myrow]\r\ndaresay\r\n\u00a0\u00a0 0.6<\/code><\/pre>\r\n \r\n![\"\frac{1}{N}\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/ql-cache\/quicklatex.com-db37feb1038a35bd0606d647338571b2_l3.png\" "\\"Rendered")
of occurring, regardless of what precedes it. Of course, this is not a realistic assumption. In actual language, the first word of a sentence narrows the number of possibilities for the second one, which in turn reduces the number of choices for the third word, and so on and so forth. We will discuss this assumption at greater length in the next chapter.<\/p>\n probability of being drawn, it is equally likely that you will draw \u201cthe\u201d and \u201cTheremin\u201d. Again, you are quite right in thinking that, actually, not all words occur with the exact same frequency. What might be less obvious than the observation that different words are used with different frequencies is how frequent different words actually are. How likely, which is to say frequent, are frequent words? How frequent are rare words? How much more frequent are frequent words than rare words? One may also take a sociolinguistic perspective, and ask: who uses how many words? Actually, how many word types are there?<\/p>\n> icegb <- scan(file.choose(),sep=\"\\n\", what=\"raw\")<\/code><\/pre>\n> icegb[1:20]<\/code><\/pre>\ntable()<\/code> command we introduced in earlier:<\/p>\n> ticegb = table(icegb)<\/code><\/pre>\n> sortticegb = sort(ticegb,decreasing=T)<\/code><\/pre>\n> sortticegb[1:100]<\/code><\/pre>\n> hist(as.vector(sortticegb[1:100]),breaks=100)\r\n> hist(as.vector(sortticegb[1:1000]),breaks=1000)\r\n> hist(as.vector(sortticegb[1:10000]),breaks=10000)<\/code><\/pre>\n> types = length(as.vector(sortticegb)) ; types\r\n[1] 33816<\/code><\/pre>\n![\"TTR=\frac{types}{tokens}\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/ql-cache\/quicklatex.com-5c87f84f2f60522be4992d6df903401c_l3.png\" "\\"Rendered")
<\/p>\n> types\/tokens\r\n[1] 0.07929615<\/code><\/pre>\n![\"TTR=\frac{tokens}{types}\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/ql-cache\/quicklatex.com-794d2f05f5712698cceba91cd8f729c4_l3.png\" "\\"Rendered")
<\/p>\n> TTR = tokens \/ types ; TTR\r\n[1] 12.61095<\/code><\/pre>\n![\"\frac{82}{59}=1.39\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/ql-cache\/quicklatex.com-eccfcc2c6c5cbd82ec25a5d4273b032d_l3.png\" "\\"Rendered")
. In this paragraph, each type occurs 1.39 times on average. Now, let us look at a very short sentence. For instance the preceding one. The TTR for \u201cnow let us look at a very short sentence\u201d is 1, because every token is of a unique type. This means that when we compare different samples with each other, as we do below, we want to take samples of a comparable size.<\/p>\n![\"rank*frequency\"](\"https:\/\/dlf.uzh.ch\/openbooks\/statisticsforlinguists\/wp-content\/ql-cache\/quicklatex.com-b3969e31d4e7d8e9004dcd5e432ed333_l3.png\" "\\"Rendered")
is nearly constant. In other words, there is an inversely proportional relationship between the rank of the type and the number of tokens this type contributes to the corpus.<\/p>\n> sortticegb[1]\r\nthe\r\n24614<\/code><\/pre>\n> sortticegb[[1]]\r\n[1] 24614<\/code><\/pre>\n> myrow = 3; freq = sortticegb[[myrow]] ; zipfconstant = freq * myrow ; zipfconstant\r\n[1] 35463<\/code><\/pre>\n> myrow = 5; freq = sortticegb[[myrow]] ; zipfconstant = freq * myrow ; zipfconstant\r\n[1] 43055\r\n> myrow = 10; freq = sortticegb[[myrow]] ; zipfconstant = freq * myrow ; zipfconstant\r\n[1] 35060\r\n> myrow = 20; freq = sortticegb[[myrow]] ; zipfconstant = freq * myrow ; zipfconstant\r\n[1] 45740<\/code><\/pre>\n> for (i in 1:30) { freq = sortticegb[[i]] ; zipfconstant[i] = freq * i}\r\n> zipfconstant\r\n\u00a0[1] 24614 27778 35463 42460 43055 46080 37436 32904 35496 35060 34738\u00a0\r\n\u00a0\u00a0\u00a0\u00a0 35808 \u00a0\u00a037791 40474 42270 44576 44727 46674 45315 45740 47649 47014\r\n[23] 45356 47256 48675 50622 51435 52276 53505 49980<\/code><\/pre>\n> for (i in 1:100) { freq = sortticegb[[i]] ; zipfconstant[i] = freq * i}\r\n> plot(zipfconstant)<\/code><\/pre>\n\n
> for (i in 1:1000) { freq = sortticegb[[i]] ; zipfconstant[i] = freq * i}\r\n> plot(zipfconstant)<\/code><\/pre>\n> for (i in 1:length(sortticegb)) { freq = sortticegb[[i]] ; zipfconstant[i] = freq * i}\r\n> plot(zipfconstant)<\/code><\/pre>\n