1.1 . Install Java

Download and install, if you don’t already have in on your computer, the Java Development Kit. No specific things to look out for during installation.

1.2 . Install cleanNLP and language model

The packages we need in R are rJava and cleanNLP. Install the developmental version of cleanNLP as using the (old) CRAN version won’t work properly.

install.packages("rJava")
devtools::install_github("statsmaths/cleanNLP")

library(cleanNLP)
library(dplyr)

After loading the package, you can pass an argument to download different language models. The default is set to English so I’m not going to pass anything to the function.

cnlp_download_corenlp()

This will take some time.

3.1 . With tidytext

tidytext is a convenient means to perform text analysis. package. Luckily, free resources are available such as Tidytext that will serve as a structured, useful guide.

library(tidytext)

This package includes some functions that are easy to use. We used the corenlp to POS tag the text but if we didn’t need that, we could have just tokenized using the unnest_tokens() function as I have done in the previous post.

unnest_token() first takes the data frame (txt_cor). The default setting breaks the text into words (i.e., tokenizes) and creates a new data frame. We need to provide the name of the column for this new data frame (output, I named it word) and the column that includes the text data (input, which is text).

tidy_tok <- txt_cor %>% unnest_tokens(word, text)

Analyzing n-grams is done with the same function. We can just provide different values to generate a table of n-grams. The first following code takes the corpus and creates a new data frame (tidy_bi) with the column bigram that contains the bigram. token = "ngrams" and n =2 will extract two-word sequences. The second code will create a list of trigrams.

tidy_bi <- txt_cor %>% unnest_tokens(bigram, text, token = "ngrams", n = 2)
tidy_tri <- txt_cor %>% unnest_tokens(trigram, text, token = "ngrams", n = 3)

Coming from a linguistics perspective, I find it potentially problematic that the bigrams include word chunks that are not meaningful, especially for qualitative text analysis. What I mean is that, for example, the last word of sentence #1 and the first word of sentence #2, “it dramatically”, are treated as a bigram. Same applies to words within a sentence that are separated by commas or other punctuation. Consider the first couple sentences from our corpus:

“Two men, one ring, only one can leave. Dramatic it may be but…”

“men one” is not meaningful, “ring only” is not meaningful. Punctuation serves specific purposes in writing, and ignoring them might fail to deliver meaningful results. Crossing such borders can also lead to misleading results. The meaningful, uninterrupted n-grams are called “CollGrams” by some researchers (Bestgen & Granger, 2014)¹.

3.2 . Manually Creating Bigrams and Trigrams

For this reason, I’ll go back to the annotated data we created earlier. To inspect sequences of words, we can use the lead() function from the dplyr package, to create new columns that contain information regarding the next row of each word.

txt_df <- txt_tab %>% 
  mutate(second_word = lead(word), second_upos = lead(upos), second_pos = lead(pos), 
         third_word = lead(word, 2), third_upos = lead(upos, 2), third_pos = lead(pos, 2))

In the newly created columns in light green are information pertaining to the next word, and the columns in light blue are that of the second next word.

To clean this data, we’ll execute the following code. The unite() function from the tidyr package concatenates the word and second_word to show the biagram. Although where punctuation occurs can be of interest itself (e.g., marking clauses, inserting phrases, etc.), in this post I’ll filter out the bigrams that include any punctuation marks to only consider two- or three-word sequences that co-occur without any interruption.

library(tidyr)

txt_bi <- txt_df %>% unite(bigram, word, second_word, sep = " ") %>% 
  filter(!second_upos == ".", !upos == ".") %>% select(1, 4:9)

txt_tri <- txt_df %>% unite(trigram, word, second_word, third_word, sep = " ") %>% 
  filter(!third_upos == ".", !second_upos == ".", !upos == ".") %>% select(1, 4:11)

3.3 . Example Analysis: Be + words

What’s the most common part of speech that comes after the “be” verb? What does it say about the role of the “be” verb and the constituent that follows?

txt_bi %>% filter(lemma == "be") %>% count(second_upos, sort = TRUE)

Largely, “be” is most frequently followed by another verb. Looking at the POS tag reveals a bit more information.

txt_bi %>% filter(lemma == "be") %>% count(second_pos, sort = TRUE)

The “be” verb most frequently co-occurs with another verb in the past participle form (i.e., VBN), so presumably the 34 occurrences are passive constructions, in which “be” serves as an auxiliary.

It’s almost certain that a determiner starts a noun phrase, thus in 27 cases the “be” verb is a main verb and is followed by a noun phrase complement.

txt_bi %>% filter(lemma == "be", second_pos == "VBN") %>% select(bigram)

## # A tibble: 34 x 1
##    bigram        
##    <chr>         
##  1 been made     
##  2 be banned     
##  3 is argued     
##  4 been won      
##  5 be banned     
##  6 was put       
##  7 be prepared   
##  8 are surrounded
##  9 are trained   
## 10 are paid      
## # ... with 24 more rows

The third most frequently co-occuring tag is the adverb. The question is then, what follows an adverb? Considering that verb past participle is the category that appears most frequently after “be”, it could be that an adverb is inserted between these two verbs (be + adverb + past participle; e.g., is actually made).

It is also possible that the adverb is part of an adjective phrase (e.g., is really important), which in turn, may or may not constitute a noun phrase (e.g., is really an important ). Let’s dig a little deeper by looking at the trigrams.

txt_tri %>% 
    filter(lemma == "be", second_pos == "RB") %>% 
    count(third_pos, sort = TRUE) %>% 
    mutate(percent = round(n*100/sum(n), 1))

## # A tibble: 10 x 3
##    third_pos     n percent
##    <chr>     <int>   <dbl>
##  1 JJ            9    34.6
##  2 DT            4    15.4
##  3 RB            4    15.4
##  4 VBG           2     7.7
##  5 VBN           2     7.7
##  6 CD            1     3.8
##  7 IN            1     3.8
##  8 RBR           1     3.8
##  9 TO            1     3.8
## 10 WRB           1     3.8

It appears that many of the “be + adverb” sequences (34.6%) are followed by an adjective, such as:

txt_tri %>% 
  filter(lemma == "be", second_pos == "RB", third_pos == "JJ") %>% 
  select(trigram)

## # A tibble: 9 x 1
##   trigram             
##   <chr>               
## 1 are already few     
## 2 were not stupid     
## 3 is not right        
## 4 is very popular     
## 5 is always much      
## 6 is clearly aware    
## 7 is hardly surprising
## 8 is very likely      
## 9 be very severe

However, to determine the exact structure, we need to go further. We can look at the sentence where each trigram occurs.

As a very rough summary, we can say that out of 127 cases where the verb “be” was used, 40 were auxiliary be, consisting passive voice constructions and progressive forms. In other times, the verb was frequently followed by a noun phrase (at least 31 times) or an adjective phrase that may or may not be embedded in a noun phrase.