Web Scraping with R: A Great Resource for Language Learning and Teaching

1 . Introduction

Recently, I helped a colleague scrape text from Wikipedia for a class project. If I didn’t know any web scraping technique, I would have copied and pasted however many number of Wiki entries I needed. This would have required going through at least four steps for each entry: 1) searching, 2) copying the content, 3) pasting it into a word processor, and 4) removing unwanted parts such as pictures. If I did this more than 5 times, I would already start feeling tired.

Web scraping can provide language data that can be used for a myriad of purposes in this world. For second language learning and teaching, it can be used to create small corpora from which we can generate lists of vocabulary, collocations, and/or lexical bundles. Such data would be particularly useful for English for Specific Purposes (ESP), English for Academic Purposes (EAP), and genre-based approaches.

In this post, I introduce how to scrape texts from the web along with (very skimpy) ideas for teaching. I have three examples. The first one is the longest because I explain how to inspect HTML information on websites.

1.1 . Tools Needed

The package required for web scraping is rvest. I’ll also use the Google Chrome web browser to view page sources and inspect elements. However, Firefox also has this feature.

2 . Example 1: Scraping Webpages

2.1 . Wikipedia entries

I was helping with scraping Wikipedia entries of American actors and actresses, with which my colleague ultimatley aimed to create a corpus and extract lexical bundles. To collect a decent amount of language that is necessary for generating a list of lexical bundles, we scraped 210 articles which resulted in about a million word corpus.

First, I asked my colleague to make a list of actors that he wanted, because he only wanted well-known actors to ensure their wiki page has enough language. With Wikipedia, the web address consists of one fixed part, “https://en.wikipedia.org/wiki/” and another part which is the name or title of the person/thing/topic. Therefore, I needed a list of names to generate links and access each page. The following codes were used to read the list and put together the links.

library(rvest)
# Set working directory and read the file with actor list
# The data must have names that are capitalized (e.g., Brad Pitt)
df <- readxl::read_excel("Alist.xlsx")

# Create a list of the names: Take the first column and repace space with underscore
df$list <- gsub("\\s", "_", df[[1]])

# Create a list of web addresses for Wikipedia entries
df$urls <- paste0("https://en.wikipedia.org/wiki/", df$list)
## # A tibble: 5 x 3
##   Name           list           urls                                       
##   <chr>          <chr>          <chr>                                      
## 1 Brad Pitt      Brad_Pitt      https://en.wikipedia.org/wiki/Brad_Pitt    
## 2 Jim Carrey     Jim_Carrey     https://en.wikipedia.org/wiki/Jim_Carrey   
## 3 Robert De Niro Robert_De_Niro https://en.wikipedia.org/wiki/Robert_De_Ni~
## 4 Jack Nicholson Jack_Nicholson https://en.wikipedia.org/wiki/Jack_Nichols~
## 5 Tom Hanks      Tom_Hanks      https://en.wikipedia.org/wiki/Tom_Hanks

To extract the exact information you want on a web page, you need to study its HTML structure. This is probably the most time-consuming process. The structures also differ from website to website so my codes won’t work as are here for web scraping a different website. Here’s a general idea and a brief explanation of how to get what you need. This document I found online also has a good description of how web scraping with R works.

First, go to the web page that exemplifies your target content. With this example, I’ll go to the Wikipedia page of Matthew McConaughey to study the structure of a Wikipedia entry.

If you’re using Google Chrome, right-click on your mouse and click on ‘Inspect’ (Keyboard shortcut: Ctrl + Shift + I). This will open up a new pane on the right side of your screen with three sub-panes (Firefox has the same panes, just horizontally organized). Click on the small icon (a square with a pointer) at the top left corner of the newly opened pane to select the element from the web page (Keyboard shortcut: Ctrl + Shift + C). Once this icon is activated, move your mouse around on the content page until you see that what you want to scrape is highlighted in blue (see the picture below). Left-click on your mouse, and this will also highlight the corresponding line on the inspector pane on the right side.

Expand the line by clicking the small arrow (at the start of the line) and view the nested structure. In this case, I don’t want the little line right under title of the article that says “From Wikipedia, the free encyclopedia”. What I want is the body of the content, and this is under \<div id="mw-content-text" lang="en" ...\>.

What I find easier to look at is the texts that appear at the bottom of the top sub-pane. This shows the structure horizontally. Starting on the left side, from html, body, div#..., down to where you’re currently at.

I took the string #mw-content-text, which is the id of the section that I want. The next command reads the html file from the web and extracts the target node. Don’t forget to include the hashtag, which indicates that it’s an id (as in \<div id=\>).

read_html("https://en.wikipedia.org/wiki/Matthew_McConaughey") %>% 
    html_nodes("#mw-content-text")
## {xml_nodeset (1)}
## [1] <div id="mw-content-text" lang="en" dir="ltr" class="mw-content-ltr" ...

If successful, it should have an output. If it says xml_nodeset (0), then it means nothing has been grabbed.

Moving on, under this node mw-content-text, I only want the text, not the pictures, not the table of contents, not the information in the box. When a web page consists of paragraphs, texts usually appear after a <p>, which denotes a new paragraph. If you’re not sure, click on the little selector icon and hover over a paragraph to see what the dark bubble says (which should say “p” as in the picture below), or what becomes highlighed on the inspector pane (\<p\>...\</p\>). So I pipeline another code to get at this node:

read_html("https://en.wikipedia.org/wiki/Matthew_McConaughey") %>% 
    html_nodes("#mw-content-text") %>% html_nodes("p")
## {xml_nodeset (25)}
##  [1] <p><b>Matthew David McConaughey</b> (<span class="nowrap"><span cla ...
##  [2] <p>In the 2000s, McConaughey became best known for starring in <a h ...
##  [3] <p>McConaughey achieved critical success in 2013 and 2014. In 2013, ...
##  [4] <p>Outside of film \nIn 2016, McConaughey became the creative direc ...
##  [5] <p>Matthew David McConaughey<sup id="cite_ref-2" class="reference"> ...
##  [6] <p>McConaughey's mother and late father married each other three ti ...
##  [7] <p>McConaughey moved to <a href="/wiki/Longview,_Texas" title="Long ...
##  [8] <p>McConaughey began working in television commercials, including o ...
##  [9] <p>After some smaller roles in <i><a href="/wiki/Angels_in_the_Outf ...
## [10] <p>By the early 2000s, he was frequently cast in <a href="/wiki/Rom ...
## [11] <p>McConaughey starred in the 2005 feature film <i><a href="/wiki/S ...
## [12] <p>McConaughey recognized that his "lifestyle, living on the beach, ...
## [13] <p>In 2012, McConaughey starred alongside <a href="/wiki/Channing_T ...
## [14] <p>In 2013, he portrayed <a href="/wiki/Ron_Woodroof" title="Ron Wo ...
## [15] <p>During this time, McConaughey recorded a public service announce ...
## [16] <p>In 2014, he shared star billing with <a href="/wiki/Woody_Harrel ...
## [17] <p>On November 17, 2014, McConaughey received a star on the <a href ...
## [18] <p>McConaughey also played Cooper, a widowed father and astronaut,  ...
## [19] <p>In 2016, McConaughey was hired as <a href="/wiki/Creative_direct ...
## [20] <p>McConaughey starred in the <a href="/wiki/Stephen_King" title="S ...
## ...

Now we have captured all nodes relevant to the information we want. However, the codes here so far only read the meta-language of the HTML and not the actual text we want to grab. To extract the text: add html_text() to the pipeline.

read_html("https://en.wikipedia.org/wiki/Matthew_McConaughey") %>% 
    html_nodes("#mw-content-text") %>% html_nodes("p") %>% html_text()
##  [1] "Matthew David McConaughey (/m<U+0259>'k<U+0252>n<U+0259>he<U+026A>/; born November 4, 1969)  is an American actor and producer. He first gained notice for his breakout role in the coming-of-age comedy Dazed and Confused (1993), before going on to appear in the slasher film Texas Chainsaw Massacre: The Next Generation (1994), A Time to Kill (1996), the comedy film Larger than Life (1996), Steven Spielberg's historical drama Amistad (1997), the science fiction drama Contact (1997), the comedy EDtv (1999), and the war film U-571 (2000).\n"                                                                                                                                                                                                                                                                                                                                                                               
##  [2] "In the 2000s, McConaughey became best known for starring in romantic comedies,[1] including The Wedding Planner (2001), How to Lose a Guy in 10 Days (2003), Failure to Launch (2006), Fool's Gold (2008), and Ghosts of Girlfriends Past (2009). Since 2011, he has preferred dramatic roles, including The Lincoln Lawyer (2011), Bernie (2011), Killer Joe (2011), The Paperboy (2012), Mud (2012), Magic Mike (2012), The Wolf of Wall Street (2013), Interstellar (2014), The Sea of Trees (2016), Free State of Jones (2016).\n"                                                                                                                                                                                                                                                                                                                                                                                                       
##  [3] "McConaughey achieved critical success in 2013 and 2014. In 2013, McConaughey portrayed Ron Woodroof, a cowboy diagnosed with AIDS in the biographical film Dallas Buyers Club, which earned him the Academy Award, Critics' Choice Movie Award, Golden Globe Award, and Screen Actors Guild Award, all for Best Actor, among other awards and nominations. In 2014, he starred as Rust Cohle in the first season of HBO's crime drama anthology series True Detective, for which he won the Critics' Choice Television Award and TCA Award, and was nominated for the Primetime Emmy Award, Golden Globe Award, and Screen Actors Guild Award.\n"                                                                                                                                                                                                                                                                                            
##  [4] "Outside of film \nIn 2016, McConaughey became the creative director for Kentucky Bourbon Distillery, Wild Turkey. He has since produced and appeared in multiple commercials.\n"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
##  [5] "Matthew David McConaughey[2] was born on November 4, 1969, in Uvalde, Texas.[3][4] His mother, Mary Kathleen \"Kay\"/\"KMac\" (née McCabe), is a former kindergarten teacher and published author who taught McConaughey.[5] She was originally from Trenton, New Jersey.[6][7][8] His father, James Donald \"Jim\" McConaughey,[9][6] was born in Mississippi in 1929 and raised in Louisiana,[10] where he ran an oil pipe supply business; he played for the Kentucky Wildcats and the Houston Cougars college football teams.[6][11] In 1953, Jim was drafted in the 27th round by the NFL's Green Bay Packers. He was released before the season began and never played an official league game in the NFL.[7][12][13][14][15]"                                                                                                                                                                                                         
##  [6] "McConaughey's mother and late father married each other three times, having divorced each other twice.[16][17] He has two older brothers, Michael and Patrick (who was adopted).[18][19][10] Michael, nicknamed \"Rooster\", is a self-made millionaire who stars in the CNBC docu-series West Texas Investors Club.[20] and as of 2018 stars in the A&E reality show Rooster & Butch with Wayne (Butch) Gilliam. McConaughey's ancestry includes English, German, Irish, Scottish, and Swedish, with his Irish roots being from the Cavan/Monaghan area.[3][21] He is a relative of Confederate brigadier general Dandridge McRae.[22][23][24] He had a Methodist upbringing.[25][26][27]"                                                                                                                                                                                                                                                  
##  [7] "McConaughey moved to Longview, Texas, in 1980,[6] where he attended Longview High School. He lived in Australia for a year, in Warnervale, New South Wales, as a Rotary exchange student in 1988.[28] He attended the University of Texas at Austin (UT-Austin), where he joined Delta Tau Delta fraternity.[29] He began in the fall of 1989 and graduated in the spring of 1993 with a Bachelor of Science in Radio-Television-Film.[30] His original plan had changed as he wanted to attend Southern Methodist University until one of his brothers told him that private school tuition would have been a burden on the family's finances. He also had planned to attend law school after graduation from college,[31] but he realized he was not interested in becoming a lawyer.[6]"                                                                                                                                                  
##  [8] "McConaughey began working in television commercials, including one for the Austin, Texas daily newspaper, the Austin American-Statesman, which is credited as his first speaking role. The line, \"How else am I gonna keep up with my 'Horns?\" – a reference to his beloved Texas Longhorns sports teams – gave the local community a look at the young actor before he was cast in Richard Linklater's film Dazed and Confused.[32] In 1992, he was cast as \"Joe\" in Trisha Yearwood's music video \"Walkaway Joe\".[33]"                                                                                                                                                                                                                                                                                                                                                                                                               
##  [9] "After some smaller roles in Angels in the Outfield, Texas Chainsaw Massacre: The Next Generation, Boys on the Side, and the television series Unsolved Mysteries, McConaughey's big break came as the lawyer Jake Brigance in the film A Time to Kill (1996), based on the John Grisham novel of the same name.[34] In the late 1990s, McConaughey was cast in leading roles in more movies, including Contact, Amistad, The Newton Boys, EDtv and U-571.[4]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
## [10] "By the early 2000s, he was frequently cast in romantic comedies, including The Wedding Planner and How to Lose a Guy in 10 Days, both of which were successful at the box office.[35] During this period, he appeared as a firefighter in the low-budget film Tiptoes, with Kate Beckinsale, in Two for the Money as a protégé to Al Pacino's gambling mogul, and in Frailty with Bill Paxton, who also directed.[4][36]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
## [11] "McConaughey starred in the 2005 feature film Sahara, along with Steve Zahn and Penélope Cruz.[37] Prior to the release of the film, he promoted it by sailing down the Amazon River and trekking to Mali.[38] That same year, McConaughey was named People magazine's \"Sexiest Man Alive\" for 2005.[39] In 2006, he co-starred with Sarah Jessica Parker in the romantic comedy Failure to Launch and as Marshall head football coach Jack Lengyel in We Are Marshall. McConaughey also provided voice work in an ad campaign for the Peace Corps in late 2006.[40] He replaced Owen Wilson in Ben Stiller's Tropic Thunder after Wilson's suicide attempt.[41] On January 21, 2008, McConaughey became the new spokesman for the national radio campaign, \"Beef: It's What's for Dinner\", replacing Sam Elliott.[42][43]"                                                                                                               
## [12] "McConaughey recognized that his \"lifestyle, living on the beach, running with my shirt off, doing romantic comedies\" had caused him to be typecast for certain roles, and he sought dramatic work with other themes.[44] He said:\n"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
## [13] "In 2012, McConaughey starred alongside Channing Tatum in Magic Mike, based on Tatum's early life, which was directed by Steven Soderbergh.[45] He returned to his East Texas roots, working again with director Richard Linklater on Bernie, playing district attorney Danny Buck Davidson.[46] In June 2012, McConaughey was invited to join the Academy of Motion Picture Arts and Sciences.[47]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
## [14] "In 2013, he portrayed Ron Woodroof in Dallas Buyers Club. The role of a rodeo rider who discovers he has AIDS and struggles to get treatment required him to lose nearly 50 lb (22 kg).[48] The film earned McConaughey many acting awards, including the Screen Actors Guild Award for Outstanding Performance by a Male Actor in a Leading Role, the Golden Globe Award for Best Actor – Drama, and the Academy Award for Best Actor.[49][50] His co-star Jared Leto won the Academy Award for Best Supporting Actor, making Dallas Buyers Club the first film since Mystic River (2003) to win both awards.[51][52] He also featured in Martin Scorsese's The Wolf of Wall Street as Mark Hanna, an early boss of Jordan Belfort.[53]"                                                                                                                                                                                                    
## [15] "During this time, McConaughey recorded a public service announcement in Austin, Texas for the LBJ Presidential Library.[54] In April 2014, Time magazine included McConaughey in its annual Time 100 as one of the \"Most Influential People in the World\".[55] In August 2014, the Lincoln Motor Company signed a multi-year collaboration with McConaughey for an ad campaign. The commercials, directed by Nicolas Winding Refn (Drive), featured McConaughey as a storyteller driving around in Lincoln's MKC crossover.[56] Shortly after the commercials debuted in September 2014, they were parodied by Ellen DeGeneres, Conan O'Brien, Jim Carrey, South Park, and Saturday Night Live. Overall sales for Lincoln increased by 25 percent one month after the ads debuted.[57] The series of commercials starring McConaughey continued until October 2015; during this period he also endorsed the MKZ sedan and MKX crossover.\n"
## [16] "In 2014, he shared star billing with Woody Harrelson in HBO's crime drama anthology series True Detective.[58] For his role as Rust Cohle, he won the Critics' Choice Television Award for Best Actor in a Drama Series.[59] He was also nominated for the Primetime Emmy Award for Outstanding Lead Actor in a Drama Series, which he lost to Bryan Cranston and the Golden Globe Award for Best Actor – Miniseries or Television Film.[60][61][60] With his first Oscar win and the critical acclaim received for True Detective, \"McConaughey seems to be tapping into something essential, remaining himself while stretching, getting older while staying the same age.\" Critic Rachel Syme dubbed his recognition and performances while taking on more complex, dramatic roles as \"The McConaissance\".[62]"                                                                                                                       
## [17] "On November 17, 2014, McConaughey received a star on the Hollywood Walk of Fame, located on 6931 Boulevard [63]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
## [18] "McConaughey also played Cooper, a widowed father and astronaut, alongside Anne Hathaway, Jessica Chastain, Michael Caine, Mackenzie Foy, and Casey Affleck in Christopher Nolan's science fiction film Interstellar (2014).[64] After finishing Gus Van Sant's 2015 film The Sea of Trees with Ken Watanabe,[65] in 2016, McConaughey starred in two films, Free State of Jones and Gold, and voiced leading characters in two animated films, Kubo and the Two Strings and Sing.\n"                                                                                                                                                                                                                                                                                                                                                                                                                                                         
## [19] "In 2016, McConaughey was hired as creative director and celebrity spokesman for Wild Turkey's latest campaign, to bring in more women and more international customers.[66][67]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
## [20] "McConaughey starred in the Stephen King adaptation The Dark Tower (2017), as Walter Padick. His upcoming roles include true life gangster drama White Boy Rick,[68]Harmony Korine's The Beach Bum,[69] and opposite Diane Lane and Anne Hathaway, in the erotic thriller Serenity.[citation needed]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
## [21] "Before McConaughey met Camila Alves in 2006, he had been in relationships with Penélope Cruz and then Sandra Bullock. He and Alves became engaged on December 25, 2011, and were married in a private Catholic ceremony[70] on June 9, 2012, in Austin, Texas, where they reside.[71][72] Together, they have three children: sons Levi Alves McConaughey (born July 7, 2008)[73] and Livingston Alves McConaughey (born December 28, 2012),[74] and daughter Vida Alves McConaughey (born January 3, 2010).[75]"                                                                                                                                                                                                                                                                                                                                                                                                                            
## [22] "McConaughey is a Christian and attends a non-denominational church.[76]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
## [23] "McConaughey started the just keep livin foundation, which is \"dedicated to helping teenage kids lead active lives and make healthy choices to become great men and women\".[77] On February 25, 2016, McConaughey received the Creative Conscience award from unite4:humanity for his work with his foundation.[78]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
## [24] "McConaughey rescued various pets stranded after the flooding of New Orleans from Hurricane Katrina.[79]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
## [25] "McConaughey is teaching classes at his alma mater, UT-Austin, the first session of which was about the filming of the 2016 film Free State of Jones.[80] For the second session, he will team with The Hunger Games director Gary Ross and university lecturer Scott Rice.[80] The college course, part of the Moody College of Communications, will be open to juniors and seniors. There are no prerequisites or fees to take the weekly course.[80]"

You can see that this code has extracted the 25 paragraphs that appear in Matthew McConaughey’s wiki.

A shortcut/alternative is to use XPath. You can obtain the XPath by selecting the line that corresponds to the target element on the inspector pane, then by right clicking the mouse to choose: Copy - Copy XPath.

The XPath I got here, //*[@id="mw-content-text"]/div/p, specifies that I want all texts under the node \<p\> nested under this particular \<div\> node.

read_html("https://en.wikipedia.org/wiki/Matthew_McConaughey") %>% 
    html_nodes(xpath = '//*[@id="mw-content-text"]/div/p') %>% 
    html_text()
##  [1] "Matthew David McConaughey (/m<U+0259>'k<U+0252>n<U+0259>he<U+026A>/; born November 4, 1969)  is an American actor and producer. He first gained notice for his breakout role in the coming-of-age comedy Dazed and Confused (1993), before going on to appear in the slasher film Texas Chainsaw Massacre: The Next Generation (1994), A Time to Kill (1996), the comedy film Larger than Life (1996), Steven Spielberg's historical drama Amistad (1997), the science fiction drama Contact (1997), the comedy EDtv (1999), and the war film U-571 (2000).\n"                                                                                                                                                                                                                                                                                                                                                                               
##  [2] "In the 2000s, McConaughey became best known for starring in romantic comedies,[1] including The Wedding Planner (2001), How to Lose a Guy in 10 Days (2003), Failure to Launch (2006), Fool's Gold (2008), and Ghosts of Girlfriends Past (2009). Since 2011, he has preferred dramatic roles, including The Lincoln Lawyer (2011), Bernie (2011), Killer Joe (2011), The Paperboy (2012), Mud (2012), Magic Mike (2012), The Wolf of Wall Street (2013), Interstellar (2014), The Sea of Trees (2016), Free State of Jones (2016).\n"                                                                                                                                                                                                                                                                                                                                                                                                       
##  [3] "McConaughey achieved critical success in 2013 and 2014. In 2013, McConaughey portrayed Ron Woodroof, a cowboy diagnosed with AIDS in the biographical film Dallas Buyers Club, which earned him the Academy Award, Critics' Choice Movie Award, Golden Globe Award, and Screen Actors Guild Award, all for Best Actor, among other awards and nominations. In 2014, he starred as Rust Cohle in the first season of HBO's crime drama anthology series True Detective, for which he won the Critics' Choice Television Award and TCA Award, and was nominated for the Primetime Emmy Award, Golden Globe Award, and Screen Actors Guild Award.\n"                                                                                                                                                                                                                                                                                            
##  [4] "Outside of film \nIn 2016, McConaughey became the creative director for Kentucky Bourbon Distillery, Wild Turkey. He has since produced and appeared in multiple commercials.\n"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
##  [5] "Matthew David McConaughey[2] was born on November 4, 1969, in Uvalde, Texas.[3][4] His mother, Mary Kathleen \"Kay\"/\"KMac\" (née McCabe), is a former kindergarten teacher and published author who taught McConaughey.[5] She was originally from Trenton, New Jersey.[6][7][8] His father, James Donald \"Jim\" McConaughey,[9][6] was born in Mississippi in 1929 and raised in Louisiana,[10] where he ran an oil pipe supply business; he played for the Kentucky Wildcats and the Houston Cougars college football teams.[6][11] In 1953, Jim was drafted in the 27th round by the NFL's Green Bay Packers. He was released before the season began and never played an official league game in the NFL.[7][12][13][14][15]"                                                                                                                                                                                                         
##  [6] "McConaughey's mother and late father married each other three times, having divorced each other twice.[16][17] He has two older brothers, Michael and Patrick (who was adopted).[18][19][10] Michael, nicknamed \"Rooster\", is a self-made millionaire who stars in the CNBC docu-series West Texas Investors Club.[20] and as of 2018 stars in the A&E reality show Rooster & Butch with Wayne (Butch) Gilliam. McConaughey's ancestry includes English, German, Irish, Scottish, and Swedish, with his Irish roots being from the Cavan/Monaghan area.[3][21] He is a relative of Confederate brigadier general Dandridge McRae.[22][23][24] He had a Methodist upbringing.[25][26][27]"                                                                                                                                                                                                                                                  
##  [7] "McConaughey moved to Longview, Texas, in 1980,[6] where he attended Longview High School. He lived in Australia for a year, in Warnervale, New South Wales, as a Rotary exchange student in 1988.[28] He attended the University of Texas at Austin (UT-Austin), where he joined Delta Tau Delta fraternity.[29] He began in the fall of 1989 and graduated in the spring of 1993 with a Bachelor of Science in Radio-Television-Film.[30] His original plan had changed as he wanted to attend Southern Methodist University until one of his brothers told him that private school tuition would have been a burden on the family's finances. He also had planned to attend law school after graduation from college,[31] but he realized he was not interested in becoming a lawyer.[6]"                                                                                                                                                  
##  [8] "McConaughey began working in television commercials, including one for the Austin, Texas daily newspaper, the Austin American-Statesman, which is credited as his first speaking role. The line, \"How else am I gonna keep up with my 'Horns?\" – a reference to his beloved Texas Longhorns sports teams – gave the local community a look at the young actor before he was cast in Richard Linklater's film Dazed and Confused.[32] In 1992, he was cast as \"Joe\" in Trisha Yearwood's music video \"Walkaway Joe\".[33]"                                                                                                                                                                                                                                                                                                                                                                                                               
##  [9] "After some smaller roles in Angels in the Outfield, Texas Chainsaw Massacre: The Next Generation, Boys on the Side, and the television series Unsolved Mysteries, McConaughey's big break came as the lawyer Jake Brigance in the film A Time to Kill (1996), based on the John Grisham novel of the same name.[34] In the late 1990s, McConaughey was cast in leading roles in more movies, including Contact, Amistad, The Newton Boys, EDtv and U-571.[4]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
## [10] "By the early 2000s, he was frequently cast in romantic comedies, including The Wedding Planner and How to Lose a Guy in 10 Days, both of which were successful at the box office.[35] During this period, he appeared as a firefighter in the low-budget film Tiptoes, with Kate Beckinsale, in Two for the Money as a protégé to Al Pacino's gambling mogul, and in Frailty with Bill Paxton, who also directed.[4][36]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
## [11] "McConaughey starred in the 2005 feature film Sahara, along with Steve Zahn and Penélope Cruz.[37] Prior to the release of the film, he promoted it by sailing down the Amazon River and trekking to Mali.[38] That same year, McConaughey was named People magazine's \"Sexiest Man Alive\" for 2005.[39] In 2006, he co-starred with Sarah Jessica Parker in the romantic comedy Failure to Launch and as Marshall head football coach Jack Lengyel in We Are Marshall. McConaughey also provided voice work in an ad campaign for the Peace Corps in late 2006.[40] He replaced Owen Wilson in Ben Stiller's Tropic Thunder after Wilson's suicide attempt.[41] On January 21, 2008, McConaughey became the new spokesman for the national radio campaign, \"Beef: It's What's for Dinner\", replacing Sam Elliott.[42][43]"                                                                                                               
## [12] "McConaughey recognized that his \"lifestyle, living on the beach, running with my shirt off, doing romantic comedies\" had caused him to be typecast for certain roles, and he sought dramatic work with other themes.[44] He said:\n"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
## [13] "In 2012, McConaughey starred alongside Channing Tatum in Magic Mike, based on Tatum's early life, which was directed by Steven Soderbergh.[45] He returned to his East Texas roots, working again with director Richard Linklater on Bernie, playing district attorney Danny Buck Davidson.[46] In June 2012, McConaughey was invited to join the Academy of Motion Picture Arts and Sciences.[47]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
## [14] "In 2013, he portrayed Ron Woodroof in Dallas Buyers Club. The role of a rodeo rider who discovers he has AIDS and struggles to get treatment required him to lose nearly 50 lb (22 kg).[48] The film earned McConaughey many acting awards, including the Screen Actors Guild Award for Outstanding Performance by a Male Actor in a Leading Role, the Golden Globe Award for Best Actor – Drama, and the Academy Award for Best Actor.[49][50] His co-star Jared Leto won the Academy Award for Best Supporting Actor, making Dallas Buyers Club the first film since Mystic River (2003) to win both awards.[51][52] He also featured in Martin Scorsese's The Wolf of Wall Street as Mark Hanna, an early boss of Jordan Belfort.[53]"                                                                                                                                                                                                    
## [15] "During this time, McConaughey recorded a public service announcement in Austin, Texas for the LBJ Presidential Library.[54] In April 2014, Time magazine included McConaughey in its annual Time 100 as one of the \"Most Influential People in the World\".[55] In August 2014, the Lincoln Motor Company signed a multi-year collaboration with McConaughey for an ad campaign. The commercials, directed by Nicolas Winding Refn (Drive), featured McConaughey as a storyteller driving around in Lincoln's MKC crossover.[56] Shortly after the commercials debuted in September 2014, they were parodied by Ellen DeGeneres, Conan O'Brien, Jim Carrey, South Park, and Saturday Night Live. Overall sales for Lincoln increased by 25 percent one month after the ads debuted.[57] The series of commercials starring McConaughey continued until October 2015; during this period he also endorsed the MKZ sedan and MKX crossover.\n"
## [16] "In 2014, he shared star billing with Woody Harrelson in HBO's crime drama anthology series True Detective.[58] For his role as Rust Cohle, he won the Critics' Choice Television Award for Best Actor in a Drama Series.[59] He was also nominated for the Primetime Emmy Award for Outstanding Lead Actor in a Drama Series, which he lost to Bryan Cranston and the Golden Globe Award for Best Actor – Miniseries or Television Film.[60][61][60] With his first Oscar win and the critical acclaim received for True Detective, \"McConaughey seems to be tapping into something essential, remaining himself while stretching, getting older while staying the same age.\" Critic Rachel Syme dubbed his recognition and performances while taking on more complex, dramatic roles as \"The McConaissance\".[62]"                                                                                                                       
## [17] "On November 17, 2014, McConaughey received a star on the Hollywood Walk of Fame, located on 6931 Boulevard [63]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
## [18] "McConaughey also played Cooper, a widowed father and astronaut, alongside Anne Hathaway, Jessica Chastain, Michael Caine, Mackenzie Foy, and Casey Affleck in Christopher Nolan's science fiction film Interstellar (2014).[64] After finishing Gus Van Sant's 2015 film The Sea of Trees with Ken Watanabe,[65] in 2016, McConaughey starred in two films, Free State of Jones and Gold, and voiced leading characters in two animated films, Kubo and the Two Strings and Sing.\n"                                                                                                                                                                                                                                                                                                                                                                                                                                                         
## [19] "In 2016, McConaughey was hired as creative director and celebrity spokesman for Wild Turkey's latest campaign, to bring in more women and more international customers.[66][67]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
## [20] "McConaughey starred in the Stephen King adaptation The Dark Tower (2017), as Walter Padick. His upcoming roles include true life gangster drama White Boy Rick,[68]Harmony Korine's The Beach Bum,[69] and opposite Diane Lane and Anne Hathaway, in the erotic thriller Serenity.[citation needed]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
## [21] "Before McConaughey met Camila Alves in 2006, he had been in relationships with Penélope Cruz and then Sandra Bullock. He and Alves became engaged on December 25, 2011, and were married in a private Catholic ceremony[70] on June 9, 2012, in Austin, Texas, where they reside.[71][72] Together, they have three children: sons Levi Alves McConaughey (born July 7, 2008)[73] and Livingston Alves McConaughey (born December 28, 2012),[74] and daughter Vida Alves McConaughey (born January 3, 2010).[75]"                                                                                                                                                                                                                                                                                                                                                                                                                            
## [22] "McConaughey is a Christian and attends a non-denominational church.[76]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
## [23] "McConaughey started the just keep livin foundation, which is \"dedicated to helping teenage kids lead active lives and make healthy choices to become great men and women\".[77] On February 25, 2016, McConaughey received the Creative Conscience award from unite4:humanity for his work with his foundation.[78]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
## [24] "McConaughey rescued various pets stranded after the flooding of New Orleans from Hurricane Katrina.[79]"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
## [25] "McConaughey is teaching classes at his alma mater, UT-Austin, the first session of which was about the filming of the 2016 film Free State of Jones.[80] For the second session, he will team with The Hunger Games director Gary Ross and university lecturer Scott Rice.[80] The college course, part of the Moody College of Communications, will be open to juniors and seniors. There are no prerequisites or fees to take the weekly course.[80]"

The results are same as with the first method. Because each paragraph becomes one element in R, I’ll collapse all paragraphs in my final code with the paste function.

# Loop through the names and collect texts
for (n in 1:nrow(df)) {
    # scrape text from the content body
    df$text[[n]] <- read_html(df$urls[[n]]) %>% 
        html_nodes(xpath = '//*[@id="mw-content-text"]/div/p') %>% 
        html_text() %>% 
        paste0(collapse = "")
    # print progress to detect if and where a problem occurs
    print(paste(df$list[[n]], "- Complete")) 
}

# Clean the text by removing the footnote numbers
df$text <- gsub("\\[\\d{1,3}\\]", " ", df$text)

Because my colleague wanted to use AntConc in the next step, I saved each text as its own text file.

# Save the text for each actor in .txt format. Use the name as filename.
for (m in 1:length(df$text)) {
    write(df$text[[m]], paste0(df$list[[m]], ".txt"))
}

2.2 . More ideas

There has been a lot of research to create vocabulary or phrase lists for a specific genre or sub-genre (often relevant to academic writing). The more interesting and perhaps more useful method for teaching would be putting such word lists into context to show what sorts of words are used where in a written text in what genre. In other words, what’s the language that is typically used in a specific context? For instance, Cortes (2013)1 analyzed lexical bundles that are frequently used to serve certain purposes in the introduction section of research articles (as implied in the title, “The purpose of this study is to: Connecting lexical bundles and moves in research article introductions”).

In the scraping process above, I collapsed all paragraphs together but texts can be categorized differently to find which words or collocations are frequently used in the first paragraph, in the last paragraph, etc., so that learners can make connections between language and its function. When teaching or giving a task that requires writing a movie review, reviews written by real people can be scraped from the Internet and analyzed as learning materials. Such summary of large, linguistic data will allow learners to see the relationship between linguistic choice and the context (e.g., register, genre, task).

3 . Example 2: Scraping Online Newspapers

In this example, I scrape recent articles from the Opinions section of The Washington Post. Your result will be different from what appears on this post because this list of articles is constantly being updated.

3.1 . Op-Eds from The Washington Post

In the first example, someone had to manually create a list to generate URLs for web pages. This time, the URLs themselves will be extracted from a web page. With this particular example, it was actually a little more complicated because the method I described above for getting the HTML elements didn’t work as it was supposed to (works fine with other websites). I thus had to experiment a lot to pull it off. It may be because it’s an RSS feed page. Here’s how it went down:

wp_url <- "http://feeds.washingtonpost.com/rss/opinions"
wp_html <- read_html(wp_url)

wp_links <- wp_html %>% 
    html_nodes("item") %>% 
    html_text()

The only node that worked was item so I just used what I got. The output was a chunk of information containing the URL to the article, date, and time for each article:

## [1] "https://www.washingtonpost.com/opinions/my-son-was-shot-by-park-police-officers-were-still-waiting-for-justice/2019/01/11/f73db62a-14fe-11e9-803c-4ef28312c8b9_story.html Fri, 11 Jan 2019 04:37:00 EST"

I needed to remove the date and time information and only keep the link to the article. Upon inspection, I found that the link and date and time information were separated by a space, so I removed everything after the first space. Then I made a data frame with the link so I can store the content there. Note that this link column needs to be character (rather than factor) to avoid error in a later stage.

# Remove unnecessary information
wp_links <- gsub(" .*", "", wp_links)
wp_table <- data.frame(link = wp_links)
wp_table$link <- as.character(wp_table$link)

I used the RSS feed page to obtain links to recent articles, but in order to grab texts from these articles, I need to view an article and find out how to extract the text from the it. I’ll loop through 5 articles as an example here.

wp_ex <- dplyr::slice(wp_table, 1:5)

for (n in 1:nrow(wp_ex)) {
    # get the title
    wp_ex$title[[n]] <- read_html(wp_ex$link[[n]]) %>% 
        html_nodes("h1") %>% 
        html_text()
    
    wp_ex$text[[n]] <- read_html(wp_ex$link[[n]]) %>% 
        html_nodes("#article-body") %>% 
        html_nodes("p") %>% 
        html_text() %>% 
        paste0(collapse = "")
}

The title of the article appeared under the node \<h1\> and the article was under \<div id="article-body"\>. I used this information to store the title and content of each article.

3.2 . More ideas

News opinion pieces can be thought of as a type of argumentative writing, and so you may want to do move analysis combined with collocations. You can use sites such as Google News, to select a specific category and scrape news on that specific topic to create a vocabulary/phrase list for English for Specific Purposes. This will assist learners with learning frequently used expressions which they are likely to encounter when studying or working.

4 . Example 3: Scraping Blogs

Blogs on specialized topics are great sources of language data. If the posts are well categorized and/or tagged, you can easily navigate to a page where the posts of your interest are listed (just as the RSS feed page), and you can go from there for web scraping.

4.1 . The Big Bang Theory transcripts

In my last example, I scrape the transcripts for the TV series, the Big Bang Theory.

This blog has all transcripts from 10 seasons. I went through a similar process of inspecting the websites to first extract the links to all pages then extract the text from each page. This blog conveniently lists links to all posts on the widget that appears on your right.

Here, getting the URL is different from the second example (which, I don’t know how it worked). The current example is what usually would be the case with the URLs that exist as meta-data. I extracted the attribute <a href> by using html_attr("href").

bbt_url <- "https://bigbangtrans.wordpress.com"
bbt_html <- read_html(bbt_url)

bbt_links <- bbt_html %>% 
    html_nodes(xpath = '//*[@id="pages-2"]/ul/li/a') %>% 
    html_attr("href")

Because the links include an introduction post that I don’t need, I excluded the first link when I saved the links as a data frame. Also, change the link column to characters.

bbt_table <- data.frame(link = bbt_links[-1])
bbt_table$link <- as.character(bbt_table$link)

I will experiment with the first 5 episodes. Note that I’m switching between HTML and XPath to give some different examples.

bbt_ex <- dplyr::slice(bbt_table, 1:5)

for (n in 1:nrow(bbt_ex)) {
    bbt_ex$text[[n]] <- read_html(bbt_ex$link[[n]]) %>% 
        html_nodes(".entrytext") %>% 
        html_nodes("p") %>% 
        html_text() %>% 
        paste0(collapse = "")
}

4.2 . More ideas

What I like about this example is that, although scripted, it provides spoken language data which is rarer than written. Thinking in terms of gathering spoken language (or at least something close), you can try finding transcripts of TV shows, movies, news, and other broadcasts.

The Big Bang Theory corpus created here could be used to identify which episode has the target grammar or vocabulary you want to teach your English learners. For example, if I want to show a clip that has one or more instances of using a specific idiom or a phrasal verb, I can use this corpus I’ve built to locate which episode has that particular phrase - which can be a huge time saver.

Further linguistic analysis could provide a useful list of collocations, phrasal verbs, or other expressions, as well as how they are used in conversations. After coming up with a list of collcations, (either with software such as AntConc or methods such as this n-gram analysis), you can give the list to learners and ask them to make a conversation using these words. After that, you can watch how these expressions are actually used in the TV show.

With further text processing, corpora of different characters can also be built. The Big Bang Theory is probably too difficult with all the technical jargon, but if your students regularly watch a TV show, such as the timeless classic, Friends, you can give them a list of words and phrases a character often uses and have students engage in writing scripts and role playing.

5 . Summary

  • Reasons to utilize web scraping for language teaching
    • You can instantly collect a lot of language
    • You can provide learners with language that is frequently used in real life
  • For meaningful teaching (beyond offering a vocabulary list):
    • Come up with a topic or genre that is useful for learners
    • Use the scraped data to show how language is used for what purposes
    • Have students reconstruct the genre or conversation using the word list and make comparisonss

This post was inspired by the course, LLT 809: Teaching Second Language Reading and Writing (Instructor: Dr. Charlene Polio). Please feel free to share your thoughts!


  1. Cortes, V. (2013). The purpose of this study is to: Connecting lexical bundles and moves in research article introductions. Journal of English for Academic Purposes, 12, 33-43. http://dx.doi.org/10.1016/j.jeap.2012.11.002.

Related

Previous
comments powered by Disqus