Chapter 17 Using Regular Expressions

17.1 RegEx basics

A regular expression (RegEx) maybe be simple or complicated ones. All of them look foreign until you learn how to read them.

Regexes encode patterns. The simplest patterns can be taken literally. For instance, "able" will match any string with “able” in it, e.g., “You are able …”, “She is capable”, “The motion is tabled”. But "able" will not match “Able to do what?” Capital letters are distinct from lower case, although this can be turned off in many functions with the argument ignore.case = TRUE, as used in the previous examples based on BabyNames.

More complicated patterns involve punctuation that has a special meaning. These characters have a special meaning: ., ^, $, [, |, ], {, }

For example, ".ble" is a regex that matches any set of four letters ending with “ble”. This will match all the strings matched by "able" but others as well: “sugar is soluble in water”, “money is fungible”, “Bible reading for today”.

Simple patterns:

• A single . means “any character.”
• A character, e.g., b, means just that character.
• Characters enclosed in square brackets, e.g., [aeiou] means any one of those characters. (So, [aeiou] is a pattern describing a vowel.)
• The ^ inside square brackets means “any except these.” So, a consonant is [^aeiou]

Alternatives: A vertical bar means “either.” For instance, the regex "rain|snow|sleet|hail" will match any of those four forms of precipitation.

Two simple patterns in a row, means those patterns consecutively. Example: "M[aeiou]" means a capital M followed by a single lower-case vowel.

Repeats

• A pattern followed by a + means “zero or more times.”

  So, "M[aeiou]+" matches “Miocene”, “Mr.”, “Ms”, “Mrs.”, “Miss”, and “Man”, among many others.

• A simple pattern followed by a ? means “zero or one times.”

  So, "M[aeiou]?[^aeiou]" will not match “Miocene”. (The regex means: M followed by at most one vowel, followed by a consonant. “Miocene” has two vowels following the M.)

• A simple pattern followed by a * means “one or more times.”

  "M[aeiou]*" will match “Miocene” and “Miss”, but not “Mr.”

• A simple pattern followed by {2} means “exactly two times.”

  Similarly, {2,5} means between two and five times, {6,} means six times or more. For instance, [aeiou]{2} means “exactly two vowels in a row” and will not match “Miss” or “Mr” or “Madam”.

Start and end of strings:

• ^ at the beginning of a regular expression means “the start of the string”
  
• $ at the end means “the end of the string.”

Extraction:

Some functions that process regexes will extract the specific character sequence that matches the specified sequence. To specify which part of the match is to be extracted, put it inside open and close parentheses. For example, the regex "([aeiouAEIOU]{4})" in following statement will match all names with 4 consecutive vowels of either upper or lower case and extract those vowels as a new variable.

BabyNames %>%
  group_by(name) %>%
  summarise(total = sum(count)) %>%
  mutate(match = str_extract(string = name, pattern = "([aeiouAEIOU]{4})")) %>%
  filter( !is.na(match)) %>%
  arrange(desc(total)) 

Table 17.1: “Baby names with 4 consecutive vowels captured by the regex "([eaiouAEIOU]\\{4\\})"”

On occasion, you will want to use one of the regex control characters in a literal sense, for example . to mean “period” and not “any one character.” You can do this by “escaping” that character by preceeding it with a double backslash such as \\. in this case. For instance, in the regex "^\\$|¥|£|¢$" about currency symbols, the first $ is intended to refer to the dollar currency sign. But on its own in a regex, $ refers to the end of a character string. We therefore escape the first dollar sign using \\$ to be taken literally as a currency symbol. To create a regex involving the period found at the end of sentences, you need to escape the ., like this: "\\.$"

17.2 Example tasks with regular expressions.

Get rid of percent signs and commas in numerals

Numbers often come with comma separators or unit symbols such as % or $. For instance, Table 17.2 shows part of a table about public debt from Wikipedia.

Table 17.2: Public debt (from Wikipedia)

To use these numbers for computations, they must be cleaned up.

Debt %>% 
  mutate(debt = gsub(pattern = "[$,%]|billion", replacement = "", debt),
         percGDP = gsub(pattern = "[,%]", replacement = "", percGDP)) 

Table 17.3: Remove comma separators, etc. to create a pure number.

Remove a currency sign

currency <- c("$100.95", "45¢")
gsub(pattern = "^\\$|€|¥|£|¢$", replacement = "", currency)

## [1] "100.95" "45"

Remove leading or trailing spaces

string <- "   My name is Julia     "
gsub(pattern = "^ +| +$", replacement = "", string)

## [1] "My name is Julia"

How often do names end in vowels?

NameList %>%
  filter( grepl( "[aeiou]$", name ) ) %>% 
  group_by( sex ) %>% 
  summarise( total = sum(total) )

Table 17.4: The number of babies given names ending in a vowel.

Girls’ names are almost five times as likely to end in vowels as boys’ names.

regex <- ".*([aeiou])$"
NameList %>% 
  filter(grepl(pattern = regex, name)) %>% 
  mutate(vowel = gsub(pattern = regex, replacement = "\\1", name)) %>%
  group_by( sex, vowel ) %>% 
  summarise( total = sum(total) ) %>%
  arrange( desc(total) ) %>%
  pivot_wider(names_from = sex, values_from = total)

17.3 Exercises

Problem 17.1: Using the BabyNames data table, find the 10 most popular names according to each of these criteria.

• Boys’ names ending in a vowel.
• Names with a space (like Lee Ann)
• Names ending with “joe” (like BettyJoe)

Problem 17.2: Construct a regular expression that matches names that sound like “Caitlin” — A k or c followed by one or more vowels followed by tl then one or more vowels, and ending with n.

Problem 17.3: Here is a character string with a regular expression:

"([2-9][0-9]{2})[- .]([0-9]{3})[- .]([0-9]{4})"

To explain the first bit …

• [2-9] means "one digit from 2 to 9.
• [0-9] refers to one digit from 0 to 9.
• [0-9]{2} refers to two consecutive digits, 0 to 9.
• [2-9][0-9]{2} means one digit 2 to 9 followed by two digits 0 to 9
• [- .] means "any of the characters dash, space, period, just once.
• The parentheses refer to the matching contents to be extracted. The whole expression has the structure (stuff)[- .](more stuff)[- .](still more stuff). The three sets of parentheses mean to extract those three pieces from strings that match.

Explain what familiar kinds of strings the entire general expression would match. (Hint: Call me maybe.) What components of those strings are being extracted?

Problem 17.4: Consider this regular expression (ignore line breaks):

"(A[LKSZRAP]|C[AOT]|D[EC]|F[LM]|G[AU]|HI|I[ADLN]|
K[SY]|LA|M[ADEHINOPST]|N[CDEHJMVY]|O[HKR]|P[ARW]|
RI|S[CD]|T[NX]|UT|V[AIT]|W[AIVY])"
    

1. How long will the strings be that match the pattern?
2. How many different strings will match?
3. People living in the United States may be able to figure out what the pattern is meant to express. Give it a try.

Problem 17.5: A list of names from the Bible can be accessed like this:

BibleNames <- 
  readr::read_csv("https://mdbeckman.github.io/dcSupplement/data/BibleNames.csv")

Using the names in BibleNames,

1. Which names have any of these words in them: “bar”, “dam”, “lory”?
2. Which names end with those words?

You need only show a few.

Problem 17.6: A list of names from the Bible can be accessed like this:

BibleNames <- 
  readr::read_csv("https://mdbeckman.github.io/dcSupplement/data/BibleNames.csv")

Using BibleNames, make an informative plot showing, year-by-year the proportion of all baby names that are Bible-related.

Problem 17.7: A list of names from the Bible can be accessed like this:

BibleNames <- 
  readr::read_csv("https://mdbeckman.github.io/dcSupplement/data/BibleNames.csv")

Using BabyNames, make a data table showing the total number of babies, adding up over all the years, given each Bible-related name. Display the ten most popular names.