basics.R

#' ---
#' format:
#'   ipynb:
#'     eval: false
#'     ipynb-title-block: "title-block.md"
#' knitr:
#'  opts_chunk:
#'    collapse: true
#'    comment: "#>"
#'    R.options:
#'      knitr.graphics.auto_pdf: true
#' jupyter:
#'  kernelspec:
#'    name: "ir"
#'    language: "R"
#'    display_name: "R"
#' ---
#'
#' #### Introduction to Computational Social Science with R
#'
#' # Introduction & Basics – Day 4: Computational Text Analysis
#'
#' Chung-hong Chan (GESIS – Leibniz Institute for the Social Sciences)
#'
#' Paul Balluff (GESIS – Leibniz Institute for the Social Sciences)
#'
#' How can we parse, load, and represent text data such that a computer can understand it?
#' This session covers the basics of computational text analysis, where we represent text as "Bag-of-Words", meaning that we split longer texts into single words (tokens) and show how this seemingly simple representation is the foundation for more complex techniques.
#' We introduce common packages alongside a running example of short social media posts.
#'
#'
#' ## Learning objectives
#'
#' By the end of this session, you should be able to
#'
#' - Load text data into R
#' - Understand and use tokenization
#' - Run basic analysis with meta data
#' - Apply pre-processing
#' - Transform text into a Document-Term-Matrix (or Document-Feature-Matrix)
#' - Know most important features of the `quanteda` package
#'
#' ## Packages & Data
#'
#' We use the [Donald Trump Tweets Dataset](https://www.thetrumparchive.com/) which covers all of his tweets from 2009 until 2024.
#'
#' Define the path of our data

data_path <- "data/trump.json"

#' In this session we will use the following packages: `jsonlite`, `quanteda`, `quanteda.textstats`, `quanteda.textplots`, `tidyverse` and `lubridate`.
#'
#' `quanteda` is a commonly used package for text analysis in R, it provides various functions to read, transform, process, and analyse text data.
#'
#' To install the packages on your own machine: `install.packages(c("jsonlite", "quanteda", "quanteda.textstats", "quanteda.textplots", "lubridate", "tidyverse"))`.
#'
#' Install R packages (on Colab)

if (Sys.getenv("COLAB_RELEASE_TAG") != "") {
    options(
        repos = c(
            CRAN = sprintf(
                "https://packagemanager.posit.co/cran/latest/bin/linux/jammy-%s/%s",
                R.version["arch"],
                substr(getRversion(), 1, 3)
            )
        )
    )
    install.packages(c(
        "jsonlite",
        "quanteda",
        "quanteda.textstats",
        "quanteda.textplots",
        "lubridate",
        "tidyverse"
    ))
    data_path <- "https://raw.githubusercontent.com/gesiscss/datasets/refs/heads/main/trump.json"
    options(repr.matrix.max.rows = 10)
    options(repr.matrix.max.cols = 7)
}

#' ### Load required R packages
#| message: false
library(jsonlite)
library(lubridate)
library(tidyverse)
library(quanteda)
library(quanteda.textstats)
library(quanteda.textplots)

#' ### Exploring Trump tweets
#'
tweets <- read_json(data_path, simplifyVector = TRUE) %>% as_tibble()
tweets

#' All column names
colnames(tweets)

#' To get a glimpse: dplyr::glimpse(), look at `source`, `text`, and `created_at`
glimpse(tweets)

#' ### Let's do some warm-up exercises
#' 1. Calculate the percent of tweets that are retweets (`is_retweet`)
#' 2. Find out the tweet with the higest number of retweets (`retweet_count`)
#' 3. Find out all types of `source`
#| eval: false
tweets

#' 4. Find out the mean retweet count of each `sources`
#| eval: false
tweets

#' ### Main task
#' For the main task, we will consider only tweets that has a source of either "Twitter for iPhone" or "Twitter for Android", at well as tweets that are not retweets

tweets %>% nrow()
tweets <- tweets %>%
    dplyr::filter(
        source %in% c("Twitter for iPhone", "Twitter for Android") & !is_retweet
    ) %>%
    mutate(android = source == "Twitter for Android")
tweets %>% nrow()

#' Our task is first to do some exploratory analyses on the `text` column of all his tweets, and then to do a comparative analysis between tweets from an iPhone and from an Android device.

tweets %>% dplyr::select(id_str, text)

#' A `corpus` object is a nice way to organize text content

tweets_corpus <- corpus(tweets$text, docnames = tweets$id_str)
tweets_corpus

#' We can also add `docvars` (document-level variables)
docvars(tweets_corpus, "android") <- tweets$android
tweets_corpus

#' Accessing `docvars`
#| eval: false
docvars(tweets_corpus, "android")
# Or tweets_corpus$android

#' Assigning many `docvars` at once
docvars(tweets_corpus) <- dplyr::select(tweets, -id_str, -text, -source)

#| eval: false
docvars(tweets_corpus, "is_retweet")

#' ### Representing the text data
#' At the current form, the text data is not easy to analyze. The text data have to be reformatted to a representation that can be analyzed. There are many ways to represent text. In this section, we will represent the text data first as `unigram`. We will explore different way to present the text in this `unigram` format.
#' Consider the following text data (We modified the classic sentence slightly)
simple_sentence <- "The quick brown fox, a slow fox, and 67 other foxes 🦊 jump over the lazy dog and jumped FoX."
simple_sentence

#' Suppose we would like to do a simple computation task of counting how many time the token "fox" occur in this sentence. The simple way is to split the sentence by the white space. This way, it is easier to do the frequency analysis, e.g., `table(split_sentence)`
split_sentence <- strsplit(simple_sentence, " ")[[1]]
split_sentence


#' Let introduce some terms. A *token* is the unit of text for analysis. In the case of *uni*gram` representation, a token is a *single* word. The process of breaking down a piece of text to tokens is called *tokenization*. In the above example, we *tokenize* the text by spliting the sentence by the white space. White space in the English language represents separation between two unigrams. [^1] Therefore, a white space is a separator, or a delimiter. Usually in English, the before and after of some prounciations can also be considered delimiters, e.g. full stop, period. We did not consider them in the above example. Therefore, the above sentence can also be tokenized into "the" "quick" "brown" "fox" **","** "a" "slow" ...
#'
#' [^1]: Please note that I do not say "a separation between two words". Lingustically speaking, there is no universal accepted definition of what a word is. For instance, some would say "ice cream" is one single word, while some says it is a compound word with two single words.
#'
#' We will use `quanteda` to tokenize the same sentence. It is almost the same, only the output is not a vector, but a `tokens` object. [^2]
#'
#' [^2]: For more advanced learners: You could explore yourself what would happen when you run this: `corpus(simple_sentence) %>% tokens() %>% table()`. You can also try to educate yourself by investigating why the output looks like this.

corpus(simple_sentence) %>% tokens()

#' There are many options for `tokens()`. See `?tokens`. We will do some manipulations that is usually grouped under the umbrella of "token normalization".
#' Usually, the punctuations are also not that important
corpus(simple_sentence) %>% tokens(remove_punct = TRUE)

#' We also want to remove all symbols and numbers
corpus(simple_sentence) %>%
    tokens(remove_punct = TRUE, remove_symbols = TRUE, remove_numbers = TRUE)

#' Another idea is "stemming". "foxes" (plural) and "fox" are considered two different words. If we want to make them the same, we can apply stemming (turning all words into their stems, e.g, the stem of "foxes" is "fox").
corpus(simple_sentence) %>%
    tokens(
        remove_punct = TRUE,
        remove_symbols = TRUE,
        remove_numbers = TRUE
    ) %>%
    tokens_wordstem(language = "english")

#' Casing is also not that important: The -> the
corpus(simple_sentence) %>%
    tokens(
        remove_punct = TRUE,
        remove_symbols = TRUE,
        remove_numbers = TRUE
    ) %>%
    tokens_wordstem(language = "english") %>%
    tokens_tolower()

#' Now, let's do the same with `tweets_corpus`

tweets_tokens <- tweets_corpus %>%
    tokens(
        remove_punct = TRUE,
        remove_symbols = TRUE,
        remove_numbers = TRUE
    ) %>%
    tokens_wordstem(language = "english") %>%
    tokens_tolower()

#' With the `tokens` object, we can already do some analyses, such as kwic (Keyword in context)
kwic(tweets_tokens, pattern = "democrat", window = 7)[1:10, ]

#' Usually, we will not do analysis with the `tokens`. We usually change it further to Document-Term Matrix (dtm, or Document-Feature Matrix, dfm, in `quanteda`'s lingo). Suppose there is a corpus like this:
#'
#' **Doc 1**: berlin is the capital of germany
#'
#' **Doc 2**: paris is the capital of france
#'
#' **Doc 3**: tokyo is the capital of japan
#'
#' **Doc 4**: the cat is weird
#'
#' **Doc 5**: berlin berlin
#'
#' **Doc 6**: paris is nice
#'
#' **Doc 7**: paris is nice
#'
#' **Doc 8**: paris is nice
#'
#' **Doc 9**: paris is nice
#'
#' **Doc 10**: berlin is weird
#'
#' The unique token types are: "berlin", "is", "the", "capital", "of", "germany", "paris", "france", "tokyo", "japan", "cat", "weird", "nice". The representation of the above corpus as a document-term matrix is:
#'
#' |    | berlin | is | the | capital | of | germany | paris | france | tokyo | japan | cat | weird | nice |
#' |----|--------|----|-----|---------|----|---------|-------|--------|-------|-------|-----|-------|------|
#' | 1  | 1      | 1  | 1   | 1       | 1  | 1       | 0     | 0      | 0     | 0     | 0   | 0     | 0    |
#' | 2  | 0      | 1  | 1   | 1       | 1  | 0       | 1     | 1      | 0     | 0     | 0   | 0     | 0    |
#' | 3  | 0      | 1  | 1   | 1       | 1  | 0       | 0     | 0      | 1     | 1     | 0   | 0     | 0    |
#' | 4  | 0      | 1  | 1   | 0       | 0  | 0       | 0     | 0      | 0     | 0     | 1   | 1     | 0    |
#' | 5  | 2      | 0  | 0   | 0       | 0  | 0       | 0     | 0      | 0     | 0     | 0   | 0     | 0    |
#' | 6  | 0      | 1  | 0   | 0       | 0  | 0       | 1     | 0      | 0     | 0     | 0   | 0     | 1    |
#' | 7  | 0      | 1  | 0   | 0       | 0  | 0       | 1     | 0      | 0     | 0     | 0   | 0     | 1    |
#' | 8  | 0      | 1  | 0   | 0       | 0  | 0       | 1     | 0      | 0     | 0     | 0   | 0     | 1    |
#' | 9  | 0      | 1  | 0   | 0       | 0  | 0       | 1     | 0      | 0     | 0     | 0   | 0     | 1    |
#' | 10 | 1      | 1  | 0   | 0       | 0  | 0       | 0     | 0      | 0     | 0     | 0   | 1     | 0    |
#'
#' This data structure is great for doing frequency-based analysis. For instance, the total frequency of the token "is" is just the sum of the that column. One important feature of a Document-Term Matrix is that majority of the cells are zero. In other words, this is a sparse matrix.
#'
some_texts <- c(
    "berlin is the capital of germany",
    "paris is the capital of france",
    "tokyo is the capital of japan",
    "berlin berlin",
    "the cat is weird",
    "paris is nice",
    "paris is nice",
    "paris is nice",
    "paris is nice",
    "berlin is weird"
)

example_dfm <- corpus(some_texts) %>% tokens() %>% dfm()
example_dfm

#' Print the entirety

example_dfm %>% print(max_ndoc = 10, max_nfeat = 20)

#' With a DTM/DFM, we can do some basic counting
example_dfm %>% ndoc()
example_dfm %>% nfeat()
example_dfm %>% topfeatures()

#' **Excercise 1:** Slightly more complicated counting; Explain to yourself what are they using the token "berlin"; and consult the help files.
example_dfm %>% featfreq()
example_dfm %>% docfreq()

#' **Excercise 2:** Convert `tweets_tokens` to `tweets_dfm`
#| eval: false
tweets_tokens

#| output: false
#| echo: false
tweets_dfm <- tweets_tokens %>% dfm()

#' **Excercise 3:** Do the three basic counting of `tweets_dfm`
#| eval: false
tweets_dfm

#' `dfm` can be subset using the `dfm_subset()` function
tweets_dfm %>% dfm_subset(android)

tweets_dfm %>% dfm_subset(!android)

#' **Excercise 4:** What are the top features of tweets from iPhone and Android repsecitively? Do you find the results useful?
tweets_dfm

#' There are many words that are not so useful for frequency analysis, e.g. prepositions, pronouns. By removing these words, it would make some analytically more important terms (usually nouns, verbs, and adjectives) more salient. The words that do not serve much purpose analytically are called "stopwords". `quanteda` provides lists of stopwords for different languages.
stopwords(language = "english") ### the same as stopwords()

#' We can use the function `dfm_remove` to remove specific words, including stopwords
tweets_dfm %>% dfm_remove(pattern = stopwords(), valuetype = "fixed")

#' **Excercise 5:** Repeat Excercise 4 but with stopwords removed
#| eval: false
tweets_dfm

#' **Excercise 6:** Try the function `quanteda.textstats::textstat_frequency()`
#| eval: false
tweets_dfm

#' **Excercise 7:** Try the function `quanteda.textstats::textstat_keyness()`
#| eval: false
tweets_dfm

#' **Excercise 8:** Try the function `quanteda.textplots::textplot_keyness()`
#| eval: false
tweets_dfm

#' ### Problem Set (with increasing difficulty)
#' 1. What did Trump tweet repeatedly?
#' 2. Do you know other functions or methods that work like `ndoc()` and `nfeat()` for a matrix? Do they work for a `dfm` object?
#' 3. Why this doesn't work? `kwic(tweets_tokens, pattern = "hilary", window = 7)`
#' 4. Is there any difference in the readability of tweets from an iPhone and an Android?