The general idea

Data transfer is highly controlled. The key notions are authentication and protocol.

Downloading toots with rtoot

There are several packages that run an interface with twitter: rtweet, RTwitterAPI, streamR and twitteR.
But since Auth V2, we will need RTwitterV2! But this only runs on R v4.2!
Documentation: https://github.com/MaelKubli/RTwitterV2.
Recent packages are better because firms update their API policies (and access), thus old protocols sometimes do not work!
Unfortunately, the Twitter API is no longer free!
Hence, in this notebook, we will test the competitor: mastodon!
The package for this will be rtoot.

First things first

First, the packages. Download…

if(!require(rtoot)){install.packages("rtoot")}

… and activate.

library(tidyverse)
library(plotly)
library(rtoot)

Authentication

Second: authentication You have to choose a particular instance of the network. Personally, I am registered on “sciences.social”, the largest one is “mastodon.social” (see https://mastodonservers.net/servers/top) => Write the answer without the quotation marks and choose a public tocken

rtoot::auth_setup(
  instance = "mastodon.social",
  type = "public"
)

Token of type "public" for instance mastodon.social is valid

<mastodon bearer token> for instance: mastodon.social of type: public

# get_timeline_hashtag(hashtag = "rstats", 
#                      instance = "mastodon.social",
#                      limit = 200)

Authentication can be an important part of the process. For more info on that:
- https://cran.r-project.org/web/packages/googlesheets/vignettes/managing-auth-tokens.html
- https://httr.r-lib.org/reference/index.html (section Authentication)
- https://blog.r-hub.io/2021/01/25/oauth-2.0/

Extraction

If no error appears, we are ready to query. Depending on the number of requested tweets, this can take some time.

There are different types of queries that the packages allows.
For instance, below we use the get_timeline_hashtag function to access toots that include one particular term, the “hashtag”.

search_term <- "election"
toots <- get_timeline_hashtag(hashtag = search_term, 
                              instance = "mastodon.social",
                              limit = 2000)

Text mining

References

The reference book is: https://www.tidytextmining.com
A great interactive tutorial: https://juliasilge.shinyapps.io/learntidytext/
And the package is:

if(!require(tidytext)){install.packages("tidytext", repos = "https://cloud.r-project.org/")}
library(tidytext)

Data retrieval

Now, let’s move forward to simple text analysis. First, we need to prepare the data! (as usual)

tokens <- toots %>% 
    select(id, content) %>%             # Keeps only id and text/content of the tweet
    unnest_tokens(word, content)        # Creates tokens!
tokens

Let’s have a look at word frequencies.

tokens %>%
    count(word, sort = TRUE)

This is polluted by small words. Let’s filter that (FIRST METHOD).

tokens %>% mutate(length = nchar(word))

Data frequencies

Now let’s omit the small words (smaller than 5 characters).
NOTE: all the thresholds below depend on the sample!

tokens %>%
    mutate(length = nchar(word)) %>%
    filter(length > 4) %>%             # Keep words with length larger than 4
    count(word, sort = TRUE) %>%       # Count words
    head(21) %>%                       # Keep only top 12 words
    ggplot(aes(y = reorder(word,n), x = n)) + geom_col() + ylab("Words") + theme_bw()

A better way to proceed is to remove “stop words” like “a”, “I”, “of”, “the”, etc (SECOND METHOD). Also, it would make sense to remove the search item and “https”.

data("stop_words")
tidy_tokens <- tokens %>% 
    anti_join(stop_words)                    # Remove unrelevant terms
tidy_tokens %>%
    count(word, sort = TRUE) %>%             # Count words
    head(20) %>%                             # Keep only top 15 words
    ggplot(aes(y = reorder(word,n), x = n)) + geom_col() + ylab("Words") + theme_bw()

Problem: strange characters remain. We are going to remove them by converting the text to ASCII format and omit NA data.

new_stop_words <- c("https", "span", "class", "href", "target", "_blank", "rel", "tag",
                    "mastodon.social", "ellipsis", "mastodon.online", "mstdn.social", "amp",
                    "http", "invisible", "03", search_term, tolower(search_term), "d0", "src",
                    "tags", "mention", "noreferrer", "noopener", "nofollow", "hashtag", "translate",
                    "www", "url", "die", "der", "und", "a", "p", "br", "1", "2", "01", "02")
tidy_tokens <- tokens %>% 
    anti_join(stop_words) %>%                            # Remove unrelevant
    mutate(word = iconv(word, from = "UTF-8", to = "ASCII")) %>% # Put in latin format
    na.omit() %>%                                        # Remove missing
    filter(nchar(word) > 2,                              # Remove small words
           !(word %in% new_stop_words)  # search_term defined above
    )
tidy_tokens %>%
    count(word, sort = TRUE) %>%         # Count words
    head(30) %>%                         # Keep only top words
    ggplot(aes(y = reorder(word,n), x = n)) + geom_col() + ylab("Words") + theme_bw()

Perfect!

Word cloud

This data can also be shown with a word cloud. We simply use the wordcloud package: https://cran.r-project.org/web/packages/wordcloud/index.html

The package wordcloud2 adds a few features: https://cran.r-project.org/web/packages/wordcloud2/vignettes/wordcloud.html

if(!require(wordcloud)){install.packages("wordcloud")}
library(wordcloud)
cloud_data <- tidy_tokens %>% count(word)
wordcloud(words = cloud_data$word, 
          freq = cloud_data$n, min.freq = 10,
          max.words = 82, random.order = FALSE, rot.per = 0.15, 
          colors = brewer.pal(8, "Dark2"))

Third party data and basic text mining