collect_text() creates a network from free text, such as speeches, articles, posts, novels, or the lines of a play. It splits the text into units, such as documents or sentences, and ties each unit to the terms it uses, or to the names it mentions. The result is a two-mode network of units and terms, from which a network of terms that co-occur, or of units that share terms, is a projection: see to_mode().

Where the text records who speaks (and to whom), the result is instead a one-mode network of speakers and what they mention.

collect_text(
  text,
  unit = "document",
  terms = NULL,
  stopwords = NULL,
  from = NULL,
  to = NULL,
  ties = c("counts", "tokens"),
  token = NULL,
  lexicon = NULL
)

Arguments

text

The text, as one or more documents. This can be:

  • a character vector, each element of which is one document, named by its names if it has any;

  • a data frame with a 'text' column, each row of which is one document, named by its 'doc_id', 'document', or 'id' column if it has one; further columns become attributes of the units;

  • a data frame with one token in each row, such as from tidytext::unnest_tokens() or udpipe::udpipe(), with a 'doc_id', 'document', or 'id' column naming the document of each. Paragraph and sentence units need 'paragraph_id' and 'sentence_id' columns, as {udpipe} gives.

unit

The unit that ties to terms. One of "document" (the default), "paragraph", "sentence", or an integer for a moving window of that many words. The first mode is named after it, e.g. "sentences".

terms

The terms to collect. By default NULL, every word. Otherwise a character vector or a named list of aliases (see Terms).

stopwords

Words to drop. A character vector, or a data frame with a 'word' column, such as tidytext::stop_words. By default NULL, none.

from

The speaker of each document. The name of a column of text, or a vector with one value for each document. The speaker is the 'from' of the ties of each unit in the document. By default NULL, no speakers, which gives a two-mode network.

to

The addressee of each document, given in the same way as from. The addressee is the 'to' of the ties of each unit in the document. Needs from.

ties

How ties are recorded. Either "counts" (the default), one tie for each unit and term, with the number of times the term is used in the unit as its weight, or "tokens", one tie for each use of the term.

token

The column of a token data frame that holds the term, such as "lemma" for {udpipe} output. By default NULL, the 'word' or else the 'token' column. Filter the rows first to keep only some parts of speech, e.g. dplyr::filter(tokens, upos == "NOUN").

lexicon

A sentiment dictionary (see Sentiment). A named numeric vector of values for words, or a data frame with a 'word' column and either a numeric 'value' column or a 'sentiment' column of "positive" (+1) and "negative" (-1). By default NULL, no sentiment.

Value

A stocnet object. A two-mode network of units and terms, or, with from, a one-mode directed network of speakers and terms.

Details

Only base R is used, so no text-mining package is needed. Words are lowercased, and a word is a run of letters and digits, possibly joined by an apostrophe, as in "let's". So punctuation and hyphens separate words. For finer control, such as lemmas or only the nouns, tokenise the text elsewhere and pass the tokens as a data frame (see text and token).

Paragraphs are separated by a blank line, and sentences end with a full stop, a question mark, or an exclamation mark that is followed by a space. Abbreviations such as "e.g." therefore end a sentence too.

Where the units are paragraphs, sentences, or windows, and there are two or more of them, each unit is stamped with its position in the text, in a time column of the ties, so that to_time() and similar can scope the network to a part of the text. Documents are not stamped, since a corpus need not be in any order. Units are positions and not waves or events, so the network does not say how one record of a tie updates another.

Terms

By default, every word is a term. Terms are then ordered alphabetically. A dictionary in terms keeps only the terms it lists, in the order it lists them. It can be a character vector of terms, or a named list, each element of which gives the aliases of one name, such as list(Harry = c("harry", "potter", "the boy who lived")). A term of several words matches where its words follow one another, and the longest match is taken where matches overlap, so "harry potter" is one mention of Harry and not two. The second mode is called "terms", or "characters" where the dictionary is a named list.

Stopwords are dropped from the terms and are not scored for sentiment, but a term of several words is still matched across them, so "bank of england" is found where "of" is a stopword.

Speakers

A text with speakers, such as a transcript or a play, gives a one-mode, directed network. With from only, each tie runs from the speaker to a term they mention. With both from and to, each tie runs from the speaker to the addressee, and the term mentioned is recorded in the 'about' column, as in gossip (see is_gossip()). A unit without a speaker (or an addressee) gives no ties. Speakers, addressees, and terms with the same label are one node, so name the dictionary entries as the speakers are named, since words are lowercased.

Sentiment

A lexicon scores each unit by the sum of the values of its words. Where units are nodes, they hold this score as a 'sentiment' attribute. Where the text has speakers, the ties from a unit of negative score are negative, so that the network is signed (positive and negative mentions, or gossip). tidytext::get_sentiments("afinn") and tidytext::get_sentiments("bing") are suitable lexicons.

Examples

speeches <- c(first = "Trade builds peace. Peace builds trade.",
              second = "War ends trade.\n\nPeace needs work.")
collect_text(speeches)
collect_text(speeches, unit = "sentence", stopwords = c("needs"))
collect_text(speeches, terms = c("peace", "war", "trade")) |>
  to_mode("terms")
#> 
#> ── # Projection of Text network ────────────────────────────────────────────────
#> # A labelled, weighted, undirected network of 3 nodes and 3 ties
#> # Transformed by projection: mode 2 (count)
#> 
#> ── Nodes 
#> # A tibble: 3 × 1
#>   label
#>   <chr>
#> 1 peace
#> 2 war  
#> 3 trade
#> 
#> ── Ties 
#> # A tibble: 3 × 3
#>    from    to weight
#>   <int> <int>  <dbl>
#> 1     1     2      1
#> 2     1     3      5
#> 3     2     3      1
#> 
play <- data.frame(speaker = c("Ann", "Bob", "Ann"),
                   addressee = c("Bob", "Ann", "Cy"),
                   text = c("Have you seen Cy?", "Cy left with Dee.",
                            "Dee is lovely, Cy is awful."))
cast <- list(Ann = "ann", Bob = "bob", Cy = "cy", Dee = "dee")
collect_text(play, terms = cast, from = "speaker")
collect_text(play, terms = cast, from = "speaker", to = "addressee",
             lexicon = c(lovely = 1, awful = -2))