library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(stringr)
library(tidytext)
library(ggplot2)
sentences <- read_csv(
"data/workforce/workforce_sentences.csv",
na = character(),
col_types = cols(
sentence_id = col_character(),
document_id = col_character(),
source_line = col_character(),
text = col_character(),
reference_label = col_character(),
uncertainty = col_character(),
annotator_id = col_character(),
rationale = col_character(),
codebook_version = col_character(),
codebook_hash = col_character(),
derived = col_character(),
transformation = col_character()
)
)
source_counts <- sentences |>
mutate(source = if_else(document_id == "F001", "training flyer", "job details")) |>
count(source, name = "rows")Counting nearby words with n-grams
See why phrase counts get sparse fast
phrases and entities
n-grams
workforce research
Learn how bigrams and trigrams count adjacent tokens in the Riverton sentences, why most of them appear only once, and why a PMI score without a count floor ranks unique pairs first.
Repeated words such as training stand out in the Riverton Workforce Lab’s short job-board lines and flyer headings. The harder question is whether neighbouring word pairs tell a clearer story.
The coordinator tries a simple phrase count before trusting it. If most phrases appear once, the list may describe these 28 sentences closely while helping less with new documents.
Note
The Riverton Workforce Lab, its job board, and its training flyer are fictional and were created for teaching.
TipWhat you will learn
This lesson shows how to:
- define an n-gram as a run of adjacent tokens;
- build bigrams and trigrams with tidytext;
- count repeated n-grams in the Riverton sentences;
- measure how many n-grams appear once;
- score repeated bigrams with pointwise mutual information; and
- explain why phrase vocabularies can generalise badly from small data.
Load the sentences
This setup chunk reads the sentence file, prepares small tables, builds n-grams with tidytext, and draws one summary chart. A tibble is a table that prints its size and column types.
| Source | Rows |
|---|---|
| job details | 22 |
| training flyer | 6 |
The file is small enough to inspect by eye. That convenience is also the reason its phrase counts should be treated as local examples.
Build adjacent-word phrases
An n-gram is a run of n adjacent tokens. A unigram has one token, a bigram has two, and a trigram has three. These counts use the same token rule as Lesson 15: lowercase the text and drop punctuation. Because each row here is one sentence, no n-gram spans two sentences; a file with whole documents in rows would need a different check.
make_ngrams <- function(n) {
if (identical(n, 1L)) {
sentences |>
select(sentence_id, text) |>
unnest_tokens(output = ngram, input = text, token = "words") |>
filter(!is.na(ngram))
} else {
sentences |>
select(sentence_id, text) |>
unnest_tokens(output = ngram, input = text, token = "ngrams", n = n) |>
filter(!is.na(ngram))
}
}
short_for_trigrams <- make_ngrams(1L) |>
count(sentence_id, name = "tokens") |>
filter(tokens < 3L)
bigrams <- make_ngrams(2L) |>
count(ngram, name = "frequency", sort = TRUE)
trigrams <- make_ngrams(3L) |>
count(ngram, name = "frequency", sort = TRUE)
top_phrases <- bind_rows(
bigrams |>
slice_head(n = 8) |>
mutate(kind = "bigram"),
trigrams |>
slice_head(n = 8) |>
mutate(kind = "trigram")
) |>
select(kind, ngram, frequency)| N-gram type | N-gram | Frequency |
|---|---|---|
| bigram | is required | 4 |
| bigram | training is | 3 |
| bigram | are required | 2 |
| bigram | experience is | 2 |
| bigram | is preferred | 2 |
| bigram | is provided | 2 |
| bigram | no prior | 2 |
| bigram | shifts are | 2 |
| trigram | training is provided | 2 |
| trigram | 12 week training | 1 |
| trigram | a daytime schedule | 1 |
| trigram | a high school | 1 |
| trigram | a medical records | 1 |
| trigram | a paid apprenticeship | 1 |
| trigram | a portfolio is | 1 |
| trigram | a valid driver’s | 1 |
The sentence Evening classes has only two tokens, so it has no trigram. The code drops that empty result before counting. The most frequent bigram is is required, with 4 appearances, and the most frequent trigram is training is provided, with 2.
Measure the sparsity
Sparsity means that many possible items have no examples or only one example. The general reason is arithmetic, not this particular table. Each new word adds at most one new word type, but it can also add a new pair and a new triple; the set of pairs a vocabulary could form grows faster than the vocabulary itself. The Riverton counts below illustrate that argument rather than prove it.
unigrams <- make_ngrams(1L) |>
count(ngram, name = "frequency", sort = TRUE)
ngram_counts <- list(unigrams, bigrams, trigrams)
ngram_summary <- tibble(
n = 1:3,
name = c("unigrams", "bigrams", "trigrams"),
distinct_items = vapply(ngram_counts, nrow, integer(1)),
seen_once = vapply(
ngram_counts,
\(counts) sum(counts$frequency == 1L),
integer(1)
),
seen_once_label = str_c(seen_once, "/", distinct_items)
)| n | Unit | Distinct units | Seen once | Seen-once fraction |
|---|---|---|---|---|
| 1 | unigrams | 96 | 72 | 72/96 |
| 2 | bigrams | 112 | 102 | 102/112 |
| 3 | trigrams | 96 | 95 | 95/96 |
The 28 sentences contain 96 distinct unigrams, 112 distinct bigrams, and 96 distinct trigrams. The seen-once fractions are 72/96 for unigrams, 102/112 for bigrams, and 95/96 for trigrams. These numbers describe only this teaching corpus.
Show the plotting code
source("R/lesson-figures.R")
plot_data <- ngram_summary |>
mutate(
seen_once_share = seen_once / distinct_items,
bar_label = str_c(
seen_once, " of ", distinct_items, " (", round(100 * seen_once_share), "%)"
),
name = factor(name, levels = c("trigrams", "bigrams", "unigrams"))
)
ggplot(plot_data, aes(x = seen_once_share, y = name)) +
geom_col(fill = lesson_colours[["accent"]], width = 0.62) +
geom_text(
aes(label = bar_label),
hjust = 0,
nudge_x = 0.015,
family = lesson_font,
size = 10,
size.unit = "pt",
colour = lesson_colours[["body"]]
) +
scale_x_continuous(expand = expansion(mult = c(0, 0.32))) +
labs(
title = "Longer n-grams are almost all one-offs",
subtitle = "Distinct n-grams seen exactly once in the 28 Riverton sentences",
x = NULL,
y = NULL
) +
theme_lesson(grid = "none") +
theme(axis.text.x = element_blank())
The chart focuses on the quantity the table asks the reader to compare: how much of each list appears only once.
Score pairs against chance
Raw frequency likes common words. Pointwise mutual information, or PMI, asks whether two adjacent words appear more often than they would if each word were chosen independently. For a pair (w1, w2):
PMI = log2( P(w1, w2) / (P(w1) * P(w2)) )
The probabilities here are token shares in these 28 sentences: 153 unigram tokens and 125 bigram tokens. A pair of words that each appear once, and that appear together once, gets a high score because the denominator is tiny. That is a property of the formula, not evidence that the pair is a stable phrase.
unigram_tokens <- make_ngrams(1L)
bigram_tokens <- make_ngrams(2L)
unigram_n <- nrow(unigram_tokens)
bigram_n <- nrow(bigram_tokens)
unigram_counts <- unigram_tokens |>
count(ngram, name = "word_n")
pmi_bigrams <- bigram_tokens |>
separate_wider_delim(
ngram,
delim = " ",
names = c("word1", "word2"),
too_many = "merge"
) |>
count(word1, word2, name = "pair_n") |>
left_join(unigram_counts, by = join_by(word1 == ngram)) |>
rename(word1_n = word_n) |>
left_join(unigram_counts, by = join_by(word2 == ngram)) |>
rename(word2_n = word_n) |>
mutate(
ngram = str_c(word1, word2, sep = " "),
pmi = log2(
(pair_n / bigram_n) /
((word1_n / unigram_n) * (word2_n / unigram_n))
)
)
hapax_singletons <- pmi_bigrams |>
filter(pair_n == 1L, word1_n == 1L, word2_n == 1L) |>
arrange(ngram)
repeated_pmi <- pmi_bigrams |>
filter(pair_n >= 2L) |>
arrange(desc(pmi), desc(pair_n), ngram)
hapax_singleton_pmi <- log2(
(1 / bigram_n) / ((1 / unigram_n) * (1 / unigram_n))
)
is_required <- pmi_bigrams |>
filter(ngram == "is required")| Bigram | Pair count | PMI |
|---|---|---|
| 50 pounds | 1 | 7.55 |
| able to | 1 | 7.55 |
| an essential | 1 | 7.55 |
Thirty-seven of the 102 one-time bigrams are pairs of words that each appear once. They all receive PMI 7.55. The most frequent bigram, is required (4 times), scores 3.04 because is appears 13 times and required appears 7 times. A floor of two pair counts leaves ten bigrams.
repeated_pmi_table <- repeated_pmi |>
transmute(
ngram,
pair_n,
pmi = round(pmi, 2)
)| Bigram | Pair count | PMI |
|---|---|---|
| no prior | 2 | 6.55 |
| shifts are | 2 | 5.55 |
| the job | 2 | 5.23 |
| is preferred | 2 | 3.85 |
| is provided | 2 | 3.85 |
| work is | 2 | 3.85 |
| are required | 2 | 3.74 |
| training is | 3 | 3.11 |
| is required | 4 | 3.04 |
| experience is | 2 | 2.85 |
With that floor, no prior leads and is required falls to ninth. Both lists still describe 28 teaching sentences. PMI does not repair sparsity; it changes which sparse pairs look interesting. For matches in running text, Lesson 25 uses a concordance instead of a score.
What to remember
- An n-gram counts adjacent tokens after a chosen token rule.
- The Riverton rows contain 112 distinct bigrams and 96 distinct trigrams.
- A two-token sentence contributes no trigram.
- PMI without a count floor ranks unique word pairs first.
- Counts from 28 sentences illustrate sparsity; they do not estimate phrase use in workforce writing.
Report the repeated phrases as local clues, then stop. The useful result here is a small list of repeats paired with a warning about sparse phrase counts.