library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(tokenizers)
library(stringr)
library(udpipe)
library(digest)
sentences <- read_csv(
"data/workforce/workforce_sentences.csv",
na = character(),
col_types = cols(
sentence_id = col_character(),
document_id = col_character(),
source_line = col_character(),
text = col_character(),
reference_label = col_character(),
uncertainty = col_character(),
annotator_id = col_character(),
rationale = col_character(),
codebook_version = col_character(),
codebook_hash = col_character(),
derived = col_character(),
transformation = col_character()
)
)
learning_curve <- read_csv(
"data/treebank/tagger-learning-curve.csv",
na = character(),
col_types = cols(
training_sentences = col_integer(),
training_tokens = col_integer(),
held_out_tokens = col_integer(),
correct_tokens = col_integer(),
accuracy = col_double()
)
)
metadata <- read_csv(
"data/treebank/treebank-metadata.csv",
na = character(),
col_types = cols(
artifact = col_character(),
description = col_character(),
source = col_character(),
source_url = col_character(),
retrieved_on = col_character(),
license = col_character(),
fingerprint = col_character()
)
)
builder_script_path <- "data-raw/build-treebank-tagger.R"
builder_seed_match <- read_lines(builder_script_path) |>
str_subset("^set.seed\\(") |>
str_match("^set.seed\\((\\d+)\\)")
builder_seed <- as.integer(builder_seed_match[1, 2])
rng_settings <- RNGkind()
learning_curve_path <- "data/treebank/tagger-learning-curve.csv"
learning_curve_fingerprint <- digest(
paste(read_lines(learning_curve_path), collapse = "\n"),
algo = "sha256",
serialize = FALSE
)
expected_curve_fingerprint <- metadata |>
filter(artifact == "tagger-learning-curve.csv") |>
pull(fingerprint)
metadata_display <- metadata |>
filter(
artifact %in% c(
"en_ewt-train-excerpt.conllu",
"en_ewt-held-out.conllu",
"en_ewt-500-tagger.udpipe",
"tagger-learning-curve.csv"
)
) |>
mutate(fingerprint_start = str_sub(fingerprint, 1, 12)) |>
select(artifact, source, retrieved_on, license, fingerprint_start)
reproducibility_table <- tibble(
item = c(
"Builder seed from script",
"Current R RNG kind",
"Current normal generator",
"Current sample generator",
"Current R version"
),
value = c(
as.character(builder_seed),
rng_settings[[1]],
rng_settings[[2]],
rng_settings[[3]],
R.version.string
)
)Tagging words by their job
Read part-of-speech tags from a small trained model
word parsing
part-of-speech tagging
workforce research
Learn how a part-of-speech tagger labels workforce text and why held-out accuracy has limits.
With sentence labels finished, attention turns to single words. The team points to a job posting and asks a smaller question: which words are the actions?
That question changes the level of analysis. The team is no longer asking whether a whole sentence is about training or schedules. It wants to know what each word is doing inside the sentence.
A part-of-speech tagger assigns a grammatical tag to each token. The tag can say that a word is a thing, an action, a description, a helper verb, a number, or punctuation. This lesson uses a UDPipe tagger trained on 500 hand-labeled sentences from the Universal Dependencies English Web Treebank. A treebank is a collection of sentences that people have labelled by hand, word by word.
Note
The Riverton Workforce Lab, its job board, and its training flyer are fictional and were created for teaching. The UDPipe model and treebank artifacts are local files bundled with this project, so the lesson does not download anything while it runs.
TipWhat you will learn
By the end of this lesson, you will be able to:
- describe common Universal POS tags in plain language;
- run a local UDPipe part-of-speech tagger on fixed tokens;
- inspect a tagging mistake in a job-posting sentence;
- read a held-out accuracy score; and
- explain why accuracy on weblog posts does not promise the same result on job postings.
Load the local files
For this read-only pass, readr opens the files, dplyr and tibble handle tables, purrr repeats token work, tokenizers split words, udpipe tags them, and digest checks file fingerprints. The model has already been trained, so the code only reads artifacts from disk and records the settings needed to describe the bundled run.
| Artifact | Source | Retrieved | License | SHA-256 start |
|---|---|---|---|---|
| en_ewt-500-tagger.udpipe | UD English EWT v2.18, released 2026-05-15 | 2026-08-28 | CC BY-SA 4.0 | 9c3907d37e9b |
| en_ewt-held-out.conllu | UD English EWT v2.18, released 2026-05-15 | 2026-08-28 | CC BY-SA 4.0 | 468cbf678026 |
| en_ewt-train-excerpt.conllu | UD English EWT v2.18, released 2026-05-15 | 2026-08-28 | CC BY-SA 4.0 | 8eb319b01a7d |
| tagger-learning-curve.csv | UD English EWT v2.18, released 2026-05-15 | 2026-08-28 | CC BY-SA 4.0 | b1c614e3cdd1 |
| Reproducibility item | Value |
|---|---|
| Builder seed from script | 20260828 |
| Current R RNG kind | Mersenne-Twister |
| Current normal generator | Inversion |
| Current sample generator | Rejection |
| Current R version | R version 4.6.1 (2026-06-24) |
The table prints the first 12 characters of each SHA-256 fingerprint so it stays readable at high zoom. The full values are in data/treebank/treebank-metadata.csv. A hash shows that a file has not changed since it was made. It says nothing about whether the numbers in the file are right. The second table mixes one build setting that is recorded in the builder script with details from the current rerun environment. The committed model does not carry build-time RNG kind, R version, or package-version metadata, so those details would need to be recorded on the next rebuild. Only the 500-sentence tagger is included in this repository, so the other four rows of the learning curve cannot be rebuilt from the committed files.
Tag fixed tokens
The model was trained to tag tokens it receives. It was not trained to split raw sentences. The code therefore tokenizes each Riverton sentence first, writes one token per line, and tells UDPipe that the input is already vertical text.
The output includes a lemma, the dictionary form of a word, the form you would look up. is has the lemma be. In the code, \(tokens) is R shorthand for a small function that handles one sentence’s tokens at a time.
tagger <- udpipe_load_model(
"data/treebank/en_ewt-500-tagger.udpipe"
)
vertical <- sentences$text |>
tokenize_words(lowercase = FALSE, strip_punct = FALSE) |>
map_chr(\(tokens) paste(tokens, collapse = "\n"))
tagged <- udpipe_annotate(
tagger,
x = vertical,
doc_id = sentences$sentence_id,
tokenizer = "vertical",
tagger = "default",
parser = "none"
) |>
as.data.frame() |>
as_tibble()
paid_sentence_tags <- tagged |>
filter(doc_id == "s001") |>
select(token, upos, lemma)| Token | POS tag | Lemma |
|---|---|---|
| Paid | PROPN | Paid |
| 12 | NUM | 12 |
| - | PUNCT | - |
| week | NOUN | week |
| training | NOUN | training |
| is | AUX | be |
| provided | VERB | provid |
| . | PUNCT | . |
The sentence has eight tokens after keeping punctuation. Two outputs are wrong in a way a reader can judge. Paid is not a proper noun. It describes the training, and the tagger likely followed the capital letter at the start of the sentence. The lemma provid for provided is not a word. The model guessed a stem from a suffix rule.
Read the accuracy number
This score is for 13 held-out political weblog posts from the same treebank, not for job postings. Held-out means text kept back from training so it can be reserved for scoring. The posts in both treebank excerpts are weblog documents from 2003 to 2006, which strengthens the domain-transfer warning.
train_path <- "data/treebank/en_ewt-train-excerpt.conllu"
held_out_path <- "data/treebank/en_ewt-held-out.conllu"
read_gold_tokens <- function(path) {
udpipe_read_conllu(path) |>
as_tibble() |>
filter(!str_detect(token_id, "[^0-9]"), !is.na(upos))
}
train_newdoc_ids <- read_lines(train_path) |>
str_subset(fixed("# newdoc id = ")) |>
str_remove(fixed("# newdoc id = "))
held_newdoc_ids <- read_lines(held_out_path) |>
str_subset(fixed("# newdoc id = ")) |>
str_remove(fixed("# newdoc id = "))
treebank_years <- str_match(
c(train_newdoc_ids, held_newdoc_ids),
"_ENG_(\\d{4})"
)[, 2] |>
as.integer()
train_gold <- read_gold_tokens(train_path) |>
transmute(token, gold_upos = upos)
held <- read_gold_tokens(held_out_path) |>
transmute(sentence_id, token, gold_upos = upos)
lookup <- train_gold |>
count(token, gold_upos, name = "n") |>
arrange(token, desc(n), gold_upos) |>
group_by(token) |>
slice(1) |>
ungroup() |>
select(token, baseline_upos = gold_upos)
held_vertical <- held |>
group_by(sentence_id) |>
summarise(text = paste(token, collapse = "\n"), .groups = "drop")
held_predictions <- udpipe_annotate(
tagger,
x = held_vertical$text,
doc_id = held_vertical$sentence_id,
tokenizer = "vertical",
tagger = "default",
parser = "none"
) |>
as.data.frame() |>
as_tibble() |>
group_by(doc_id) |>
mutate(token_row = row_number()) |>
ungroup() |>
transmute(
sentence_id = doc_id,
token_row,
tagger_upos = upos
)
scored <- held |>
left_join(lookup, by = "token") |>
mutate(
baseline_upos = coalesce(baseline_upos, "NOUN"),
seen_in_training = token %in% train_gold$token
) |>
group_by(sentence_id) |>
mutate(token_row = row_number()) |>
ungroup() |>
left_join(held_predictions, by = c("sentence_id", "token_row")) |>
mutate(
baseline_correct = baseline_upos == gold_upos,
tagger_correct = tagger_upos == gold_upos,
easy_gold_tag = gold_upos %in% c("PUNCT", "DET")
)
scored_500 <- learning_curve |>
filter(training_sentences == 500L) |>
mutate(
calculated_accuracy = correct_tokens / held_out_tokens,
accuracy_percent = round(calculated_accuracy * 100, 2)
)
learning_curve_display <- learning_curve |>
mutate(
accuracy_percent = round(accuracy * 100, 2)
)
gain_100_to_500 <- learning_curve |>
filter(training_sentences %in% c(100L, 500L)) |>
summarise(
points = diff(accuracy) * 100,
.groups = "drop"
) |>
pull(points)
gain_500_to_2000 <- learning_curve |>
filter(training_sentences %in% c(500L, 2000L)) |>
summarise(
points = diff(accuracy) * 100,
.groups = "drop"
) |>
pull(points)
training_ratio_100_to_500 <- 500L / 100L
training_ratio_500_to_2000 <- 2000L / 500L
accuracy_summary <- scored |>
summarise(
held_out_tokens = n(),
lookup_correct = sum(baseline_correct),
lookup_accuracy = lookup_correct / held_out_tokens,
tagger_correct = sum(tagger_correct),
tagger_accuracy = tagger_correct / held_out_tokens,
gain_points = (tagger_accuracy - lookup_accuracy) * 100,
seen_tokens = sum(seen_in_training),
seen_percent = seen_tokens / held_out_tokens * 100,
easy_tokens = sum(easy_gold_tag),
easy_percent = easy_tokens / held_out_tokens * 100,
.groups = "drop"
)
content_summary <- scored |>
filter(!easy_gold_tag) |>
summarise(
held_out_tokens = n(),
lookup_correct = sum(baseline_correct),
lookup_accuracy = lookup_correct / held_out_tokens,
tagger_correct = sum(tagger_correct),
tagger_accuracy = tagger_correct / held_out_tokens,
.groups = "drop"
)
accuracy_comparison <- tibble(
scope = c(
"All held-out tokens",
"All held-out tokens",
"Excluding PUNCT and DET",
"Excluding PUNCT and DET"
),
system = c(
"Lookup baseline",
"500-sentence tagger",
"Lookup baseline",
"500-sentence tagger"
),
correct = c(
accuracy_summary$lookup_correct,
accuracy_summary$tagger_correct,
content_summary$lookup_correct,
content_summary$tagger_correct
),
tokens = c(
accuracy_summary$held_out_tokens,
accuracy_summary$held_out_tokens,
content_summary$held_out_tokens,
content_summary$held_out_tokens
),
accuracy = round(correct / tokens, 4)
)| Training sentences | Training tokens | Held-out tokens | Correct tokens | Accuracy | Accuracy percent |
|---|---|---|---|---|---|
| 100 | 2311 | 4007 | 3301 | 0.8238 | 82.38 |
| 250 | 5539 | 4007 | 3496 | 0.8725 | 87.25 |
| 500 | 11595 | 4007 | 3641 | 0.9087 | 90.87 |
| 1000 | 21857 | 4007 | 3720 | 0.9284 | 92.84 |
| 2000 | 39802 | 4007 | 3778 | 0.9429 | 94.29 |
| Scope | System | Correct | Tokens | Accuracy |
|---|---|---|---|---|
| All held-out tokens | Lookup baseline | 3178 | 4007 | 0.7931 |
| All held-out tokens | 500-sentence tagger | 3641 | 4007 | 0.9087 |
| Excluding PUNCT and DET | Lookup baseline | 2307 | 3123 | 0.7387 |
| Excluding PUNCT and DET | 500-sentence tagger | 2766 | 3123 | 0.8857 |
The lookup baseline gives each word its most frequent tag in the 500 training sentences and guesses NOUN for words it has not seen. It gets 3,178 of 4,007 tokens right, or 0.7931. The trained tagger gets 3,641 of 4,007 right, or about 91 percent, which buys about 11.6 points over the lookup table. In the held-out posts, 3,140 of 4,007 tokens, or 78.4 percent, are words that also appear in the training sentences.
The easy tags matter. PUNCT and DET are 884 of the 4,007 held-out tokens, or 22.1 percent. Excluding them, the trained tagger scores 2,766 of 3,123, or 0.8857, and the lookup baseline scores 2,307 of 3,123, or 0.7387. The easy tokens flatter both systems.
On this one learning curve, accuracy rises quickly at first and then more slowly. Moving from 100 to 500 training sentences buys about 8.5 points. Moving from 500 to 2,000 buys about 3.4 more. The curve cannot separate more labeled sentences from broader topic coverage, because each step adds new blog posts as well as new sentences. The intervals are not the same scale: 100 to 500 is a five-fold increase, while 500 to 2,000 is a four-fold increase.
Keep the limits attached
The score is one number from one run over thirteen blog posts. Tokens inside a post are not independent, because vocabulary, topic, and author repeat within the post. The real spread is wider than the token count suggests. The builder script sets set.seed(20260828). The RNG and R-version rows above describe the current rerun environment, not the build-time environment of the committed model. This page still reports the bundled run only and does not add a confidence interval, because it does not retrain or rescore on new samples or domains.
The Riverton sentences are job postings and flyer lines, a different kind of writing. This lesson does not claim the same rate there. On the Riverton text, we can point to individual mistakes a reader can judge.
Universal POS tags are a useful set of bins for this task. Other projects can choose a different tagset.
What to remember
- A part-of-speech tagger assigns a grammatical tag to each token.
- Universal POS tags are compact labels such as
NOUN,VERB, andADJ. - The local tagger labeled 178 Riverton tokens.
- Its held-out weblog-post accuracy is 3,641 of 4,007 tokens, or about 91 percent.
- Job-posting errors must be checked directly; the weblog-post score does not transfer by promise.
The team can use the tagger to make a first action-word list, then inspect the mistakes before treating the labels as evidence.