Finding phrases with readable rules

Score a credential pattern before trusting it

phrases and entities
rule-based matching
workforce research
Learn how simple string rules find credential phrases, how concordances help inspection, and why rules need maintenance.

Credential requirements are easy to miss in a short job-board feed. The Riverton Workforce Lab starts with a readable rule, then checks whether the rule answers the same question a reviewer would ask.

That readability is useful only if the rule is checked. A pattern can find real mentions, miss wording it did not include, and catch lines that use the same word for a different purpose.

Note

The Riverton Workforce Lab, its job board, and its training flyer are fictional and were created for teaching.

TipWhat you will learn

After this lesson, you should be able to:

  • write small phrase rules with stringr;
  • extract the matched credential word;
  • apply a written hand-marking rule for required credentials;
  • count false positives and false negatives without treating the counts as a rate; and
  • use a concordance to inspect matches in context.

Write the first rules

The setup chunk reads the Riverton sentences, builds rule-result tables, handles regular expressions, and prepares a concordance. In stringr, regular expressions use the ICU engine, which defines how boundaries and letter classes behave.

library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(stringr)
library(quanteda)

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

credential_terms <- c("certificate", "certification", "licen[cs]e")
credential_pattern <- regex(
  str_c("\\b(", str_c(credential_terms, collapse = "|"), ")\\b"),
  ignore_case = TRUE
)

rule_results <- sentences |>
  mutate(
    rule_match = str_detect(text, credential_pattern),
    matched_word = str_extract(text, credential_pattern),
    hand_required_credential = sentence_id %in% c("s005", "s017")
  )

matched_rows <- rule_results |>
  filter(rule_match | hand_required_credential) |>
  select(sentence_id, text, matched_word, rule_match, hand_required_credential)
Rule matches and hand markings for required credentials
Sentence ID Text Matched word Rule matched Hand-marked required credential
s005 A high school diploma is required. NA FALSE TRUE
s006 The employer pays for certification training. certification TRUE FALSE
s013 A medical records certificate is preferred. certificate TRUE FALSE
s017 A valid driver’s license is required. license TRUE TRUE
s024 DATA SUPPORT CERTIFICATE CERTIFICATE TRUE FALSE

The rule searches for certificate, certification, and either spelling of license. The lesson author hand-marked each sentence against this written definition: mark the sentence if it says the employer requires a formal qualification issued by a school, licensing body, or certifying organisation, and do not mark a sentence that merely mentions one. Under that rule, s005 and s017 qualify. One reviewer applying one rule to 28 sentences gives a worked example, not a reference standard.

Score the rule

A false positive is a sentence the rule flags but the hand marking rejects. A false negative is a sentence the hand marking accepts but the rule misses. The rule and the hand marking are not asking the same question: the pattern looks for a credential word, while the hand marking asks whether a credential is required. Nothing in the pattern tests for requirement, so it can fire on preferred and on a heading.

score_counts <- rule_results |>
  summarise(
    true_positive = sum(rule_match & hand_required_credential),
    false_positive = sum(rule_match & !hand_required_credential),
    false_negative = sum(!rule_match & hand_required_credential),
    .groups = "drop"
  ) |>
  pivot_longer(
    cols = everything(),
    names_to = "result",
    values_to = "sentences"
  )

false_positive_ids <- rule_results |>
  filter(rule_match & !hand_required_credential) |>
  pull(sentence_id)
false_negative_ids <- rule_results |>
  filter(!rule_match & hand_required_credential) |>
  pull(sentence_id)

problem_examples <- tibble(
  result = c("false positive", "false negative"),
  sentence_id = c("s006", "s005"),
  explanation = c(
    "The line says the employer pays for certification training, not that certification is required.",
    "The line says a diploma is required, but diploma was not in the rule."
  )
)
Sentence-level counts for the first credential rule
Result Sentences
true_positive 1
false_positive 3
false_negative 1
One false positive and one false negative
Result Sentence ID Explanation
false positive s006 The line says the employer pays for certification training, not that certification is required.
false negative s005 The line says a diploma is required, but diploma was not in the rule.

This is a bad rule for required credentials: 3 of its 4 hits are wrong, and it misses 1 of the 2 hand-marked positives. Two positives cannot measure how well a rule works. These counts only show failure modes: certificate and certification can appear without a requirement, while diploma is absent from the pattern.

Inspect matches in context

A concordance is a list of every place a word occurs in a text. The usual layout, called keyword in context, or KWIC, puts the matched word in the middle with a few words on each side.

riverton_corpus <- corpus(setNames(sentences$text, sentences$sentence_id))
riverton_tokens <- tokens(riverton_corpus, remove_punct = TRUE)

credential_kwic <- kwic(
  riverton_tokens,
  pattern = credential_terms,
  window = 4,
  valuetype = "regex",
  case_insensitive = TRUE
) |>
  as.data.frame() |>
  as_tibble() |>
  select(docname, pre, keyword, post)
Concordance lines for the credential rule
Sentence ID Before Keyword After
s006 The employer pays for certification training
s013 A medical records certificate is preferred
s017 A valid driver’s license is required
s024 DATA SUPPORT CERTIFICATE

The KWIC display makes the maintenance problem visible. The token-level regex uses the same credential terms as the scoring pattern: it finds certification training, a certificate that is preferred, a license that is required, and a flyer heading.

What to remember

  • A readable rule can still answer the wrong question.
  • This first credential rule has 1 true positive, 3 false positives, and 1 false negative.
  • Two hand-marked positives are enough for a worked example, not for a performance estimate.
  • KWIC lines help reviewers see why a pattern fired.

Use this pattern only as a review aid. Its next revision should test requirement language before widening the credential word list.

Sources