Finding relations between named things

Extract directed triples without treating co-occurrence as evidence

systems
relation extraction
workforce research
Learn how a small dependency-pattern extractor turns linked Riverton mentions into relation triples with status and evidence spans.

Tomas has a stack of short notices and one question: which organisation offers which credential, and which credential does a job require? A word search gives him fragments. It finds required, but it cannot tell whether the sentence says a licence is required, preferred, or not required.

Relation extraction finds a statement that two entities stand in a named, directed relationship. The output is a triple such as ORG-0002 offers CRD-0002, plus a status and a pointer back to the sentence that said it.

This lesson keeps the example small. It links mentions with the Riverton alias table, inspects the parser links, and compares the extractor with a co-occurrence baseline that only knows two mentions shared a sentence.

Note

The Riverton Workforce Lab, its job board, and its training flyer are fictional and were created for teaching.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • define a relation schema before extracting triples;
  • keep entity IDs, direction, status, and offsets with each triple;
  • explain why two linked mentions in one sentence are not automatically related;
  • mark negated and hedged statements separately from asserted ones;
  • compare pair detection with triple extraction without mixing their units; and
  • say why a same-author reference is only a demonstration.

Define the relation schema

A schema says which relation types the extractor is allowed to produce. It also says which entity type can appear on each side. This matters because direction is part of the claim: an organisation offers a credential; the credential does not offer the organisation.

library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(purrr)
library(stringr)
library(digest)
library(spacyr)
source("R/use-spacy.R")

relation_schema <- tribble(
  ~relation, ~subject_type, ~object_type, ~direction, ~status_values, ~what_counts,
  "offers", "organisation", "credential",
  "organisation -> credential", "asserted, negated, hedged",
  "The sentence says the organisation offers, may offer, or does not offer the credential.",
  "requires", "organisation", "credential",
  "organisation -> credential", "asserted, negated, hedged",
  "The sentence says the organisation requires, may require, or does not require the credential."
)
Relation schema used before any extraction output
Relation Subject type Object type Direction Status values What counts
offers organisation credential organisation -> credential asserted, negated, hedged The sentence says the organisation offers, may offer, or does not offer the credential.
requires organisation credential organisation -> credential asserted, negated, hedged The sentence says the organisation requires, may require, or does not require the credential.

The schema is small on purpose. It does not try to extract every fact in a job notice. It only checks two relations that later lessons can put into a knowledge base.

Read the source records and aliases

The 28 Riverton sentences are the verbatim source records used in earlier lessons. The alias table supplies IDs for organisations, places, and credentials. The extractor uses those IDs for arguments; named entity recognition is not used to choose the arguments.

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

entities <- read_csv(
  "data/riverton/riverton-entities.csv",
  na = character(),
  col_types = cols(
    entity_id = col_character(),
    canonical_name = col_character(),
    entity_type = col_character(),
    description = col_character()
  )
) |>
  select(entity_id, canonical_name, entity_type)

aliases <- read_csv(
  "data/riverton/riverton-aliases.csv",
  na = character(),
  col_types = cols(
    entity_id = col_character(),
    alias = col_character()
  )
) |>
  left_join(entities, by = "entity_id")

metadata <- read_csv(
  "data/riverton/riverton-reference-metadata.csv",
  na = character(),
  col_types = cols(
    artifact = col_character(),
    description = col_character(),
    source = col_character(),
    license = col_character(),
    created_on = col_character(),
    fingerprint = col_character()
  )
)

alias_hash <- digest(
  paste(read_lines("data/riverton/riverton-aliases.csv"), collapse = "\n"),
  algo = "sha256",
  serialize = FALSE
)
Riverton alias table used to find relation arguments
Entity ID Canonical name Type Alias
CRD-0001 Forklift Operator Licence credential Forklift Operator Licence
CRD-0001 Forklift Operator Licence credential forklift certification
CRD-0002 Data Support Certificate credential DATA SUPPORT CERTIFICATE
CRD-0002 Data Support Certificate credential Data Support Certificate
LOC-0001 Riverton place Riverton
LOC-0001 Riverton place Riverton, Calder
LOC-0002 Riverton place Riverton
LOC-0002 Riverton place Riverton, Tidewater
LOC-0003 Bellhaven place Bellhaven
ORG-0001 Riverton Workforce Lab organisation Riverton Workforce Lab
ORG-0001 Riverton Workforce Lab organisation Workforce Lab
ORG-0001 Riverton Workforce Lab organisation the Lab
ORG-0002 Riverton Skills Centre organisation RSC
ORG-0002 Riverton Skills Centre organisation Riverton Skills Centre
ORG-0002 Riverton Skills Centre organisation Skills Centre
ORG-0003 Marrow County Transit organisation MCT
ORG-0003 Marrow County Transit organisation Marrow County Transit

The table includes two rows for the alias Riverton, which is an ambiguity the lesson leaves unresolved. For relation arguments, the examples below use organisation and credential aliases that link to one ID.

Inspect the 28 verbatim sentences once

The job-board and flyer lines are useful for inspection, but they do not contain two distinct linked entity mentions in one sentence. The table below is the only place this lesson prints all 28 source sentences. It shows why none of these lines can produce complete relation triples.

The matcher behind the table looks for every alias in each sentence. It tries longer aliases first, so Riverton Skills Centre is claimed as one organisation before Riverton alone can match inside it, and it records the first and last character of every match. An alias shared by two IDs is kept and marked ambiguous rather than resolved by guessing.

Show the code that finds alias mentions 
escape_regex <- function(value) {
  str_replace_all(value, "([.\\\\|()\\[\\]{}^$*+?])", "\\\\\\1")
}

find_mentions <- function(document_id, sentence_id, text) {
  candidates <- aliases |>
    mutate(normalized_alias = str_to_lower(alias, locale = "en")) |>
    group_by(normalized_alias, alias) |>
    summarise(
      candidate_count = n_distinct(entity_id),
      entity_id = paste(sort(unique(entity_id)), collapse = "; "),
      canonical_name = paste(sort(unique(canonical_name)), collapse = "; "),
      entity_type = paste(sort(unique(entity_type)), collapse = "; "),
      .groups = "drop"
    ) |>
    mutate(alias_length = nchar(alias)) |>
    arrange(desc(alias_length), alias)

  kept <- vector("list", nrow(candidates))
  used <- matrix(numeric(0), ncol = 2)
  kept_index <- 0L

  for (row in seq_len(nrow(candidates))) {
    pattern <- regex(
      paste0("(?<![[:alnum:]])", escape_regex(candidates$alias[[row]]), "(?![[:alnum:]])"),
      ignore_case = TRUE
    )
    locations <- str_locate_all(text, pattern)[[1]]

    if (nrow(locations) == 0L) {
      next
    }

    for (hit in seq_len(nrow(locations))) {
      span <- locations[hit, ]
      overlaps <- nrow(used) > 0L &&
        any(span[1] <= used[, 2] & span[2] >= used[, 1])

      if (!overlaps) {
        kept_index <- kept_index + 1L
        kept[[kept_index]] <- tibble(
          document_id = document_id,
          sentence_id = sentence_id,
          entity_id = candidates$entity_id[[row]],
          canonical_name = candidates$canonical_name[[row]],
          entity_type = candidates$entity_type[[row]],
          link_status = if_else(candidates$candidate_count[[row]] > 1L, "ambiguous", "linked"),
          mention = str_sub(text, span[1], span[2]),
          start = span[1],
          end = span[2]
        )
        used <- rbind(used, span)
      }
    }
  }

  if (kept_index == 0L) {
    return(tibble(
      document_id = character(),
      sentence_id = character(),
      entity_id = character(),
      canonical_name = character(),
      entity_type = character(),
      link_status = character(),
      mention = character(),
      start = integer(),
      end = integer()
    ))
  }

  kept[seq_len(kept_index)] |>
    list_rbind() |>
    arrange(start, end, entity_id)
}

verbatim_mentions <- pmap(
  sentences |>
    select(document_id, sentence_id, text),
  find_mentions
) |>
  list_rbind()

verbatim_inspection <- sentences |>
  mutate(
    linked_mentions = map_chr(
      sentence_id,
      \(id) {
        rows <- verbatim_mentions |>
          filter(sentence_id == id)

        if (nrow(rows) == 0L) {
          "none"
        } else {
          paste(
            paste0(
              rows$mention,
              " (",
              rows$entity_id,
              if_else(rows$link_status == "ambiguous", ", ambiguous", ""),
              ")"
            ),
            collapse = "; "
          )
        }
      }
    ),
    schema_cue = case_when(
      str_detect(str_to_lower(text), "\\boffer|\\bprovided|\\bincluded|\\bpays") ~ "offer-like wording",
      str_detect(str_to_lower(text), "\\brequired|\\bmust|\\bneed") ~ "require-like wording",
      TRUE ~ "none"
    ),
    polarity_read_by_hand = case_when(
      sentence_id %in% c("s002", "s027") ~ "negated",
      TRUE ~ "asserted or not applicable"
    ),
    modality_read_by_hand = case_when(
      sentence_id %in% c("s010", "s013") ~ "preferred, not asserted as required",
      uncertainty == "needs_review" ~ "needs review",
      TRUE ~ "plain"
    )
  ) |>
  select(
    sentence_id,
    document_id,
    text,
    linked_mentions,
    schema_cue,
    polarity_read_by_hand,
    modality_read_by_hand
  )
Inspection table for all 28 verbatim Riverton sentences
Sentence ID Document ID Text Linked mentions Schema cue Polarity read by hand Modality read by hand
s001 J001 Paid 12-week training is provided. none offer-like wording asserted or not applicable plain
s002 J001 No prior data experience is required. none require-like wording negated plain
s003 J001 Evening schedules are available. none none asserted or not applicable plain
s004 J001 Applicants need basic spreadsheet skills. none require-like wording asserted or not applicable plain
s005 J002 A high school diploma is required. none require-like wording asserted or not applicable plain
s006 J002 The employer pays for certification training. none offer-like wording asserted or not applicable plain
s007 J002 Rotating night shifts are part of the job. none none asserted or not applicable plain
s008 J002 Workers must be able to lift 50 pounds. none require-like wording asserted or not applicable plain
s009 J003 A portfolio is required. none require-like wording asserted or not applicable plain
s010 J003 Six months of experience is preferred. none none asserted or not applicable preferred, not asserted as required
s011 J003 Remote work is available two days each week. none none asserted or not applicable plain
s012 J003 Each new hire receives a mentor. none none asserted or not applicable plain
s013 J004 A medical records certificate is preferred. none none asserted or not applicable preferred, not asserted as required
s014 J004 The position uses a daytime schedule. none none asserted or not applicable plain
s015 J004 On-the-job training is provided. none offer-like wording asserted or not applicable plain
s016 J005 This is a paid apprenticeship. none none asserted or not applicable plain
s017 J005 A valid driver's license is required. none require-like wording asserted or not applicable plain
s018 J005 The work is outdoors and includes local travel. none none asserted or not applicable plain
s019 J006 Two years of customer service experience are required. none require-like wording asserted or not applicable plain
s020 J006 Weekend shifts are required. none require-like wording asserted or not applicable needs review
s021 J006 Product training is included. none offer-like wording asserted or not applicable plain
s022 J006 Clear written communication is an essential skill. none none asserted or not applicable plain
s023 F001 RIVERTON SKILLS OPEN HOUSE RIVERTON (LOC-0001; LOC-0002, ambiguous) none asserted or not applicable plain
s024 F001 DATA SUPPORT CERTIFICATE DATA SUPPORT CERTIFICATE (CRD-0002) none asserted or not applicable needs review
s025 F001 Paid training stipend none none asserted or not applicable plain
s026 F001 Evening classes none none asserted or not applicable plain
s027 F001 No prior experience required none require-like wording negated plain
s028 F001 Apply by October 15 none none asserted or not applicable plain

The important negative result is concrete: these 28 lines do not give the extractor complete entity pairs. A page or posting could supply a missing employer, but that would be a page-structure assumption, not sentence evidence; this lesson does not use that assumption. The polarity and modality columns are read by hand so the table can flag lines worth inspecting.

Write a same-author reference before extraction

The next examples are extractor test sentences. They are constructed to test the parser patterns and are not claims about the Riverton handbook or knowledge base. The reference below is written before the extractor runs. The same author wrote the sentences, the reference, and the rules, so the score is only a demonstration.

constructed_sentences <- tribble(
  ~sentence_id, ~document_id, ~source, ~text, ~teaching_case,
  "c01", "T001", "extractor test sentence", "Riverton Skills Centre offers the Data Support Certificate.", "active offer",
  "c02", "T002", "extractor test sentence", "The Data Support Certificate is offered by Riverton Skills Centre.", "passive offer",
  "c03", "T003", "extractor test sentence", "Riverton Skills Centre does not offer the Forklift Operator Licence.", "verb negation",
  "c05", "T005", "extractor test sentence", "Riverton Skills Centre offers the Data Support Certificate and the Forklift Operator Licence.", "coordinated objects",
  "c06", "T006", "extractor test sentence", "Riverton Skills Centre may offer the Forklift Operator Licence next year.", "hedged offer",
  "c07", "T007", "extractor test sentence", "Marrow County Transit runs a bus to Riverton Skills Centre.", "same-sentence distractor",
  "c08", "T008", "extractor test sentence", "The Workforce Lab counted job ads that require the Forklift Operator Licence.", "relative-clause distractor",
  "c09", "T009", "extractor test sentence", "Marrow County Transit no longer requires the Forklift Operator Licence.", "negation under an adverb",
  "c11", "T011", "extractor test sentence", "No Forklift Operator Licence is required by Marrow County Transit.", "determiner no on argument",
  "c13", "T013", "extractor test sentence", "Marrow County Transit is reported to require the Forklift Operator Licence.", "reported requirement"
)

reference_triples <- tribble(
  ~sentence_id, ~relation, ~subject_id, ~object_id, ~status,
  "c01", "offers", "ORG-0002", "CRD-0002", "asserted",
  "c02", "offers", "ORG-0002", "CRD-0002", "asserted",
  "c03", "offers", "ORG-0002", "CRD-0001", "negated",
  "c05", "offers", "ORG-0002", "CRD-0002", "asserted",
  "c05", "offers", "ORG-0002", "CRD-0001", "asserted",
  "c06", "offers", "ORG-0002", "CRD-0001", "hedged",
  "c09", "requires", "ORG-0003", "CRD-0001", "negated",
  "c11", "requires", "ORG-0003", "CRD-0001", "negated",
  "c13", "requires", "ORG-0003", "CRD-0001", "hedged"
)
Constructed extractor test sentences, not Riverton facts
Sentence ID Document ID Source Extractor test sentence Teaching case
c01 T001 extractor test sentence Riverton Skills Centre offers the Data Support Certificate. active offer
c02 T002 extractor test sentence The Data Support Certificate is offered by Riverton Skills Centre. passive offer
c03 T003 extractor test sentence Riverton Skills Centre does not offer the Forklift Operator Licence. verb negation
c05 T005 extractor test sentence Riverton Skills Centre offers the Data Support Certificate and the Forklift Operator Licence. coordinated objects
c06 T006 extractor test sentence Riverton Skills Centre may offer the Forklift Operator Licence next year. hedged offer
c07 T007 extractor test sentence Marrow County Transit runs a bus to Riverton Skills Centre. same-sentence distractor
c08 T008 extractor test sentence The Workforce Lab counted job ads that require the Forklift Operator Licence. relative-clause distractor
c09 T009 extractor test sentence Marrow County Transit no longer requires the Forklift Operator Licence. negation under an adverb
c11 T011 extractor test sentence No Forklift Operator Licence is required by Marrow County Transit. determiner no on argument
c13 T013 extractor test sentence Marrow County Transit is reported to require the Forklift Operator Licence. reported requirement
Same-author reference triples written before the extractor runs
Sentence ID Relation Subject ID Object ID Status
c01 offers ORG-0002 CRD-0002 asserted
c02 offers ORG-0002 CRD-0002 asserted
c03 offers ORG-0002 CRD-0001 negated
c05 offers ORG-0002 CRD-0002 asserted
c05 offers ORG-0002 CRD-0001 asserted
c06 offers ORG-0002 CRD-0001 hedged
c09 requires ORG-0003 CRD-0001 negated
c11 requires ORG-0003 CRD-0001 negated
c13 requires ORG-0003 CRD-0001 hedged

The reference has no row for the bus sentence or the job-ad counting sentence. Both contain more than one linked mention, but neither states one of the schema relations between those mentions. The test-sentence IDs skip some numbers; no test sentence was removed after the extractor ran.

Parse and align mentions to tokens

spaCy supplies dependency links. The code aligns each alias-table mention to the tokens covered by its character offsets, then chooses the token whose head points outside the mention as the mention head.

pipeline <- use_project_spacy()
pipeline_info <- spacy_pipeline_version()

parsed <- spacy_parse(
  setNames(constructed_sentences$text, constructed_sentences$sentence_id),
  pos = TRUE,
  lemma = TRUE,
  entity = TRUE,
  dependency = TRUE,
  additional_attributes = "idx"
) |>
  as_tibble() |>
  mutate(
    token_id = as.integer(token_id),
    head_token_id = as.integer(head_token_id),
    token_start = idx + 1L,
    token_end = idx + nchar(token)
  )

constructed_mentions <- pmap(
  constructed_sentences |>
    select(document_id, sentence_id, text),
  find_mentions
) |>
  list_rbind()

head_for_mention <- function(current_sentence_id, current_start, current_end) {
  covered <- parsed |>
    filter(
      doc_id == current_sentence_id,
      token_start >= current_start,
      token_end <= current_end
    )

  external <- covered |>
    filter(!head_token_id %in% covered$token_id)

  if (nrow(external) == 0L) {
    max(covered$token_id)
  } else {
    external$token_id[[1]]
  }
}

mention_heads <- constructed_mentions |>
  mutate(
    head_token_id = pmap_int(
      list(sentence_id, start, end),
      head_for_mention
    )
  ) |>
  ungroup()

mention_head_table <- mention_heads |>
  left_join(
    parsed |>
      transmute(sentence_id = doc_id, head_token_id = token_id, head_token = token, head_dep = dep_rel),
    by = c("sentence_id", "head_token_id")
  ) |>
  select(sentence_id, entity_id, mention, start, end, head_token, head_dep)
Alias-table mentions aligned to spaCy dependency heads
Sentence ID Entity ID Mention Start End Mention head Head relation
c01 ORG-0002 Riverton Skills Centre 1 22 Centre nsubj
c01 CRD-0002 Data Support Certificate 35 58 Certificate dobj
c02 CRD-0002 Data Support Certificate 5 28 Certificate nsubjpass
c02 ORG-0002 Riverton Skills Centre 44 65 Centre pobj
c03 ORG-0002 Riverton Skills Centre 1 22 Centre nsubj
c03 CRD-0001 Forklift Operator Licence 43 67 Licence dobj
c05 ORG-0002 Riverton Skills Centre 1 22 Centre nsubj
c05 CRD-0002 Data Support Certificate 35 58 Certificate dobj
c05 CRD-0001 Forklift Operator Licence 68 92 Licence conj
c06 ORG-0002 Riverton Skills Centre 1 22 Centre nsubj
c06 CRD-0001 Forklift Operator Licence 38 62 Licence dobj
c07 ORG-0003 Marrow County Transit 1 21 Transit nsubj
c07 ORG-0002 Riverton Skills Centre 37 58 Centre pobj
c08 ORG-0001 Workforce Lab 5 17 Lab nsubj
c08 CRD-0001 Forklift Operator Licence 52 76 Licence dobj
c09 ORG-0003 Marrow County Transit 1 21 Transit nsubj
c09 CRD-0001 Forklift Operator Licence 46 70 Licence dobj
c11 CRD-0001 Forklift Operator Licence 4 28 Licence nsubjpass
c11 ORG-0003 Marrow County Transit 45 65 Transit pobj
c13 ORG-0003 Marrow County Transit 1 21 Transit nsubjpass
c13 CRD-0001 Forklift Operator Licence 50 74 Licence dobj

The offsets are one-based character positions in the unmodified sentence text. That lets the triple point back to the exact evidence instead of to a rebuilt token string.

In the head-relation column, nsubj marks an active subject, dobj a direct object, nsubjpass a passive subject, pobj an object of a preposition, and conj a coordinated item. Lesson 18 introduces dependency parses.

Extract directed triples

The extractor looks for offer and require triggers. It handles active subjects, passive agents, coordinated objects, not on a verb, No on an argument, no longer under an adverb, and hedging from may. The reported requirement is a visible miss for this small pattern set.

Show the extractor code 
trigger_tokens <- parsed |>
  filter(lemma %in% c("offer", "require"), pos == "VERB") |>
  transmute(
    sentence_id = doc_id,
    trigger_token_id = token_id,
    trigger = token,
    trigger_lemma = lemma,
    trigger_dep = dep_rel,
    trigger_head_token_id = head_token_id
  )

argument_roles <- mention_heads |>
  left_join(
    parsed |>
      transmute(sentence_id = doc_id, token_head_id = head_token_id, head_token_id = token_id, dep_rel),
    by = c("sentence_id", "head_token_id")
  ) |>
  select(
    sentence_id, document_id, entity_id, canonical_name, entity_type,
    mention, start, end, head_token_id, dep_rel, token_head_id
  )

object_heads <- argument_roles |>
  filter(entity_type == "credential") |>
  select(
    sentence_id,
    object_id = entity_id,
    object_name = canonical_name,
    object_mention = mention,
    object_start = start,
    object_end = end,
    object_head_id = head_token_id,
    object_dep = dep_rel,
    object_token_head_id = token_head_id
  )

active_triples <- trigger_tokens |>
  inner_join(
    argument_roles |>
      filter(entity_type == "organisation", dep_rel == "nsubj") |>
      select(
        sentence_id,
        subject_id = entity_id,
        subject_name = canonical_name,
        subject_mention = mention,
        subject_start = start,
        subject_end = end,
        subject_head_id = head_token_id,
        subject_token_head_id = token_head_id
      ),
    by = c("sentence_id", "trigger_token_id" = "subject_token_head_id")
  ) |>
  inner_join(
    object_heads |>
      filter(object_dep %in% c("dobj", "conj")),
    by = "sentence_id",
    relationship = "many-to-many"
  ) |>
  filter(
    pmap_lgl(
      list(sentence_id, object_dep, object_token_head_id, trigger_token_id),
      \(current_sentence_id, current_object_dep, current_object_token_head_id, current_trigger_token_id) {
        direct_object_heads <- object_heads$object_head_id[
          object_heads$sentence_id == current_sentence_id &
            object_heads$object_token_head_id == current_trigger_token_id
        ]

        current_object_token_head_id == current_trigger_token_id ||
          (current_object_dep == "conj" &&
            current_object_token_head_id %in% direct_object_heads)
      }
    )
  )

passive_agents <- parsed |>
  filter(dep_rel == "agent") |>
  select(sentence_id = doc_id, agent_token_id = token_id, trigger_token_id = head_token_id)

passive_triples <- trigger_tokens |>
  inner_join(
    object_heads |>
      filter(object_dep == "nsubjpass"),
    by = c("sentence_id", "trigger_token_id" = "object_token_head_id")
  ) |>
  inner_join(passive_agents, by = c("sentence_id", "trigger_token_id")) |>
  inner_join(
    argument_roles |>
      filter(entity_type == "organisation", dep_rel == "pobj") |>
      select(
        sentence_id,
        subject_id = entity_id,
        subject_name = canonical_name,
        subject_mention = mention,
        subject_start = start,
        subject_end = end,
        subject_head_id = head_token_id,
        subject_token_head_id = token_head_id
      ),
    by = c("sentence_id", "agent_token_id" = "subject_token_head_id")
  )

raw_triples <- bind_rows(
  active_triples,
  passive_triples
) |>
  mutate(
    relation = recode(trigger_lemma, offer = "offers", require = "requires")
  )

status_for_trigger <- function(current_sentence_id, trigger_token_id, subject_head_id, object_head_id) {
  sentence_tokens <- parsed |>
    filter(doc_id == current_sentence_id)

  verb_negated <- any(
    sentence_tokens$head_token_id == trigger_token_id &
      sentence_tokens$dep_rel == "neg"
  )

  advmod_ids <- sentence_tokens$token_id[
    sentence_tokens$head_token_id == trigger_token_id &
      sentence_tokens$dep_rel == "advmod"
  ]
  adverb_negated <- any(
    sentence_tokens$head_token_id %in% advmod_ids &
      sentence_tokens$dep_rel == "neg"
  )

  argument_det_no <- any(
    sentence_tokens$head_token_id %in% c(subject_head_id, object_head_id) &
      sentence_tokens$dep_rel == "det" &
      str_to_lower(sentence_tokens$lemma) == "no"
  )

  hedged_aux <- any(
    sentence_tokens$head_token_id == trigger_token_id &
      sentence_tokens$dep_rel == "aux" &
      str_to_lower(sentence_tokens$lemma) %in% c("may", "might", "could")
  )

  trigger_row <- sentence_tokens |>
    filter(token_id == trigger_token_id)
  reported_parent <- nrow(trigger_row) == 1L &&
    trigger_row$dep_rel == "xcomp" &&
    sentence_tokens$lemma[
      match(trigger_row$head_token_id, sentence_tokens$token_id)
    ] %in% c("report", "say")

  case_when(
    verb_negated | adverb_negated | argument_det_no ~ "negated",
    hedged_aux | reported_parent ~ "hedged",
    TRUE ~ "asserted"
  )
}

extracted_triples <- raw_triples |>
  mutate(
    status = pmap_chr(
      list(sentence_id, trigger_token_id, subject_head_id, object_head_id),
      status_for_trigger
    )
  ) |>
  left_join(
    constructed_sentences |>
      select(sentence_id, document_id, text),
    by = "sentence_id"
  ) |>
  mutate(
    evidence_start = 1L,
    evidence_end = nchar(text),
    evidence_text = str_sub(text, evidence_start, evidence_end)
  ) |>
  select(
    sentence_id,
    document_id,
    relation,
    subject_id,
    object_id,
    status,
    subject_start,
    subject_end,
    object_start,
    object_end,
    evidence_start,
    evidence_end,
    evidence_text
  ) |>
  arrange(sentence_id, relation, subject_id, object_id, status)
Extracted triples with status and evidence offsets
Sentence ID Document ID Relation Subject ID Object ID Status Subject start Subject end Object start Object end Evidence start Evidence end Evidence text
c01 T001 offers ORG-0002 CRD-0002 asserted 1 22 35 58 1 59 Riverton Skills Centre offers the Data Support Certificate.
c02 T002 offers ORG-0002 CRD-0002 asserted 44 65 5 28 1 66 The Data Support Certificate is offered by Riverton Skills Centre.
c03 T003 offers ORG-0002 CRD-0001 negated 1 22 43 67 1 68 Riverton Skills Centre does not offer the Forklift Operator Licence.
c05 T005 offers ORG-0002 CRD-0001 asserted 1 22 68 92 1 93 Riverton Skills Centre offers the Data Support Certificate and the Forklift Operator Licence.
c05 T005 offers ORG-0002 CRD-0002 asserted 1 22 35 58 1 93 Riverton Skills Centre offers the Data Support Certificate and the Forklift Operator Licence.
c06 T006 offers ORG-0002 CRD-0001 hedged 1 22 38 62 1 73 Riverton Skills Centre may offer the Forklift Operator Licence next year.
c09 T009 requires ORG-0003 CRD-0001 negated 1 21 46 70 1 71 Marrow County Transit no longer requires the Forklift Operator Licence.
c11 T011 requires ORG-0003 CRD-0001 negated 45 65 4 28 1 66 No Forklift Operator Licence is required by Marrow County Transit.

The negation check is deliberately broader than “look for not under the verb.” It catches no longer through the adverb and No Forklift Operator Licence through the determiner on the credential argument.

Compare pairs separately from triples

The co-occurrence baseline pairs every two distinct linked mentions in a constructed sentence. It does not assign a relation type or direction, so those fields are not applicable. The extractor is scored twice: once for entity-pair detection and once for labelled triples.

Show the pair and triple scoring code 
cooccurrence_pairs <- constructed_mentions |>
  select(sentence_id, document_id, entity_id) |>
  distinct() |>
  group_by(sentence_id, document_id) |>
  filter(n() >= 2L) |>
  summarise(
    pairs = list(combn(sort(entity_id), 2, simplify = FALSE)),
    .groups = "drop"
  ) |>
  unnest_longer(pairs) |>
  transmute(
    sentence_id,
    document_id,
    first_id = map_chr(pairs, 1),
    second_id = map_chr(pairs, 2),
    relation = "not applicable",
    direction = "not applicable"
  )

reference_pairs <- reference_triples |>
  transmute(
    sentence_id,
    first_id = pmin(subject_id, object_id),
    second_id = pmax(subject_id, object_id)
  ) |>
  distinct()

extractor_pairs <- extracted_triples |>
  transmute(
    sentence_id,
    first_id = pmin(subject_id, object_id),
    second_id = pmax(subject_id, object_id)
  ) |>
  distinct()

score_pairs <- function(system_pairs, system_name) {
  full_join(
    system_pairs |> mutate(predicted = TRUE),
    reference_pairs |> mutate(reference = TRUE),
    by = c("sentence_id", "first_id", "second_id")
  ) |>
    mutate(
      predicted = coalesce(predicted, FALSE),
      reference = coalesce(reference, FALSE),
      system = system_name,
      result = case_when(
        predicted & reference ~ "true positive",
        predicted & !reference ~ "false positive",
        !predicted & reference ~ "false negative",
        TRUE ~ "true negative"
      )
    ) |>
    filter(result != "true negative")
}

pair_results <- bind_rows(
  score_pairs(cooccurrence_pairs |> select(sentence_id, first_id, second_id), "co-occurrence"),
  score_pairs(extractor_pairs, "dependency extractor")
) |>
  arrange(system, sentence_id, first_id, second_id)

triple_results <- full_join(
  extracted_triples |>
    select(sentence_id, relation, subject_id, object_id, status) |>
    mutate(extractor = TRUE),
  reference_triples |>
    mutate(reference = TRUE),
  by = c("sentence_id", "relation", "subject_id", "object_id", "status")
) |>
  mutate(
    extractor = coalesce(extractor, FALSE),
    reference = coalesce(reference, FALSE),
    cooccurrence_relation = "not applicable",
    cooccurrence_direction = "not applicable",
    result = case_when(
      extractor & reference ~ "true positive",
      extractor & !reference ~ "false positive",
      !extractor & reference ~ "false negative",
      TRUE ~ "true negative"
    )
  ) |>
  filter(result != "true negative") |>
  arrange(sentence_id, relation, subject_id, object_id, status)
Co-occurrence pairs have no relation type or direction
Sentence ID Document ID First ID Second ID Relation Direction
c01 T001 CRD-0002 ORG-0002 not applicable not applicable
c02 T002 CRD-0002 ORG-0002 not applicable not applicable
c03 T003 CRD-0001 ORG-0002 not applicable not applicable
c05 T005 CRD-0001 CRD-0002 not applicable not applicable
c05 T005 CRD-0001 ORG-0002 not applicable not applicable
c05 T005 CRD-0002 ORG-0002 not applicable not applicable
c06 T006 CRD-0001 ORG-0002 not applicable not applicable
c07 T007 ORG-0002 ORG-0003 not applicable not applicable
c08 T008 CRD-0001 ORG-0001 not applicable not applicable
c09 T009 CRD-0001 ORG-0003 not applicable not applicable
c11 T011 CRD-0001 ORG-0003 not applicable not applicable
c13 T013 CRD-0001 ORG-0003 not applicable not applicable
Unlabelled entity-pair results for both systems
Sentence ID First ID Second ID Predicted Reference System Result
c01 CRD-0002 ORG-0002 TRUE TRUE co-occurrence true positive
c02 CRD-0002 ORG-0002 TRUE TRUE co-occurrence true positive
c03 CRD-0001 ORG-0002 TRUE TRUE co-occurrence true positive
c05 CRD-0001 CRD-0002 TRUE FALSE co-occurrence false positive
c05 CRD-0001 ORG-0002 TRUE TRUE co-occurrence true positive
c05 CRD-0002 ORG-0002 TRUE TRUE co-occurrence true positive
c06 CRD-0001 ORG-0002 TRUE TRUE co-occurrence true positive
c07 ORG-0002 ORG-0003 TRUE FALSE co-occurrence false positive
c08 CRD-0001 ORG-0001 TRUE FALSE co-occurrence false positive
c09 CRD-0001 ORG-0003 TRUE TRUE co-occurrence true positive
c11 CRD-0001 ORG-0003 TRUE TRUE co-occurrence true positive
c13 CRD-0001 ORG-0003 TRUE TRUE co-occurrence true positive
c01 CRD-0002 ORG-0002 TRUE TRUE dependency extractor true positive
c02 CRD-0002 ORG-0002 TRUE TRUE dependency extractor true positive
c03 CRD-0001 ORG-0002 TRUE TRUE dependency extractor true positive
c05 CRD-0001 ORG-0002 TRUE TRUE dependency extractor true positive
c05 CRD-0002 ORG-0002 TRUE TRUE dependency extractor true positive
c06 CRD-0001 ORG-0002 TRUE TRUE dependency extractor true positive
c09 CRD-0001 ORG-0003 TRUE TRUE dependency extractor true positive
c11 CRD-0001 ORG-0003 TRUE TRUE dependency extractor true positive
c13 CRD-0001 ORG-0003 FALSE TRUE dependency extractor false negative
Labelled triple results for the dependency extractor
Sentence ID Relation Subject ID Object ID Status Extractor Reference Co-occurrence relation Co-occurrence direction Result
c01 offers ORG-0002 CRD-0002 asserted TRUE TRUE not applicable not applicable true positive
c02 offers ORG-0002 CRD-0002 asserted TRUE TRUE not applicable not applicable true positive
c03 offers ORG-0002 CRD-0001 negated TRUE TRUE not applicable not applicable true positive
c05 offers ORG-0002 CRD-0001 asserted TRUE TRUE not applicable not applicable true positive
c05 offers ORG-0002 CRD-0002 asserted TRUE TRUE not applicable not applicable true positive
c06 offers ORG-0002 CRD-0001 hedged TRUE TRUE not applicable not applicable true positive
c09 requires ORG-0003 CRD-0001 negated TRUE TRUE not applicable not applicable true positive
c11 requires ORG-0003 CRD-0001 negated TRUE TRUE not applicable not applicable true positive
c13 requires ORG-0003 CRD-0001 hedged FALSE TRUE not applicable not applicable false negative

The pair table and triple table answer different questions. Co-occurrence can ask whether the right two IDs appeared together. It cannot answer which relation holds, which way the arrow points, or whether the sentence denied the claim. The reported-verb test sentence is also visible as a false negative for this small dependency-pattern extractor.

Where modern systems fit

Current relation-extraction papers often use large language models, prompts, or trained classifiers. Those systems still need the same schema, argument spans, status policy, and evidence checks. Published comparisons disagree by domain and scoring method, so this page does not make a performance claim about them.

Many current pipelines ask a language model to return each triple as JSON that must match a schema. A schema guarantee checks the format, not the content: a reply can parse perfectly and still name the wrong entity or drop a negation. Asking for the answer before any reasoning can also lower accuracy, so a schema that includes a reasoning field should put it first. Keep this page’s evidence offsets and status column, and score the output against a reference written before the model runs. The guide Using a language model in a reproducible analysis covers the rest of the record-keeping.

What to remember

  • Relation extraction produces typed, directed triples, not loose word matches.
  • The argument IDs in this lesson come from the alias table, not from NER.
  • No on an argument and no longer under an adverb can change triple status.
  • Co-occurrence pairs have no relation type or direction.
  • This same-author score is a demonstration, not a benchmark.
  • Every triple needs sentence, document, and offset evidence.

The safe habit is to treat candidate pairs as leads for review and schema triples as claims that need sentence evidence.

Sources