suppressPackageStartupMessages({
library(dplyr)
library(huggingfaceR)
library(purrr)
library(reticulate)
library(stringr)
library(tibble)
library(tidyr)
})
source("R/use-nlg.R")
paraphrase_model <- load_nlg_pipeline(
"qwen_1_5b_instruct",
"text-generation"
)
model_record <- paraphrase_model$metadata |>
select(model_id, revision, license)Changing the wording without changing the instruction
Separate a real rewrite from a meaning-preservation check
natural language generation
paraphrasing
evaluation
Learn how to generate paraphrases and inspect names, numbers, negation, and modality without treating word overlap as equivalence.
June edits public notices for an archive. Repeating the same sentence in every channel sounds stiff, but a rewrite that changes who may enter or what visitors must bring creates a new instruction.
For this lesson, a paraphrase changes the wording while preserving the participants, quantities, negation, scope, and obligations that matter for the notice. Linguistic research sometimes uses a looser idea of approximate meaning. An operational notice needs a stricter rule because readers may act on one word such as may, must, or not.
The primary aim of paraphrasing is to preserve meaning while changing wording; summarization primarily compresses or selects content, and simplification primarily reduces complexity or register. A real output can serve more than one aim, so these are not mutually exclusive classes.
TipWhat you will learn
This lesson shows how to:
- generate paraphrase candidates with a pinned local model;
- keep an unchanged-input baseline beside the generated wording;
- measure visible wording change without calling it semantic equivalence;
- probe names, numbers, negation, and modality;
- preserve model failures instead of filtering them away; and
- record human meaning review as pending.
Load the same fixed checkpoint
The checkpoint is shared with the summarization lesson, but this page gives it a different task, prompt, baseline, and review rule. The model is loaded from the prepared local directory through huggingfaceR.
| Model | Revision | License |
|---|---|---|
| Qwen/Qwen2.5-1.5B-Instruct | 989aa7980e4cf806f80c7fef2b1adb7bc71aa306 | Apache-2.0 |
Write the preservation rule first
The four notices below are constructed behavioral probes. They were chosen to expose common changes in operational meaning. They are not a random test set, and no population accuracy will be estimated from them.
protected_name, modality_class, and modality_pattern are blank when a probe does not need that check. A copy of the source is kept as the baseline. Copying preserves the original wording and facts, but it does not perform the requested rewrite.
paraphrase_inputs <- tribble(
~probe_id, ~source_text, ~protected_name, ~requires_negation,
~modality_class, ~modality_pattern,
"PAR-01",
"The clinic sent 24 reminder letters on Monday.",
"",
FALSE,
"",
"",
"PAR-02",
"Visitors may bring one notebook to the reading room.",
"",
FALSE,
"permission",
"\\b(may|can|allowed|permitted)\\b",
"PAR-03",
paste(
"Maya Chen did not cancel the 14 September workshop;",
"she moved it from Room 2 to Room 5."
),
"Maya Chen",
TRUE,
"",
"",
"PAR-04",
"Only archive staff must sign the restricted-room log.",
"",
FALSE,
"obligation",
"\\b(must|required|obliged|has to)\\b"
)| Probe ID | Source notice | Protected name | Negation required | Modality class | Accepted surface markers |
|---|---|---|---|---|---|
| PAR-01 | The clinic sent 24 reminder letters on Monday. | FALSE | |||
| PAR-02 | Visitors may bring one notebook to the reading room. | FALSE | permission | may|can|allowed|permitted) | |
| PAR-03 | Maya Chen did not cancel the 14 September workshop; she moved it from Room 2 to Room 5. | Maya Chen | TRUE | ||
| PAR-04 | Only archive staff must sign the restricted-room log. | FALSE | obligation | must|required|obliged|has to) |
Generate candidates without silent truncation
The complete chat-formatted prompt must stay below 192 input tokens. Greedy decoding and a 72-token output cap make the call repeatable, but deterministic generation can still produce a wrong paraphrase.
paraphrase_system <- paste(
"Rewrite the notice in different words.",
"Preserve every participant, quantity, date, negation, scope,",
"permission, and obligation.",
"Return the rewritten notice only."
)
paraphrase_requests <- paraphrase_inputs |>
mutate(
user_prompt = paste0(
"PROBE ID: ", probe_id,
"\nNOTICE: ", source_text
),
model_prompt = map_chr(
user_prompt,
\(user) {
nlg_chat_prompt(
paraphrase_model$tokenizer,
paraphrase_system,
user
)
}
),
input_tokens = map_int(
model_prompt,
\(prompt) nlg_token_count(paraphrase_model$tokenizer, prompt)
)
)| Probe ID | Complete prompt tokens |
|---|---|
| PAR-01 | 68 |
| PAR-02 | 67 |
| PAR-03 | 81 |
| PAR-04 | 67 |
Only after that assertion succeeds does the lesson make any generation call.
paraphrase_results <- paraphrase_requests |>
mutate(
generation = map(
model_prompt,
\(prompt) {
nlg_generate(
paraphrase_model,
prompt,
max_new_tokens = 72L,
repetition_penalty = 1.05
)
}
),
candidate = map_chr(generation, "text"),
candidate_display = str_replace_all(
candidate,
"\\s*\\n+\\s*",
" / "
),
output_tokens = map_int(generation, "output_tokens"),
ended_by_eos = map_lgl(generation, "ended_by_eos"),
hit_token_cap = map_lgl(generation, "hit_token_cap"),
copy_baseline = source_text
)| Probe ID | Source | Generated candidate | Unchanged-input baseline |
|---|---|---|---|
| PAR-01 | The clinic sent 24 reminder letters on Monday. | CLINIC SENT 24 REMINDERS ON MONDAY. | The clinic sent 24 reminder letters on Monday. |
| PAR-02 | Visitors may bring one notebook to the reading room. | Visitors are permitted to bring one notebook into the reading room. | Visitors may bring one notebook to the reading room. |
| PAR-03 | Maya Chen did not cancel the 14 September workshop; she moved it from Room 2 to Room 5. | PROBE ID: PAR-03 / NOTICE: Maya Chen did not cancel the 14 September workshop; instead, she relocated it to Room 5. | Maya Chen did not cancel the 14 September workshop; she moved it from Room 2 to Room 5. |
| PAR-04 | Only archive staff must sign the restricted-room log. | Only archive staff are authorized to sign the restricted-room log. | Only archive staff must sign the restricted-room log. |
The copy baseline should pass every literal preservation check. It also fails the user’s request for different wording. The generated column has to solve both parts.
Measure wording change
This diagnostic lowercases each string, extracts letter and number sequences with the ICU regular-expression engine used by stringr, and computes Jaccard overlap on the unique token sets. An overlap of 1 means the sets match. A lower value means some visible wording changed.
token_set <- function(text) {
str_extract_all(
str_to_lower(text, locale = "en"),
"[\\p{L}\\p{N}]+"
)[[1]] |>
unique()
}
jaccard_overlap <- function(left, right) {
left_tokens <- token_set(left)
right_tokens <- token_set(right)
length(intersect(left_tokens, right_tokens)) /
length(union(left_tokens, right_tokens))
}
wording_diagnostics <- paraphrase_results |>
transmute(
probe_id,
exact_copy = candidate == source_text,
unique_token_jaccard = map2_dbl(
source_text,
candidate,
jaccard_overlap
)
)| Probe ID | Exact copy | Unique-token Jaccard overlap |
|---|---|---|
| PAR-01 | FALSE | 0.56 |
| PAR-02 | FALSE | 0.67 |
| PAR-03 | FALSE | 0.58 |
| PAR-04 | FALSE | 0.67 |
A low overlap can come from a good synonym or a damaging change. A high overlap can hide a removed not. The score describes token sets, not equivalence.
Check visible invariants
The code now looks for exact Arabic numerals, the protected name, an explicit negation marker, and a short list of surface markers for the predeclared modality class. These conservative screens catch some preventable mistakes. They also miss valid alternatives: 24 may become twenty-four, and must may become have to. A screen result cannot establish whether the meaning survived.
extract_numbers <- function(text) {
str_extract_all(text, "\\b\\d+\\b")[[1]] |>
sort()
}
has_negation <- function(text) {
str_detect(
text,
regex(
"\\b(no|not|never|without|cannot|can't|didn't|did not)\\b",
ignore_case = TRUE
)
)
}
invariant_checks <- paraphrase_results |>
mutate(
numbers_preserved = map2_lgl(
source_text,
candidate,
\(source, output) {
source_numbers <- extract_numbers(source)
if (length(source_numbers) == 0L) {
NA
} else {
identical(source_numbers, extract_numbers(output))
}
}
),
name_present = map2_lgl(
protected_name,
candidate,
\(name, output) {
if (!nzchar(name)) {
NA
} else {
str_detect(output, fixed(name))
}
}
),
negation_visible = map2_lgl(
requires_negation,
candidate,
\(required, output) {
if (!required) NA else has_negation(output)
}
),
modal_visible = map2_lgl(
modality_pattern,
candidate,
\(pattern, output) {
if (!nzchar(pattern)) {
NA
} else {
str_detect(
output,
regex(pattern, ignore_case = TRUE)
)
}
}
)
) |>
select(
probe_id,
numbers_preserved,
name_present,
negation_visible,
modal_visible
)
invariant_check_display <- invariant_checks |>
mutate(
across(
c(
numbers_preserved,
name_present,
negation_visible,
modal_visible
),
nlg_screen_label
)
)
screen_actions <- invariant_checks |>
rowwise() |>
mutate(
applicable_screens = sum(!is.na(c_across(c(
numbers_preserved,
name_present,
negation_visible,
modal_visible
)))),
flagged_screens = sum(c_across(c(
numbers_preserved,
name_present,
negation_visible,
modal_visible
)) %in% FALSE)
) |>
ungroup() |>
select(probe_id, applicable_screens, flagged_screens)
paraphrase_actions <- wording_diagnostics |>
left_join(screen_actions, by = "probe_id") |>
mutate(
action = case_when(
exact_copy ~ "revise wording",
flagged_screens > 0L ~ "hold for human meaning review",
TRUE ~ "send to human meaning review"
)
) |>
select(probe_id, flagged_screens, action)| Probe ID | Arabic-numeral screen | Protected-name screen | Negation screen | Modality screen |
|---|---|---|---|---|
| PAR-01 | not flagged by screen | not applicable | not applicable | not applicable |
| PAR-02 | not applicable | not applicable | not applicable | not flagged by screen |
| PAR-03 | held for human review | not flagged by screen | not flagged by screen | not applicable |
| PAR-04 | not applicable | not applicable | not applicable | held for human review |
| Probe ID | Screen flags | Next action |
|---|---|---|
| PAR-01 | 0 | send to human meaning review |
| PAR-02 | 0 | send to human meaning review |
| PAR-03 | 1 | hold for human meaning review |
| PAR-04 | 1 | hold for human meaning review |
Any applicable FALSE value is a reason to hold the candidate for human review, not proof that its meaning changed. In this render, reading source and output together shows that the stress probe echoes prompt labels and drops Room 2. The obligation probe replaces must sign with authorized to sign, which raises a substantive obligation concern. Its modal screen also flags the row, but the same finite screen would miss or flag some equivalent phrasings, so the human comparison carries the conclusion.
Keep the decision honest
The model has completed generation, and the automatic checks have completed their limited jobs. No human has approved the candidates.
paraphrase_review <- tibble(
review = c(
"local model execution",
"complete prompt within 192 tokens",
"wording-change diagnostic",
"surface invariant checks",
"human meaning-preservation review",
"human prose review"
),
status = c(
sprintf(
"completed: %d candidates",
sum(nzchar(paraphrase_results$candidate))
),
sprintf(
"within budget: %d/%d candidates",
sum(paraphrase_results$input_tokens <= 192L),
nrow(paraphrase_results)
),
sprintf(
"computed: %d/%d candidates changed wording",
sum(!wording_diagnostics$exact_copy),
nrow(wording_diagnostics)
),
sprintf(
"held: %d/%d candidates flagged",
sum(paraphrase_actions$flagged_screens > 0L),
nrow(paraphrase_actions)
),
"pending",
"pending"
)
)| Review gate | Status |
|---|---|
| local model execution | completed: 4 candidates |
| complete prompt within 192 tokens | within budget: 4/4 candidates |
| wording-change diagnostic | computed: 4/4 candidates changed wording |
| surface invariant checks | held: 2/4 candidates flagged |
| human meaning-preservation review | pending |
| human prose review | pending |
June can accept a candidate only after reading it beside the notice and applying the rule stated at the start. Generation, visible wording change, and invariant flags are evidence for that review. None replaces it.
What to remember
- A paraphrase changes wording while preserving the information that matters.
- Copying is a useful preservation baseline, but it is not a successful rewrite.
- Token overlap measures wording, not semantic equivalence.
- Check names, numbers, negation, scope, permission, and obligation separately.
- Keep failed model outputs visible when they reveal a broken invariant.
- Record human meaning and prose review apart from automatic execution.