Natural language processing in R, one idea at a time

A map of natural language processing tasks, with lessons written for readers who are new to the field.

Computers work with human language constantly and mostly out of sight: a search box, a spam filter, a translation, a reply that arrives too fast to have been typed. The field that studies how they do it is called natural language processing, usually shortened to NLP. It covers dozens of connected tasks, from reading a file without corrupting it to generating a paragraph. Each tile below is one of them.

The tasks

Related tasks share a column, and the columns run roughly from getting text in, on the left, through cutting it into usable pieces, to models, generation, systems, and visual explanation. The arrangement is editorial: position tells you what a task is filed near and nothing else about it. Laying the field out at once means a lesson can be found by where it sits rather than by remembering its name. All 81 lessons are written.

Start here

For a course, read in number order because the R code builds on earlier lessons. For a specific problem, open any available lesson; each page introduces its own NLP idea and points out the assumptions it needs. If a language model will do part of the work, read the guide Using a language model in a reproducible analysis. It shows how to record the model and check its labels against a sample that a person reviewed.

Read lesson

Source and training dataLanguage structureModels and analysisLanguage generationLanguage systemsVisualization

Select a solid-bordered tile to open its lesson. If the whole map does not fit, use the buttons, arrow keys, or horizontal scrolling.

Source and training dataSource data loading

Source and training dataTraining data generation

Language structureWord parsing

Language structureWord processing

Language structurePhrases and entities

Language structureEntity enrichment

Language structureSentences and paragraphs

Language structureDocuments

Models and analysisModel development

Models and analysisClassification

Models and analysisSignals and discovery

Models and analysisSimilarity

Language generationNatural language generation

Language systemsSystems

VisualizationInformation visualization

Reading it as a course. Start at task 1 and work forward. Each lesson assumes the R introduced before it, so the code builds.

Solving one problem. Open the stage that matches what is in front of you: source and training data, language structure, models and analysis, language generation, language systems, visualization. Every lesson states the idea it needs, so you can start in the middle.

Looking one thing up. Find the task by where it sits in the map and read that page on its own. Each one names its limits and what it cannot tell you.

What this map is

This is a teaching aid, not a scientific taxonomy or a complete account of the field. It mixes tasks, methods, objectives, resources, services, applications, and work done across a model’s life. Each tile names its item type so those differences stay visible. Some inherited group names are broad or imperfect, and where that happens the lesson says so.

Why these lessons exist

Your director asks how many of last quarter’s 40,000 customer comments are complaints, because next quarter’s support budget depends on the answer. You hand the comments to an AI assistant. It answers in seconds: 22 percent.

The number might be right. The question is how anyone would know.

Start with the word. Before anything can be counted, someone has to say what a complaint is. Is a blunt question a complaint? A compliment with one sharp clause? A report of a broken link? Some answer gets applied to all 40,000 comments, in every run, whether or not a person chose it. A more capable model does not settle this, because the difficulty is not what the model can do. It is what you meant.

Then the margin around it. “About 22 percent” needs the “about” to come from somewhere. One way to investigate how often an automatic answer is wrong is to compare it with a hand-checked sample drawn from the same material. What that comparison supports depends on how the sample was drawn and what the checking rules mean. Some methods instead estimate error from agreement among automatic labellers, but those estimates depend on assumptions about how their errors relate. Checking those assumptions can still require hand-reviewed examples.

Underneath both is the text itself. Every number about a body of writing is really a number about the version of that writing which arrived. Characters get mangled between systems. Rows split at a stray comma. A scanned page reads a digit as a letter. Sometimes that raises an error. Often it does not, and the answer changes while continuing to look reasonable. Lessons 1, 3 and 7 each produce one of these failures on purpose, so you can see what one looks like from the inside.

Whatever a tool absorbs, something is left over. Here what is left over is deciding what to measure, on which material, and when to disbelieve the result. That part has not moved, because it is where the tool gets its instructions and where its output is accepted or rejected.

You can still buy the work. Labelled data is for sale, annotation vendors are easy to hire, and a model will label almost anything you give it, while refusing a few things. Each of those arrives with a number describing how much its labels agree. Reading that number and knowing what it conceals are different skills, and in my experience the second comes from having produced such a number yourself, on your own untidy text, and having watched it flatter you.

None of this says you must do the work by hand. A tool may be useful for drafting, summarising, or exploring unfamiliar text while still needing checks that match the decision. What the parts named here describe is not a fixed capability gap but the shape of a measurement: the definition, the comparison, and the text that actually arrived. The useful amount of checking depends on your text and on the cost of a wrong answer; this page cannot set it for you.

The guide Using a language model in a reproducible analysis works through that shape on a constructed collection. It records a local model as an instrument, checks its labels against a random sample reviewed by a person, and corrects an estimate for the errors that sample reveals.

The first thirteen lessons cover getting text out of files, web pages, and scans without damaging it, then building labels you can defend. Lessons 14 through 42 follow the text through tokens and vocabularies, word forms and grammar links, phrases and entities, sentence and paragraph boundaries, and whole-document cleaning, comparison, metadata, and language identification. Together they show where each transformation helps and what information it can lose.

Lessons 43 through 68 train, test, and explain models, then use them to classify, discover, compare, and generate text, keeping a baseline or a stated limit beside each result. Lessons 69 through 74 combine those parts into small systems: a relation extractor, a knowledge base filled from announcements, a question answerer that cites a handbook, a scripted enrollment conversation, a search index, and a document review with a recall estimate. Lessons 75 through 81 show results to people through an app, annotated text, charts, maps, and graphs, and ask what each picture can honestly claim.

About this site

Available means a lesson source exists and its project-recorded automated checks passed. Those passed values are project assertions, not independent certification. Manual accessibility and final human sign-off remain pending, and the separate review fields are kept in data/lesson_reviews.csv. Practice in this field moves quickly, so where a lesson describes current methods rather than settled ground, it says so and gives a date. The rules a lesson has to meet before it appears here are written down in CONTRIBUTING.md and RESEARCH_STANDARDS.md.

This repository builds and checks the source but does not deploy it. The canonical published location is workforcefutures.net/learn/nlp/. A successful repository build establishes a rendered artifact, not that the canonical host is current or that pending manual reviews have passed.

The task names and groupings are adapted from Rob van Zoest’s Periodic Table of NLP Tasks. That page states no licence and invites reuse. The task names themselves are standard terms in the field; the grouping, wording, interface, and lessons here are separate work, and any error in them belongs to this project. If the original author prefers different terms, this site will follow them.