Natural language processing in R, one idea at a time

A map of natural language processing tasks, with lessons written for readers who are new to the field.

Computers work with human language constantly and mostly out of sight: a search box, a spam filter, a translation, a reply that arrives too fast to have been typed. The field that studies how they do it is called natural language processing, usually shortened to NLP. It covers dozens of connected tasks, from reading a file without corrupting it to generating a paragraph. Each tile below is one of them.

The tasks

Related tasks share a column, and the columns run roughly from getting text in, on the left, through cutting it into usable pieces, to models, generation, systems, and visual explanation. The arrangement is editorial: position tells you what a task is filed near and nothing else about it. Laying the field out at once means a lesson can be found by where it sits rather than by remembering its name. Sixty-two of the eighty-one lessons are written. The rest are planned, and their tiles stay visible so you can see what is coming.

Start here

For a course, read in number order because the R code builds on earlier lessons. For a specific problem, open any available lesson; each page introduces its own NLP idea and points out the assumptions it needs.

Read lessonLesson planned

Source and training dataLanguage structureModels and analysisLanguage generationLanguage systemsVisualization

Select a solid-bordered tile to open its lesson. Dashed tiles are planned. If the map is wider than the window, use the buttons, arrow keys, or horizontal scrolling.

Source and training dataSource data loading

Source and training dataTraining data generation

Language structureWord parsing

Language structureWord processing

Language structurePhrases and entities

Language structureEntity enrichment

Language structureSentences and paragraphs

Language structureDocuments

Models and analysisModel development

Models and analysisClassification

Models and analysisSignals and discovery

Models and analysisSimilarity

Language generationNatural language generation

63objectiveNext-token predictionPlanned
64applicationReport writingPlanned
65taskMachine translationPlanned
66taskAbstractive summarizationPlanned
67taskParaphrasingPlanned
68taskLong-text generationPlanned

Language systemsSystems

69taskRelation extractionPlanned
70taskQuestion answeringPlanned
71applicationChatbot dialoguePlanned
72applicationSemantic search indexingPlanned
73taskKnowledge base populationPlanned
74applicationE-discovery and media monitoringPlanned

VisualizationInformation visualization

75applicationInteractive app creationPlanned
76applicationAnnotated text visualizationPlanned
77applicationWord cloudsPlanned
78applicationWord embedding visualizationPlanned
79applicationEvents on a timelinePlanned
80applicationLocations on a mapPlanned
81applicationKnowledge graph visualizationPlanned

What this map is

This is a teaching aid, not a scientific taxonomy or a complete account of the field. It mixes tasks, methods, objectives, resources, services, applications, and work done across a model’s life. Each tile names its item type so those differences stay visible. Some inherited group names are broad or imperfect, and where that happens the lesson says so.

Why these lessons exist

Your director asks how many of last quarter’s 40,000 customer comments are complaints, because next quarter’s support budget depends on the answer. You hand the comments to an AI assistant. It answers in seconds: 22 percent.

The number might be right. The question is how anyone would know.

Start with the word. Before anything can be counted, someone has to say what a complaint is. Is a blunt question a complaint? A compliment with one sharp clause? A report of a broken link? Some answer gets applied to all 40,000 comments, in every run, whether or not a person chose it. A more capable model does not settle this, because the difficulty is not what the model can do. It is what you meant.

Then the margin around it. “About 22 percent” needs the “about” to come from somewhere. One way to investigate how often an automatic answer is wrong is to compare it with a hand-checked sample drawn from the same material. What that comparison supports depends on how the sample was drawn and what the checking rules mean. Some methods instead estimate error from agreement among automatic labellers, but those estimates depend on assumptions about how their errors relate. Checking those assumptions can still require hand-reviewed examples.

Underneath both is the text itself. Every number about a body of writing is really a number about the version of that writing which arrived. Characters get mangled between systems. Rows split at a stray comma. A scanned page reads a digit as a letter. Sometimes that raises an error. Often it does not, and the answer changes while continuing to look reasonable. Lessons 1, 3 and 7 each produce one of these failures on purpose, so you can see what one looks like from the inside.

Whatever a tool absorbs, something is left over. Here what is left over is deciding what to measure, on which material, and when to disbelieve the result. That part has not moved, because it is where the tool gets its instructions and where its output is accepted or rejected.

You can still buy the work. Labelled data is for sale, annotation vendors are easy to hire, and a model will label almost anything you give it, while refusing a few things. Each of those arrives with a number describing how much its labels agree. Reading that number and knowing what it conceals are different skills, and in my experience the second comes from having produced such a number yourself, on your own untidy text, and having watched it flatter you.

None of this says you must do the work by hand. A tool may be useful for drafting, summarising, or exploring unfamiliar text while still needing checks that match the decision. What the parts named here describe is not a fixed capability gap but the shape of a measurement: the definition, the comparison, and the text that actually arrived. The useful amount of checking depends on your text and on the cost of a wrong answer; this page cannot set it for you.

The first thirteen lessons cover getting text out of files, web pages, and scans without damaging it, then building labels you can defend. Lessons 14 through 42 follow the text through tokens and vocabularies, word forms and grammar links, phrases and entities, sentence and paragraph boundaries, and whole-document cleaning, comparison, metadata, and language identification. Together they show where each transformation helps and what information it can lose.

About this site

Available means a lesson source exists and its project-recorded automated checks passed. Those passed values are project assertions, not independent certification. Manual accessibility and final human sign-off remain pending, and the separate review fields are kept in data/lesson_reviews.csv. Practice in this field moves quickly, so where a lesson describes current methods rather than settled ground, it says so and gives a date. The rules a lesson has to meet before it appears here are written down in CONTRIBUTING.md and RESEARCH_STANDARDS.md.

This repository builds and checks the source but does not deploy it. The canonical published location is workforcefutures.net/learn/nlp/. A successful repository build establishes a rendered artifact, not that the canonical host is current or that pending manual reviews have passed.

The task names and groupings are adapted from Rob van Zoest’s Periodic Table of NLP Tasks. That page states no licence and invites reuse. The task names themselves are standard terms in the field; the grouping, wording, interface, and lessons here are separate work, and any error in them belongs to this project. If the original author prefers different terms, this site will follow them.