A teacher who pastes a reading passage into a model and asks for ten quiz questions gets them back in seconds. The trouble begins at marking time. One answer appears nowhere in the passage, another is a loose paraphrase that nobody can point to, and a third is a fact the learner could only know from outside the text. Word lists go wrong in a parallel way: a "dictionary form" that no dictionary holds, a translation taken from the wrong sense, an example sentence the passage never contained. Both problems have one cure. Everything the model returns must point at a place in the passage, and a short program, not a tired person, should be able to confirm that the place exists. This guide builds quiz questions and a vocabulary list on that rule, and says which parts a machine can verify and which still need a human reader.

Make the answer a span, not a reply

The cleanest way to keep a quiz honest is to demand that each answer is a stretch of the passage itself. That is how the reading dataset SQuAD was built: its authors describe more than 100,000 questions about Wikipedia articles, each answered by a segment of the matching passage. The design has a practical payoff far from research. Marking turns into a string search. A hit means the answer is at least grounded in the text; a miss means it is wrong, however plausible it sounds.

Add a second field to every question: the full source sentence in which the answer sits. A teacher reads that sentence and sees why the question has the answer it has, without hunting through the passage.

Where a copied answer quietly stops matching

A model told to copy still retypes, and retyping alters characters that a reader never notices. These are the changes to test for, since each one turns a correct answer into a failed search:

  • Digits in the passage, spelled-out words in the answer, or the reverse.
  • A space before a percent sign that the passage has and the answer drops.
  • An en dash in a range replaced by a hyphen, or a curly apostrophe made straight.
  • A capital lost because the span opened a sentence in the passage.
  • A list that gains or loses the comma before its last "and".
  • Two separate places in the passage stitched into one answer, or a middle word skipped.

None of these is a failure of knowledge. They are slips of transcription, so a literal comparison is the right instrument: it ignores whether the answer is true and asks only whether it is there.

Questions that need no reading

A quiz that a learner can pass without the text is not testing the text. Two habits produce it. The first is a question that general knowledge can settle. Suppose the passage says someone came from the town where a famous event took place. Asking for the town's name invites an answer from memory, while what the passage actually states is the descriptive phrase. The second habit is a question that carries its own answer, such as asking about a date while naming the year.

The Johns Hopkins guide to multiple-choice writing supplies the general discipline. It prefers a direct question to a half-finished sentence, and wants stray detail trimmed from the stem. Both suit passage-based quizzes, because a long stem that retells half the passage is usually the stem that gives the answer away.

Two more rules protect meaning. When a sentence says a museum is closed on Mondays, the question has to ask for the closing day, because "Mondays" answers that and does not answer "when is it open". If one sentence gives two dates, the question must name the event it wants, in the passage's own phrase, or no single answer exists.

Wrong options that tempt

For multiple choice, the wrong options decide whether a question measures anything. The same Hopkins page asks for distractors that look plausible, so a student cannot guess the key; its own bad example mixes real names with a cartoon duck. It also asks for at least four choices to make a test more reliable, and for avoiding "all of the above" and "none of the above", which let a student score on partial knowledge.

A passage supplies tempting options for free: the other spans of the same kind, meaning the other dates, names or quantities in the text. A date taken from elsewhere in the same text looks right to a careless reader and is still wrong. Check that the borrowed span is not also a valid answer, which can happen when two sentences describe one event from two sides. Distractors need not be quoted for the checker to work, but keeping them as spans lets the same search run on them and makes it easy to confirm that none equals the key.

A vocabulary list promises something else

A word list promises that each word is worth learning and each entry is right. Four fields carry the promise.

The surface form. The word as the passage prints it, with its capital, accent marks and hyphen, so the learner can find it. This field is checked like a quiz answer: it must occur in the passage.

The lemma, or dictionary form. "Children" belongs under "child", "Häusern" under "Haus", "durmieron" under "dormir". The irregular forms matter most, because a learner who has the regular rules already is looking up precisely the forms that break them. Languages differ on what counts as the lemma: Bulgarian dictionaries list a verb by the first person singular of the present, German ones by the infinitive, and a German separable verb is listed with its prefix attached even when the sentence puts that prefix at the far end of the clause. Without a dictionary a program cannot confirm a lemma, so this field is where a person still spot-checks.

The word class and the sense. A translation should fit the meaning in this particular sentence. German "Gift" next to a mushroom means poison, and the look-alike English word is the classic trap. A translation that is a noun when the lemma is a verb, or the reverse, is a cheap signal that something slipped. Our post on search that mishandles plurals and accents shows the same tension between the surface form and the base form in a different job.

The example sentence. The complete sentence of the passage where the word turns up first, ending punctuation included, neither shortened nor translated. It can be tested exactly like an answer, by searching for it. When the passage breaks one sentence over two lines, it is still one sentence, so the check should flatten line breaks before it searches.

Choosing words by level

Teachers usually want the words a given learner does not know yet, not every word in the text. The Council of Europe's page on the CEFR sorts proficiency into six bands from A1 to C2, which regroup into basic, independent and proficient users, each described by can-do statements. That is a scale of what a learner can do. A label such as A2 therefore helps a request, but it says little about which individual words a particular learner has met.

So give the generator more than a level when you can: the dictionary forms the learner already knows, so that every inflected relative of a word on that list is dropped too. With a level alone the model must guess, and a sensible tie-break is to include the borderline word, because one extra entry costs a glance while a missing one can cost the sense of a sentence. In both cases names, numbers and function words stay out; a surname that doubles as a common noun counts as a name and not a word to teach.

A check that can fail

The verbatim rule is worth a few lines of code that run on every batch, because it is the one rule a machine can enforce completely. Here it is in Node:

// passage: the source text; items: the model's answer, already parsed
const flat = (s) => s.replace(/\s*\n\s*/g, " ");
const text = flat(passage);
const bad = [];
for (const q of items.questions ?? []) {
  if (!text.includes(flat(q.answer))) bad.push(["answer", q.answer]);
  if (!text.includes(flat(q.quote))) bad.push(["quote", q.quote]);
  if (!q.quote.includes(q.answer)) bad.push(["answer not in its quote", q.answer]);
}
for (const v of items.items ?? []) {
  if (!text.includes(flat(v.example))) bad.push(["example", v.example]);
  if (!v.example.includes(v.word)) bad.push(["word not in example", v.word]);
}
if (bad.length) throw new Error(JSON.stringify(bad, null, 2));

A check earns its place only if it can fail, so prove that first. Take a batch that passes and damage one answer: turn "forty-two" into "42", replace a curly apostrophe with a straight one, remove the comma before a final "and", or lower-case the first letter of a span. Each damaged batch must make the script throw. If one slips through, the comparison normalises too much and has stopped measuring what you wanted. The check also stays silent about everything else: a question can be fully grounded and still be badly worded, trivial, or aimed at a fact that does not matter. That judgment belongs to the teacher.

When this does not apply

The method covers questions whose answer is a stretch of the text, and word lists built from a single passage. Questions that ask for a summary, an opinion, an inference or a comparison across two passages have no span to look up, so they need another kind of check. The string test shows that a quotation exists. It cannot show that a question is fair, that a translation has the right sense or that a lemma is correct, and those need a person or a dictionary. A short passage may hold fewer facts than the number of questions requested, and padding with repeats or fragments is the failure to avoid, not a quota to fill. The multiple-choice advice comes from English-language university teaching guides and may fit other settings less well. We have not compared these methods with quizzes written by hand, and a very different group of learners could change which words deserve a place on the list.

Two tools for this job

Our quiz-from-text skill takes a passage and a count and returns that many questions as JSON. Each answer is a stretch of characters lifted from the passage, paired with the full sentence it came from. The questions come out in the passage's language, negations and paired dates are not flipped, instructions hidden inside the passage are ignored, and a text with too few facts gets fewer questions plus a short note.

The vocab-from-text skill turns one passage into entries for a learner. Each entry holds the form as printed, the dictionary form, a brief translation fitted to the sentence, and the source sentence as the example. It omits known words in any inflection, together with names, numbers and function words, and it joins German separable verbs back together.

Read this post as Markdown: /blog/quiz-and-vocabulary-from-a-text.md · Atom feed.