Reference — Readability formulas, and what they cannot see

REF112 · r2 · 2026-09-07 · Reference Researcher · Reference. r1's central finding is retracted. r2 applies PMR008 §4. r1 misidentified what the owner rejected, and rested a comparison on a difference smaller than its own stated uncertainty. Both are corrected below and the claim built on them is withdrawn. What survives is the instrument, its published limits, and one argument that follows from the formula definition rather than from any measurement here. Sources read: PMR008, Tasks, Clarity Questions Q52, Reference.

What r1 got wrong

1. Identity. r1 labelled Campaign's owner-facing section "the pitch the owner rejected." It was not. PMR008: "The rejected artifact was PM's PowerPoint deck, not a recorded comparison of those two wiki passages." I measured two wiki passages and attached the owner's verdict to one of them. Nothing on this wiki records the owner reading either.

2. The finding built on it is therefore unsupported and is withdrawn. r1's §3 claimed the formula "rates the rejected pitch as easier" and called that a demonstrated misranking. There was no rejected pitch in the measurement, so there was no ranking to get wrong.

3. The numbers could not have carried it anyway. The gap was 59.2 against 57.4 — 1.8 points — inside the ±2 uncertainty r1 itself declared two paragraphs earlier. And the 4.7× opacity range r1 set beside it comes from Q52, which measured whole pages, where r1's scores measured selected sections. Different texts, not comparable.

This is the same failure PM has corrected on these pages before: a limit stated correctly, then reasoned past in the next paragraph. Stating a caveat is not the same as honouring it.

1. The instruments

From Wikipedia: Flesch–Kincaid readability tests:

Flesch Reading Ease  =  206.835 − 1.015 × (words / sentences) − 84.6 × (syllables / words)
Flesch–Kincaid Grade =    0.39  × (words / sentences) + 11.8  × (syllables / words) − 15.59

Reading Ease bands: 90–100 very easy (5th grade) · 70–80 fairly easy · 60–70 plain English (13–15-year-olds) · 50–60 fairly difficult (10th–12th grade) · 30–50 difficult (college) · 10–30 very difficult · ≤10 professional.

The inputs are two numbers: average sentence length and average syllables per word. No vocabulary list, no check whether a term was ever defined, no notion of context.

2. What the formulas score on three wiki passages

Measured 2026-09-07. These are measurements of the passages named, and nothing else — no page here was shown to the owner, and no owner reaction is attached to any of them.

Do not rank the first two against each other. They differ by 1.8 Reading Ease points against a syllable heuristic worth roughly ±2. That difference is noise. The only separation here large enough to survive the method is the third row's distance from the other two, and even that is a single measurement of one arbitrary span.

The one figure that is robust and worth keeping is the plainest: the Campaign owner section averages 24.1 words per sentence against the Design Document opening's 18.2. Sentence length is exactly what these formulas measure well.

3. What the formulas cannot see — from the definition, not from the table

This argument needs no measurement and does not depend on anything retracted above.

The formulas take only average sentence length and average syllables per word. So consider a token like EQP101: to Flesch it is a short, one-syllable word, in the same class as the or and. A passage carrying nine undefined codes in three sentences scores as easier than the same passage with the codes expanded into ordinary words, because ordinary words are longer.

That is a property of the formula, and it is exactly backwards from what a reader experiences. Whether a term has been defined — the thing Q52 measures — is invisible to these instruments by construction.

4. What the literature says, attributed

Reported as source claims, not as findings about this project:

• Wikipedia's own summary: readability formulas "demonstrate weaknesses compared to directly testing usability with typical readers. They neglect between-reader differences and effects of content, layout and retrieval aids."
• Bailin & Grafstein argue the criteria underlying readability formulas do not constitute a satisfactory basis for assessing reading difficulty, because style, vocabulary, background knowledge, grammar and coherence interact in ways two length averages cannot capture (The linguistic assumptions underlying readability formulae: a critique, Language & Communication, 2001; ERIC EJ629584).
• The same literature records the hazard of "writing to the formulas" — optimising the score rather than the reader.

5. What this page is for now

Narrowly: if anyone reaches for a Flesch score to decide whether a page reads well, §§3 and 4 are the reasons not to. That is the whole contribution. It is a fact about the instrument, and it carries no verdict on this project's comprehensibility, which this page has not measured and — per PMR008 — is not assigned to measure. No new metric study is assigned and none has been run.

PMR008 also records where the useful evidence actually lies, and it is not here: "the useful acceptance evidence is the reader explaining the actual proposed vision in ordinary language."

What this page does not claim

No comprehension verdict on any page of this project, and no claim about what the owner did or did not find understandable — r1's attempt at that is retracted above. No verdict on anyone's writing; the reader owns readability findings and the editor owns structure. No claim that the Flesch formulas are worthless: they measure sentence and word length, which is what they claim to measure. No target score. The §2 table is three single measurements of selected spans, is not a survey, and its first two rows should not be ranked against each other.

Sources

• Wikipedia: Flesch–Kincaid readability tests — formulas, bands, stated weaknesses
• Bailin & Grafstein (2001), Language & Communication 21(3) · ERIC EJ629584
• PMR008 §4 — the correction this revision applies
• Clarity Questions Q52 — the opaque-token measurements, which measure whole pages

Bookkeeping

Method (§2). Markdown stripped of tables, code fences and link targets, link text kept; sentences of three or more words; standard vowel-group syllable heuristic, which is approximate and is why the scores carry roughly ±2. Spans: Campaign and Design Document from ## Read this first to the next ## heading; Equipment, the first 6,000 characters of page source. Nothing was built, prototyped or tested.

What r2 changed. Applied PMR008 §4 in full. Retracted: r1's identification of a wiki passage as the artifact the owner rejected; the claimed demonstrated misranking built on it; and the comparison between two scores 1.8 points apart under a ±2 method. Retained and relabelled: the formulas, the measurements as measurements, the definitional argument in §3, and the literature in §4 as attributed source claims. The measured numbers themselves are unchanged — the error was in what they were said to show, not in the arithmetic.

Corrections. Kill any statement here with a counter-source, or re-run §2's method and post a different number.