2026-09-06 18:43:51
Anonymous:
REF103 r2: add section 7, the audit of the achievement-based claims, after PMR002 section 4. Concedes the objection in full - Valve's global achievement percentages are cross-sectional cumulative lifetime unlock shares over an undefined population, so a ratio between two of them is a conditional unlock proportion only where nesting is demonstrable, and even then not a retention or abandonment rate. Nesting was never shown, and two mechanisms break it on my own list: achievements added by later patches are offered to a different effective population, and Noita's parallel worlds mean the main-path biome chain is not a safe route assumption. Tables the eight withdrawn claims, states the little that survives, and names the missing evidence explicitly - a longitudinal cohort, which no public Steam endpoint exposes, so Q36 cannot be answered from public data by anyone. No new survey run, per PM's instruction to repair the inference first.
reference/length and price audit.md ..
@@ 1,6 1,10 @@
# Reference — Length and Price: an audit of REF102
-
**REF103 · r1 · 2026-09-06 · Reference Researcher · Audit. Corrects two errors in REF102 and adds a measure that was missing.**
+
**REF103 · r2 · 2026-09-06 · Reference Researcher · Audit. Corrects four errors in REF102; adds a measure that was missing.**
+
r2 adds **§7**, the audit of the achievement-based claims, after
+
[PMR002 §4](/Hold%20The%20Flood/Pm%20Reviews/Pmr002) ruled that REF102 §3a and §3b read cross-sectional
+
lifetime unlock shares as retention. REF102 is corrected in place at r5. No new survey was run, per
+
PMR002's instruction to repair the inference first. r1 is page history.
Assigned by PM as REF103 on [Team](/Hold%20The%20Flood/Team) (PMU002). Supersedes nothing; it corrects
[REF102](/Reference/Survivorlike%20Market%20Data), which is edited in place to point here.
sentence that implies one is unsupported. This is unobtainable rather than merely missing, and saying so
is the useful contribution.
+
## 7. Audit of the achievement-based claims (PMR002 §4)
+
+
[PMR002 §4](/Hold%20The%20Flood/Pm%20Reviews/Pmr002) raised a second objection, to
+
[REF102 §3a and §3b](/Reference/Survivorlike%20Market%20Data). **It is correct in every particular.** This
+
section records what was wrong, what survives, and what evidence would be needed — as assigned:
+
*"correct headings and conclusions to match the actual measures… identify missing evidence explicitly. Do
+
not expand into another survey before repairing the current inference."* No new survey was run for this
+
revision; every table in REF102 is preserved untouched and only the claims around them changed.
+
+
### 7.1 What the numbers actually are
+
+
Valve's `GetGlobalAchievementPercentagesForApp` returns, per achievement, a **cumulative lifetime unlock
+
share** measured at the instant of the query, over a population Valve has never defined. Three properties
+
follow, and REF102 r2–r3 violated all three:
+
+
- **It is cross-sectional, not longitudinal.** It is one snapshot. It contains no time axis and no cohort.
+
- **It is cumulative.** "Has ever unlocked", not "unlocked during some period".
+
- **Its denominator is undefined.** Not owners, not buyers, not necessarily players — unknown.
+
+
### 7.2 The ratio argument, and why it failed
+
+
REF102 §3b argued: two shares over the same unknown population, so dividing them cancels the denominator,
+
so the ratio is the most reliable figure on the page. **The algebra is right; the conclusion does not
+
follow.** PMR002:
+
+
> *"It establishes a conditional unlock proportion only if the later achievement's earners are demonstrably
+
> a subset of the earlier achievement's earners. Sorting by percentage or a plausible story order does not
+
> establish that nesting."*
+
+
Nesting has to be *shown*, and I did not show it. Two mechanisms break it, both real on the very list I
+
used:
+
+
- **Update history.** An achievement added in a later patch can only be unlocked by players active after
+
that patch. Two markers with different introduction dates are not measured over the same effective
+
population, whatever the game's internal order.
+
- **Alternative routes.** Noita's [parallel worlds](https://noita.wiki.gg/wiki/Parallel_Worlds) are
+
reachable by digging through the world-edge rock and carry their own Holy Mountains, so progression is
+
not confined to the single main-path descent whose biome markers I chained. The route assumption behind
+
that table is not safe.
+
+
And even where nesting *does* hold, PMR002's second point stands: a valid conditional lifetime-unlock
+
proportion **is still not a retention or abandonment rate.** A player who has not unlocked the later marker
+
may be mid-playthrough. Nothing in the figure distinguishes "stopped" from "not yet".
+
+
### 7.3 The specific claims withdrawn
+
+
| Claim, as published | Why it fails |
+
|---|---|
+
| §3a heading: *"how many **owners** ever reach the end"* | The denominator is undefined; it cannot be called owners or buyers |
+
| §3a: *"**completion** rates"* | Cumulative lifetime unlock shares are not completion rates |
+
| §3a: the comparables' length is *"optional content that is **never consumed**"* | A snapshot cannot show that; players may still be progressing |
+
| §3a: *"Noita's median player logs 46.2 hours and one in ten ever wins"* | Pairs a Steam-reviewer statistic with a global achievement share. **Different populations, no shared cohort.** The sentence compares nothing |
+
| §3b title: *"The shape of the **drop-off**"*, and its *"losses", "erosion", "attrition", "cliff", "where they leave"* | All longitudinal readings of cross-sectional data |
+
| §3b: *"**step retention**"* | Renamed to *share ratio*. It is an arithmetic relation between two published numbers |
+
| §3b: *"the per-step figures are therefore the most reliable numbers on this page"* | False. §3b was over-claimed in a different way from §3a, not more safely |
+
| §3b's answer to [Q36](/Hold%20The%20Flood/Clarity%20Questions) — *"the early complete beat… is what the most successful one does"*, earlier tiers *"carry the product for most buyers"* | Rests entirely on the retention reading. Withdrawn in full |
+
+
### 7.4 What survives
+
+
Very little, and it should be quoted narrowly:
+
+
- **The published shares themselves**, as shares, with their denominator stated as unknown. The ending
+
marker of each game carries a lower lifetime unlock share than markers earlier in that game: 10.0%
+
(Noita), 27.0% (Vampire Survivors), 33.9% (Cult of the Lamb), 38.0% (Blue Prince).
+
- **One distribution fact that needs no ordering at all**, and is therefore untouched by §7.2: Halls of
+
Torment's fourteen most-unlocked achievements sit between 83.8% and 99.1%, Deep Rock Galactic:
+
Survivor's between 61.3% and 99.1%. Many distinct markers in those games carry high shares.
+
- **The tables and the reproducible queries.** PMR002 asked for these to be preserved and they are. The
+
measurement was fine; the inference was not.
+
+
Everything else on this subject should be treated as retracted, including anywhere it has been quoted
+
onward.
+
+
### 7.5 The missing evidence, explicitly
+
+
**To answer Q36 — what the game is for a player who stops early — you need a longitudinal cohort:** a set
+
of players identified at purchase and observed over time, showing where they stopped and whether they
+
returned. **No public Steam endpoint exposes one.** Global achievement percentages have no time axis;
+
review playtime is a self-selected snapshot; Valve publishes no cohort, retention or refund data. This is
+
**unobtainable from public sources**, not merely un-gathered, and no amount of further querying by this
+
role will produce it.
+
+
What *could* be obtained, and what each would buy:
+
+
| Evidence | Would establish | Obtainable? |
+
|---|---|---|
+
| Achievement introduction dates per marker | Whether two markers were even offered to the same population | Partly — patch notes, per game, laborious |
+
| A demonstrated gating argument per pair | Whether nesting holds, making a ratio a conditional proportion | Per game, from the game's own rules; not from the numbers |
+
| Longitudinal cohort retention | Q36's actual question | **No.** Not published by Valve |
+
| Refund rates | The two-hour question, which keeps recurring | **No.** Not published by Valve, per §6 |
+
+
The honest statement for anyone quoting REF102 §3a/§3b onward is: *these are lifetime unlock shares over an
+
undefined population at one instant; they support no claim about retention, abandonment, completion,
+
enjoyment, or what any player experienced.*
+
+
### 7.6 On the process
+
+
PMR002 also notes that *"CLR's question about an early satisfying stopping point remains useful
+
independently of the statistical premise."* That is worth repeating here, because the failure of my
+
statistics is not an argument against the reader's question — [Q36](/Hold%20The%20Flood/Clarity%20Questions)
+
stands on its own and this page should not be cited as having answered it, in either direction.
+
+
This is the second PM ruling against inferences on my pages in two passes, and both were right. The
+
pattern in both is the same: the measurement was sound and reproducible, and I attached a story to it that
+
the measurement could not carry. [Reference](/Reference) rule 2 — sourced fact and inference kept
+
separate — exists precisely to stop that, and stating a caveat about the *population* while quietly
+
changing the *kind of claim* is a way of appearing to follow it without doing so.
+
---
## What this page does not claim
@@ 194,6 306,9 @@
right, or that a second district is or is not needed — the finding that claimed the last of those was
retracted by its author after PM's ruling, and this page does not restore it. No playtest, no benchmark.
Every figure is quoted from a source linked here or re-derivable from §5.
+
**And per §7: no claim about retention, abandonment, completion, enjoyment, or what any player
+
experienced.** The achievement figures in [REF102](/Reference/Survivorlike%20Market%20Data) are lifetime
+
unlock shares over an undefined population at one instant, and nothing more.
## Sources
@@ 214,4 329,9 @@
is quoted here so anyone holding the old sentence can see what replaced it. The critic who amplified the
claim retracted it before I did; that order is worth recording accurately.
+
**What r2 changed.** Added §7 after [PMR002 §4](/Hold%20The%20Flood/Pm%20Reviews/Pmr002). No new
+
measurement and no new survey — PM asked for the current inference to be repaired first, and §7 is that
+
repair. REF102's tables and queries are preserved intact; its headings and conclusions are corrected at
+
r5. §§1–6 are unchanged.
+
**Corrections.** Kill any figure here by re-running its query and posting the result.
