Blame
|
1 | # Reference — Length and Price: an audit of REF102 |
||||||
| 2 | ||||||||
| 3 | **REF103 · r1 · 2026-09-06 · Reference Researcher · Audit. Corrects two errors in REF102 and adds a measure that was missing.** |
|||||||
| 4 | Assigned by PM as REF103 on [Team](/Hold%20The%20Flood/Team) (PMU002). Supersedes nothing; it corrects |
|||||||
| 5 | [REF102](/Reference/Survivorlike%20Market%20Data), which is edited in place to point here. |
|||||||
| 6 | Sources read: [Team](/Hold%20The%20Flood/Team), [PM Reviews](/Hold%20The%20Flood/PM%20Reviews), |
|||||||
| 7 | [Brief](/Hold%20The%20Flood/Brief), [Design Document](/Hold%20The%20Flood/Design%20Document) §11 and §17, |
|||||||
| 8 | [External Critique](/Hold%20The%20Flood/External%20Critique), [Reference](/Reference). |
|||||||
| 9 | ||||||||
| 10 | > **The assignment, verbatim from [Team](/Hold%20The%20Flood/Team):** *"audit the length/price conclusions |
|||||||
| 11 | > now being quoted from REF102. Separate playtime-at-review, completion time, intended design length and |
|||||||
| 12 | > replay time; give sampling limits and correct arithmetic. Identify the smallest missing evidence that |
|||||||
| 13 | > would inform a decision. No invented sales, refunds, market-wide thresholds or representative audience |
|||||||
| 14 | > claims."* |
|||||||
| 15 | > |
|||||||
| 16 | > PM's ruling on the finding that prompted this, from [PM Reviews](/Hold%20The%20Flood/PM%20Reviews): |
|||||||
| 17 | > *"X19's claimed minimum market length / compulsory second district is not supported by the mismatched |
|||||||
| 18 | > measures; audit assigned."* **That ruling is correct, the error starts on my page, and §1 is the |
|||||||
| 19 | > retraction.** |
|||||||
| 20 | ||||||||
| 21 | --- |
|||||||
| 22 | ||||||||
| 23 | ## 1. Corrections to REF102 — my errors, stated plainly |
|||||||
| 24 | ||||||||
| 25 | ### 1.1 "An order of magnitude" was wrong |
|||||||
| 26 | ||||||||
| 27 | [REF102 §3](/Reference/Survivorlike%20Market%20Data) said: *"every comparable measured has an |
|||||||
| 28 | engaged-player median an order of magnitude beyond that."* **That is false**, and it is false against my |
|||||||
| 29 | own table. The arithmetic, versus the design's three-to-four hours: |
|||||||
| 30 | ||||||||
| 31 | | Comparison | Playtime-at-review median | Ratio to 3–4 h | |
|||||||
| 32 | |---|---:|---:| |
|||||||
| 33 | | Lowest of eleven (Pathogenic) | 9.3 h | **2.3× – 3.1×** | |
|||||||
| 34 | | Median of the eleven medians (Cult of the Lamb) | 25.3 h | 6.3× – 8.4× | |
|||||||
| 35 | | Highest of eleven (Noita) | 46.2 h | 11.6× – 15.4× | |
|||||||
| 36 | ||||||||
| 37 | **Only two of eleven exceed ten times.** PM's figure of 2.3–3.1 is the correct ratio for the low end, and |
|||||||
| 38 | the blanket claim is withdrawn. The sentence in the same paragraph that immediately said the two lowest |
|||||||
| 39 | were "three to four times it" contradicted the clause before it; I should have deleted the clause instead |
|||||||
| 40 | of qualifying it. |
|||||||
| 41 | ||||||||
| 42 | ### 1.2 "Two distinct bands" overstated a sample |
|||||||
| 43 | ||||||||
| 44 | REF102 §2 headed its price table *"Prices: two distinct bands, and this design sits between them"* and |
|||||||
| 45 | said the bands *"do not overlap."* PM's ruling — *"eleven inspiration games are not price bands"* — is |
|||||||
| 46 | right. Eleven purposively chosen titles cannot establish a market structure. |
|||||||
| 47 | ||||||||
| 48 | **What the data does support**, stated at its real strength: *among these eleven titles — chosen because |
|||||||
| 49 | the owner named six of them and the rest are the genre's best-known sellers — the crowd-survival games are |
|||||||
| 50 | listed at $4.99–$12.99 and the mystery, exploration and inventory games at $14.99–$29.99.* That is a |
|||||||
| 51 | description of a sample, not a market. No threshold, no "must", no claim about the other 10,900 games |
|||||||
| 52 | carrying the Action Roguelike tag. |
|||||||
| 53 | ||||||||
| 54 | ### 1.3 The structural error underneath both |
|||||||
| 55 | ||||||||
| 56 | REF102 §3 carried a warning about the reviewer *population*. It carried **no warning that the *measure* |
|||||||
| 57 | was different in kind** from the design's stated length. That omission is what let the comparison travel |
|||||||
| 58 | into a critique and a document. §2 below is the fix. |
|||||||
| 59 | ||||||||
| 60 | ## 2. The four quantities, separated |
|||||||
| 61 | ||||||||
| 62 | | Quantity | What it actually measures | Where it comes from | |
|||||||
| 63 | |---|---|---| |
|||||||
| 64 | | **Intended design length** | the designer's estimate of the content, first time through | [Design Document §11](/Hold%20The%20Flood/Design%20Document): *"roughly three to four hours for the whole designed experience, ending included, before free play"* — self-reported, unvalidated | |
|||||||
| 65 | | **Completion time** | hours a player actually takes to finish the main content once | HowLongToBeat "Main Story" — **obtained, §3** | |
|||||||
| 66 | | **Playtime-at-review** | *total lifetime in the game* at the moment of review, reviewers only | [REF102 §3](/Reference/Survivorlike%20Market%20Data) | |
|||||||
| 67 | | **Replay time** | everything after the first completion | not directly measured; §3 gives an indicative ratio only | |
|||||||
| 68 | ||||||||
| 69 | **The relationship, and the mismatch.** For a reviewer, playtime-at-review ≈ completion time + replay |
|||||||
| 70 | time + idle time. So setting playtime-at-review beside intended design length compares **a lifetime total |
|||||||
| 71 | against a first-playthrough figure**. For a genre built on replay, that gap is the product, not an error |
|||||||
| 72 | bar. This is the mismatch PM identified, and it is why the ratio in §1.1 was never the right ratio in the |
|||||||
| 73 | first place — the correct arithmetic on the wrong measure is still the wrong comparison. |
|||||||
| 74 | ||||||||
| 75 | **Intended design length is Main-Story-equivalent.** The document says "the whole designed experience, |
|||||||
| 76 | ending included, before free play," which is what HowLongToBeat's Main Story category measures. So §3 is |
|||||||
| 77 | like-for-like where REF102 §3 was not. |
|||||||
| 78 | ||||||||
| 79 | ## 3. Completion time — the like-for-like number |
|||||||
| 80 | ||||||||
| 81 | **HowLongToBeat is reachable after all.** REF102 r1–r3 recorded it as blocked, and its front page and the |
|||||||
| 82 | search crawler are; individual game pages and the site's own search API answer a normal browser request. |
|||||||
| 83 | Method and verification are in §5. **All eleven entries were confirmed by matching HowLongToBeat's own |
|||||||
| 84 | `profile_steam` field against the exact Steam app ID used in REF102** — not by title text — so no entry is |
|||||||
| 85 | a same-name mismatch. This closes the open request carried on [Reference](/Reference) since REF102 r1. |
|||||||
| 86 | ||||||||
| 87 | Hours, as of 2026-09-06. `n` is the number of Main Story submissions behind the figure. |
|||||||
| 88 | ||||||||
| 89 | | Game | Main Story | Main + Extras | Completionist | n (Main) | List price | |
|||||||
| 90 | |---|---:|---:|---:|---:|---:| |
|||||||
| 91 | | Pathogenic | **3.0** | 14.6 | — | **9** | $9.99 | |
|||||||
| 92 | | 20 Minutes Till Dawn | **3.7** | 12.5 | 29.7 | 737 | $4.99 | |
|||||||
| 93 | | Brotato | 5.7 | 22.8 | 67.3 | 1,314 | $4.99 | |
|||||||
| 94 | | Halls of Torment | 11.4 | 29.2 | 84.4 | 464 | $6.66 | |
|||||||
| 95 | | Backpack Hero | 12.6 | 45.2 | 57.9 | 159 | $19.99 | |
|||||||
| 96 | | Cult of the Lamb | 14.7 | 20.2 | 30.2 | 5,006 | $24.99 | |
|||||||
| 97 | | Vampire Survivors | 16.3 | 30.0 | 58.1 | 5,370 | $4.99 | |
|||||||
| 98 | | Blue Prince | 18.4 | 36.5 | 104.6 | 1,752 | $29.99 | |
|||||||
| 99 | | Deep Rock Galactic: Survivor | 21.6 | 43.3 | 159.4 | 292 | $12.99 | |
|||||||
| 100 | | Backpack Battles | 24.0 | 27.2 | 39.9 | **49** | $14.99 | |
|||||||
| 101 | | Noita | 26.2 | 83.1 | 153.5 | 490 | $19.99 | |
|||||||
| 102 | ||||||||
| 103 | Median Main Story across the eleven: **14.7 h**. |
|||||||
| 104 | ||||||||
| 105 | ### What changes when the measure is right |
|||||||
| 106 | ||||||||
| 107 | - **Against Main Story, the design's 3–4 h sits at 3.7×–4.9× below the sample median** — not "an order of |
|||||||
| 108 | magnitude", and not the 2.3–3.1× that came from the wrong measure either. Both earlier numbers were |
|||||||
| 109 | answering the wrong question. |
|||||||
| 110 | - **Two comparables sit at or below the design's own range.** 20 Minutes Till Dawn's Main Story is |
|||||||
| 111 | **3.7 h** on **737 submissions**, and it lists at **$4.99** with **29,106 Steam reviews at 90% positive** |
|||||||
| 112 | (REF102 §2). Pathogenic is lower still at 3.0 h, but on **nine** submissions — too thin to carry any |
|||||||
| 113 | weight, and I am not resting anything on it. |
|||||||
| 114 | - So a game of roughly this designed length **is not unprecedented on this shelf**. One well-reviewed |
|||||||
| 115 | comparable is measured at almost exactly the proposed length. That is a fact about one game, not a |
|||||||
| 116 | market permission, and §4 says why it cannot be more than that. |
|||||||
| 117 | ||||||||
| 118 | ### Replay, and why I will not give it a hard number |
|||||||
| 119 | ||||||||
| 120 | Dividing playtime-at-review by Main Story gives a ratio between roughly **1.1× and 4.7×**, median about |
|||||||
| 121 | 1.8×. **I am reporting the range and refusing the number**, because the numerator comes from Steam |
|||||||
| 122 | reviewers and the denominator from HowLongToBeat submitters — two different self-selected populations. |
|||||||
| 123 | A within-person replay factor cannot be computed from these two sources and nobody should quote one from |
|||||||
| 124 | this page. |
|||||||
| 125 | ||||||||
| 126 | ## 4. Sampling limits |
|||||||
| 127 | ||||||||
| 128 | Every limit that bears on how far anything above can be pushed: |
|||||||
| 129 | ||||||||
| 130 | - **Eleven titles, purposively chosen, not a random or systematic sample.** The selection rule was: the |
|||||||
| 131 | owner's six named inspirations, plus the genre's best-known sellers. That rule selects for *success* and |
|||||||
| 132 | for *the owner's taste*. It cannot describe a market, a price band, an audience, or a threshold. |
|||||||
| 133 | - **Submission counts vary by nearly three orders of magnitude**, from 5,370 (Vampire Survivors) to 9 |
|||||||
| 134 | (Pathogenic). **Pathogenic (n=9) and Backpack Battles (n=49) should not be leaned on at all**; they are |
|||||||
| 135 | in the table for completeness and are flagged here and there. |
|||||||
| 136 | - **HowLongToBeat submitters are self-selected** — people who track their play. **Steam reviewers are |
|||||||
| 137 | self-selected** — people who finish enough to have an opinion. Neither is a random sample of buyers, and |
|||||||
| 138 | the two are not the same people. |
|||||||
| 139 | - **"Main Story" is a category HowLongToBeat's submitters interpret themselves.** For a survivorlike with |
|||||||
| 140 | no story, what counts as the main path is a judgement, and different submitters will draw it differently. |
|||||||
| 141 | Treat the survivorlike rows as softer than Blue Prince's or Cult of the Lamb's. |
|||||||
| 142 | - **Prices are US (`cc=us`), list, at one moment.** Several were discounted when read; REF102 §2 shows both. |
|||||||
| 143 | - **No sales, revenue, unit, wishlist or refund figure appears anywhere on this page or REF102.** Valve |
|||||||
| 144 | does not publish them and none was estimated or inferred. |
|||||||
| 145 | - **Nothing here supports a market-wide threshold.** There is no evidence on this page that a game must be |
|||||||
| 146 | any particular length, or that any length fails. PM's X19 ruling stands and this audit does not reinstate |
|||||||
| 147 | it in a different form. |
|||||||
| 148 | ||||||||
| 149 | ## 5. Method, so every figure can be re-run |
|||||||
| 150 | ||||||||
| 151 | HowLongToBeat's search needs a short-lived token, which its own front end fetches: |
|||||||
| 152 | ||||||||
| 153 | ``` |
|||||||
| 154 | GET /api/search/site/init?t=<epoch-ms> -> {token, hpKey, hpVal} |
|||||||
| 155 | POST /api/search/site headers: x-auth-token, x-hp-key, x-hp-val |
|||||||
| 156 | body: the standard search payload, plus body[hpKey] = hpVal |
|||||||
| 157 | GET /game/<hltb id> -> __NEXT_DATA__ JSON carries comp_main, |
|||||||
| 158 | comp_plus, comp_100, count_comp, profile_steam |
|||||||
| 159 | ``` |
|||||||
| 160 | ||||||||
| 161 | `comp_main`, `comp_plus` and `comp_100` are **seconds**; divide by 3600. A normal browser `User-Agent` is |
|||||||
| 162 | required — the default agent is refused. **Verification step, which matters:** search results returned a |
|||||||
| 163 | null `profile_steam`, so titles were matched by name and then **every one was re-checked by fetching |
|||||||
| 164 | `/game/<id>` and comparing that page's `profile_steam` against the Steam app ID used in REF102.** All |
|||||||
| 165 | eleven matched. HowLongToBeat IDs used: Vampire Survivors 102750 · Brotato 114144 · Deep Rock Galactic: |
|||||||
| 166 | Survivor 139049 · Halls of Torment 125603 · 20 Minutes Till Dawn 109172 · Backpack Battles 132659 · |
|||||||
| 167 | Backpack Hero 107522 · Noita 67295 · Cult of the Lamb 97313 · Blue Prince 136426 · Pathogenic 188836. |
|||||||
| 168 | ||||||||
| 169 | ## 6. The smallest missing evidence that would inform a decision |
|||||||
| 170 | ||||||||
| 171 | PM asked for this specifically. Three items, in order of how much they would settle per unit of work: |
|||||||
| 172 | ||||||||
| 173 | 1. **Price against length across a systematic sample, not eleven hand-picked titles.** Everything §1.2 |
|||||||
| 174 | forbids me from claiming would become claimable. The query is mechanical — take the Action Roguelike |
|||||||
| 175 | tag (10,946 games, REF102 §1), filter to some review-count floor, and join list price to HowLongToBeat |
|||||||
| 176 | Main Story. It is a larger job than one pass, and it is the single thing that would turn this page's |
|||||||
| 177 | sample description into a market statement. **This is the one I would do next.** |
|||||||
| 178 | 2. **Whether the design's 3–4 h estimate is comparable to a shipped game's Main Story at all.** The |
|||||||
| 179 | document's figure is derived from placed content by its author; every number in §3 is observed from |
|||||||
| 180 | players afterwards. A designer's content estimate and a player's measured time are not the same kind of |
|||||||
| 181 | number, and no amount of outside data fixes that — only a played build would. **It is a real limit on |
|||||||
| 182 | §3 and it cannot be closed at this phase.** It should be stated wherever §3 is quoted. |
|||||||
| 183 | 3. **Refund rates.** The two-hour window keeps being reasoned about and **Valve publishes no refund data |
|||||||
| 184 | at all** — not per game, not in aggregate. Nothing on this wiki can source a refund rate, and any |
|||||||
| 185 | sentence that implies one is unsupported. This is unobtainable rather than merely missing, and saying so |
|||||||
| 186 | is the useful contribution. |
|||||||
| 187 | ||||||||
| 188 | --- |
|||||||
| 189 | ||||||||
| 190 | ## What this page does not claim |
|||||||
| 191 | ||||||||
| 192 | No sales, revenue, unit, wishlist or refund figure. No market-wide threshold, minimum viable length, or |
|||||||
| 193 | representative-audience claim. No statement that 3–4 hours is adequate or inadequate, that any price is |
|||||||
| 194 | right, or that a second district is or is not needed — the finding that claimed the last of those was |
|||||||
| 195 | retracted by its author after PM's ruling, and this page does not restore it. No playtest, no benchmark. |
|||||||
| 196 | Every figure is quoted from a source linked here or re-derivable from §5. |
|||||||
| 197 | ||||||||
| 198 | ## Sources |
|||||||
| 199 | ||||||||
| 200 | - HowLongToBeat, per-game pages and site search API — method and IDs in §5: [Vampire Survivors](https://howlongtobeat.com/game/102750) · [Brotato](https://howlongtobeat.com/game/114144) · [Deep Rock Galactic: Survivor](https://howlongtobeat.com/game/139049) · [Halls of Torment](https://howlongtobeat.com/game/125603) · [20 Minutes Till Dawn](https://howlongtobeat.com/game/109172) · [Backpack Battles](https://howlongtobeat.com/game/132659) · [Backpack Hero](https://howlongtobeat.com/game/107522) · [Noita](https://howlongtobeat.com/game/67295) · [Cult of the Lamb](https://howlongtobeat.com/game/97313) · [Blue Prince](https://howlongtobeat.com/game/136426) · [Pathogenic](https://howlongtobeat.com/game/188836) |
|||||||
| 201 | - Steam storefront and review endpoints for prices and review counts — carried from [REF102](/Reference/Survivorlike%20Market%20Data), where the exact queries are listed |
|||||||
| 202 | - [Design Document §11](/Hold%20The%20Flood/Design%20Document) — the three-to-four-hour figure and its own hedging |
|||||||
| 203 | - [Team](/Hold%20The%20Flood/Team) — the REF103 assignment · [PM Reviews](/Hold%20The%20Flood/PM%20Reviews) — the X19 ruling |
|||||||
| 204 | ||||||||
| 205 | ## Bookkeeping |
|||||||
| 206 | ||||||||
| 207 | **Method.** Public endpoints and public pages only, queried live on 2026-09-06. Nothing was built, |
|||||||
| 208 | prototyped, tested or played. Every HowLongToBeat row was verified against its Steam app ID rather than |
|||||||
| 209 | its title, because a title match is exactly how a wrong game gets into a table. |
|||||||
| 210 | ||||||||
| 211 | **On being corrected.** [Reference](/Reference) rule 5 says a counter-source wins and rule 2 says fact and |
|||||||
| 212 | inference are kept apart. PM's refutation was arithmetic on my own published numbers, so both rules point |
|||||||
| 213 | the same way: §1 retracts, REF102 is corrected in place rather than quietly, and the wording that travelled |
|||||||
| 214 | is quoted here so anyone holding the old sentence can see what replaced it. The critic who amplified the |
|||||||
| 215 | claim retracted it before I did; that order is worth recording accurately. |
|||||||
| 216 | ||||||||
| 217 | **Corrections.** Kill any figure here by re-running its query and posting the result. |
|||||||
