Unreleased and in development. There is no installer and no release yet. This file describes what the tool is and how it is meant to be judged, not a running product. What currently works, and what every figure is currently worth, is measured by the tool itself and recorded next to the data it was measured from: see the reference sets under reference/ and the release notes when releases begin.

A tool for finding out what players of a game actually think, rather than what the loudest reviews say.

Steam sorts reviews by helpfulness. If you read the top forty to judge a game, you are reading the reviews other people upvoted, which are also the longest and angriest ones. That is a measure of agreement, not of how common an opinion is.

The gap is large enough to change conclusions. Measured on real corpora, a theme can appear in half or more of the fifty Steam ranks most helpful and in a fifth of all of them. Both numbers are true. Only the second is a fact about players. The tool reports the ratio between the two as a bias factor, so the distortion is a number you can read rather than an argument you have to have.

The only way to remove the argument is to hold every review and count.

• Take one Steam app ID, or a whole list of them.
• Download every review for those games, not a sample, and bring the capture up to date later without downloading it again.
• Split each review into the points it makes and sort each point into a subject, with a measured error and an honest "cannot tell".
• Show, per subject, the words the praise uses and the complaints use, counted by reviewer.
• Build a picture of the game in a paragraph from those counts.
• Click through from any number, anywhere, to the actual reviews behind it.

Results come out as one self-contained page: every rate, the reviews behind it, and what the classifier is measured to get wrong, in a single file you can open from disk, send to someone, or print. It fetches nothing, because a corpus that never left your machine should not start leaving it the moment somebody looks at it.

You choose which model does the sorting, and how closely it reads.

The headline figure for a category is a mention rate: the share of all reviews in the corpus that say something about it. A review that covers two topics counts towards both, so mention rates across categories add up to more than 100%, and they are meant to. This is the number people usually have in mind when they ask how common a complaint is.

Where a figure is not a mention rate, it is labelled. Two others appear:

• Primary share, the share of reviews whose single main subject is that category. These are exhaustive and add up to the number of reviews.
• Claim share, used in deep reading, where the unit counted is an individual opinion rather than a whole review.

No percentage is ever shown without saying which of the three it is. The point of counting everything is lost if the denominator is ambiguous.

A review is not one opinion. "Looks incredible, runs like a slideshow, and the story is the best in the series" is three, about three different things, and a single vector for the whole review is their average: a point that belongs to none of them. So every review is first split into the separate points it makes, and a subject belongs to a point rather than to a review.

That is what keeps the arithmetic honest at both ends. A review that makes twelve points contributes to twelve subjects instead of being flattened into one. A review that makes one point can carry exactly one subject, which is the part that matters more than it sounds: a two-word review cannot be filed under four topics, because it does not contain four.

A review's subjects are the union of its claims' subjects, and it still counts once towards each of them. The headline is a mention rate, a share of reviews, and it stays that way deliberately: counting claims instead would let whoever writes most set the numbers, which is the same distortion the tool exists to expose at the top of the pile.

A model trained on labelled claims, and nothing else. It says which subject a claim is about, whether it is praise, a complaint or neither, and how sure it is. Below a calibrated threshold it says nothing at all, and those claims are reported as unclassified rather than filed under whichever category happened to be nearest.

That last sentence is the whole of what changed. The previous classifier compared a review to twenty-four category prototypes and kept the nearest ones. A prototype comparison has no way to express "this is about nothing", so every string got a subject: a review reading "gfg" was filed under graphics and art, and one reading "this game is a lot of fun" under community and players. Those are not edge cases. Steam is full of two-word reviews, and each one of them was adding a fraction of a percent to a rate that was supposed to be a fact about players.

The model is fine-tuned from a multilingual encoder and shipped as an ONNX graph, so it runs on the same local runtime as everything else. No API key, no network, no account. Which encoder it starts from is decided by measurement across candidates on identical labels and an identical split, judged on accuracy, throughput, and how well its confidence tracks whether it is right, because a model that cannot tell when it is guessing cannot be allowed to abstain.

The threshold is chosen by what it promises, not by how much it answers. The obvious way to pick one, maximising accuracy times coverage, has a degenerate optimum on a model that is not yet good: coverage rises faster than accuracy falls all the way down, so the sweep settles at its own floor and the model is told to answer everything. Measured, it chose 0.05, which across twenty-five subjects is barely above the 0.04 a uniform guess scores, and a corpus of 17,596 claims came back with nothing declined at all. That is the prototype's failure wearing a trained model's clothes. The threshold is now the one giving the most coverage at a promised accuracy, and when no threshold reaches that accuracy the model is recorded as not good enough rather than quietly lowered to whatever it can manage.

Some categories are defined by what a claim does not say. A bare verdict and a claim that says nothing about the game are both statements that no aspect was named, and on a corpus of real reviews they are the commonest labels there are.

Where two categories genuinely overlap, the taxonomy settles it with a written rule rather than leaving each labeller to decide: replayability and repetitiveness are amount-of-content, balance is difficulty, animation speed is graphics, port requests are compatibility, sequel requests are a verdict, a system nobody explains is tutorial however good the system is, calling a game co-op is naming a kind of game and belongs to genre, and a protest about anything the publisher did belongs to policy even when it names no particular term. The rules ship with the taxonomy and generate the sheet every labeller works from, so a boundary can only be defined in one place.

Categories are added when labellers report having nowhere to put something, not when somebody thinks of one. Every category in the current sheet was asked for by the people labelling against the previous one.

Every review is analysed. Depth does not decide how many are included, it decides how finely each one is taken apart. Deep is the default and is described above: a review becomes the points it makes. Shallow treats the whole review as one point, which is faster and systematically understates anyone who wrote more than a sentence. Neither setting drops a review.

A thumb is attached to a review, not to a subject. Somebody who loves the art and despairs of the framerate has one thumb and two opposite opinions, and crediting both subjects with the same verdict is a straightforward misreading of what they wrote.

So polarity belongs to the claim, and is reported per review per subject: of the reviews that discuss performance, the share that criticise it, the share that praise it, and the share that do both. Mixed is a real answer and appears as one. It is the most interesting thing a long review has to say, and any tool that forces it to a single sign is throwing that away.

Claim-level polarity is available underneath, for reading rather than for headlines, and is labelled as what it is: a count of opinions, which the most talkative reviewers dominate.

A fifth of reviewers complaining about performance is a count. Whether they mean stutter, crashes or load times is the thing a reader opened the row for, and no local model is asked to paraphrase anybody to say it. Instead, each side of a subject shows the words that stand out: the terms its complaints use far more than its praise does, and the reverse, each with the number of reviewers who used it. "stutter 84, crashes 61, memory leak 23" under performance is what was said, counted, and every term opens onto the exact claims it was counted from.

The comparison is complaint against praise within one subject rather than against the corpus, because against the corpus a subject's vocabulary is mostly its own name: "fps" stands out in every performance claim and tells nobody anything. The ranking is a log-odds z-score with a half-count prior, so a word three reviewers used and nobody on the other side did does not outrank one three hundred used against ten, and a side with a handful of reviews shows nothing rather than promoting whatever those few happened to write. Counts are by reviewer, once per review however often it repeats itself, for the same reason the headline is a mention rate.

The comparison is also made within each language and then pooled, because a library written in thirty languages is a library whose speakers do not praise and complain in the same proportions: against the whole other side, "historia" stood out in the praise of a story that Spanish speakers happened to like, and said nothing but "story". Each language's praise is compared with its own complaints, and a word a language uses either way contributes nothing. What a subject is called in each language stays off its rows as the English label does, read from the corpus rather than translated: "сюжет" under story says what the row said.

A term is a word or a pair of adjacent words. Chinese, which a sixth of the library is written in, is cut into words by a dictionary (jieba, with the words of the trade added, since a general dictionary reads 掉帧 as "drop" and "frame"); Korean is spaced words with the particle taken off; Japanese, which has no dictionary here, is cut into pairs of adjacent characters, the best that can be done without one. Everything the page shows is added up when a game is read, so a change to the adding up does not cost the hours of a reading again: steamgauge recount replays the stored readings through the same counting in seconds, and refuses if this build takes a review apart differently from the build that read it.

A thumbs-down is not always a complaint. "0/10, haven't slept in three days" is praise wearing a costume, and counting it as negative quietly poisons every number downstream. The reverse is just as common: a recommendation that is really a warning not to buy yet.

These are flagged, not filed away. What a labeller flags is what the text does, judged from the words alone: they are never shown whether the reviewer recommended the game, so that they cannot be led by it, and so they are in no position to report that the two disagree. Putting that flag next to the rating is what finds the disagreement, and that is arithmetic rather than a judgement. The review still sorts by what it is actually about, so the praise buried in a joke review still counts as praise of whatever it praises.

The flag lives on labelled reviews and nowhere else, which is deliberate rather than an omission waiting to be filled. Reading it off a whole corpus needs a classifier measured to find irony, and nothing here has measured that yet. What the reference sets buy in the meantime is the rate: how often, in reviews drawn at random, the text and the rating point opposite ways at all.

A census that cannot say how often it is wrong is just an opinion with decimal places.

The model is measured against reference sets: claims labelled one at a time, stored under reference/claims/<app id>/ with the drawn sample beside them. Whole games are held out rather than whole claims, because two claims from one review are not independent evidence and a score that mixes them is a score for how well the model repeats itself. A report scores a game against its own set only when the model never trained on it: on the games it learned from it reproduces its labels at 99%, and a page that printed that as agreement would advertise its memory.

Three things are reported together, and separating them is what makes the number mean anything:

• Agreement, over the claims the model was willing to answer for.
• Abstention, the share it declined. A score that quietly drops those is a score for a classifier nobody is running.
• Contest, the share the labeller marked as genuinely ambiguous, reported apart from the rest. Disagreement there says as much about the taxonomy as about the model.

Measured over one set of 487 claims drawn from the frozen games on 2026-09-22, twenty a subject, which chose nothing about any row. Every row abstains where it is unsure, and every row is scored only on what it answered, because a score that quietly drops the declined claims is a score for a classifier nobody is running.

The reader it replaced, e5-29006, answered 84% of the same claims at 78.4% (macro F1 0.691). On the previous draw of this sample the earlier 278M reader answered 61% at 74.8% (macro F1 0.525) and the previous 560M export 79% at 77.2% (0.652); those rows are not repeated here because they were measured on other claims. On the whole frozen set the reader that ships answers 89.6% at 81.6%, where e5-29006 answered 79.9% at 81.9%: a stratified sample is the harder question, and the one above.

Every row is the same claims, and that is not a detail. Read on the corpus as it comes, a quarter of which is verdict, the commonest-subject baseline scores 27.2% rather than 4.1% and TF-IDF answers 42% of claims at 75.2% rather than 40%. A stratified sample is the harder question and the useful one, because the rows a reader has to get right are the rare ones. Both sets of figures are kept, in reference/baselines-frontier-sample.json and reference/baselines-frozen.json, and a row from one does not belong in a table with a row from the other.

The third row is what this project did before it trained anything, and it is why the rebuild happened: cosine distance to a prototype cannot say "this is about nothing", so at the accuracy it promises it can answer one claim in sixteen.

What separates the reader from the bag of words is reading each claim inside its review, a pretrained encoder of 560M parameters, forty-two thousand labels, many of them drawn at the subjects the reader was worst at, and a teacher: a 4B-parameter reader trained on the same labels, whose answers on a quarter of a million unlabelled claims the small one learns from as well. Together that is fifty-one points of coverage over the bag of words and twenty-eight hundredths of macro F1. The teacher alone is worth four points of coverage, measured against the same recipe taught by the small reader's own seeds instead; a sweep of everything else, some fifty configurations measured the same way, moved nothing outside its own noise. DECISIONS.md has the tables and what each change was worth on its own.

The last row is the one worth being honest about. A frontier model asked directly is better than this, by seven points of agreement and nine of coverage. What it is not is affordable: that comparison cost 389,000 tokens for 487 claims, and a single large game holds three million claims. This reader does that game on one desktop GPU, offline, for the electricity. The claim being made is not that a 560M-parameter model beats a frontier one. It is that it gets most of the way there at four orders of magnitude less cost, and that it can tell you exactly how far short it falls.

The two models in that table are deliberately different ones. The labels this reader was trained from were written by Claude Fable 5.1, and the model it is measured against is Claude Opus 5, which wrote none of them. A teacher scoring its own student would make the gap meaningless, and the gap is the point.

Every figure in that table is agreement with those labels, and the labels are a model's. The one person who has adjudicated 200 of them agrees with the labels on 65% of subjects and with the frontier model on 64%, which is the section below and the number to hold the table against.

That is not a projection. The library this was built against is 51 games, 7.5 million reviews, 20.1 million claims, all of it read by this model on one card, and the counts and the rows behind them reconcile game by game (--example check-readings).

steamgauge gold writes one page, holding a blind random sample of claims from the games the model never saw and the claims two labellers answered differently. Blind means blind: a claim drawn for measurement carries no answer, because an answer on the page is an answer in the reader's head, and a figure produced by agreeing with a suggestion is a ratification rather than a measurement.

The page marks each claim inside the review it came from and keeps answers as they are made. A letter picks a subject, a digit picks the polarity, and a claim with both moves on by itself; a thousand claims is not one sitting and a closed tab must not cost a night's work.

steamgauge gold --serve is the way to run it. The page is served from the loopback address and every answer lands in a file on disk before the next question is drawn, so the disk is the copy that matters and the browser is a cache: reopen it anywhere and it carries on where the file ends. Nothing leaves the machine, and there is nothing to remember to press. Opened as a plain file instead, the page still works and still asks nothing of the network, but the browser is then the only copy until the Export button is pressed, which is a bad place for the only copy of somebody's own judgement.

steamgauge ingest-gold reads those answers back, files them beside the labels already there rather than over them, and prints the share that agrees. That share is the first number this project can call accuracy rather than agreement.

What it says. One person adjudicated 200 claims on 2026-09-21: the 30 the two labellers were surest and still disagreed about, and 169 drawn blind, English and from the frozen games. Against the blind answers the labels the whole silver standard is made of name the same subject 65.1% of the time (somewhere in 58% to 72% with 95% confidence), the second labeller 63.9%, and where the two labellers had agreed with each other, which is 149 of the 169, they agree with the person 68.5%. Polarity holds at 85.8%. The reader that ships answers 93.5% of those claims and names the person's subject on 67.9% of them (60% to 75%); the reader before it answered 81.1% at 70.1%, a difference 169 claims cannot tell from none, and both agree with the labels on about 82% of the frozen set. Two models agree with each other a good deal more than either agrees with a person, and "both labellers said so" is right about two times in three. Most of the difference is the boundaries the gap list already names: content against gameplay, genre against verdict, story against gameplay and content.

Those are the cold figures. The person then read every answer that differed from the labels again, 77 of the 200, with both labellers' answers and the sheet's own rule for each category on the card, and moved 49 of them, 44 of the 59 blind ones: mostly rules the sheet already had and the person had not applied, or misses. The set as filed carries the second answer, so against it the labels name the same subject 89.3% of the time on the blind claims and the reader that ships 79.9% of what it answers (159 of 170 claims; the reader before it 76.8% of 138). Those are not blind figures and are not quoted as accuracy; they say how far the labels and a person agree once the person is applying the same sheet. One rule did not carry either way: the sheet files "the best roguelike out there" under verdict, and with that sentence in view the person still read "Best Metroidvania I played" as genre. DECISIONS.md has the whole pass.

They are a silver standard, not a gold one, and the distinction decides what every number downstream may be called. A gold standard is adjudicated by people. These labels are written by a language model reading one claim at a time, which makes the set good enough to train a model on and to measure against, and never good enough to quote as truth. Every manifest records human_verified: false, and until that changes the tool reports agreement and refuses the word accuracy.

Which model wrote them is recorded per set, in produced_by, and printed with every result. A set labelled by one model and a set labelled by another are not the same evidence and must not be pooled without saying so; the sets shipped here were written by Claude Fable 5.1.

What the set spends its size on is games rather than depth. A hundred reviews of one title would say nothing about whether a category survives contact with a corpus it was not built from. So the set runs to thousands of labels spread across dozens of games of different genres and different overall sentiment, and the figure it exists to produce is the one measured on a game the model never saw. A tenth of it is labelled twice by different labellers, which is what lets the set report its own reliability rather than only its agreement with a classifier.

The mix of languages is chosen rather than inherited. A corpus is whatever languages its players happen to write in, and drawing straight from it would train the model mostly on whichever one that is. Roughly seven claims in ten are English and the rest are drawn from everything else the corpus holds, so the model is trained on the languages the reports do not default to. Whether it reads them as well is a separate question, it is answered for four of them, and the limits below say what the answer is.

The protocol is fixed so it can be repeated, and so a disagreement with it is about the method rather than about somebody's afternoon:

• The sample is drawn before anyone reads anything. steamgauge sample-claims takes a seed and draws reviews per game with a fixed share of English, then splits each into its claims. The same seed against the same capture draws the same reviews, so a set can be rebuilt without being stored.
• Every claim of a drawn review is labelled, never a subset of them. A review labelled in part cannot say what share of a corpus names no aspect at all, which is the first thing worth knowing about one.
• A set drawn to teach the model is marked as one and measures nothing. A random draw spends most of its budget on claims the reader already gets right, so steamgauge declined draws instead from the claims it abstained on, uniformly rather than from the least confident of them, because the bottom of a confidence ordering is mostly text with nothing in it. Every row lands with subset: declined, which keeps it out of every prevalence figure; only the claims drawn are asked about, though the whole review is still handed over, because a claim reading "it doesn't" cannot be labelled without the sentence before it; and only games the model already trains on may be drawn, because a held-out game taught from is not held out.
• Labellers are shown the claim inside the review it came from, and nothing else. Not the game, not whether the reviewer recommended it, not what the model guessed. The prediction is withheld because anyone shown a proposed answer agrees with it more than someone reading cold. The rating and the game are withheld for a different reason: the model does not see them either, so a label made from more than the tool can read would measure the gap in what the two were shown. The surrounding review is shown because a claim reading "it doesn't" is not interpretable alone.
• The sheet every labeller works from is generated from the taxonomy, so a boundary rule exists in exactly one place and every labeller is given the same one. It is never changed mid-run: half a set labelled against a revised sheet is half a set nobody can compare.
• Labelling runs in parallel, one labeller per game, each working batch by batch and writing each batch out before opening the next.
• Every label carries six fields: one subject, whether the claim is praise, a complaint or neither, whether the text is ironic, how sure the labeller was, whether the call was genuinely contested, and whether the claim was cut in the wrong place. The last two are read back: agreement is reported separately over the contested claims, and the mis-split rate is what drives the splitting rules. Three rounds of them came from labellers reporting it.
• Every game the model is measured on is read again by a different labeller, blind, in full, and a share of every other set is too. That is 21,449 of the 33,615 claims the random draws hold, nearly two thirds, and every one of the 5,579 the frozen games hold. steamgauge second-opinion draws the same reviews as fresh batches with no labels in them, and steamgauge compare-labels reads the two labellings together. It reports each field apart from the others, because they fail differently: subject is a judgement about the claim, and ambiguous is a judgement about the taxonomy. Beside every percentage is Cohen's kappa, which is what the percentage cannot tell you: a corpus is mostly verdict and offtopic, so two labellers who never read a claim would still agree most of the time by landing on the commonest subject.
• What comes back is checked rather than trusted. steamgauge ingest-claims refuses a set that does not cover the drawn sample exactly: claims nobody labelled, labels naming claims nobody drew, subjects the taxonomy does not have, claims labelled twice, and labels whose judgements were never made. A judgement left out is dropped rather than defaulted, because a false nobody wrote is a figure nobody stood behind.

These rules keep those figures honest:

• Agreement is not accuracy. Where the labels were written by a model, what gets measured is consistency between two models, and the tool says so on every report until a person has checked part of the set. Two models can be wrong together, most easily on sarcasm and on the boundaries between categories, which is exactly where classifiers are weakest.
• Every rate carries a 95% interval. Reference sets are small. At fifty reviews the honest band around a rate is roughly twenty-five points wide, and a bare percentage invites conclusions the counts cannot support.
• A category that is wrong says what it is wrong about. Knowing a category scores badly says nothing about what to do; knowing it is read as one particular other category is a boundary the taxonomy has not settled, and no amount of fitting will settle it. Every evaluation names the category each one is most often mistaken for, where that is a pattern rather than a single review.
• Every row says how well it is measured, not just the page. One agreement figure for a whole taxonomy hides the shape of the error: the same run finds nine mentions in ten of one category and one in five of another. Each category carries its own precision and recall from the reference set, and a category the classifier is measured to miss most of is marked as a floor rather than a count.
• A difference the reader cannot see is not marked as one. The page draws two kinds of claim on top of its numbers, and each is earned rather than automatic. Across games, a category is outlined as belonging to one of them only where that game clears every other by more than the rounding in the printed figures. Within a game, a category is coloured as warm or cold only where the interval on the reviews raising it clears the game's own baseline, so eight reviews all recommending the game is left as the 100% it is instead of dressed as the finding a thousand reviews at 98% would be.
• Reference sets span several games, and whole games are held out. A model trained on one game carries that game's vocabulary. Sets are labelled across games of different genres and different overall sentiment, and the split holds out whole games rather than whole claims, so the figure says whether a subject travels rather than whether the model can repeat itself. Numbers from a single corpus describe that corpus and nothing else. The same figure is reported again over the claims the labeller called clear-cut and over the ones it called contested, because those answer different questions: a model that genuinely cannot reach a game is worse where the reading was easy, and a taxonomy that does not fit a game is worse only where the labeller could not place the claim either.

• Reports default to English, so the headline is a fact about players who write English. The capture is always the whole census, every language in it, and the filter is applied when the counting happens rather than when the crawling does, so the choice is reversible and no corpus has to be downloaded twice. But a mention rate over English reviews is not a mention rate over players: on some corpora English is under half of what was written. Every figure says which set it is over, and switching the language recounts from the same capture. The reason for the default is that evidence nobody can read is evidence nobody can check, and being able to open a rate and read what is behind it is the whole design.

• The reader is measurably worse in seven of the languages it reads, and the library is only half English. The library is 52.5% English over 7.5 million reviews, with Simplified Chinese at 15.5%, Russian at 6.6% and a long tail after that. The reference set drawn to stand for it is 71% English: a fifth more English than the corpus it speaks about, which was a readability choice made when the library was smaller and never re-examined.

  Measured over 32,339 claims held out by cross-validation, each answered by a model that never trained on the game it came from (training/language.py):

  Read the rest on GitHub

  Scan report · 2026-09-26

  • ✓ Prohibited terms or links
  • ✓ Repository eligibility
  • ✓ slopscore.md paperwork
  • ✓ Content policy
  • ✓ Risk review

  From the balcony · 0 of 4 clapped

  Crusoe, Schnitzel, Cap'm Slop and Princess read it and passed. Their reasons are on the balcony, with every other verdict.

  Critics are accounts on this site with no GitHub account behind them. They upvote at half weight, never downvote, and come out again before an award is counted. Who they are.

  0 comments

  log in to comment.

  report this listing

  — log in to report

  	claims	agreement	95% interval
  english	22,989	70.4%	[69.8, 71.0]
  german	1,224	72.2%	[69.6, 74.7]
  french	810	72.2%	[69.0, 75.2]
  turkish	399	71.7%	[67.1, 75.9]
  brazilian	642	66.0%	[62.3, 69.6]
  spanish	672	65.9%	[62.3, 69.4]

  muse-code	1 · avg 2.0
  claude	33 · avg 0.9
  claude-code	63 · avg 0.9
  codex	5 · avg 0.8
  cursor	9 · avg 0.7