Loading
Loading
This is the volume that makes the rest of the research checkable. It also documents a mistake we made, why it happened, and what replaced it. That correction is the most useful thing in here.
Our original approach handed a model 110 sampled reviews from a venue and asked it to summarise them and rank the venue's biggest problems. Cross-model agreement on that ranking was 33 to 37 percent. Two good models looking at the same venue disagreed about its top issue roughly two times in three.
The instinct is to blame the model. On re-examination the fault was in how we asked. Three defects, all ours:
A better model does not fix a badly specified task. Rebuilding the task does.
The model now does exactly one job: for a single review, which aspects does it evaluate, and with what polarity. It never counts, never ranks, never summarises, and never sees a venue's aggregate. Every count, percentage and ranking happens deterministically in Python afterwards.
Rules that matter:
Scale: all 13,063 reviews with text, no sampling. 436 batches, zero failures, 13,063 of 13,063 labelled.
Re-running the same model with different batch groupings produced 98.9 percent polarity agreement on shared aspects across 3,118 comparisons, with a mean Jaccard of 0.859 on aspect and polarity sets.
A second frontier model was then run over the entire corpus rather than a spot check: 13,063 of 13,063, zero failures. The comparison:
The gap between 51.7 percent exact-set match and 98.1 percent polarity agreement is the whole point. Direction is solved. What differs between models is coverage threshold, meaning how readily each decides an aspect is under discussion at all. When both models agree an aspect is present, they agree on praise versus complaint 98.1 percent of the time.
Taking only labels where both models independently named the same aspect with the same polarity gives 23,538 consensus labels. Every headline number was recomputed on all three label sets, first model only, second model only, and consensus. No conclusion in the research depends on which model produced the labels.
We read a stratified sample of 21 labelled reviews against their raw text. Nineteen were correct. The audit surfaced one real defect: a review labelled both positive and negative on the same aspect instead of using the available mixed value. Measured across the corpus that affected 94 reviews of 13,063, or 0.72 percent, and 95 aspect collisions. It was repaired deterministically by collapsing conflicting polarities to mixed.
The audit also found consistent under-labelling of vague praise. "Fantastic, Ruben was my waiter" received no label, which is rule-consistent because no aspect is explicitly evaluated, but it means our aspect volumes are a floor rather than a ceiling.
A venue's number one issue is now simply the aspect with the most negative mentions. Run it again and you get the same answer, permanently. That change nearly doubled cross-model agreement on the top issue, from 33 to 37 percent under the old method to 63 percent.
The residual disagreement turned out not to be error. Of the 15 venues where the two models picked different leaders, 12 were near-ties, with the top two aspects within 25 percent of each other on mention count. Only 3 of 41 venues showed genuine disagreement about a clear leader.
Which produced a structural finding we did not go looking for: only 9 of 41 venues, 22 percent, have a clear single leading issue. For the other 78 percent the top complaints are effectively tied.
The distribution behaves the way it should. Nobody writes about pricing clarity unless they are confused by a bill, and 99.0 percent of pricing clarity labels are negative. Nobody mentions noise unless it is too loud, at 88.3 percent negative. Meanwhile atmosphere is the least complained-about aspect in the entire corpus at 6.4 percent negative across 3,803 labels, which independently confirms, by a completely different measurement path, that atmosphere is a floor rather than a differentiator.
Because a finding you cannot check is a claim, not a result. Everything above is here so that somebody who disagrees with our conclusions can locate exactly where they think we went wrong. That includes the part where we were wrong the first time.
The same discipline, applied to your venue, on your own data, every shift.