Exact measurement, and the method we had to throw away

What was wrong with the first method?

Our original approach handed a model 110 sampled reviews from a venue and asked it to summarise them and rank the venue's biggest problems. Cross-model agreement on that ranking was 33 to 37 percent. Two good models looking at the same venue disagreed about its top issue roughly two times in three.

The instinct is to blame the model. On re-examination the fault was in how we asked. Three defects, all ours:

• We asked for priority issues without defining priority by what. Frequency? Severity? Revenue impact? Each model quietly picked a different criterion and then appeared to disagree.
• We asked the model to eyeball a share of negatives as prose, roughly a third. That is unmeasurable and unverifiable.
• We asked a single call to summarise 110 reviews and rank issues at the same time. Two tasks, one of them badly posed.

A better model does not fix a badly specified task. Rebuilding the task does.

What replaced the first method?

The model now does exactly one job: for a single review, which aspects does it evaluate, and with what polarity. It never counts, never ranks, never summarises, and never sees a venue's aggregate. Every count, percentage and ranking happens deterministically in Python afterwards.

Rules that matter:

• An aspect is labelled only on an evaluative statement, not a mention. "We ordered the steak" gets no label. "The steak was dry" gets food quality, negative.
• Polarity is judged per aspect, independent of the star rating. A one-star review saying the food was great but service was awful produces a positive food label and a negative service label.
• Confusable pairs get explicit boundaries: service speed against wait time, value for money against pricing clarity, atmosphere against noise level against music.
• An empty label set is a correct outcome. "Great!" evaluates nothing.

Scale: all 13,063 reviews with text, no sampling. 436 batches, zero failures, 13,063 of 13,063 labelled.

How were the labels validated?

Reproducibility

Re-running the same model with different batch groupings produced 98.9 percent polarity agreement on shared aspects across 3,118 comparisons, with a mean Jaccard of 0.859 on aspect and polarity sets.

Cross-model, on the full corpus

A second frontier model was then run over the entire corpus rather than a spot check: 13,063 of 13,063, zero failures. The comparison:

• 98.1 percent polarity agreement across 23,993 aspect pairs
• 29,273 aspect labels from the first model, 28,876 from the second
• Every individual aspect above 96 percent agreement; cleanliness and pricing clarity at 100 percent
• Exact set match 51.7 percent

The gap between 51.7 percent exact-set match and 98.1 percent polarity agreement is the whole point. Direction is solved. What differs between models is coverage threshold, meaning how readily each decides an aspect is under discussion at all. When both models agree an aspect is present, they agree on praise versus complaint 98.1 percent of the time.

Consensus labels

Taking only labels where both models independently named the same aspect with the same polarity gives 23,538 consensus labels. Every headline number was recomputed on all three label sets, first model only, second model only, and consensus. No conclusion in the research depends on which model produced the labels.

Human audit, and a defect we found

We read a stratified sample of 21 labelled reviews against their raw text. Nineteen were correct. The audit surfaced one real defect: a review labelled both positive and negative on the same aspect instead of using the available mixed value. Measured across the corpus that affected 94 reviews of 13,063, or 0.72 percent, and 95 aspect collisions. It was repaired deterministically by collapsing conflicting polarities to mixed.

The audit also found consistent under-labelling of vague praise. "Fantastic, Ruben was my waiter" received no label, which is rule-consistent because no aspect is explicitly evaluated, but it means our aspect volumes are a floor rather than a ceiling.

How is a venue's top issue ranked?

A venue's number one issue is now simply the aspect with the most negative mentions. Run it again and you get the same answer, permanently. That change nearly doubled cross-model agreement on the top issue, from 33 to 37 percent under the old method to 63 percent.

The residual disagreement turned out not to be error. Of the 15 venues where the two models picked different leaders, 12 were near-ties, with the top two aspects within 25 percent of each other on mention count. Only 3 of 41 venues showed genuine disagreement about a clear leader.

Which produced a structural finding we did not go looking for: only 9 of 41 venues, 22 percent, have a clear single leading issue. For the other 78 percent the top complaints are effectively tied.

Do the labels behave the way they should?

The distribution behaves the way it should. Nobody writes about pricing clarity unless they are confused by a bill, and 99.0 percent of pricing clarity labels are negative. Nobody mentions noise unless it is too loud, at 88.3 percent negative. Meanwhile atmosphere is the least complained-about aspect in the entire corpus at 6.4 percent negative across 3,803 labels, which independently confirms, by a completely different measurement path, that atmosphere is a floor rather than a differentiator.

What are the limits of the research?

• Restricted range. All 41 venues are top-10 alcohol earners in their city. Every correlation is a comparison among winners. None is causal, and none generalises to venues that are not already succeeding.
• Scope is four cities. Houston, Katy, Sugar Land and The Woodlands. The pipeline runs anywhere and the state data goes back to 2007, but we have not run it elsewhere.
• One venue excluded. One venue whose review corpus we determined was not authentic. We do not name it or its category. Its tax filings remain valid and are still used; its review data is not.
• Yelp is about 5 percent of the review blend. Houston's venues are Google-only.
• Aspect volumes are a floor. Conservative under-labelling of vague praise means true counts are somewhat higher than reported.

Why publish the method at all?

Because a finding you cannot check is a claim, not a result. Everything above is here so that somebody who disagrees with our conclusions can locate exactly where they think we went wrong. That includes the part where we were wrong the first time.