On this page
A forecast range sold as 80% held between 57% and 72% of nights when we checked it, far less often than we claimed. After a rebuild, the range held 78.4% of 111 test nights it had never seen.
Key takeaways
- Sold as 80%, the original range held 72.3% of 112 test nights and 57.1% of the 28 most recent ones.
- The rebuilt range held 78.4% of 111 test nights and 82.1% of the 28 most recent, against a label of 80%.
- The best-guess number barely changed: a typical miss was 25.9% of the night's sales, against 26.1% before. The gain is in the range.
- The rebuild used 480 nights across two venues, and the range was scored only on nights the forecast had not seen.
- The result covers two venues and one time of day, and every scored night came from one of the two venues.
What does a forecast range promise?
A forecast is a number with a range around it. When CoreTAP shows where tonight's sales are heading at 9 p.m., it also shows a range, and the range carries a label: 80%. That label is a promise. On about 8 nights in 10, the night's actual sales should land inside the range.
The promise is the useful part. A manager deciding how many people to keep on, or how much to prep, is planning against the range, not against the single number. If the range holds less often than its label says, the plan is built on certainty that was never there.
How we checked whether ours kept the promise
We took past nights and, for each one, put the range the forecast had stated next to what the venue actually sold. Then we counted the nights that fell inside. If the label says 80%, about 80% of nights should.
We scored it on nights the forecast had never seen when it was built. A forecast checked on the nights it was tuned against always looks better than it really is.
Did the range hold as often as it claimed?
| Which nights | Range claimed | Nights scored | Nights it held |
|---|---|---|---|
| All test nights | 80% | 112 | 72.3% |
| Most recent stretch | 80% | 28 | 57.1% |
The most recent stretch was the one where the forecast's single best-guess number was at its most accurate. It was also the stretch where the range was at its least reliable. A better headline number had been hiding a range that was too narrow for what it claimed.
What did we change?
We did not adjust the label. We went after the measurement first, then rebuilt until the range could be checked against results and pass.
- We started recording every forecast and scoring it against what happened. Before this, the accuracy shown to a venue was a figure fixed when the forecast was built, not a record of how it had done.
- We fixed what fed the forecast. Nine data problems turned up, such as an input that was never actually delivered and a figure that had been mapped onto the wrong quantity. Every one of them still produced plausible numbers, which is why none had been noticed.
- We judged each venue's range against that venue's own record. Venues differ in how hard they are to predict, and one shared allowance was too tight for the hard venue and too loose for the easy one.
- We let the range respond to results. Whether recent nights landed inside it feeds back into how wide the next range is drawn. In live use that step begins once ten nights have been scored.
- We set a pass mark before anything shipped. The range had to hold between 78% and 86% of test nights. It was refused more than once on the way. A version of the rebuild that held 73% of nights was not released.
What it holds now
| Which nights | Range claimed | Nights scored | Nights it held |
|---|---|---|---|
| All test nights | 80% | 111 | 78.4% |
| Most recent stretch | 80% | 28 | 82.1% |
The rebuilt forecast was built from 480 nights across two venues and scored on nights it had not seen. Its range now holds a little under what it claims, not far over and not far under. The best-guess number barely changed: a typical miss was 25.9% of the night's sales, against 26.1% for the earlier forecast on the same nights. The improvement is in the range, the part people plan against.
What this means when you read any forecast
Ask how often its range actually holds. A forecast that shows a range without a track record is asking you to trust a label. Ask to see the range next to the result for past nights, scored on nights the forecast did not train on. In our view, a range that holds well below its label can mislead more than having no range at all, because people plan against it.
The limits of this
- Two venues. We make no claim about other venues, other places or other times of year.
- The test is smaller than the build. The forecast was built from 480 nights, but the range was scored on 111 test nights, and all of them came from one of the two venues. The second venue's range was not scored separately here.
- Small counts move. The most recent stretch is 28 nights. A handful of nights going the other way would shift those percentages by several points.
- Not even across the week. The range held noticeably less often on some days of the week than others.
- One time of day. Everything on this page is the 9 p.m. forecast.
- A figure belongs to a set of nights. Scoring the earlier forecast on different versions of the data gave between 63% and 72%. We quote the original measurements and say which nights they cover.
- We changed several things at once. We cannot say which change produced the improvement.
- Live nights come next. Every forecast is now recorded and scored, so the next check is on real nights as they happen, and we will report what we find.
Frequently asked questions
What does an 80% forecast range mean?
It means that on about 8 nights in 10, the actual result should land inside the range. It is a promise about how often the range holds, which is different from how close the single best-guess number is.
How did you check whether the range kept its promise?
We compared the range the forecast had stated with what the venue actually sold, night by night, on nights the forecast had never seen when it was built, and counted how many nights landed inside.
How often did the original range hold?
Sold as 80%, it held 72.3% of 112 test nights, and 57.1% of the 28 most recent ones.
How often does the rebuilt range hold?
78.4% of 111 test nights, and 82.1% of the 28 most recent ones, against a label of 80%. It was built from 480 nights across two venues.
Does this prove the forecast works for any venue?
No. It covers two venues, one time of day, and test nights that all came from one of the two venues. We make no claim beyond that.
