Skip to content
Research5 min read

We checked our own forecast range. It held far less often than we claimed.

A forecast range is a promise about how often reality lands inside it. We measured whether ours kept that promise, night by night, found it fell well short, and rebuilt it until it held close to what it claimed.

Matty HerreraBy Matty Herrera, Founder & CEO, Delta Kinetics

Published

Delta Kinetics research card for Forecast Accuracy: a range sold as 80% held 57 to 72%

On this page
  1. Key takeaways
  2. What does a forecast range promise?
  3. How we checked whether ours kept the promise
  4. Did the range hold as often as it claimed?
  5. What did we change?
  6. What it holds now
  7. What this means when you read any forecast
  8. The limits of this
  9. Frequently asked questions
  10. Sources

Key takeaways

  • Sold as 80%, the original range held 72.3% of 112 test nights and 57.1% of the 28 most recent ones.
  • The rebuilt range held 78.4% of 111 test nights and 82.1% of the 28 most recent, against a label of 80%.
  • The best-guess number barely changed: a typical miss was 25.9% of the night's sales, against 26.1% before. The gain is in the range.
  • The rebuild used 480 nights across two venues, and the range was scored only on nights the forecast had not seen.
  • The result covers two venues and one time of day, and every scored night came from one of the two venues.

Frequently asked questions

What does an 80% forecast range mean?

It means that on about 8 nights in 10, the actual result should land inside the range. It is a promise about how often the range holds, which is different from how close the single best-guess number is.

How did you check whether the range kept its promise?

We compared the range the forecast had stated with what the venue actually sold, night by night, on nights the forecast had never seen when it was built, and counted how many nights landed inside.

How often did the original range hold?

Sold as 80%, it held 72.3% of 112 test nights, and 57.1% of the 28 most recent ones.

How often does the rebuilt range hold?

78.4% of 111 test nights, and 82.1% of the 28 most recent ones, against a label of 80%. It was built from 480 nights across two venues.

Does this prove the forecast works for any venue?

No. It covers two venues, one time of day, and test nights that all came from one of the two venues. We make no claim beyond that.

Sources

  1. Delta Kinetics, CoreTAP, the product whose forecast range was scored
  2. Delta Kinetics, research methodology: exact measurement and the method we had to throw away

See it read your floor live

Bring two weeks of trading. That is usually enough to tell a real pattern from one busy Saturday.