The same model, with and without Qloo
Every metric, interactive
Each bar is the same model answering the same questions, once alone and once with Qloo's data in its
prompt, scored against a source outside Qloo. Pink is Qloo. Hover anything for the exact numbers.
Every test where Qloo helped
The results, test by test
Each test was registered before it ran (PREREGISTRATION files, committed first), with its data
source, metric and decision rule fixed in advance. 95% intervals by bootstrap.
Restaurant hours: a model with Qloo against the same model alone
Question. Asked when a real New York restaurant closes on Friday, is a model right more often with
Qloo's place record?
Data. 30 places, mostly small independents, whose Friday closing time a person read off Google Maps.
- GPT-6 Astra
- 0/30 alone, 29/30 with Qloo
- Gemini 3.8 Flash
- 3/30 alone, 29/30 with Qloo
- Gain
- +96.7 points (90 to 100) and +86.7 points (70 to 100)
Shared fans: the clash score's input, against real listeners
Question. Does a model rank pairs of acts by the fans they really share better with Qloo's scores?
Data. 10 bills of 30 real artists each, 4,350 pairs, against the Last.fm listeners each pair shares.
- GPT-6 Astra
- 0.697 alone, 0.728 with Qloo (+0.031, 0.005 to 0.056)
- Gemini 3.8 Flash
- 0.695 alone, 0.723 with Qloo (+0.028, 0.003 to 0.056)
- Qloo alone
- 0.723, level with GPT-6 Astra, with no model call
Timetables: how close to the best possible day
Question. Build a timetable from each scorer's numbers: how much of the real clash does it remove,
from a random timetable (0) to the best possible one (1)?
Data. The same 10 bills, judged by real shared listeners.
- GPT-6 Astra
- 0.669 alone, 0.702 with Qloo (+0.033, 0.000 to 0.063)
- Qloo
- 0.663, against 0.043 for popularity alone (+0.62, significant)
Small artists: who will a real listener play?
Question. Among 35 small artists, can a scorer find the 3 to 5 a real listener actually played?
Data. 40 real Last.fm listeners (GroupLens HetRec 2011). AUC, higher is better.
- GPT-6 Astra
- 0.865 alone, 0.874 with Qloo (+0.010)
- Qloo alone
- 0.841, against 0.516 for popularity (+0.324, significant)
Consistency: the same question, asked twice
Question. Score every pair of acts on a real Lollapalooza day for shared fans, twice. How many answers
move by 10 points or more?
Data. Five real days, 903 to 946 pairs each.
- GPT-6 Astra alone
- 4 of 946, then 9, 9, 13 and 46 of 903 pairs moved
- Qloo
- 0: the same score every time, so the court's ruling is repeatable
Identity: is this the place, or the act, you mean?
Question. When a model names a real place, does Qloo find the right one?
Data. 120 New York places named in model-written plans; 44 acts on a Lollapalooza bill.
- Places
- 119 of 120 found
- Acts
- 38 of 44 matched with an ID, a photo and a description; Last.fm fills the rest
The court: fewer fans forced to choose
Question. On a real published Lollapalooza day, how much clash does the court remove without breaking
a festival rule?
Data. Lollapalooza Chicago, Thursday 2024: 44 acts, 7 stages, 946 pairs scored by Qloo.
- Clash score
- 10.85 to 5.92: 45% fewer clashes
- Clashing pairs
- 9 to 1
- Rules
- 15 rule-breaking proposals refused; 0 rules broken
Price and speed of a festival day's overlap
Question. What does it cost to score every pair of acts on a day?
Data. Lollapalooza Thursday 2024, 946 pairs.
- Qloo
- about 76 calls, cached after the first, no model needed
- GPT-6 Astra alone
- about $0.50 and several minutes, every time
28 pre-registered testsRules before results
Each protocol, metric and decision rule was committed to git before any code or result existed. Deviations are
recorded in the log. The clash test is PREREGISTRATION_28.md in this repo.
Outside referencesReal-world truth
Google Maps hours read by a person, real listeners' plays (Last.fm, GroupLens) and real published set times.
Never Qloo graded by Qloo.
Two model familiesGPT-6 Astra and Gemini
A frontier model and a small fast one, each run alone and with Qloo on the same questions, so the gain is
Qloo's and not one model's quirk.
Qloo alone and GPT-6 Astra alone are level on taste (Test 28: 0.723 and 0.697, not a
significant gap), which is why Verdict uses them together. Full protocols and readings: PREREGISTRATION_28.md
and PROJECT_LOG.md.