Composition, rubric & reliability

What the 200 items are made of, how experts grade them, and how closely two independent experts agree. Together these establish that the scores reflect a shared professional standard rather than individual opinion.

Composition by domain

Executive domain	Items	Share
Product & Tech	59	29.5%
Business	55	27.5%
Marketing	46	23.0%
Finance	40	20.0%

The grading rubric

Each model response is scored on a five-point scale across five dimensions, and every checklist criterion is marked covered / partial / missed. Scores are shown throughout this site as a percentage of the 5-point maximum (5/5 = 100%, 4/5 = 80%, …). Coverage fraction = (covered + 0.5·partial) / total.

Five scored dimensions

1
Domain Accuracy Are the facts, regulations, and figures correct for the market in question?
2
Strategic Reasoning Does the answer show sound executive judgment, not just recall?
3
Actionability Could an operator act on it, with concrete and sequenced next steps?
4
Executive Communication Is it structured and clear enough for a C-suite reader?
5
Local / Regulatory Fidelity Does it respect the specific European market and legal context?

Checklist depth

5 criteria 37.0% · 74

6 criteria 27.5% · 55

7 criteria 14.5% · 29

8 criteria 10.0% · 20

9 criteria 3.5% · 7

10 criteria 7.5% · 15

Inter-annotator reliability

All 200 items were judged independently by two domain-matched experts (6,000 paired rubric scores), so agreement is measured across the whole dataset, not a subsample. The experts are highly consistent, both with themselves and with one another. Figures carry bootstrap 95% CIs (1,000 resamples clustered over items).

0.79

95% CI 0.76–0.81

Within-expert consistency (Kendall τ): each expert's stated ranking matches a ranking rebuilt from their own rubric scores.

77%

95% CI 74–80%

of rubric scores land within ±1 point between the two independent experts.

2.3×

vs. chance

more likely than chance that both experts independently pick the same top model.

Provenance & scale

1,200

model responses

8.8M

characters of model output

2,400

per-response score rows

distinct expert annotators

Scope & method notes

Taxonomy. Round 1 covers four executive domains: Finance, Business, Marketing, and Product & Tech.
Judging. Final scoring is human (2 experts/item); any LLM in the pipeline is used only for Stage-1 QA and the Stage-2 difficulty gate, never for final grades.
Non-independence. 2 correlated judgments per item, so the effective sample is items, not judgments; every leaderboard CI is clustered over items to account for this.

See how it's made on the methodology page, or explore one item end to end in the sample item.