One item, end to end

A single Business scenario from Round 1, shown exactly as an expert annotator receives it: the executive prompt, the graded checklist, six blind model responses, and the scores, rankings, and rationales of two independent experts. Responses appear blind, labeled A to F, until you reveal which model produced each one.

52%
mean rubric score across all six answers — a deliberately hard item
60%
the strongest single answer, on the same five-point scale
22%
of the graded checklist that best answer actually covers

Executive scenario (prompt)

I am CEO of a German grocery chain with 12 billion in annual revenue, an EBITDA margin of 6%, the company is leveraged at 3.5 times their EBITDA and cannot exceed 4 times, and operates 2,000 stores across Germany, Austria, and the Czech Republic. The board has set a target EBITDA margin of 7% while growing revenue 20% over the next five years without becoming over-leveraged. You are considering expanding into Italy which has a highly fragmented grocery store environment. the top 5 players control 50% of the environment and discount penetration is growing 6-7% annually and average gross margin for discount stores is 2 percentage points higher in Germany because of more favorable conditions and less intense price competition. However labor costs are 20% higher in Northern Italy than in Eastern Germany, Austria, and the Czech Republic. our strategy team has identified three options to reach the board's goals in the next 5 years: 1-open 300 stores in Northern Italy. this would cost 4 million euros for each store and requires building a new Italian distribution network at a cost of 250 million euros 2-a joint venture expansion into Italy with an existing mid-sized regional supermarket (owns 150 stores and 2.1 billion euros in revenue). the plan would convert 100 stores to discount stores and would cost 1 million euros for each store. the joint venture would pay profits 50% once EBITDA exceeds 100 million 3-Germany first approach with small Italian test. focus 80% of capex on automation, online ordering infrastructure, and format optimization in existing German stores. run a limited Italian pilot with 40 leased stores over 3 years write a one page memo for the supervisory board and recommend one of the options.

Graded checklist

  1. 1
    States a clear recommendation (or a hybrid of two options) and explains why it rejects the other options with 1-2 reasons ● critical
  2. 2
    Quantifies Italy's 5 year contribution to revenue and EBITDA (for the recommended option) and clearly states how it moves the company towards the board's target of 20% revenue growth and 7% EBITDA margin
  3. 3
    Balances economic and social factors, specifically reputation and political risk from job reductions
  4. 4
    Identifies the top 3-5 risks (union backlash, tech underperformance, political pressure, cost overruns, etc.) and gives a concrete mitigation action for each option)
  5. 5
    Quantifies capex and EBITDA at the end of year 5 for chosen option (or hybrid option)
  6. 6
    Estimates final capex for chosen option at the end of each 5 years and is broken down into major components (Italian store openings, Italian distribution network, German automation, etc. for whichever option is recommended)
  7. 7
    Provides a comparison of labor costs impacts between entering Italy compared to a non-entry scenario, including change in total group labor cost
  8. 8
    Compares revenue, margin, capex, labor costs, EBITDA of chosen option to non-entry alternative hypothetical

Model responses

Six models answered this prompt blind, labeled A to F. Read them, then flip the switch to reveal which lab wrote each.

How the two experts ranked it

Both experts scored every answer on the five-point rubric and produced a full preference order, blind and independently. On a hard item the two orders diverge — exactly what double-judging exists to surface. Experts stay pseudonymized (A04, A43).

See how items like this are built on the methodology page, or the field-wide scores on the leaderboard.