Every project, pilots included, runs the same four checks before data reaches you:
- Calibration: Before live work, every evaluator completes calibration tasks against gold-standard answers on your rubric, and only those who meet the threshold start live work. Your feedback on each batch goes into the next calibration round.
- Agreement reporting: We monitor agreement between evaluators daily and report it weekly in your project report, using Cohen's kappa by default or the metric your dataset calls for. On projects where every task is judged twice, such as benchmarks, we compare every task; where each task is annotated once, we double-annotate a 10% overlap sample. If agreement falls below the target you set, we pause the affected evaluators within 24 hours and recalibrate them.
- Review: On every project, a reviewer checks every item for rubric adherence and consistency before it reaches you, not a sample. Reviewers come from the same practitioner bench as our authors, whose median experience on EuroExec is 7 years.
- Human-only work: Evaluators may not use AI tools to judge, write or rewrite task content, or paste your data into any external tool. Prompts and checklists must be typed: paste is blocked in our authoring tool. Uploaded documents are checked against the file's own record of how it was made (the application that wrote it, editing time and saves), and a file generated by code that no person ever opened is rejected automatically. We don't rely on AI-text detectors, which misfire on non-native English. A reviewer checks every item, and anything flagged is removed from your delivery.