What is deep annotation?

Deep annotation is structured, multi-criteria evaluation of complex content, with a written justification for every score. Most people picture data annotation as simple labeling: “Is this image a cat or a dog?” “Is this review positive or negative?” That kind of tagging is useful. Deep annotation goes several layers further.

Deep annotation means producing structured, multi-layered evaluations of complex content. You assess it across multiple dimensions, write a justification for each, and sometimes work through structured rubrics with weighted criteria and dependencies between them.

A useful comparison: simple annotation is like grading a multiple-choice test with an answer key (pretty deterministic). Deep annotation is like writing a detailed peer review of an academic paper. You’re evaluating methodology, reasoning, evidence quality, and presentation, and you’re explaining why you scored each dimension the way you did.

The “deep” refers to both the depth of analysis required and the depth of domain knowledge needed to do it well.

As AI models get deployed in higher-stakes environments (finance, legal, education, healthcare) the annotation that shapes them needs to match that complexity. A simple thumbs-up/thumbs-down doesn’t cut it when you’re training a model to summarize earnings reports or draft legal documents.

Simple labels are cheap and increasingly automated. Deep annotation needs real domain knowledge and careful human judgment, so it pays more than simple labeling. On Sovrano, rubric work in specialist domains typically pays €30 to €150+/hr, paid monthly.

Companies building AI for professional use cases need evaluators who can evaluate outputs at the same level their end-users would. If you’re training a model to analyze financial documents, you need evaluators who actually understand financial documents. If the model is generating legal analysis, you need someone who can tell the difference between a sound argument and one that misapplies legal articles.

The training data that comes out of deep annotation is more complex and more expensive to produce than simple labeled data, and it carries more signal per item. One detailed evaluation tells the model team what a binary label cannot: which criterion failed, and why.

This is the most common deep annotation task type. You receive an AI-generated output (a document, analysis, code snippet, translation, explanation) and evaluate it against a detailed rubric with multiple criteria.

How it works: Each criterion has a defined scale (usually 1-5 or 1-7), specific descriptions of what each score means, and often a weighted importance. You score each criterion individually and write a short justification for each score.

A typical rubric for evaluating an AI-generated financial analysis might look like:

• •Accuracy of calculations (weight: 30%): Are the numbers correct?
• •Appropriateness of methodology (weight: 25%): Did it use the right approach?
• •Clarity of presentation (weight: 20%): Is it well-organized and readable?
• •Completeness of risk factors (weight: 15%): Are material risks addressed?
• •Formatting and structure (weight: 10%): Does it follow standard conventions?

Your job is to score each dimension and explain each score with specific references to the content you’re evaluating. “Accuracy: 3/5” alone tells the model team nothing. “Accuracy: 3/5. Revenue figures are correct, but the EBITDA margin calculation on page 2 transposes Q2 and Q3 figures, overstating margin by 1.3%” is what makes the annotation valuable.

The key difference from RLHF: In RLHF, you’re choosing between options. In multi-criteria evaluation, you’re producing a structured analytical document. The output you create is itself a piece of training data.

How it works: You read a complex source document (an earnings report, legal filing, research paper, or technical specification) and either extract specific information into structured fields, or verify that an AI’s extraction of the same information is correct.

This is where your ability to actually understand the source material matters most. You can’t verify that an AI correctly extracted the key financial metrics from a quarterly report if you don’t know what EBITDA means or why a year-over-year revenue comparison matters. You can’t check whether the AI correctly identified the governing law clause in a contract if you don’t understand what a governing law clause does.

Typical workflow: The platform shows you the original document alongside the AI’s structured extraction. You go field by field, checking what the AI got right, what it missed, and critically, what it hallucinated (confidently stated something that isn’t in the source document at all). You flag each field as correct, incorrect, or missing, and provide corrections where needed.

How it works: You evaluate AI-generated long-form content (reports, articles, analyses, explanations) against quality standards that go well beyond factual accuracy.

You’re assessing logical flow: does the argument build coherently from premise to conclusion? You’re checking whether claims are supported by evidence or just asserted. You’re evaluating whether the content is appropriate for its intended audience. And you’re annotating specific spans, highlighting passages that contain errors, unsupported claims, or unclear reasoning, as well as sections that are particularly strong.

This is the closest deep annotation gets to academic peer review. If you’ve ever given detailed feedback on a classmate’s essay or paper, the skillset is nearly identical.

Some evaluation rubrics have built-in logic: certain criteria only apply if prior criteria are met. For example, you might first assess whether a response is factually accurate (Criterion 1), and only if it passes that threshold do you then evaluate the depth of explanation (Criterion 2). If the response fails on accuracy, the depth score is irrelevant. A deeply detailed but factually wrong answer is worse than a shallow but correct one.

This means you need to understand the logical relationships between evaluation criteria and apply them in the correct sequence. It’s structured analytical thinking, not just reading and reacting.