Inter-annotator agreement
Each fragment receives three independent annotation passes, followed by adjudication against the published guidelines and an audit of unanimous records. The figures below are measured on the independent passes as submitted, before repair and adjudication; where those passes were not kept, the notes to the per-document table say what is shown instead. Expert validation is pending.
How agreement is computed — see the guidelinesLow-agreement cases
95 entries with κ < 0.6 · 8 lowest shownκ = −0.50 is, for example, the value Fleiss’ κ takes for three passes when one of them marked a span that the other two left unlabelled.
Adjudicated entries are not listed.
188 entries have Cohen’s κ between passes below 0.8 and are not adjudicated yet.
Agreement by document
Entity-window Fleiss’ κ and span F1, averaged over a document’s fragments and measured on the independent passes as submitted, before any repair or adjudication, unless a note says otherwise. Each document has its own annotator panel, so figures are not directly comparable across documents.
| Document | Fragments | κ | Span F1 |
|---|---|---|---|
| ДБН В.2.6-98:20091 | 904 | 0.88 | 0.95 |
| ДБН Б.2.2-6:20132 | 199 | — | — |
| ДБН В.1.2-8:20213 | 89 | — | — |
| ДБН В.1.1-45:2017 | 367 | 0.92 | 0.97 |
| ДБН В.1.2-12:2026 | 133 | 0.98 | 0.99 |
| ДБН Б.2.2-1:2008 | 241 | 0.92 | 0.96 |
| ДБН В.2.6-220:2017 | 559 | 0.89 | 0.96 |
| ДБН В.1.1-25:2009 | 328 | 0.91 | 0.97 |
| ДБН В.1.2-7:2021 | 107 | 0.94 | 0.98 |
| All documents with a figure | 2,639 | 0.90 | 0.96 |
- Measured on the passes as stored: a targeted repair after annotation changed 237 of 2,712 decisions; the rest are as submitted.
- After annotation, a repair round aligned all three passes on every fragment (119 span operations over 42 fragments), and the passes as submitted were not kept. The stored agreement (1.00) therefore describes the aligned record, not agreement between annotators. Measured before the repair: κ 0.963, span F1 0.982.
- This document deliberately has no single figure: its three strata (pilot, wave 1, wave 2) measure different things, and wave 2 measures the reproducibility of the rules rather than independent judgement. The passes as submitted were not kept, and the stored agreement (1.00) describes the aligned record, not agreement between annotators.
Agreement over time
Mean κ before adjudication by the week entries were merged, and how many were merged.
