Evaluation
Benchmarks: evaluation plan
How models will be evaluated on the BUDOVA test set. No runs have happened yet: results will be published with the baseline-model release (after v1.0). The only numbers on this page are targets for our own model.
Roadmap · no results yet| Model | NER F1 | Term acc. |
|---|---|---|
BUDOVA-XLM-R-base (ours)Domain-adapted XLM-R, LoRA fine-tune on BUDOVA (planned for the baseline-model release) | target 0.891 | target 0.880 |
Liberta-UK-largeUkrainian LM with supervised NER head | to be measured | to be measured |
XLM-R-large (zero-shot)No BUDOVA fine-tune | to be measured | to be measured |
mBERT-baseMultilingual baseline | to be measured | to be measured |
GPT-4o (few-shot)5-shot prompt, no fine-tune | to be measured | to be measured |
Random baselineUniform label assignment | to be measured | to be measured |
The third-party models in the table are a comparison plan, not results: none of them has been run on the BUDOVA test set yet.
