The best NER model changed when the benchmark did

Named-entity recognition, or NER, is the part of a language system that finds spans such as a person's name, an organization or a location. It sits early in a knowledge pipeline. If the system misses an entity or gives it the wrong boundary, later steps start from damaged material.

Before implementing nauron-ner, we had no neutral reference for the available approaches. Official model scores were useful, but they came from different datasets and label systems. Reading them as one leaderboard would have meant aligning decimals, not experiments.

We built the Nauron Evaluation benchmark to close that gap. Its first reference run had a deliberately narrow job: compare open-source NER candidates under one scoring contract, in English and Polish, before deciding what deserved further work for Nauron. The same harness can later accept Nauron's own model and other extraction tasks without changing the measurement underneath them.

The run compared eight configurations from spaCy, Flair, Stanza and GLiNER. It used six datasets: CoNLL-2003, WikiNER and WikiANN in English; PolEval-2018, WikiNER and WikiANN in Polish. Together they contained 60,160 documents.

Every dataset was normalized to four core labels: person, organization, location and miscellaneous. Flat F1 was the main quality measure. A prediction counted as a match only when both the label and character boundaries matched the reference annotation. The run also measured median and p95 latency per document on the same Apple M1 Pro machine.

That controlled the scoring and runtime, but it did not make the corpora identical. CoNLL is English newswire. WikiNER and WikiANN are derived from Wikipedia through different annotation processes. PolEval asks for nested Polish entities in an NKJP-compatible form. Those differences are not noise to remove. They are the point of testing more than one benchmark.

English produced two quality winners. Flair led CoNLL and WikiANN; multilingual spaCy led WikiNER. Polish did the same: Stanza led PolEval and WikiANN, while multilingual spaCy led WikiNER. There was no model that won all three datasets in either language.

The movement inside each model matters as much as the change at the top. Flair scored 0.912 on CoNLL, 0.757 on WikiNER and 0.491 on WikiANN. The executable did not change. The corpus did. In Polish, multilingual spaCy moved from 0.372 on PolEval to 0.857 on WikiNER, then to 0.610 on WikiANN.

The report can still calculate an average winner. Across the three English datasets, Flair was strongest at 0.720. Across the three Polish datasets, Stanza led at 0.684. Those are valid summaries of this selection of tests. They are not universal properties the models carry into every future corpus.

Two comparisons made the pattern unusually clear. Flair's fast English model is trained for CoNLL-03 and publishes an F1 of 0.929 there. Our common harness measured 0.912 on CoNLL-2003. The multilingual spaCy model is trained on WikiNER and publishes 0.831; our run measured 0.811 on English WikiNER and 0.857 on Polish WikiNER.

This is what specialization looks like. Model developers train, tune and select around named datasets, then report performance on those datasets. That is normal engineering. It also creates room to foreground whichever benchmark tells the strongest story. The same model can look like the clear winner on its home ground and ordinary somewhere else.

Our results prove that the ranking depends on the benchmark. They do not prove why any publisher chose a particular number, or that a company deliberately gamed an evaluation. Historical convention, available licenses and the intended use of a model can all explain the choice. What the data does prove is enough: one favorable score cannot establish general superiority.

The broader research literature calls this the benchmark lottery. Change the tasks used for comparison and the relative performance of methods can change with them. Our NER run produced that effect in a concrete selection decision, not as an abstract warning.

A quality winner may still be the wrong operational choice. On CoNLL, Flair led at 0.912 F1 with a median latency of 42.44 ms per document. Multilingual spaCy returned in 1.38 ms and scored 0.604. On PolEval, Stanza led at 0.732 and 581.55 ms; multilingual spaCy returned in 16.01 ms and scored 0.372.

WikiNER changed that trade again. Multilingual spaCy won both quality and speed: 0.811 at 2.05 ms in English, and 0.857 at 1.68 ms in Polish. There was no universal accuracy-versus-latency frontier, either.

These timings belong to one local run with Python 3.12.10 on an Apple M1 Pro with eight logical CPUs. They compare candidates inside that environment; they are not deployment promises. The useful question is narrower: which errors can this workflow tolerate, at what throughput, on the documents and hardware it will actually use?

The current report is an open-source NER reference run. The harness contains an OpenAI adapter, but no OpenAI result appears in this committed reference artifact. It should not be described as a comparison of every NER approach. It also predates nauron-ner, so it establishes the baseline rather than declaring our own model the winner.

It measured entity recognition, not entity resolution, relation extraction, graph construction or whether an answer carries enough evidence for a decision. A correct span is only an early step in the Evidence Engine pipeline.

The data boundary matters too. The shipped run had no active OntoNotes or NKJP slice. Twenty-one long PolEval documents were truncated to 384 tokens for GLiNER. Most tracks measured flat spans; only PolEval exposed nested and overlapping annotations, and not every provider supported them. These are limits on what the numbers can carry, not footnotes to hide.

WikiNER needs one more caution. The harness made a deterministic 10% test slice from its wp3 corpus, but the repository does not establish that this slice is disjoint from the data used to train the multilingual spaCy model. Its WikiNER result demonstrates benchmark alignment, not independent out-of-domain generalization.

The report records model identifiers and library versions, but not an immutable revision or checksum for every model and dataset artifact. The committed results describe the June 2026 run. They do not guarantee that a fresh download would be byte-for-byte identical.

Before selecting an entity recognizer, we now expect the comparison to answer five questions.

1. Which languages, document types and entity labels does the production workload contain?
2. Were the candidate scores produced with the same dataset, label mapping and matching rule?
3. Does the ranking survive a second representative corpus?
4. What quality-versus-latency trade can the workflow support on its actual hardware?
5. Which datasets, providers and document shapes were excluded, skipped or truncated?

Then we run the smallest representative evaluation that can answer them. The useful output is not “Flair wins”, “Stanza wins” or “spaCy wins”. It is: under this data, ontology, matching rule and runtime, this candidate gives us this trade. Change any clause and the decision reopens.

Nauron's principle is that evidence is the contract. Model selection is not exempt. A score without its benchmark is a citation without its source.

• Nauron Evaluation: NER OSS Reference BenchmarkDashboard, full report and exported results · June 2026
• The Benchmark LotteryDehghani et al. · 2021
• Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity RecognitionTjong Kim Sang & De Meulder · 2003
• Learning multilingual named entity recognition from WikipediaNothman et al. · WikiNER dataset
• Cross-lingual Name Tagging and Linking for 282 LanguagesPan et al. · WikiANN · 2017
• PolEval 2018 NER taskOfficial challenge and evaluation
• English NER in Flair (fast model)Flair model card · CoNLL-03
• spaCy multilingual trained pipelinexx_ent_wiki_sm · WikiNER