mLateOn: A New SoTA for Multilingual ColBERT-Style Retrieval, Evaluated on HAKARI-Bench

Among multilingual ColBERT-style models, mLateOn stands out for its retrieval performance. I used the HAKARI-Bench results to write an article about the model for the Hugging Face blog.

• mLateOn: A New SoTA for Multilingual ColBERT-Style Retrieval, Evaluated on HAKARI-Bench

The rest of this post is reproduced from the Hugging Face article. Republishing the same content may not be ideal for SEO, but because it is my own work, I wanted to keep a copy here as well.

---

LightOn released mLateOn alongside mDenseOn. It combines a multilingual mmBERT encoder with ColBERT-style token matching and supports inputs up to 8,192 tokens. In HAKARI-Bench, it leads the multilingual ColBERT field, remains competitive on English retrieval, and can reuse document encodings for candidate rescoring.

The size ratio is what makes the result unusual. mLateOn has 115M active parameters and 312M total parameters. On MNanoBEIR it scores 63.33, between the recent large dense embedding models Qwen3-Embedding-8B at 62.87 and Nemotron-3-Embed-8B at 64.28.

Table of Contents

• Evaluation
  • mLateOn leads all 11 evaluated ColBERT models
  • 115M on the active-parameter frontier
  • HAKARI-Bench Overall and six benchmark results
• Evaluation details
  • Above the peer mean in all 14 MNanoBEIR languages
  • Task slices show where the average comes from
  • Multilingual mLateOn remains top-tier on English retrieval
  • Reranking with reusable document encodings
  • Short queries and long documents
• Model architecture and discussion
  • One encoder, many reusable document vectors
  • When to consider mLateOn
• Evaluation environment
• Closing thoughts
• What Is HAKARI-Bench?
• References

Evaluation

HAKARI-Bench is a lightweight framework that rebuilds existing retrieval benchmarks as small Nano-sets in one common format. Its HAKARI-Bench Overall view combines 550 task rows. mLateOn records 65.52 Macro and 63.96 Micro. Macro gives each benchmark equal weight; Micro gives each task row equal weight. mLateOn is the highest-scoring late-interaction model and the only ColBERT model in the Overall top 13.

mLateOn leads all 11 evaluated ColBERT models

The most direct first comparison is within the same retrieval family. HAKARI-Bench evaluates 11 base late-interaction models, including recent PPLX, Jina, Liquid AI, and LightOn releases. mLateOn reaches 65.52 Overall Macro, 8.33 points above the runner-up, pplx-embed-v1-late-0.6b at 57.19. It also uses about one quarter of that model's active parameters, 115.1M versus 440.6M.

Long-input support is one reason the Overall gap matters. Nine of the ten peers in this family view were evaluated at configured input limits between 299 and 518 tokens; mxbai-edge-colbert-v0-17m was configured for 7,999 and mLateOn for 8,191. These are the limits used in this evaluation, not measured ceilings for each encoder. Overall therefore reflects both matching quality and how much of a document each configuration lets a model read. The narrower MNanoBEIR and NanoBEIR-en results below show that mLateOn remains highly competitive where long-context support is less decisive.

Figure 1. HAKARI-Bench Overall Macro, nDCG@10 × 100. All 11 evaluated base late-interaction models are shown. AP is written in M, and each model's per-token vector width is written as d/token.

Table 1. Base late-interaction models in HAKARI-Bench. The field includes both multilingual and English-only releases. Overall and MNanoBEIR are Macro scores. NanoBEIR-en is the English-only 13-task view. Dim is per token and is not comparable to a dense retriever's per-text vector width.

The English column matters because it checks whether multilingual retrieval training reduced English quality. On NanoBEIR-en, multilingual LFM2.5 scores 68.70, English-only LateOn 68.68, and multilingual mLateOn 68.59. These results are effectively tied. The English section below tests that tie against a harder task-wise reference.

115M on the active-parameter frontier

Among much larger dense retrievers, mLateOn is fifth in the HAKARI-Bench Overall mean-rank order, behind two 8B and two 4B dense embedding models. It reaches 63.96 Micro with 115.1M active parameters; the next point above it on the disclosed frontier is Qwen3-Embedding-4B at 64.63 with 3.63B active parameters. mLateOn and Gemini Embedding 2 both round to 63.96 Micro. Mean rank and Borda, a 0-to-100 aggregation of within-task ranks, place mLateOn ahead, while Gemini leads on Macro.

How HAKARI-Bench counts active parameters. AP here is not the Mixture-of-Experts meaning. mLateOn is not an MoE model. HAKARI-Bench excludes only the Transformer's input word-embedding lookup table from the count. The table is still present and used during inference. For mLateOn, 311.8M total parameters minus a 196.6M lookup table gives 115.1M AP. The same rule removes 63.1% of mLateOn's total, 8.2% of Qwen3-Embedding-8B, and 6.8% of Nemotron-3-Embed-8B, so the exact table also reports total parameters. Neither count measures memory, latency, or index size.

Figure 2. Retrieval quality per active parameter. The step line uses only the 11 models with disclosed AP. Gemini Embedding 2 and text-embedding-3-large have no public AP, so their hollow marks sit in a separate strip outside the parameter axis. Overlapping points with identical AP are offset slightly. The x axis is logarithmic and the y axis is explicitly bounded from 59 to 69.

Exact scores: HAKARI-Bench Overall top 13

Rank	Model	Type	AP	Total	Borda	Macro	Micro
1	Nemotron-3-Embed-8B	dense	7,415.8M	7,952.7M	95.41	71.07	68.27
2	Qwen3-Embedding-8B	dense	6,946.1M	7,567.3M	91.51	68.12	65.75
3	Qwen3-Embedding-4B	dense	3,633.5M	4,021.8M	88.86	67.01	64.63
4	pplx-embed-v1-4B	dense	3,633.5M	4,022.5M	89.43	66.43	64.41
5	mLateOn	late interaction	115.1M	311.8M	85.90	65.52	63.96
6	Gemini Embedding 2	dense API	not disclosed	not disclosed	83.04	66.00	63.96
7	Jina Embeddings v5 text small	dense	521.2M	676.8M	84.44	64.07	62.29
8	Voyage 4 nano	dense	190.9M	346.5M	78.53	64.51	61.32
9	Jina Embeddings v5 text nano	dense	140.4M	238.9M	81.49	62.95	61.29
10	pplx-embed-v1-0.6B	dense	440.5M	596.0M	80.52	62.48	61.10
11	Harrier OSS v1 0.6B	dense	440.5M	596.0M	77.90	62.86	60.50
12	Nemotron-3-Embed-1B	dense	872.5M	1,140.9M	77.81	63.56	60.46
13	text-embedding-3-large	dense API	not disclosed	not disclosed	76.16	61.89	59.80

scroll ↔

Table 2. HAKARI-Bench Overall top 13 base models. Rank is the leaderboard's mean-rank order; Borda is a separate 0-to-100 aggregation of normalized within-task ranks, so the two columns need not be monotonic. Macro gives benchmarks equal weight; Micro gives task rows equal weight.

HAKARI-Bench Overall and six benchmark results

One aggregate can conceal a narrow specialization, so Figure 3 places HAKARI-Bench Overall beside six benchmark results for the Overall top 13. Borda is a 0-to-100 aggregation of each model's within-task rank. The selected benchmark columns profile multilingual BEIR-style retrieval, a broader multilingual mix, real-world domains, multilingual long documents, code, and multilingual question-answering retrieval. They are diagnostic views of the broader Overall result, not a decomposition of all 550 task rows.

Figure 3. HAKARI-Bench Overall Borda and six retrieval benchmark results for the Overall top 13. Every mini bar uses a zero-to-100 scale. Overall Borda aggregates task ranks; MNanoBEIR is a task Macro; the other five are query-weighted benchmark diagnostics. Green marks mLateOn's second-place MNanoBEIR and first-place NanoMLDR results. The blue outline only directs attention to the mLateOn row. LightOn discloses target-language MLDR training overlap, so NanoMLDR is not a clean zero-shot result.

Exact scores: Overall top 13 × seven views

Rank	Model	AP	Borda	MNanoBEIR	MMTEB	RTEB	MLDR	CoIR	MIRACL
1	Nemotron-3-Embed-8B	7,415.8M	95.41	64.28	67.24	83.64	66.89	93.46	86.79
2	Qwen3-Embedding-8B	6,946.1M	91.51	62.87	57.16	75.74	68.86	92.18	84.44
3	Qwen3-Embedding-4B	3,633.5M	88.86	61.80	57.48	74.67	68.72	91.86	84.27
4	pplx-embed-v1-4B	3,633.5M	89.43	62.01	58.22	76.46	63.14	92.77	83.52
5	mLateOn	115.1M	85.90	63.33	54.61	68.63	82.29	87.49	82.66
6	Gemini Embedding 2	not disclosed	83.04	60.50	54.12	77.80	59.12	96.60	85.26
7	Jina v5 text small	521.2M	84.44	60.77	56.08	72.11	53.54	90.53	83.06
8	Voyage 4 nano	190.9M	78.53	57.06	53.52	77.38	63.05	92.24	79.56
9	Jina v5 text nano	140.4M	81.49	59.81	54.16	69.72	56.41	88.67	82.70
10	pplx-embed-v1-0.6B	440.5M	80.52	59.59	54.75	73.47	51.64	91.52	83.66
11	Harrier OSS 0.6B	440.5M	77.90	57.45	58.32	68.00	54.94	88.43	81.95
12	Nemotron-3-Embed-1B	872.5M	77.81	57.36	63.76	77.46	60.23	91.57	84.36
13	text-embedding-3-large	not disclosed	76.16	57.50	52.50	67.21	52.02	82.73	79.21

scroll ↔

Table 3. Exact values used in Figure 3. AP is written in M and is unavailable for two API models. Overall Borda is a rank aggregation, not nDCG. The five non-MNanoBEIR benchmark columns are query-count-weighted diagnostics within each benchmark.

MNanoBEIR makes the parameter gap especially visible. mLateOn scores 63.33, above Qwen3-Embedding-8B at 62.87 and below Nemotron-3-Embed-8B at 64.28. These are recent large dense embedding models. mLateOn has about 1/60 and 1/64 of their active parameters, respectively; by total parameters, the ratios are about 1/24 and 1/26. Dense and late-interaction models output differently shaped representations, so these quality scores do not imply equal indexing or search cost.

mLateOn's strongest results are concentrated in multilingual BEIR-style retrieval and long-document retrieval: it ranks second on MNanoBEIR and first on NanoMLDR within the Overall top-13 cohort. Its ranks are ninth on NanoMMTEB-v2, tenth on NanoMIRACL, eleventh on NanoRTEB, and twelfth on NanoCoIR, showing that the advantage depends on the task distribution. In practice, I would validate it separately for MIRACL-style multilingual question-answering retrieval and code-heavy retrieval.

Evaluation details

Above the peer mean in all 14 MNanoBEIR languages

MNanoBEIR balances 13 BEIR task types across 14 languages, so the next question is whether its 63.33 average remains consistent across languages. Figure 4 compares mLateOn with the mean of the other 12 Overall top-13 models. mLateOn is above that peer mean in all 14 languages. The collapsed table also lists the single best peer for each language. Nemotron-3-Embed-8B is that peer in 13 languages and Qwen3-Embedding-8B in English. mLateOn exceeds this strong individual reference in Italian, French, Portuguese, and Spanish.

Figure 4. MNanoBEIR language Macro, nDCG@10 × 100. The signed mLateOn point is compared with the unweighted mean of the other 12 models. Each value averages the same 13 tasks. All 14 deltas are positive; the legend retains the article's shared positive, neutral, and negative thresholds. The bounded 54-to-72 axis is explicit; exact best-peer values remain in the table below.

Exact language scores and deltas

Language	mLateOn	Peer mean	Best peer	Δ vs best
English	68.59	67.33	70.40	-1.80
Italian	66.04	61.47	65.26	+0.78
French	65.43	61.57	65.29	+0.14
Portuguese	65.05	61.27	64.17	+0.88
Spanish	64.59	61.11	64.14	+0.45
German	63.94	61.20	65.13	-1.19
Swedish	63.62	59.12	64.34	-0.72
Vietnamese	63.60	60.38	64.23	-0.63
Norwegian	62.78	59.02	63.30	-0.52
Japanese	61.80	59.95	63.52	-1.72
Serbian	61.44	56.47	63.48	-2.04
Arabic	60.87	57.05	61.52	-0.65
Korean	60.54	58.31	63.18	-2.65
Thai	58.39	56.93	62.55	-4.17

scroll ↔

LightOn's retrieval training uses English plus Arabic, French, German, Italian, Norwegian, Portuguese, Spanish, and Swedish. Japanese, Korean, Serbian, Thai, and Vietnamese were not added at this stage, although mmBERT had broader multilingual pretraining. mLateOn remains above the peer mean in all five; the best individual peer still leads in each. This combination supports broad multilingual coverage and gives a clear target for further improvement.

Task slices show where the average comes from

Language coverage is only one axis. By task, mLateOn records nine clear gains and three near-ties within one point against the mean of the other 12 Overall top-13 models. Its largest margins are ClimateFEVER at +10.12 and HotpotQA at +9.20. SCIDOCS is the only task where the peer mean leads, by 2.66 points.

Figure 5. MNanoBEIR task Macro across 14 language cells, nDCG@10 × 100. Green and red require a gap of at least 1.0 point; white marks a focal near-tie inside ±1.0. The bounded 30-to-95 axis is explicit.

Exact task scores and deltas

Task	mLateOn	Other top-13 mean	Delta
FEVER	92.78	88.98	+3.80
HotpotQA	90.07	80.87	+9.20
Quora Retrieval	88.65	88.61	+0.05
SciFact	77.99	75.59	+2.40
Natural Questions	67.98	65.73	+2.25
DBpedia	66.98	62.66	+4.32
MS MARCO	57.56	53.44	+4.12
FiQA	55.19	51.39	+3.80
ArguAna	54.08	53.54	+0.54
Touché	54.01	50.03	+3.98
ClimateFEVER	44.69	34.57	+10.12
SCIDOCS	38.62	41.28	-2.66
NFCorpus	34.74	34.41	+0.33

scroll ↔

FEVER and ClimateFEVER retrieve evidence for factual claims, while HotpotQA requires multiple pieces of evidence. A token-level matcher can preserve several local clues that a single-vector bottleneck must compress, so these gains fit one plausible advantage of late interaction. SCIDOCS retrieves cited papers from a paper title; that relation can be topical rather than an explicit token match. This makes SCIDOCS a useful case for further validation on scientific-document retrieval, while one Nano task is too small to characterize the model's scientific retrieval quality more broadly.

Multilingual mLateOn remains top-tier on English retrieval

On NanoBEIR-en, the English-only 13-task view, multilingual mLateOn scores 68.59. Multilingual LFM2.5-ColBERT-350M scores 68.70 and English-only LateOn scores 68.68, so the three model-level averages are effectively tied. Figure 6 makes the comparison harder by taking the best score from all 10 other evaluated ColBERT models separately for each task. No single peer attains this entire envelope. mLateOn still leads Natural Questions and stays within one point on ArguAna, FiQA, HotpotQA, and NFCorpus.

Figure 6. NanoBEIR-en, nDCG@10 × 100, on an explicitly bounded 25-to-100 axis. The hollow point is the best score among all 10 other evaluated base ColBERT models for each task. Green marks a mLateOn lead of at least 1.0 point, white a near-tie, and gray a gap to the synthetic task-wise oracle. No single peer achieves every oracle score.

Exact NanoBEIR-en task scores and oracle gaps

Task	mLateOn	Best peer	Delta	Best-peer model
FEVER	95.90	97.02	-1.12	LateOn
Quora Retrieval	94.17	97.95	-3.78	LateOn
HotpotQA	93.21	93.76	-0.55	LFM2.5-ColBERT-350M
SciFact	81.13	83.53	-2.41	pplx-embed-v1-late-0.6b
Natural Questions	80.05	78.85	+1.21	LFM2.5-ColBERT-350M
DBpedia	70.39	73.27	-2.88	ColBERT-Zero
MS MARCO	69.29	72.69	-3.40	ColBERT-Zero
FiQA	60.88	61.21	-0.33	GTE-ModernColBERT-v1
Touché	60.58	62.26	-1.68	mxbai-edge-colbert-v0-32m
ArguAna	58.04	58.18	-0.14	pplx-embed-v1-late-0.6b
ClimateFEVER	46.54	51.90	-5.35	LFM2.5-ColBERT-350M
SCIDOCS	41.45	45.05	-3.60	pplx-embed-v1-late-0.6b
NFCorpus	40.10	40.80	-0.70	ColBERT-Zero

scroll ↔

LightOn's full-BEIR evaluation through MTEB reports the opposite sub-point ordering: mLateOn at 57.56 and English LateOn at 57.22. HAKARI-Bench uses NanoBEIR-en instead of full BEIR, so I do not treat either small margin as a stable ranking. The useful result is that a multilingual model sits level with the highest-scoring late-interaction systems on this English view, including the English-only LateOn. Figure 6 raises the bar further by constructing a different best peer for every task.

Reranking with reusable document encodings

In HAKARI-Bench, reranking simulates the second stage of a search system. Every supported retrieval task has one fixed top-100 candidate pool, built by fusing BM25 and dense retrieval with Reciprocal Rank Fusion. Dedicated pair rerankers and retrieval models rescore those same candidates, so HAKARI-Bench can measure reranking on every task rather than maintaining a separate, smaller reranker suite. Across the stored evaluation, the pool contains 86.63% of relevant items; reranking can change their order but cannot recover relevant documents outside the pool.

mLateOn reuses its document token vectors and applies MaxSim to each candidate. A pair reranker instead reads the query and document together, so document-side computation is repeated for every pair. Qwen3-Reranker-0.6B is an LLM-style point-wise pair reranker rather than a classical BERT cross-encoder; BGE Reranker v2 M3 is a cross-encoder.

On the full HAKARI-Bench reranking view, mLateOn scores 66.21 Macro, between Qwen3-Reranker-0.6B at 67.27 and the other 12 dedicated rerankers. The order changes with aggregation on the same tasks: mLateOn leads Qwen on Micro, 64.56 to 64.23, and ranks above it by Borda. Among the selected references in Figure 7, mLateOn leads BGE Reranker v2 M3 by 3.97, GTE multilingual reranker by 4.04, and Jina Reranker v2 multilingual by 5.01. Nemotron-3-Embed-8B in the full panel, and Jina Embeddings v5 text small in the short-query panel, show how reusable dense retrieval models score the same fixed candidates.

The short-query view is even more favorable. Restricting the task-level mean query length to at most 200 characters leaves 438 reranking tasks. mLateOn ranks second overall by Borda, behind Nemotron-3-Embed-8B, and is ahead of both BGE and Qwen on Borda, Macro, and Micro. It exceeds BGE by 3.81 Borda points and Qwen by 6.17. Figure 7 places this result directly below the full reranking view so the change in ordering is visible.

Figure 7. HAKARI-Bench reranking on the same fixed hybrid candidate pool. The first panel selects mLateOn, the five highest-scoring dedicated rerankers, Ettin 150M as a similar-AP reference, and Nemotron as a dense reference. It is sorted by full-view Macro and omits rank badges because leaderboard rank uses a different aggregate. The second shows the top five short-query Borda results, with each badge giving its rank on that 438-task slice. Blue identifies mLateOn; every comparison model is gray; all bars start at zero. Families differ in online computation.

Table 4. HAKARI-Bench reranking with task-level mean query length ≤ 200 characters. Every model rescores the same fixed candidates. Macro gives benchmarks equal weight; Micro gives task rows equal weight.

A 115M-active ColBERT-style model scoring above these dedicated pair rerankers on a broad short-query slice matters because its document representations remain reusable. This table measures ranking quality, not latency; candidate depth, vector compression, MaxSim implementation, batching, and hardware still decide the practical trade-off.

Short queries and long documents

The reranking result suggested checking the same 200-character filter in first-stage retrieval. It retains 438 of 550 Overall tasks, or 80%. On those tasks, mLateOn ranks third by Borda, behind the two evaluated 8B dense models and ahead of both 4B dense models. On the complementary 112 tasks with longer mean queries it ranks 16th. This identifies short-query retrieval as the better-supported operating range and longer-query workloads as a separate validation target.

Figure 8. Top-five short-query retrieval results by HAKARI-Bench Borda. Rank badges give position in the complete 73-model field on this slice. The filter uses task-level mean characters, not per-query buckets or tokenizer tokens. Bars start at zero.

Table 5. Retrieval on tasks with mean query length ≤ 200 characters. This descriptive filter leaves 438 tasks.

Now add mean document length of at least 2,000 characters. This leaves 49 of 550 Overall tasks, or 9%. mLateOn ranks first by Borda at 94.22, ahead of Nemotron 8B at 93.25 and Qwen 8B at 92.86. Thirteen of those tasks are NanoMLDR, and LightOn discloses that target-language MLDR data was used during fine-tuning. Remove MLDR, and mLateOn remains second across 36 tasks, between Nemotron 8B and Qwen 8B.

Figure 9. Top-five short-query, long-document retrieval results. Rank badges give position in the complete 73-model field on each slice. The first panel uses mean query length ≤ 200 characters and mean document length ≥ 2,000 characters. The second removes all MLDR tasks. Both thresholds are task-level mean character counts. They are descriptive filters, not token limits or controlled length interventions.

Table 6. Exact leading results for the two Figure 9 slices. MLDR removal changes the rank from first to second; mLateOn remains between the two 8B dense models by Borda.

These two views establish a useful operating result: under the explicit short-query and long-document definition, mLateOn ranks first overall and remains second after the known MLDR overlap is removed. The remaining 36 tasks mix domains, languages, and retrieval intents, so isolating how much of the result comes from document length would require a controlled follow-up.

NanoLongEmbed provides a separate benchmark-defined long-context check rather than a slice derived from character-length filters. It contains two synthetic and four real-world tasks with varying document lengths and target information placed at different positions. mLateOn scores 80.09, second among the evaluated learned models and 0.47 below Nemotron-3-Embed-8B. BM25 scores 82.17 on the same six Nano tasks, showing that literal matching remains a strong baseline here. Together with the filtered result, this supports long-context competence without isolating document length as the cause.

Model architecture and discussion

The evaluation suggests two separate advantages that should not be conflated. The 115M AP figure describes the non-lookup encoder capacity under HAKARI-Bench's accounting rule. The late-interaction design describes how the model stores and compares token vectors. A small encoder can make document encoding attractive, while a multi-vector index can still be large.

One encoder, many reusable document vectors

A dense retriever creates one vector for a document. A ColBERT-style late-interaction model creates one vector per scored token. In mLateOn, query and document text pass through the same encoder with [Q] and [D] role prefixes, and each token becomes a 128-dimensional vector. For every query token, MaxSim selects its most similar document token. The released PyLate configuration sums those maxima into one relevance score.

Figure 10. ColBERT-style late interaction. Query and document token vectors are encoded separately, every query token is compared with every document token, and standard MaxSim sums one maximum per query token. The example values are illustrative, not measured output. mLateOn follows this style; its released configuration disables query expansion and uses an empty token skiplist.

Representation size, training recipe, and projection head

Representation size. The two 8B dense models in this article emit one 4,096-dimensional vector per text. mLateOn emits 128 dimensions per scored token, so “128 dimensions” is not a smaller version of the same representation. Below 32 token vectors, the raw scalar count is lower than one 4,096-dimensional vector; above 32 it grows linearly with document length. At fp16, 512 token vectors contain about 128 KiB before index overhead, versus 8 KiB for one 4,096-dimensional dense vector. Compression, pruning, special tokens, and index structures change the actual bytes. mLateOn's smaller encoder can reduce document-encoding work; index size and search cost remain deployment variables to measure on the intended corpus.

Long-context supervision. Contrastive pretraining uses 32-token queries and 300-token documents. Supervised fine-tuning then raises both limits to 8,192 tokens and adds MLDR because most collected retrieval pairs are short. Long-input support is therefore trained with a retrieval objective, not enabled only by changing a configuration limit.

Training and inference scores. Contrastive pretraining uses standard MaxSim. Supervised fine-tuning uses MeanMaxSim, which divides the sum of per-query-token maxima by query length to reduce length bias. The released checkpoint returns to standard summed MaxSim. Figure 10 shows the released inference operation, not a free choice between sum and mean.

Projection head. mLateOn transforms each 768-dimensional token representation from mmBERT with a residual MLP whose intermediate width is 1,536, then projects it to the 128-dimensional vector used for ColBERT-style retrieval. This intermediate transformation provides trainable capacity for shaping a retrieval-specific representation before the final compression to 128 dimensions. In the technical report, I did not find a stated reason for choosing 1,536 or an ablation comparing projection-head designs, so I treat this role as an architectural interpretation rather than an isolated explanation for the benchmark result.

When to consider mLateOn

mLateOn is worth evaluating when the following conditions match the intended system. The first case I would test is offline document encoding with limited compute. Its full checkpoint has 312M parameters, of which 115M are non-lookup encoder weights. That is more than an order of magnitude smaller than the 4B and 8B dense checkpoints in this comparison, making it a plausible way to precompute high-quality document representations without a high-end GPU. Parameter count is not a wall-clock measurement, however, and this article does not benchmark encoding throughput.

• multilingual retrieval quality per active parameter matters;
• documents can be encoded before queries arrive;
• the retrieval engine can store and search token-level multi-vector representations with MaxSim;
• for a large document collection, a production-grade multi-vector engine is available or can be adopted, since a conventional one-vector ANN index is not a drop-in substitute;
• queries are usually short; mLateOn is third on the 438 tasks averaging at most 200 characters, while the complementary 112 longer-query tasks provide a separate validation target;
• one model needs to cover first-stage retrieval, multilingual search, long documents, and fixed-candidate rescoring.

The representation-size arithmetic is detailed in the collapsed architecture note above. Compression and pruning can change the deployed ratio. NanoCoIR sits outside mLateOn's strongest benchmark results, so I would run a corpus-specific evaluation before adopting it for code-heavy retrieval. A large dense model may still be the better engineering choice when one-vector indexing simplicity dominates. A pair reranker may be preferable when the candidate set is small and full joint query-document reasoning matters more than reusing document encodings.

Evaluation environment

These results were built on 2026-08-12. The HAKARI-Bench results revision is 15fc1ad3; the database hash is a318a514; the evaluation code revision is 217c8723; and the mLateOn checkpoint revision is 35391e36. Comparisons use base variants.

Closing thoughts

mLateOn is the strongest multilingual ColBERT-style model in this HAKARI-Bench evaluation. It leads all 11 evaluated late-interaction models, remains essentially level with the best English ColBERT systems, ranks third on short-query retrieval, and ranks first on the short-query, long-document view. After MLDR is removed, it remains second. On short-query reranking it is second overall and ahead of the evaluated pair rerankers BAAI/bge-reranker-v2-m3 and Qwen/Qwen3-Reranker-0.6B.

With 115M active parameters, mLateOn scores above Qwen3-Embedding-8B and below Nemotron-3-Embed-8B on MNanoBEIR. That is a retrieval-quality result across different system architectures, but it is still an unusually strong outcome for the encoder size.

LightOn published the training datasets, training code, model weights, and technical report. Researchers can inspect the recipe, test alternative explanations, and build on the model. My thanks to the LightOn team for releasing the whole package.

What Is HAKARI-Bench?

HAKARI-Bench is a lightweight evaluation infrastructure that rebuilds existing retrieval benchmarks as small Nano-sets. It puts BM25, dense, sparse, late-interaction, and reranking systems into a common task format so that retrieval architectures and efficiency settings can be compared under shared conditions. It also stores a fixed hybrid candidate pool for every task, which makes it possible to evaluate dedicated rerankers and retrieval models on the same second-stage problem.

The views in this article show another part of HAKARI-Bench: the same results can be measured from different angles by selecting benchmark suites or tasks and by filtering on task metadata, including mean query and document character lengths. The short-query and long-document analyses above use those filters. The name HAKARI refers to ⚖️ 秤 (hakari), the Japanese word for a weighing scale. Instead of forcing every model-selection question into one aggregate score, the benchmark lets me weigh models under conditions that match the retrieval problem.

For example, the Web UI can directly open the result view filtered to short queries and long documents. When the UI controls are not enough, the public result warehouse can also be analyzed directly with DuckDB and SQL. That is how I performed the MLDR-excluded sensitivity check used in this article.

You can explore the current results on the HAKARI-Bench leaderboard, read the paper, and inspect the implementation on GitHub. If HAKARI-Bench is useful for your work, I would be glad to see you browse the leaderboard, compare models for your retrieval setting, or use the evaluation toolkit for your own model.

References

mLateOn primary sources

1. Raphaël Sourty, Antoine Chaffin, Paulo Roberto Moura Junior, and Amélie Chatelain. DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search, 2026.
2. LightOn. mDenseOn with the mLateOn: Open Multilingual, Long-Context, and Code Retrieval Models, 2026.
3. LightOn. lightonai/mLateOn model card.
4. LightOn. Multilingual pre-training curated data, English pre-training curated data, and the mDenseOn & mLateOn collection of released fine-tuning datasets.
5. LightOn. PyLate.

Retrieval architecture and evaluation

6. Omar Khattab and Matei Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, 2020.
7. Nandan Thakur et al. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models, 2021.
8. Zeta Alpha. NanoBEIR collection, 2024.
9. Xinyu Zhang et al. Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages, 2023.
10. Shitao Xiao et al. MLDR: Multilingual Long-Document Retrieval dataset, 2024.
11. Dawei Zhu et al. LongEmbed: Extending Embedding Models for Long Context Retrieval, 2024.
12. Yanzhao Zhang et al. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models, 2025.

Relevant evaluated models

13. 8B dense comparators: Qwen. Qwen3-Embedding-8B; NVIDIA. Nemotron-3-Embed-8B-BF16.
14. ColBERT-family peers: LightOn. LateOn; Perplexity AI. pplx-embed-v1-late-0.6b; Liquid AI. LFM2.5-ColBERT-350M.
15. Reranking comparators: Qwen. Qwen3-Reranker-0.6B; BAAI. bge-reranker-v2-m3.

HAKARI-Bench

16. Paper: HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditions, 2026.
17. HAKARI-Bench. Leaderboard.
18. HAKARI-Bench. Repository.