Introducing saihai: Japanese Decision Models and a Japanese Decision Benchmark

I have released saihai (采配), a Decision Model trained with a substantial amount of Japanese data, in the same category as TypeSafe AI's System One Decision Model Jev. The training datasets are also available.

• saihai-v1-8b
• saihai-v1-8b-gguf
• saihai-v1-310m, a smaller encoder model

saihai-8b is an 8B Decision Model fine-tuned from llm-jp-4.1-8b-thinking, an open LLM with strong Japanese capabilities. I hoped that using llm-jp would help it draw on knowledge specific to Japan. Like its base model, it is licensed under Apache 2.0.

Alongside the regular weights, I have released GGUF versions for use with llama.cpp's recently added /v1/systemone API endpoint.

The smaller saihai-310m uses an encoder architecture rather than a decoder LLM. It follows the approach used in bekko-system-one and is based on the Japanese reranker ruri-v3-reranker-310m. It also uses Apache 2.0, the same license as ruri.

The second half of this article covers the architecture, training, controlled experiments, and quantization measurements in more detail.

A Japanese Decision Model benchmark

To evaluate performance in Japanese, I created japanese-v1, a Japanese benchmark suite for S1MB (System One Mosaic Benchmark). It combines 19 tasks from existing Japanese datasets, including natural language inference, with eight synthetic tasks generated using GPT-6.1-Sol for this benchmark. Results for all 27 tasks are available on the S1MB leaderboard.

Tasks fall into three categories according to their answer type:

• Noul: a binary yes/no decision.
• Choice: selection from multiple candidates.
• Score: a rating on discrete levels such as 0, 1, 2, 3, and so on.

The table below includes the Jev and GPT-6 Luna API models and a selection of models with at most 12B parameters. See the leaderboard for the full results, including 27B models. The best value in each column, excluding Jev and GPT-6 Luna, is shown in bold.

saihai-8b falls short of Jev, but I think it performs reasonably well for an 8B model. There is an important qualification: to improve its handling of Japanese tasks, I trained saihai on the training splits of some datasets also used in the benchmark. This makes higher benchmark scores easier to achieve. I removed overlap with evaluation data as far as possible; controlled experiments examining this effect appear later in the article.

Here is how to read the table:

• Noul, Choice, and Score average the scores by answer type across all 27 tasks: 19 from existing datasets and eight synthetic tasks. The three General columns use only the eight synthetic tasks, averaged by type.
• Task Avg is the equally weighted mean of Noul, Choice, and Score. The General columns are not added separately to it.
• Values range from 0 to 100 and are rounded to two decimal places. They are baseline-adjusted scores, not raw accuracy.

saihai-8b compares favorably with models of similar size on specific Japanese tasks, particularly tasks originally written in Japanese, as the later comparisons show. Still, it trails Winnow 12B and Jebadiah 9B v2 on synthetic Noul tasks and many models on synthetic Choice tasks, with differences on Score as well. Strength on familiar task types does not necessarily translate into flexibility across varied inputs.

OpenAI's GPT-6 Luna through the Decisions API scored 55.00 overall, close to Jev. It exceeded Jev on Noul (62.10) and General Noul (67.11). The API refused one decision in the Japanese evaluation; that refusal was counted as incorrect.

The smaller saihai-310m scored 30.19 overall, leaving a substantial gap to decoder models with several billion parameters. It did much better than the similarly sized encoder Laya multilingual (5.29), however. Its Choice score was 40.99, with relatively good results on JCommonsenseQA (89.58), KUCI (82.67), and JNLI (79.00). For Noul, it also handled CommonsenseMoralJA (74.67) and synthetic Diverse (65.33) reasonably well. It barely solved the synthetic knowledge and mathematical tasks, and its General Score was 0.00. It also used much more training data from the benchmark's source datasets than saihai-8b did, which makes its corresponding scores easier to raise.

Large decoder models currently attract most of the attention in Decision Models. I hope that small encoders will eventually offer sufficient performance for many uses too.

Using existing Japanese datasets

I assigned each existing task to Noul, Choice, or Score according to which answer type fit it best. Evaluation generally uses the original authors' test sets, with some exceptions involving validation sets, training sets, or unsplit collections. Each task contains at most 300 examples. IVRy's article on evaluating Jev with Japanese benchmarks provides a clear explanation of this approach in Japanese.

I converted the data into instruction/state structures suitable for Decision Models. Where training splits were available, I also prepared training data and used it for saihai. Some training splits contain examples that also appear in test data; I removed these as far as possible.

For the rest of this article, same-source training data means training splits from the same source datasets as the benchmark tasks. The last column below gives the number of training examples used from those splits, not the number of evaluation examples.

A few details matter here:

• Exceptions to using test sets: JCoLA uses the original validation set; JSeM uses a collection without a train/test split; NanoJMTEB uses its evaluation split; JMMLU Japan has no original split specified; and SOBACO Culture uses its original training split. GPQA also uses its original evaluation question collection. These are not all author-defined test sets.
• JSICK uses Choice in training and Score in evaluation, but I treat them as belonging to the same source family.
• JMMLU Japan contains 75 examples each for Japanese history, idioms, civics, and geography. SOBACO Culture contains 300 Japanese-culture examples.
• Retrieval (NanoJMTEB) is a Choice task based on JMTEB retrieval benchmarks, with one positive and three to five non-positive candidates. Licensing varies by row because it combines multiple sources.
• Licenses depend on the underlying material; the table is a summary. Consult the original sources for subject-specific JMMLU restrictions and attribution or redistribution requirements for retrieval data.

Synthetic datasets

Existing datasets measure performance on particular tasks. To examine whether models can choose appropriate answers across somewhat more flexible inputs, I generated additional tasks using GPT-6.1-Sol. I refer to them as generalization indicators for convenience, although they do not establish broad generalization.

There are eight tasks across three families, totaling 2,400 examples.

Diverse has no Score task. An uncertainty task I created earlier is no longer included in the evaluation. I assigned the synthetic data a CC BY-NC 4.0 license because it was generated using an OpenAI model. The answers have not undergone independent, blinded human verification, so results on these eight tasks alone cannot establish generalization to every use case.

Evaluation on the English benchmark

Although saihai targets Japanese, its training data includes substantial English material. I therefore also evaluated it on S1MB's english-v1, which contains 137 tasks. Its General columns average six synthetic tasks from the Diverse and Contextual families, two per answer type. The columns have the same meaning as in the Japanese table, but the task composition differs, so scores cannot be compared directly between the two suites.

This table adds the English encoders bekko-system-one-v0-400m, Laya, and Laya typed-decisions to the models above. Bold indicates the best value excluding Jev and GPT-6 Luna.

saihai-8b scored 50.59 overall, placing third among the seven decoder models other than the API models. Noul was relatively strong at 59.96, while Score was lower at 32.29. General Noul and Choice both exceeded 90. Despite its Japanese focus, the model appears to retain useful English capabilities.

GPT-6 Luna scored 49.21, ranking lower than it did in Japanese. Its API refused 17 decisions in English, all counted as incorrect, which contributed somewhat to the result.

saihai-310m scored 40.59, below bekko-system-one-v0-400m's 50.60. I used only about 10% of the bekko-system-one dataset for its English training data. General Noul and Score dropped particularly sharply.

Even so, its overall score was more than 2.5 times those of Laya (13.36) and Laya typed-decisions (15.00), with substantially higher Noul, Choice, and Score results. Its General columns were closer to the Laya models, with General Score slightly lower. Flexible inputs remain difficult for small encoders.

Many decoder models cluster around 90 on English General Noul and Choice, making those tasks less useful for distinguishing them. The scores are not directly comparable across languages, but Jev's Japanese General Noul score of 64.67 suggests that the new Japanese synthetic tasks leave more room to distinguish models.

Building the saihai models

The two models use different architectures, so I will describe them separately.

saihai-8b

Both the training code and training dataset are public.

Architecture

saihai-8b is a decoder LLM trained using an approach common among current Decision Models. There is no universally established architecture for these models yet. I chose a structure compatible with the format used by Perplexity's pplx-decider-v1-27b, which llama.cpp supports through /v1/systemone.

The training and inference process works as follows:

1. Associate each example's candidates with a target probability distribution. Correct answers and source metadata are excluded from the model input.
2. Serialize the instruction, state, and candidates as JSON and insert them into the base model's chat template. Assign each candidate a code such as A or B that the tokenizer represents as a single token.
3. Feed the input up to the point immediately before the answer begins, and extract the hidden state of the last valid token.
4. Compute candidate logits with a bias-free FP32 linear head, then apply softmax over the candidates used in that question. Initialize the head from the original LM head rows corresponding to candidate-code tokens.
5. Train LoRA parameters and the candidate head using cross entropy against the target probability distribution. The backbone is BF16. Full fine-tuning would not fit in memory and would be slower, so I use LoRA.
6. For release, merge LoRA and write the trained head back into the corresponding candidate-code rows of the original LM head. Inference reads candidate probabilities from a single forward pass, without generating text or reasoning.

Candidate codes follow the same rules as llama.cpp's pplx-decider support: single-token codes selected from A–Z and AA–ZZ, supporting up to 255 candidates. Noul orders candidates as false, then true; Score supports 2–10 levels. This matches the probability readout mechanism rather than reproducing the entire pplx training method.

Training data

The training data combines three sources:

• Datasets used to train existing English Decision Models, including material extracted from existing datasets and synthetic data.
• Training splits from the source datasets of the Japanese benchmark.
• Synthetic Japanese data generated using llm-jp-4.1-32b-a3b-thinking.

There are 33,532 examples in total.

Training used LoRA with rank 16 and alpha 32 and took about 3.62 hours on an RTX 5090, excluding evaluation and quantization. Other settings were one epoch, effective batch size 16, maximum input length 8,192 tokens, and seed 42. I evaluated a fixed validation set of 279 examples ten times during training, but released the final checkpoint rather than the checkpoint with the best validation result.

The model weights' Apache 2.0 license is separate from the licenses of the individual training-data sources.

saihai-310m

saihai-310m is a small encoder trained using the bekko-system-one approach and its training code.

• Training code and procedure
• Training dataset

Architecture

The architecture is the same as bekko-system-one. It processes the instruction and state once as a shared prefix, then reuses each layer's keys and values when processing each candidate, or criterion. Each candidate attends only to the shared prefix and itself, and its representation is used to compute a score. This avoids processing a long state again for every candidate, allowing efficient inference with a small model. The bekko-system-one article explains it in more detail.

With bekko-system-one, initializing from a trained reranker worked better than starting from a plain encoder. The 68M model in particular improved in both cross-entropy loss and task accuracy. Following that result, I based saihai-310m on the Japanese reranker ruri-v3-reranker-310m. This is a reranker trained from ruri-v3-pt-310m, itself based on modernbert-ja-310m. The model has 315M parameters, or 236M excluding embeddings.

saihai-310m does not support llama.cpp's /v1/systemone. Load it through Transformers' AutoModel with trust_remote_code=True, using a state/questions input format similar to saihai-8b. One difference is that saihai-8b raises an error for overly long inputs, while saihai-310m truncates them before inference.

Training data and settings

The following counts represent individual decisions within examples.

The seven Japanese source datasets are CommonsenseMoralJA, JCommonsenseQA, JNLI, JSTS, KUCI, Mintaka, and JaNLI. Whereas saihai-8b used at most 400 examples per dataset, saihai-310m used all selected training data, resulting in much larger counts: KUCI alone contributes 69,527 decisions. This likely raises scores on the corresponding tasks more than it does for saihai-8b. As with saihai-8b, evaluation questions and their related data were excluded.

I updated all parameters without LoRA, using batch size 512 for one epoch. Training took about two hours and 45 minutes. The encoder learning rate was 1e-5, and the head learning rate was 2e-4. Input limits were 4,096 tokens for the query and 2,048 for candidates.

S1MB evaluation used the same settings as bekko-system-one to truncate long inputs and allocate the input-length budget.

Controlled experiments

Because training includes tasks from the benchmark's source datasets, it is reasonable to ask how much of the performance comes from that overlap in task families. I ran two controlled experiments for saihai-8b and then compared the version without same-source training data against other Decision Models.

With and without same-source training data

Alongside LLM-jp 4.1 8B, I trained three sizes of Qwen3.5, each with and without same-source training data. These are their S1MB Task Avg scores.

Including same-source training data improved most models, but Qwen3.5 4B scored 0.43 points higher without it. The effect is therefore not uniformly positive.

To see where the differences came from, I separated the ten tasks with corresponding training data from the other 17: nine existing-dataset tasks without same-source training and eight synthetic tasks. Each group uses the simple mean of its task scores. Differences are with minus without, so positive values favor including the data.

For LLM-jp 8B, the ten corresponding tasks gained 8.02 points, while the other 17 gained only 0.15. Every model showed a large improvement on the ten corresponding tasks. Training on a dataset's training split clearly makes its task family easier. There is a confound, however: removing these examples reduced the training set from 33,532 to 29,643 examples and the number of updates from 2,096 to 1,853, so the smaller amount of training may also matter.

Comparing base models

I expected llm-jp to offer strengths in understanding Japanese domains. To investigate, I trained several Qwen3.5 models on the same data.

For this comparison, I divided the 19 existing-dataset tasks into two groups:

• 12 Japanese-original tasks: datasets created originally in Japanese rather than translated. This includes language tasks such as inference and semantic similarity as well as knowledge about Japan: CommonsenseMoralJA, JamC-QA, Jamp, JaNLI, JCoLA, JCommonsenseQA, JCulture-MCQ, JNLI, JSTS, KUCI, JMMLU Japan, and SOBACO Culture.
• 7 translated/mixed tasks: datasets based on translations or combinations of multiple sources, including many general-knowledge questions from MMLU-ProX, MMMLU, and GPQA. The full group is MMLU-ProX, MMMLU, GPQA, Mintaka, JSICK, JSeM, and NanoJMTEB.

These groups are an analytical choice for this article, not a definition of Japanese ability as a whole. Task Avg is S1MB's overall metric; the other columns are simple averages over each task group.

The slightly larger Qwen3.5 9B performed better overall, but LLM-jp 8B was slightly ahead on the 12 Japanese-original tasks. Qwen3.5 9B led by 6.83 points on translated/mixed tasks. This suggests that LLM-jp's strengths help with questions originally written in Japanese. LLM-jp 8B also scored higher overall than Qwen3.5 4B and 0.8B, though there are exceptions by task and answer type. LoRA ranks and learning rates differed between models, so these are not comparisons of the base models alone.

Comparing saihai-8b without same-source training against other models

Finally, I compared LLM-jp 8B trained without same-source training data against other Decision Models. The 4 cultural-knowledge tasks are JamC-QA, JCulture-MCQ, JMMLU Japan, and SOBACO Culture. They are a subset of the 12 Japanese-original tasks.

Even without same-source training, saihai-8b placed second to Jev among these eight models on the Japanese-original tasks, ahead of Winnow 12B, whose underlying base is gemma-4-12B-it. It ranked fourth on the eight synthetic tasks, behind Jev, Winnow, and Jebadiah. Winnow still led it overall, and Clef scored higher on the four cultural-knowledge tasks. On translated/mixed tasks, saihai-8b placed sixth of eight, losing ground on general knowledge. These results do not support a broad claim that it knows more about everything related to Japan.

Here, “without same-source training” means excluding that data from additional fine-tuning. I have not established that there is no semantic overlap with evaluation data or that the base model never encountered the material during pretraining.

Quantization and llama.cpp support

saihai-8b is available in six GGUF formats: BF16, Q8_0, Q6_K, Q4_K_M, Q3_K_M, and Q2_K. I evaluated all six on the full 27-task, 8,067-example S1MB japanese-v1 suite using llama.cpp's native /v1/systemone. All used S1MB's standard llama.cpp adapter, four concurrent requests, 8,192 tokens per slot, and cache reuse.

Argmax agreement is the proportion of decisions whose highest-probability candidate matches GGUF BF16, not accuracy. Score tasks are evaluated using the expected value over numeric levels, so scores can change even when argmax agrees.

Q8_0 retained nearly all of BF16's overall score, losing about 0.03 points. Q4_K_M lost only about 0.66 points, but argmax agreement was 92.10%. Similar benchmark scores do not guarantee identical behavior in applications that use the probabilities directly. Q3_K_M reduces the file to about 4.37 GB with a 2.56-point loss, while Q2_K loses 11.46 points. Q8_0 and Q4_K_M look like practical options, offering either near-equivalent performance or a smaller footprint with limited benchmark degradation.

I also measured throughput and peak GPU memory on an RTX 5090, evaluating the same 8,067 examples with four concurrent requests.

Quantization increased throughput by roughly 1.1–1.3 times over BF16. Q8_0 used about 12 GiB of VRAM, and Q4_K_M used less than 10 GiB. Speeds were fairly similar across Q8_0 and lower-bit formats; Q2_K was slower than the other quantized versions. Q8_0 and Q4_K_M look like reasonable choices from this perspective too.

These are end-to-end evaluation throughputs, including HTTP communication, scoring, and saving results, not per-request latency. Each was measured once, in a separate run from the quality evaluation. The weights match the release, but the speed runs used GGUF files from before the metadata update for 64K context support. VRAM figures are observed usage, including driver overhead and related allocations.

The Transformers version scored 49.77, compared with 49.53 for GGUF BF16. Transformers used saihai's own input conversion, whereas GGUF used S1MB's standard adapter, with differences in representations such as Score. This is not a quantization loss.

Two additional qualifications:

• The model is configured for 64K context, but this evaluation contains short tasks. It does not establish decision quality on long, 64K-token inputs.
• Measurements used llama.cpp revision 3d65c90 with the LLM-jp tokenizer fix from PR #28864 applied. Without the patch, tokenization differs slightly. I expect the fix to be incorporated into llama.cpp in the future.

Reflections on Decision Models and benchmarks

The S1MB English leaderboard already includes more than 100 Decision Models built by individuals and companies, with some scoring above Jev. Building a model that appears to work reasonably well is quite approachable with a capable base model and straightforward fine-tuning. saihai-8b took less than four hours to fine-tune on a single RTX 5090.

What impresses me about Jev, though, is its generalization: in my experience, it chooses appropriate answers across a wide range of problems. I do not think current benchmarks, including S1MB, capture that adequately. TypeSafe AI's terms for Jev's preview release prohibited publishing benchmark results; that clause has since been removed. I wondered whether the concern was that benchmark scores could misrepresent performance. Diogo Almeida also commented on X about a gap between models optimized for benchmarks and models built with reliability in mind—a gap that benchmarks themselves do not reveal.

I agree that a benchmark covering only a narrow slice of behavior cannot establish broad generalization. I would welcome a benchmark from TypeSafe that covers a wider range of Decision Model use cases. My speculation is that they may have internal evaluations, and that publishing them would also reveal the use cases they consider important. I do not know their reasoning, but I can imagine why they might prefer to keep those evaluations private while building an early advantage.

Does that make S1MB japanese-v1 pointless? I think the answer is mixed. A shared benchmark is useful even when it captures only particular aspects of performance. Taking it at face value, however, risks overlooking generalization. Please treat these results as reference points. The evaluation that really matters is one you build around your own use cases.

My current impression is that around 4B parameters may be a useful balance between Decision Model performance and inference cost. In the future, models below 1B may be sufficient too. I like small models, especially encoders, and would enjoy seeing compact models that do exactly what their applications need. saihai-310m is not there yet, but I am releasing it as a step in that direction.

References

• Evaluating Jev on Japanese benchmarks (Japanese)
• Base model: llm-jp-4.1-8b-thinking
• Japanese synthetic-data generator: llm-jp-4.1-32b-a3b-thinking
• saihai training code
• saihai training data
• bekko-system-one-v0 release article
• saihai-310m training documentation
• saihai-310m training data
• saihai-310m base model: ruri-v3-reranker-310m
• S1MB evaluation code
• S1MB Japanese leaderboard
• S1MB English leaderboard
• S1MB English benchmark data
• GPT-6 Luna: OpenAI Decisions API
• S1MB evaluation results
• pplx-decider-v1-27b
• llama.cpp Decision readout implementation