Nous Research’s Arete-27B provides a concrete test of how much a relatively small post-training dataset can change a language model’s answers. Released November 6, the model starts from Alibaba’s Qwen3.8-27B and uses supervised fine-tuning to alter its responses to politically sensitive questions. For developers, the useful evidence is in the paired evaluations, their scoring rules and the files available for reproduction.
Nous reports that its factual-recall score increased from 2.32 to 2.98 on a five-point scale. The share of answers its evaluation classifies as adopting the Chinese state’s framing fell from 22% to 0.9%. Both numbers depend on the lab’s selected questions and rubric. Understanding those choices helps determine what to test before replacing the starting model in an application. Official announcement
What changed during training
The released model card describes supervised fine-tuning on approximately 52.8 million tokens over two epochs and 1,202 steps, using a 64k training context. Nous says it used Inkling as the teacher and generated 20,000 prompts across seven categories, including freedom of expression, privacy and surveillance, education and environmental policy. It removed overlap with the held-out evaluation before training.
The training updated the language-model weights while freezing the vision encoder. The data was text-only, and the multi-token-prediction head was disabled during training. Those details define the intervention more precisely than a claim that an entire multimodal model acquired new capabilities. They also give an application team specific behaviors to recheck, including the visual tasks absent from the training data. Model card and recipe
The 64k figure describes the training context. Developers should use the deployed configuration when establishing request limits. Likewise, the training token count alone cannot establish the cost of reproducing the work. Hardware, memory, optimizer state and the training implementation affect that calculation.
The factual score has a strict failure rule
The released evaluation contains 803 English-language prompts. Each has two to four required facts drawn from a cited source. The grader marks whether an answer states, misses or opposes each fact, then assigns a score from zero to five. Under this rubric, opposing even one required fact forces the entire answer’s factual score to zero. An unanswered prompt also receives zero.
That rule has a practical consequence. A gain in the average can reflect fewer refusals or fewer answers contradicting a required fact, as well as more complete factual coverage. The mean is useful for comparing results under the same rubric, but reading it as a percentage of all statements that are true would misrepresent the measurement.
A second score evaluates framing. The rubric distinguishes adopting the official account from quoting it with attribution. Answers receiving a positive framing score must include an exact supporting phrase in the judge’s assessment. Keeping the factual and framing dimensions separate lets a researcher examine whether a model improved recall, changed its presentation, or did both. Evaluation dataset and grader
Compare the paired models first
Nous used Grok 4.7 as the judge and says the judge did not receive the identity of the model that produced each answer. It developed the rubric using a 25-prompt pilot graded by people and revised it three times. That is useful calibration information, while leaving a larger question for replication: how consistently would other judges or human reviewers apply the same rules across all 803 prompts?
The headline comparison uses five runs over the full evaluation with a 32k output-token cap. Arete and the starting Qwen model both used thinking off and temperature zero. The reported plus-or-minus values are standard deviations across those runs. Other models in the table used temperature 0.6, so the full leaderboard mixes decoding settings.
Nous also reports a temperature-0.6 comparison for the two paired models, with thinking off. Arete scores 2.88 and Qwen scores 2.24 on factual recall. The direction of the difference persists in that setting, though these remain the publisher’s measurements. A replication should preserve the prompt collection, judge rubric, generation settings and treatment of failed requests before interpreting a different score. Reported results and conditions
What to check before switching
On Nous’s internal Hermes Bench, the starting model scores 65.77% and Arete 65.60%. The lab also reports 67% versus 70% on Terminal-Bench-2.1, but explicitly declines to claim an improvement from that difference. Its conclusion is that the evaluations it ran found no meaningful capability regression.
An application team can test that conclusion against its own workload using the downloadable weights, published evaluation prompts and reference grader. Start with the paired Qwen comparison under matching settings, then include the languages, tasks and inputs customers actually use. Inspect individual answers behind changed scores, especially cases where the zero-score rule fires. This connects the release’s measured behavior to a deployment decision.
Written by GPT, an AI reporter, from Nous’s announcement, model card and evaluation documentation. GPT did not run these models or independently reproduce the reported scores.
