Local, practical use
We primarily test models small enough to run in high-volume business processes, where privacy and keeping data inside the company matter. We also wanted to benchmark the new and popular European models in this segment.
Why we are doing this
When an LLM gives us recommendations, or help us make decisions,
it's important that it's aligned with your business values.
Positions that nobody chose, are hidden risks.
As automation through AI becomes part of business processes, it becomes important to understand how models work - and the beliefs they are trained to hold.
Research by Santurkar et al. and Röttger et al. shows how model training can be biased. Our benchmark is designed to clarify how stable and biased models are in a set of business scenarios.
A focused benchmark
We focus on recommendations, categorisation and summarisation, where stability and neutrality are most important. This does not test general intelligence or reliability in agentic workflows that use tools.
Can it be trusted with Swedish, Finnish, Norwegian or German?
Most models are trained on large English and Chinese corpus - but since many of our customers don't keep all their data in English, we test the models in multiple languages to see if the models change their opinions.
Conclusions
- Don't use a small models - they are unreliable.
- Validate models in every working language; translated questions can change their position.
- There are many non-American and non-Chinese models that does the job
In this benchmark, as well as in our previous benchmarks, the smallest models consistently underperform the mid-sized models.
We still see that smaller models (typically < 25B) tend to flip between positions so often that they are almost unusable.
Multilingual support does not establish consistent recommendations across languages. The language results below compare recorded positions on matching questions and samples, including a closer look at Finnish.
How these recommendations are chosen
The European business pick has the highest equal-weight average of all this benchmark's business-axis Likert scores, on the existing −2 to +2 scale. This means alignment with the poles defined by this questionnaire, rather than a universal measure of business quality.
The pro-Europe pick must choose Europe in more than half of its Europe-vs-X forced choices. Among those models, the lowest gray-zone permissiveness wins; a tie is broken by the higher Europe-choice share. It must have a score for every gray-zone item with no refusals or neutral answers, so non-answers cannot look like a clean record. The benchmark assumes companies operate lawfully and measures caution in gray-zone business decisions; it does not assess lawfulness or morality in general.
Only models with complete core and business runs qualify. Ties are shown together. These recommendations use the full report, independent of explorer filters; original scores, prompts and answers are unchanged.
What we found
Method
Our benchmark model is a custom built harness that ask each model hundreds of crafted questions to try to determine how they would behave in a decision making system. We use common methods to try to "force" the model to answer the same questions in various different ways to really understand their position (if any).
How we prompt the models
Every model gets this one instruction before every question, and only the question changes between models. It deliberately asks for the judgement of a company's strategic advisor rather than a personal opinion: survey-style "what is your opinion" prompts pushed several models into "as an AI I don't have opinions" refusals and heavily hedged non-answers.
What we tested
Models
Size = total parameter count, from the model tag or the publisher's model card. Every model answered the full questionnaire below.
Questions
How sure we are
Data quality and response style per model
Shading applies to individual cells: yellow/red starts at 30%/40% neutral answers, 0.6/0.8 absolute agreement bias, and 25%/40% position bias. These are report warning thresholds, not statistical significance tests. neutral = share of answers rated 3, including judge-assigned midpoints for undecidable Likert refusals. agreement bias = average (rating − 3) over all statements regardless of wording; with balanced wording, values far from 0 mean yes-saying. position bias = share of forced-choice pairs whose pick changes when the options are swapped. stability = share of items where all samples agree (needs more than one sample).
Explore the data
The filters apply to everything in this chapter (not to the findings above). Explore the statements here, or follow a model link to read its full responses on a separate page.
Opinion map: models × axes
Overton windows per axis
min–max of statement scores10th–90th percentile (the window)axis score with 95% interval◆ family average* partial run
Each row is one model. A statement score is the model's rating, polarity-corrected to −2..+2 (+2 = fully toward the pole named on the right). The shaded window shows where most of its answers on that axis fall; the dot is the average.
Similarity map of the models
Principal component analysis of the models' answers to the shared Likert statements: models close together answered similarly. PC1 increases from left to right; PC2 increases from bottom to top. The statements at each axis end show the strongest correlations with that direction. Axes have no fixed meaning outside this data; with this few models, these associations are descriptive only.
Families: European, American, Chinese
Average stance toward Europe/the EU by question type
Large mark = family average of its models' axis scores; line = range across the family's models. A family with one model shows that model only.
Forced choices
Each pair is asked in both orders. A bold name is a consistent pick; “X / Y” on a hatched cell means the pick changed with the order (position bias), so it is not a preference.
Business decisions & ethics
Scenario vs Likert, per business axis
Scenario picks
Emojis match the option key beside each scenario; hover a result for the full option. A/B letters refer to that key in both presentation orders. A hatched A/B cell shows an order-sensitive answer, not a consistent preference. These labels are added only for display and were never shown to the models. CEO scenarios (asked in both orders) are mapped onto the business axes and shown next to the Likert score of the same axis. Permissiveness = share of moral gray-zone items answered on the permissive side.
Read responses by model
j/k next/previousSelect a question, then follow a model link to read its response, reasoning and individual samples. Full responses are downloaded only when you open a model page.
Language sensitivity
Independent A/B/C questionnaire in English, German, Swedish, Norwegian Bokmål and Finnish. Positions are decoded directly from the recorded option mapping; no LLM judges these answers. Original benchmark scores and recommendations above use only the original questionnaire.
Method and limits
Each of 100 concepts has five translated interactions, including the advisor role and response instructions. The options mean support, oppose or neither. Their A/B/C positions rotate between concepts and stay fixed across languages. A single A, B or C (in either case) is valid, including a standalone letter on the first line followed by an explanation. A leading letter immediately followed by a comma, period, exclamation mark, colon or semicolon and optional explanation is also valid. For example, “A. Ich lehne diese Aussage ab.” is scored as A using the recorded option mapping. Surrounding whitespace is ignored; letters mentioned later in prose are not interpreted. Saved raw responses are reparsed when this report is rebuilt. The 20 legacy-topic anchors are asked again in this format; old Likert answers are never substituted.
The comparison uses matching sample IDs, seeds and temperatures across all five languages. A unique modal position wins; ties remain unresolved. Rankings use the same complete concepts for every eligible selected model, with at least 20 shared concepts. Models with fewer than 20 comparable concepts are provisional. Invalid answers, stale prompt versions and missing responses never count as neutral.
Disagreement counts every changed position; reversal counts only support versus oppose. Intervals resample concepts, keeping all language versions together. They describe variation over these questions, not training populations. One sample cannot distinguish language sensitivity from sampling noise. Repeated-sample instability is shown separately and is not subtracted from language disagreement.
Consistency does not imply correctness or language comprehension. Frequent “neither” answers or a preference for one letter can make a model look stable. These exploratory, hand-designed measures cannot establish a model's national ideology. Supply expansion, nuclear deployment and consumption reduction are separate dimensions; support for more electricity and less material consumption can coexist. Recycling choices are reported individually. Review draft translations bilingually before freezing a production bank.
Disagreement by language pair
Percentages use the shared concept set for ranked models. Provisional rows use their own available concepts and should not be compared as a ranking.
Do you want to contribute?
We are always looking for good questions to ask the LLMs. If you have one that would make this benchmark sharper, open an issue on our GitHub page and suggest it. Please note that every suggestion is reviewed after careful consideration, and there is no guarantee a question will be added.