Kev on option order
Kev is an open-source a fast decision model. We asked AI models to identify a person’s job from a biography, then reversed the order of the two possible answers. “Surgeon or physician?” became “Physician or surgeon?” The biography stayed the same. We counted how often the models chose a different job.
This page shows Kev’s results for these tests. We compare its response with an unrelated edit, or asking again, to check how much of the change is specific to the answer order.
Every decision, most biased first
Kev's effect beyond the control edit for each decision. Each is shown with its 95% range.
Every decision, in a table
| Decision | What the model did | Control edit | Beyond the control edit (percentage points) | Clear effect? | Texts tested |
|---|---|---|---|---|---|
| surgeon or physicianlargest | 5.80% [4.07, 8.21] how often the answer changes when the two options swap places | 1.38% [0.88, 2.15] the same biography with one neutral wording change, such as "also" for "as well" | 4.42 [1.92, 7.32] | a clear effect | 500 |
| |||||
| nurse or physician | 2.40% [1.38, 4.15] how often the answer changes when the two options swap places | 1.05% [0.64, 1.72] the same biography with one neutral wording change, such as "also" for "as well" | 1.35 [−0.35, 3.51] | no clear effect | 500 |
| |||||
| teacher or professor | 5.00% [3.41, 7.28] how often the answer changes when the two options swap places | 1.03% [0.58, 1.84] the same biography with one neutral wording change, such as "also" for "as well" | 3.97 [1.57, 6.70] | a clear effect | 500 |
| |||||
| paralegal or attorney | 2.60% [1.53, 4.40] how often the answer changes when the two options swap places | 1.10% [0.64, 1.87] the same biography with one neutral wording change, such as "also" for "as well" | 1.50 [−0.35, 3.75] | no clear effect | 500 |
| |||||
Release v1.49.1, published .
Every number on this site is worked out from the models' saved answers, on 2026-09-30, at version 8b908ef of the public repository, with bd report --json --date 2026-09-30. You can download every number in one data file. About this site.
Where the answers come from: studies/<task>-<cue>.jsonl (re-run from the saved answers with bd replay).
Biased-Decisions is MIT-licensed (source). Bias in Bios (De-Arteaga et al., 2019) is MIT-licensed on the Hugging Face Hub. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.