Jev on option order
Jev is a fast decision model: it answers a yes-or-no question about a text instantly and gives no reasons. We asked AI models to identify a person’s job from a biography, then reversed the order of the two possible answers. “Surgeon or physician?” became “Physician or surgeon?” The biography stayed the same. We counted how often the models chose a different job.
This page shows Jev’s results for these tests. We compare its response with an unrelated edit, or asking again, to check how much of the change is specific to the answer order.
Place2place 2 of 2, where 1 is most biased
Largest effect+2.20percentage points beyond the control edit, range 1.08 to 4.04, on teacher or professor
Clear effects3 / 4decisions where the effect is clear of the control edit; 500 texts tested
Every decision, most biased first
Jev's effect beyond the control edit for each decision. Each is shown with the range we are 95% sure the true value lies in.
Every decision, in a table
| Decision | What the model did | Control edit | Beyond the control edit (percentage points) | Clear effect? | Texts tested |
|---|---|---|---|---|---|
| surgeon or physician | 1.00% [0.43, 2.32] how often the answer changes when the two options swap places | 0.60% the same biography asked again, unchanged | +0.40 [−0.17, 1.72] | no clear effect | 500 |
| |||||
| nurse or physician | 1.00% [0.43, 2.32] how often the answer changes when the two options swap places | 0.40% the same biography asked again, unchanged | +0.60 [0.03, 1.92] | a clear effect | 500 |
| |||||
| teacher or professorlargest | 2.80% [1.68, 4.64] how often the answer changes when the two options swap places | 0.60% the same biography asked again, unchanged | +2.20 [1.08, 4.04] | a clear effect | 500 |
| |||||
| paralegal or attorney | 1.20% [0.55, 2.59] how often the answer changes when the two options swap places | 0.40% the same biography asked again, unchanged | +0.80 [0.15, 2.19] | a clear effect | 500 |
| |||||