
Post: How to Redesign an Assessment to Resist AI Gaming: A Judgment-First Method
An assessment resists AI gaming when there is no fixed answer to reverse-engineer. You replace correctness with judgment: messy scenarios, competing priorities, missing information, and questions with no clean right answer. The deliverable becomes the reasoning, which AI assistance can’t shortcut. This is the assessment-redesign core of the AI resume screening rebuild.
Before You Start
Accept the premise first: any assessment with fixed correct answers is now solvable by a candidate with a tool in another tab, so a high score no longer differentiates. If you haven’t confirmed your scores stopped predicting performance, run the screening-to-hire audit before redesigning. Pull a real, messy situation from your own operation to build the scenario around.
Step 1: Replace the Right Answer With a Real Dilemma
Take a genuine situation from the role and strip it to its hard core: competing priorities, a deadline, and missing information. The candidate’s task is to decide and defend, not to retrieve a correct answer. Because there’s no key, there’s nothing for AI to reverse-engineer. A fixed-answer question asks “what is the right way to handle a payroll discrepancy” and has one correct response a tool will retrieve in seconds. A dilemma asks “the discrepancy surfaced an hour before the run, you can delay the whole payroll or push it with a known error and fix it next cycle, and you cannot reach the controller — what do you do and why.” There is no key for the second question because the answer depends on judgment the candidate has to supply themselves, which is exactly what no tool can hand them.
- Use a situation your team actually faced, so the texture is real and hard to fake.
- Build in a tradeoff with no clean resolution, where every choice gives something up.
- Withhold a piece of information on purpose, forcing the candidate to decide under the uncertainty the real job carries.
Step 2: Ask for the Decision and the Sacrifice
The prompt should require two things: what they’d decide and what they’d give up to do it. Naming the sacrifice is where judgment lives — it forces the candidate to show they understand the cost of their choice. Anyone can pick an option; the signal is whether they see what that option costs. A candidate who writes “I’d delay the payroll” has told you little. A candidate who writes “I’d delay the payroll, accepting that several hundred people get paid late and I’ll spend the morning fielding angry calls, because a known wrong number going out is worse than a late right one” has shown you they reason about consequences, not just outcomes. The sacrifice is the tell, because pretending a hard decision is costless is the surest sign someone has not actually made one.
- Require an explicit decision — no fence-sitting, no “it depends” without a chosen path.
- Require the tradeoff they accept, stated plainly as what they are giving up.
- Reward candor about the downside; the candidate who names the pain of their own choice is the one who has thought it through.
Step 3: Build a Reasoning Rubric, Not an Answer Key
Score three dimensions: did they identify the real tradeoff, did they make a defensible decision, and did they reason coherently to it? Two candidates can reach opposite conclusions and both score well — you’re measuring thinking, not agreement. This is the hardest mental shift for a scoring team, because a rubric feels safe and a rubric that allows opposite answers to both pass feels like chaos. It is not chaos; it is the point. One candidate delays the payroll and one pushes it, and if both correctly identified the tradeoff and reasoned coherently to their choice, both demonstrated the judgment you are hiring for. The moment you decide one of those answers is “the right one,” you have quietly rebuilt the answer key and handed the advantage back to whoever guesses your preference best.
- Define what strong reasoning looks like — naming the real tension, weighing both sides, committing to a defensible path.
- Allow multiple defensible conclusions, scoring the quality of the thinking rather than the side it landed on.
- Score the path, not the destination; the route a candidate took to their answer is the thing you are actually measuring.
Step 4: Add Intentional Ambiguity to the Wording
Leave the scenario slightly open so candidates must state their assumptions. How they handle the ambiguity — whether they name what they assumed and why — is itself signal. AI-generated answers rarely surface their own assumptions. A real professional reading an underspecified problem instinctively fills the gaps and says so: “I’m assuming this is a salaried payroll, not hourly, because that changes whether the error is recoverable.” That instinct — to notice what is missing and make the assumption explicit — is a hallmark of someone who has done the work, and it is precisely what a generated answer skips, because a tool tends to answer the question as written rather than interrogate what the question left out. The ambiguity you build in is a trap for thoughtlessness and a stage for real judgment.
- Ask candidates to state assumptions explicitly, making the gap-filling visible.
- Score the quality of those assumptions — are they reasonable, and do they reveal domain understanding?
Step 5: Pair the Assessment With a Live Follow-Up
Confirm the written reasoning in a short conversation. Ask “why that over the alternative?” Borrowed reasoning collapses under live follow-up; lived reasoning deepens. The combination is far stronger than either alone — route candidates into a structured phone screen. The written answer and the spoken answer are two samples of the same thinking, and a person who actually reasoned their way to a written conclusion can extend it live without strain, because they are describing their own thought. A person who submitted reasoning they did not generate stalls the instant you push past the script — they cannot tell you why they rejected the alternative, because they never considered it. Five minutes of “walk me through that decision again” separates the two more reliably than any detection tool, because it asks the candidate to think rather than to retrieve.
- Probe the written answer live, asking the candidate to defend the choice they already made.
- Compare written and spoken reasoning for consistency; a real author’s two versions agree and deepen, a borrowed one’s diverge and thin.
How to Know It Worked
You’ll see divergence return. Where fixed-answer scores clustered near the ceiling, judgment-based answers spread across a real range — because reasoning under ambiguity reflects who the candidate is. A fixed-answer test under AI produces a wall of perfect scores that tells you nothing; a judgment test produces a real distribution, with strong reasoners pulling clear of weak ones the way they should. When your assessment scores start predicting interview and on-the-job performance again — when the people who scored well on the dilemma turn out to be the people who handle real dilemmas well — the redesign succeeded, and you have an instrument that measures the candidate instead of their toolkit.
Common Mistakes
- Leaving a hidden right answer in the scenario. If a “best” answer exists, it’s reverse-engineerable. Keep the tradeoff genuinely open.
- Scoring agreement instead of reasoning. Penalizing a defensible minority answer rebuilds the answer key you just removed.
- Skipping the live follow-up. The conversation is what confirms the written reasoning was the candidate’s own.
Expert Take
The instinct when AI breaks your test is to make the test harder. That fails — a harder fixed-answer test is still solvable in another tab, just with more steps. The escape is to change what you’re measuring, from correctness to judgment. The day your assessment has no answer key is the day AI assistance stops being an advantage, because there’s nothing left to retrieve. Build the dilemma, score the reasoning, and confirm it live. That’s a test a chatbot can’t take for anyone.
Next Step
Compare this approach to detection in automated scoring vs human phone screens, and read the pillar guide for the full rebuild.

