One open-ended judgment question gives you early signal AI can’t fake. You add a single prompt asking for a specific decision under incomplete information, score the reasoning against a short rubric, and route answers to a human. There’s no fixed answer to reverse-engineer. This is step one in the screening rebuild after the audit.

Before You Start

Confirm your application form supports a long-text field and that your ATS can route that field to a reviewer. If you haven’t yet, run the screening-to-hire audit first so you know the resume stage needs help. Sequence matters here: the audit gives you the evidence that justifies adding friction to your application, and adding friction without that evidence invites pushback from anyone watching completion rates. Decide too who will read the answers and how their time gets protected, because a judgment question with no committed reviewer becomes a field nobody scores. The setup is light — one field, one rubric, one routing rule — but each piece has to exist before the first real candidate hits the form, or the answers pile up unread and the signal you built evaporates.

Step 1: Write the Prompt

Use a prompt with no clean answer: “Describe a judgment call you made with incomplete information — what did you decide and why?” The power is that it demands a specific real event. Generic AI text produces plausible-but-hollow answers that fall apart against specificity. Compare it to the prompt it replaces. “Why are you a good fit for this role?” has an obvious target a candidate can hit perfectly with a tool in thirty seconds, because the question is asking to be flattered. “Describe a judgment call with incomplete information” has no target — there is no model answer to converge on, only the candidate’s own history to draw from. The mechanism is that you are asking for an event, not an opinion. An opinion can be generated; a specific event has to be supplied, and a specific event carries the texture that separates a real applicant from a polished one. Keep the wording concrete and avoid any phrasing that hints at the answer you want.

Step 2: Add It as a Required Field

Place it as a required long-text field on the application, after the basics. Keep it to one question — a single strong prompt beats a wall of them and respects the candidate’s time. Position matters: put it after name, contact, and resume upload so a candidate has already committed before they reach it, which lifts completion. Making it required is the point of the whole exercise, because an optional judgment question gets skipped by exactly the rushed-but-strong applicants you most want to surface. Resist the urge to add a second or third prompt. Each extra question shaves your completion rate and dilutes the signal, and one well-chosen prompt scored carefully tells you more than five answered carelessly. The mechanism is focus: a single field everyone must complete produces one clean, comparable answer per candidate, which is the unit your reviewers can actually score.

Step 3: Build a Three-Point Rubric

Score on three things: did they name a specific situation, did they identify the real tradeoff, and did they defend the decision? A defensible-but-imperfect answer with clear reasoning outscores a flawless-sounding answer with no specifics. The rubric is what keeps the question honest, because without it reviewers drift toward rewarding polish — the smooth, confident answer that says nothing. Make each criterion a simple yes-or-no so two reviewers land on the same score. Specificity asks whether a real, dated, located event appears or whether the answer floats in generalities. Tradeoff asks whether they named what they gave up, since a decision with no cost is not a judgment call. Defense asks whether the reasoning holds together. The mechanism is that a written rubric forces attention onto substance the candidate had to live and away from fluency a tool can manufacture, which is precisely the inversion the question exists to create.

Step 4: Route Answers to a Human

Use your ATS automation hooks to send each answer to the right reviewer. Automate the routing and the reminders with a platform like Make.com — keep the reading and scoring human. Evaluation is a judgment, and judgment stays with a person. The split is the entire discipline of the approach. A webhook can move the answer to the right reviewer’s queue, nudge them if it sits for a day, and post the score back to the ATS, and none of that touches the evaluation itself. The line you never cross is letting the machine read and grade the answer, because the moment a model scores a judgment response, you have handed candidates a target to optimize against and rebuilt the gameable surface you removed. The mechanism is that logistics is rule-bound and repeatable while evaluation is contextual, so you automate the first completely and protect the second completely.

Step 5: Use It to Prioritize, Not Knockout

Treat a strong answer as a reason to fast-track to a structured phone screen, not as a hard gate. The question gives early signal on how a candidate thinks; the screen confirms it live. The reason to prioritize rather than reject is that a single written answer is one data point, and one data point is enough to move someone up the queue but not enough to end their candidacy. Someone who writes a flat answer on a rushed lunch break can still be excellent on a call, and you do not want to lose them to a hard cutoff on a paragraph. So a strong answer earns a fast track to the conversation where you ask the follow-ups, and a weak one simply does not jump the line. The mechanism is using the question for what it is good at — cheap early signal — without overloading it with a finality it cannot bear.

How to Know It Worked

Within a few weeks you’ll see clear separation: specific, reasoned answers versus generic filler. When your reviewers can reliably tell the two apart, the question is doing its job — surfacing thinking the resume hid. The strongest confirmation comes downstream: the candidates who wrote sharp answers should also be the ones who hold up on the structured phone screen, and that consistency tells you the early signal predicts the live one. Watch your reviewers’ scores converge as well, because two people landing on the same rating for the same answer means the rubric is carrying the judgment rather than personal taste. If instead every answer reads the same, the prompt is too easy to optimize and needs more ambiguity. A working question divides the field; a broken one flattens it.

Common Mistakes

• Asking a question with a “right” answer. That’s reverse-engineerable. Keep it open.
• Adding five questions. One strong prompt outperforms a battery and won’t tank completion rates.
• Auto-scoring the answer. The whole point is human judgment — don’t hand it back to a model.

Next Step

Pair the question with a live conversation: run a 15-minute structured phone screen. For the full picture, see the pillar guide.