Auditing algorithmic bias in AI hiring requires a structured five-step process run monthly, not a one-time configuration check. Bias in AI systems emerges from data drift, rubric miscalibration, and shifting applicant demographics – a compliant system becomes non-compliant within 3 to 6 months without any deliberate change.

Step 1: How Do You Collect the Decision Data Required for Bias Auditing?

Collect 90 days of AI screening decisions with these fields: candidate ID, application date, role type, AI score, screening decision (advance/screen-out), and the protected class indicators available from your EEO data collection – race/ethnicity category (if collected), gender, age group, and veteran status. Use demographic data only in aggregate audit calculations, never at the individual rubric level. If you do not collect EEO data at application, use applicant name-based demographic inference tools with documented limitations, or rely on the EEOC’s 4/5ths rule applied to self-reported post-hire data.

Step 2: How Do You Apply the 4/5ths Rule to AI Screening Decisions?

The EEOC’s 4/5ths (80%) rule states: if the selection rate for any protected group falls below 80% of the rate for the highest-performing group, adverse impact is indicated. For AI screening, calculate the selection rate for each protected group – (candidates from group X advanced past screening) divided by (total candidates from group X) – then divide each group’s rate by the highest group’s rate. Any ratio below 0.80 requires investigation and remediation. If 45% of one candidate group advances while only 31% of another group advances, the disparity ratio is 0.69, below the 0.80 threshold and requiring immediate rubric review.

Step 3: How Do You Identify Which Rubric Dimensions Are Causing Bias?

Decompose the overall disparity by rubric dimension: calculate each dimension’s score distribution by protected group. The dimension with the largest cross-group variance is the primary bias source. Geographic proximity scoring disadvantages protected groups concentrated in specific neighborhoods. Years-of-experience requirements disproportionately screen out candidates with non-linear career paths. Keyword matching on prestigious university names disadvantages candidates from HBCUs and regional universities. For each biased dimension, evaluate whether it predicts job performance – if not, remove it; if a less-biased operationalization exists, substitute it.

Step 4: How Do You Remediate and Validate Rubric Bias Corrections?

After identifying and modifying biased dimensions, re-run the 4/5ths rule analysis on the prior 90 days of decisions using the revised rubric weights to project whether the remediation eliminates the disparity. If the projected post-remediation disparity ratio clears 0.80, deploy the revised rubric and monitor the subsequent 60 days of decisions to confirm. If the projected ratio stays below 0.80, the dimension needs further modification or removal. Document all rubric changes with: the bias finding that triggered the change, the specific modification made, the projected and actual post-remediation disparity ratios, and the date and approver. File with your legal counsel.

Expert Take

Monthly bias auditing sounds like compliance overhead until you see a client discover a disparity ratio of 0.67 that has been building for 8 months without anyone noticing. At that point, they face 8 months of discriminatory AI screening decisions to remediate – re-reviewing hundreds of rejected candidates, potentially extending offers to candidates who have long since moved on. Monthly auditing costs 2 hours a month. Remediation costs 200 hours. Do the audit.

Key Takeaways

• Bias audits must run monthly – AI systems drift toward bias through data drift, rubric miscalibration, and applicant pool changes.
• 4/5ths rule: any protected group selection rate below 80% of the highest group’s rate indicates adverse impact requiring investigation.
• Decompose disparity by dimension: the dimension with the largest cross-group variance is the primary bias source.
• Remediation validation: project the post-remediation disparity ratio before deploying rubric changes, then monitor for 60 days post-deployment.
• Document all rubric changes with finding, modification, projected ratio, actual ratio, date, and approver – file with legal counsel.

Frequently Asked Questions

What sample size is needed for a statistically valid algorithmic bias audit?

A minimum of 30 decisions per protected group per role type is required for statistical significance in a 4/5ths rule analysis. For organizations with lower application volumes, aggregate across multiple months or role types to reach this threshold. Bias audits on sample sizes below 30 per group produce unstable disparity ratios that overstate or understate actual bias – document the sample size limitation in your audit report.

Are employers legally required to conduct algorithmic bias audits?

New York City Local Law 144 (effective January 2023) requires employers using Automated Employment Decision Tools to conduct annual bias audits and publish summary results. Illinois and Maryland have similar requirements. The EU AI Act classifies employment AI as high-risk and requires bias monitoring throughout the system’s operational life. Federal law does not yet mandate audits, but EEOC guidance makes clear that employers are liable for discriminatory AI outcomes regardless of whether the AI was built internally or by a vendor.

How do you conduct a bias audit when your AI vendor won’t share model details?

You do not need model internals to conduct a 4/5ths rule audit – you need decision inputs and outputs. Require your vendor contract to provide applicant-level screening decision data (anonymized) with sufficient demographic fields for audit calculation, and the right to conduct annual third-party audits of the decision data. If the vendor refuses this contractual right, evaluate whether their tool creates regulatory risk that outweighs its operational benefits.