Auditing algorithmic bias in AI hiring requires a structured five-step process run monthly — not a one-time configuration check — because bias in AI systems emerges from data drift, rubric miscalibration, and changing applicant demographics that transform a compliant system into a non-compliant one over 3–6 months without any deliberate change. Here is the complete audit methodology. See the Explainable AI for Fair Hiring guide for the XAI transparency tools this audit process uses.

Step 1: How Do You Collect the Decision Data Required for Bias Auditing?

Collect 90 days of AI screening decisions with: candidate ID, application date, role type, AI score, screening decision (advance/screen-out), and the protected class indicators available from your EEO data collection — race/ethnicity category (if collected), gender, age group, and veteran status. Do not use individual candidate-level demographic data in your rubric; use it only in the aggregate audit calculation. If you do not collect EEO data at application, use applicant name-based demographic inference tools (with documented limitations) or rely on the EEOC’s 4/5ths rule applied to self-reported post-hire data.

Step 2: How Do You Apply the 4/5ths Rule to AI Screening Decisions?

The EEOC’s 4/5ths (80%) rule: if the selection rate for any protected group is less than 80% of the selection rate for the group with the highest rate, adverse impact is indicated. For AI screening, calculate the selection rate for each protected group: (candidates from group X advanced past screening) / (total candidates from group X). Divide each group’s rate by the highest group’s rate. Any ratio below 0.80 requires investigation and remediation. Example: if 45% of white male candidates advance and only 31% of Black female candidates advance, the disparity ratio is 0.69 — below the 0.80 threshold and requiring immediate rubric review.

Step 3: How Do You Identify Which Rubric Dimensions Are Causing Bias?

Decompose the overall disparity by rubric dimension: calculate each dimension’s score distribution by protected group. The dimension with the largest cross-group variance is the primary bias source. Common findings: geographic proximity scoring disadvantages protected groups concentrated in specific neighborhoods; years-of-experience requirements disproportionately screen out candidates from protected groups with non-linear career paths; keyword matching on prestigious university names disadvantages candidates from HBCUs and regional universities. For each biased dimension, evaluate: is this dimension predictive of job performance (if not, remove it), or is there a less biased operationalization (if so, substitute it).

Step 4: How Do You Remediate and Validate Rubric Bias Corrections?

After identifying and modifying biased dimensions: re-run the 4/5ths rule analysis on the prior 90 days of decisions using the revised rubric weights to project whether the remediation eliminates the disparity. If the projected post-remediation disparity ratio is above 0.80, deploy the revised rubric and monitor the subsequent 60 days of decisions to confirm. If the projected ratio is still below 0.80, the dimension needs further modification or removal. Document all rubric changes with: the bias finding that triggered the change, the specific modification made, the projected and actual post-remediation disparity ratios, and the date and approver. File with your legal counsel.

Expert Take — Jeff Arnold, 4Spot Consulting™

Monthly bias auditing sounds like compliance overhead until you see a client discover a disparity ratio of 0.67 that has been building for 8 months without anyone noticing. At that point, they have 8 months of discriminatory AI screening decisions to remediate — re-reviewing hundreds of rejected candidates, potentially extending offers to candidates who have long since moved on. Monthly auditing costs 2 hours a month. Remediation costs 200 hours. Do the audit.

Key Takeaways

• Bias audits must run monthly — AI systems drift toward bias through data drift, rubric miscalibration, and applicant pool changes.
• 4/5ths rule: any protected group selection rate below 80% of the highest group’s rate indicates adverse impact requiring investigation.
• Decompose disparity by dimension: the dimension with the largest cross-group variance is the primary bias source.
• Remediation validation: project post-remediation disparity ratio before deploying rubric changes; monitor for 60 days post-deployment.
• Document all rubric changes with finding, modification, projected ratio, actual ratio, date, and approver — file with legal counsel.

Frequently Asked Questions

What sample size is needed for a statistically valid algorithmic bias audit?

A minimum of 30 decisions per protected group per role type is required for statistical significance in a 4/5ths rule analysis. For organizations with lower application volumes, aggregate across multiple months or role types to reach this threshold. Bias audits on sample sizes below 30 per group produce unstable disparity ratios that overstate or understate actual bias — document the sample size limitation in your audit report.

Are employers legally required to conduct algorithmic bias audits?

New York City Local Law 144 (effective January 2023) requires employers using Automated Employment Decision Tools to conduct annual bias audits and publish summary results. Illinois and Maryland have similar requirements. EU AI Act classifies employment AI as high-risk and requires bias monitoring throughout the system’s operational life. Federal law does not yet require audits but EEOC guidance makes clear that employers are liable for discriminatory AI outcomes regardless of whether the AI was built internally or by a vendor.

How do you conduct a bias audit when your AI vendor won’t share the model details?

You do not need model internals to conduct a 4/5ths rule audit — you need decision inputs and outputs. Require your vendor contract to provide: applicant-level screening decision data (anonymized) with sufficient demographic fields for audit calculation, and the right to conduct annual third-party audits of the decision data. If the vendor refuses this contractual right, evaluate whether their tool creates regulatory risk that outweighs its operational benefits.