The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Evaluate a machine-learning hiring tool against the job it will actually be used for, not against the vendor’s product description. Before deployment, document what the model measures and how much its output shapes the decision. Confirm which audit and notice duties apply, test whether qualified applicants with disabilities are screened out, give candidates a working accommodation path, and tie all of that evidence to the exact tool version you will run. For uses covered by New York City’s Local Law 144, the statutory core is a bias audit no more than one year before use, a public audit summary, and candidate notice at least 10 business days before the tool is used.
Settle whether the tool is covered before testing it
Coverage decides which duties you inherit, so answer it first. New York City’s rules for automated employment decision tools (AEDTs) are in New York City Administrative Code § 20-871. The definition has three working parts: a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence; a simplified output such as a score, classification, or recommendation; and use that substantially assists or replaces discretionary employment decisions.
A vendor’s product label does not settle any of those points. Assess the workflow as it runs. Find out whether recruiters start from the output, whether a threshold decides who advances, and whether anyone reviews candidates the tool scored low. Write that down, because it is the record you will need if the deployment is ever questioned.
| Output form | Where it shows up in the pipeline | Question to answer |
|---|---|---|
| Score | A numeric rating attached to each candidate | Does a cutoff or a minimum value decide who advances? |
| Ranking | An ordered shortlist presented to recruiters | Do reviewers begin at the top and stop reading once they have enough names? |
| Classification | An advance or reject flag, or a pass or fail label | Does a person have to confirm or override the flag before the candidate is removed? |
| Recommendation | A note such as “interview” or “not a fit” | How often is the recommendation followed? Track the rate rather than assuming it. |
New York City: obligations before first use
Local Law 144 applies to covered tools used to screen candidates or employees for employment decisions in New York City. According to the city’s Department of Consumer and Worker Protection (DCWP), enforcement began July 5, 2023, per its AEDT page. Work through the following steps in order. Each one should be complete before the tool scores a city candidate.
#1 Best Overall
- Confirm the use is covered. The notice and disclosure duties apply to city-resident candidates and employees. Establish from actual use whether the tool substantially assists or replaces a discretionary decision.
- Obtain a bias audit dated no more than one year before use. The audit must be conducted by an independent auditor and must apply to the tool you are deploying.
- Publish or confirm the audit summary. The most recent summary and its distribution date must be publicly available before the tool is used.
- Give notice at least 10 business days before use. The notice must state that an AEDT will be used, list the job qualifications and characteristics it assesses, and let candidates request an alternative selection process or an accommodation.
- Prepare the data disclosure. Within 30 days of a written request, you must provide or publish the data types collected, their sources, and your retention policy.
- Recheck steps 2 through 4 when the tool version or the use changes. An audit that covers a different version or a different job context may no longer satisfy the one-year, same-tool test.
What a bias audit summary should let you verify
Do not accept a summary as a headline number. Check the parts that determine whether it describes your deployment:
- Audit date. It must fall within one year before your planned use. Calculate the expiry date and put it on the deployment calendar.
- Auditor independence. The statute requires an independent auditor. Ask who the auditor is and what relationship, if any, it has with the vendor.
- Data source. Ask whether the audit used historical applicant data or test data. The city’s audit rules allow test data when historical data is insufficient, so a clean summary built on test data says less about your applicant pool.
- Impact ratios. Summaries report impact ratios across sex and race/ethnicity categories, as the DCWP AEDT page describes the program. Check which categories were used and the selection rates behind each ratio, not only the ratio itself.
- Version and scope. Match the audit’s tool version or distribution date to what you will run, including the job families and locations it covers.
- Stated limitations. Read what the auditor says the audit does not cover. A limitation you accept without reading is still a limitation.
A posted summary is not proof of compliance
The New York State Office of the State Comptroller reviewed enforcement of Local Law 144 in a report issued December 2, 2025. Among 32 companies it reviewed, DCWP identified one issue. The Comptroller’s own review found at least 17 potential instances of non-compliance. The same report says DCWP received only two AEDT complaints during the period examined, July 2023 through June 2025.
Rank #2
- Used Book in Good Condition
These figures describe one sample and one enforcement period. They are not a market-wide non-compliance rate, and two complaints do not measure how often violations occur. What they do show is that the existence of a posted summary or notice tells you little on its own. Check each element against the statute, and treat the gap between the two agencies’ counts as a reason to verify rather than to assume.
Disability access: test the assessment, not only its accuracy
The Americans with Disabilities Act applies to employers’ selection, testing, and promotion decisions. The Department of Justice’s guidance on algorithms, artificial intelligence, and disability discrimination in hiring says employers should examine hiring technologies before use, and regularly while in use, for whether they screen out qualified people with disabilities who could perform essential job functions with or without accommodation. The guidance says tests should measure the relevant job skill, not an unrelated sensory, manual, or speaking impairment. Employers must provide reasonable accommodations unless doing so would cause undue hardship.
Recommended Free Tools
The question this creates is not whether the model is accurate on average. It is whether a qualified applicant with a disability can get through the process and be assessed on the job skill itself. An overall pass rate can look healthy while one group is consistently blocked at one step.
The EEOC and DOJ’s joint announcement of May 12, 2022 highlighted three concerns: accommodation processes, screening out qualified people with disabilities, and technology that prompts prohibited disability-related inquiries or medical examinations. In the EEOC’s press release, EEOC Chair Charlotte A. Burrows said: “New technologies should not become new ways to discriminate.”
Test each assessment step against an essential job function
- Start from the job’s essential functions and write down the skill each assessment step is meant to measure.
- Flag modality barriers: audio-only prompts, recorded video, timed interfaces, game mechanics, speaking requirements, and mouse-only interaction. For each one, ask whether the barrier measures a job skill or an unrelated ability.
- Run every step with the assistive technology a candidate is likely to use, including a screen reader, keyboard-only navigation, captions, magnification, and extended time. Record where the flow stops or where scoring changes.
- For each failure, either remove the barrier or provide an alternative that measures the same skill.
- Document how you know the alternative measures that same skill. Its existence alone does not show that it is equivalent.
Check labels and proxies for exclusion
DOJ warns that comparing candidates to current successful employees can perpetuate exclusion when disabled people were historically left out of those roles. If your training labels come from past hires or past performance ratings, ask who was in that pool and who was never in it. Review each input for a documented job-relevance justification. A feature that is only a statistical correlate of past success is a risk to explain, not evidence that it belongs in the model.
Make the accommodation path work in practice
Define a single accommodation channel, a named service owner, a response-time target, and an alternative process that measures the same skill. DOJ’s guidance gives accessible alternatives to interview software as an example of the kind of accommodation to expect. Test the path before candidates depend on it: submit a request yourself, time the response, and confirm the alternative can actually be completed. Log each request, the decision made, and the reason for it. If an accommodation is refused, the file should show the undue-hardship analysis.
Best Value
- Used Book in Good Condition
Vendor evidence to request
Ask for evidence tied to the configuration you intend to run, not to the product line in general.
- The audit date, scope, and the exact tool version or distribution date the audit covers.
- The audit methodology, the population it describes, and the job context it assumed.
- Known limitations stated by the auditor or the vendor.
- A current data inventory listing the data types collected, their sources, and retention periods, so a written request can be answered within 30 days.
- Accessibility test records showing which assistive technologies were run and which steps failed.
- A model and configuration change log, including default thresholds and which settings your team is permitted to change.
- A documented rollback or disable procedure, and the role authorized to invoke it.
- Confirmation of whether the audit covers your configuration or only a generic version of the product.
Human review and override
The controls below are engineering practices. They are drawn from the statute’s deployment requirements and DOJ’s call to examine hiring technologies before and during use, but neither source lists them item by item.
- Reviewer inputs. Define what evidence the reviewer sees alongside the output, and whether they can see the features behind a score or only the score itself.
- Override authority. State whether a reviewer can override an output, and record every override with a reason code. Overrides that never occur are also a signal worth checking.
- Error escalation. Give reviewers and candidates one route for reporting a suspected error, and set a deadline for investigating it.
- Accommodation handoff. Specify how an accommodation request reaches the service owner without passing through the scoring step.
- Pause authority. Name who can stop the tool for a job family or location, and what happens to candidates already in the queue.
- Reviewer training. Teach reviewers the tool’s known limitations and the job-relevance basis for each assessed skill.
Renewed evaluation after changes
Re-evaluate when anything that shapes the output or the decision changes. Whether a particular change requires a new bias audit under Local Law 144 is a legal question for counsel. The statute’s test is an audit no more than one year old that applies to the tool being used.
| Trigger | What to re-check |
|---|---|
| Model or vendor version change | Whether the audit and accessibility testing still cover this version; ask counsel whether a new audit is needed |
| Job criteria or assessed skills change | The mapping from each assessment step to an essential function; the notice text |
| Score thresholds or cutoffs change | Selection outcomes since the change; rollback authority and its owner |
| Training data or input features change | Label sources, proxy review, and the justification for any new input |
| New role, location, or applicant population | Whether the tool is now covered in a city or population the audit did not describe |
| Accommodation requests or complaints cluster at one step | That step’s barrier analysis and the alternative process for it |
Comparing tools on five axes
Use the same five axes to compare candidate tools, so procurement decisions rest on the same evidence across vendors.
Quick Recap
| Axis | What to compare | Evidence that settles it |
|---|---|---|
| Job relevance | Whether assessed skills map to the role, and whether your team can explain the construct | The essential-function mapping from the disability access section, and label review records |
| Outcome evidence | Audit date, scope, and whether the audited version matches the one you will deploy | The audit summary and its distribution date, checked against the NYC steps above |
| Accessibility | Whether qualified applicants can complete the process with assistive technology or an accommodation | Your own assistive technology test logs and the accommodation request test |
| Transparency | Whether you can describe the tool’s use, the qualifications it assesses, its data types and sources, and its retention practices | The written data inventory and the notice text |
| Operational control | Whether humans can inspect and challenge results, handle accommodations, investigate complaints, and roll back changes | The override and escalation records, the change log, and the named rollback owner |
Where this checklist stops
- Jurisdiction. The legal detail here is specific to New York City. Other state, local, and non-U.S. rules are outside this checklist and need their own review.
- Currency. City code pages can lag newer rules, and enforcement practice can change. Check the current text of § 20-871 and DCWP’s current guidance, and have qualified counsel confirm how the law applies to your facts.
- Federal guidance status. DOJ states that its AI guidance is informal and nonbinding. Treat it as a description of how DOJ reads the ADA, not as a rule. Federal agencies have revised AI-related web guidance in recent years, so confirm that the cited pages are still posted before you rely on them in a procurement file.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
