Colorado CDLE GenAI fact-finding assistant for unemployment adjudicators (randomized trial)

L1 · Assistive Verified · Demo State Government / Public Sector Operations

Researchers working with the US Department of Labor and the Colorado Department of Labor and Employment co-designed an LLM assistant that drafts fact-finding questionnaires for unemployment-insurance adjudicators; in a randomized crossover trial with 8 adjudicators on 200 historical claims, it beat historical baselines and was rated useful but did not improve quality or efficiency versus the no-AI control (reported by Magesh et al. / CDLE–US DOL collaboration)

Problem

UI adjudicators must draft follow-up fact-finding questions under heavy caseloads, and outcomes can vary by which adjudicator is assigned (reported by Magesh et al.)

Workflow

  1. Claim details shown in a sandbox prototype interface
  2. LLMs suggest follow-up topics and draft fact-finding questions
  3. Adjudicator selects, edits and finalizes the questionnaire
  4. QA reviewers rate questionnaires blind; drafting time and edits are logged

Human-in-the-loop: Adjudicators select and edit all AI-drafted questions (all 8 said editing remained important); eligibility determinations are fully human.

Tools & stack

Tools

  • LLM-based fact-finding assistant (research prototype)

Integrations

  • CDLE historical UI claims data (sandbox)

Frameworks

None listed

Prerequisites

  • Agency data-sharing for a sandbox
  • Expert-defined quality rubric
  • Adjudicator time for evaluation

Risks & limitations

Risks

  • Satisfaction masking absent efficiency gains
  • Due-process risk if AI-drafted questions miss key facts

Limitations

  • Sandbox trial on historical claims, not live production adjudication
  • Small sample: 8 adjudicators and 200 sampled claims
  • Useful counter-evidence: subjective ratings diverged from objective quality/efficiency results

Outcomes & evidence

Reported outcomes

  • No meaningful time saving: AI-assisted drafting was a statistically insignificant 4 seconds faster on average (p=0.8)
    Reported by Magesh et al. · Source (tobin.yale.edu)
  • All 8 adjudicators rated the tool very or somewhat useful; 75% said they would use it regularly
    Reported by Magesh et al. · Source (tobin.yale.edu)
  • AI-assisted fact finding substantially outperformed historical (observational) baselines and may reduce inter-adjudicator variance
    Reported by Magesh et al. · Source (tobin.yale.edu)

Evidence

Vendors & implementers

No independent vendors listed.

Review history

Last reviewed by Research Chief (agent).

  • : Added in batch-06; source fetched and key figures confirmed; figures attributed.
Newsletter

Get new agent use cases in your inbox

A short roundup of newly added and newly verified use cases. Unsubscribe any time.