Typed Decision Bench · tasks · Workflow control

`cfpb-complaint-product-queue`

support inbox triage · finance · 199 items · primary question type choice. Source liri-uzh/cfpb-complaints-mini (CC0-1.0 (CFPB public domain)); labels are outcome labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
Jev	83.5	79.4%	199 / 199
System One Scorer	82.6	77.9%	199 / 199
decider-2b	79.7	72.9%	199 / 199
OpenJev	78.9	76.9%	199 / 199
JevFish	78.8	71.4%	199 / 199
NanoJev	63.4	43.7%	199 / 199

Sample 1 of 6 · workflow-control:cfpb-5d3ac3f2cba6 · 179 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "complaint": "I am reaching out due to an ongoing issue I am having with Bank of America. Ive had a personal account, and I receive my direct deposit from my job directly to this account. However, when I tried to log in to my online banking I received and error message. When I called customer service, after spending almost 30 minutes being passed around the phone, they advised my account is being closed and they could not give me any information. The communication from the bank has not been clear and I have received conflicting information. I feel misled, taken advantage of and pushed to the side. This is unfair as the funds from my job are what I use to support my family. I tried explaining this to no avail.\n\nThe way Ive been unfairly treated by Bank of America is completely unacceptable. I received a call from one of their reps advising that someone …"
}

Question product_queue · choice · primary (ranked)

Read the `complaint` narrative. Choose the single financial product line the consumer is complaining about, i.e. the specialist queue that should own this ticket. Personal details are redacted as XXXX.

7 options

bank_account — Checking or savings account, or bank account services (deposits, overdrafts, holds, closures, ATM/debit issues)
credit_card — Credit card or prepaid card (charges, rewards, card fees, card fraud, card billing disputes)
credit_reporting — Credit reporting agencies, credit repair, or other consumer reports (incorrect report data, disputes with bureaus, identity-theft entries)
debt_collection — A third-party or first-party debt collector's conduct or a debt the consumer says is not theirs
money_transfer — Money transfer, virtual currency, or money services (wires, P2P apps, remittances, crypto, check cashing)
mortgage — Home mortgage loans (origination, servicing, escrow, modification, foreclosure)
student_loan — Federal or private student loans (servicing, repayment plans, forgiveness, collections on student debt)

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: bank_account

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`bank_account`	correct	100.0%	1.000	`bank_account` 100.0% `debt_collection` 0.0% `credit_card` 0.0% `credit_reporting` 0.0% `money_transfer` 0.0% `mortgage` 0.0%
System One Scorer	`bank_account`	correct	90.0%	0.994	`bank_account` 90.0% `debt_collection` 5.0% `money_transfer` 1.2% `mortgage` 1.2% `credit_card` 1.0% `student_loan` 0.9%
decider-2b	`bank_account`	correct	99.7%	1.000	`bank_account` 99.7% `credit_card` 0.1% `money_transfer` 0.1% `credit_reporting` 0.1% `debt_collection` 0.0% `mortgage` 0.0%
OpenJev	`bank_account`	correct	99.8%	1.000	`bank_account` 99.8% `credit_card` 0.2% `debt_collection` 0.0% `money_transfer` 0.0% `credit_reporting` 0.0% `student_loan` 0.0%
JevFish	`bank_account`	correct	99.8%	1.000	`bank_account` 99.8% `money_transfer` 0.2% `credit_card` 0.0% `mortgage` 0.0% `credit_reporting` 0.0% `student_loan` 0.0%
NanoJev	`bank_account`	correct	35.7%	0.717	`bank_account` 35.7% `debt_collection` 35.1% `money_transfer` 14.5% `credit_reporting` 7.6% `credit_card` 4.2% `student_loan` 2.0%

Sample 2 of 6 · workflow-control:cfpb-855d7037c1ba · 183 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "complaint": "XXXX XXXX XXXX XXXX XXXX XXXX XXXX, AZ XXXX SSN : XXXX XXXX : XX/XX/XXXX Dear Sir/Madam , I hope this letter finds you well. I am writing to bring to your attention serious concerns I have regarding the inaccurate and unverified information that has been adversely affecting my credit profile. I kindly request your assistance in resolving these matters and enforcing the relevant laws to rectify the violations that have had a profound impact on my financial well-being and personal life. \n\n1. Inaccurate Reporting of Late Payments : It has come to my attention that several creditors have reported inaccurate late payment information on my credit report, specifically : XXXX XXXX XXXX  : Account number : XXXX Please correct/update this inaccurate information on my credit report. To the best of my knowledge, this account has never been paid …"
}

Question product_queue · choice · primary (ranked)

7 options

bank_account — Checking or savings account, or bank account services (deposits, overdrafts, holds, closures, ATM/debit issues)
credit_card — Credit card or prepaid card (charges, rewards, card fees, card fraud, card billing disputes)
credit_reporting — Credit reporting agencies, credit repair, or other consumer reports (incorrect report data, disputes with bureaus, identity-theft entries)
debt_collection — A third-party or first-party debt collector's conduct or a debt the consumer says is not theirs
money_transfer — Money transfer, virtual currency, or money services (wires, P2P apps, remittances, crypto, check cashing)
mortgage — Home mortgage loans (origination, servicing, escrow, modification, foreclosure)
student_loan — Federal or private student loans (servicing, repayment plans, forgiveness, collections on student debt)

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: credit_reporting

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`credit_reporting`	correct	99.0%	1.000	`credit_reporting` 99.0% `credit_card` 1.0% `student_loan` 0.0% `mortgage` 0.0% `money_transfer` 0.0% `debt_collection` 0.0%
System One Scorer	`credit_reporting`	correct	67.2%	0.912	`credit_reporting` 67.2% `debt_collection` 25.9% `credit_card` 2.1% `mortgage` 1.6% `student_loan` 1.1% `money_transfer` 1.1%
decider-2b	`credit_reporting`	correct	99.0%	1.000	`credit_reporting` 99.0% `credit_card` 0.7% `debt_collection` 0.1% `mortgage` 0.1% `student_loan` 0.0% `bank_account` 0.0%
OpenJev	`credit_reporting`	correct	100.0%	1.000	`credit_reporting` 100.0% `credit_card` 0.0% `debt_collection` 0.0% `bank_account` 0.0% `mortgage` 0.0% `money_transfer` 0.0%
JevFish	`credit_reporting`	correct	99.4%	1.000	`credit_reporting` 99.4% `credit_card` 0.5% `debt_collection` 0.1% `bank_account` 0.0% `mortgage` 0.0% `student_loan` 0.0%
NanoJev	`credit_reporting`	correct	28.9%	0.681	`credit_reporting` 28.9% `debt_collection` 27.2% `credit_card` 19.4% `money_transfer` 11.7% `bank_account` 7.2% `student_loan` 4.4%

Sample 3 of 6 · workflow-control:cfpb-96faf572a428 · 164 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "complaint": "Why is it that, despite my diligent efforts to bring these inaccuracies to your attention, your bureau continues to report the same late payments, casting a shadow over my creditworthiness? I was the victim and Im not sure why the CRA is not complying with the law. Im very confused why this one seems to ignore my requests. I am left questioning the sincerity of your commitment to aiding consumers in maintaining accurate credit profiles. The fact that the inaccurate late payments persist suggests either a monumental failure in your investigative processes or a deliberate negligence that borders on betrayal. \nXXXX XXXX XXXX Those payments were made on time. I am requesting that the late payment markings be updated to reflect my accurate payment history. \nI demand an immediate and thorough investigation into this matter, along with a …"
}

Question product_queue · choice · primary (ranked)

7 options

bank_account — Checking or savings account, or bank account services (deposits, overdrafts, holds, closures, ATM/debit issues)
credit_card — Credit card or prepaid card (charges, rewards, card fees, card fraud, card billing disputes)
credit_reporting — Credit reporting agencies, credit repair, or other consumer reports (incorrect report data, disputes with bureaus, identity-theft entries)
debt_collection — A third-party or first-party debt collector's conduct or a debt the consumer says is not theirs
money_transfer — Money transfer, virtual currency, or money services (wires, P2P apps, remittances, crypto, check cashing)
mortgage — Home mortgage loans (origination, servicing, escrow, modification, foreclosure)
student_loan — Federal or private student loans (servicing, repayment plans, forgiveness, collections on student debt)

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: debt_collection

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`credit_reporting`	wrong	0.0%	0.000	`credit_reporting` 100.0% `credit_card` 0.0% `student_loan` 0.0% `mortgage` 0.0% `money_transfer` 0.0% `debt_collection` 0.0%
System One Scorer	`credit_reporting`	wrong	6.2%	0.172	`credit_reporting` 88.0% `debt_collection` 6.2% `credit_card` 1.9% `mortgage` 1.8% `money_transfer` 0.7% `student_loan` 0.7%
decider-2b	`credit_reporting`	wrong	0.1%	0.006	`credit_reporting` 99.4% `credit_card` 0.2% `mortgage` 0.1% `debt_collection` 0.1% `student_loan` 0.1% `bank_account` 0.0%
OpenJev	`credit_reporting`	wrong	0.0%	0.000	`credit_reporting` 100.0% `credit_card` 0.0% `bank_account` 0.0% `debt_collection` 0.0% `money_transfer` 0.0% `student_loan` 0.0%
JevFish	`credit_reporting`	wrong	0.0%	0.006	`credit_reporting` 99.4% `credit_card` 0.5% `mortgage` 0.0% `student_loan` 0.0% `bank_account` 0.0% `debt_collection` 0.0%
NanoJev	`debt_collection`	correct	28.7%	0.676	`debt_collection` 28.7% `credit_reporting` 27.4% `credit_card` 21.4% `money_transfer` 11.7% `bank_account` 4.9% `student_loan` 4.8%

Sample 4 of 6 · workflow-control:cfpb-9863f619d389 · 62 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "complaint": "Commencing on or around XX/XX/, I fell victim to two multi-layered scam operations run by XXXX XX/XX/XXXXXXXX ) which involved me making deposits for a total amount of XXXX USD from my JPMorgan Chase bank account to fraudulent investment firm ( s )."
}

Question product_queue · choice · primary (ranked)

7 options

bank_account — Checking or savings account, or bank account services (deposits, overdrafts, holds, closures, ATM/debit issues)
credit_card — Credit card or prepaid card (charges, rewards, card fees, card fraud, card billing disputes)
credit_reporting — Credit reporting agencies, credit repair, or other consumer reports (incorrect report data, disputes with bureaus, identity-theft entries)
debt_collection — A third-party or first-party debt collector's conduct or a debt the consumer says is not theirs
money_transfer — Money transfer, virtual currency, or money services (wires, P2P apps, remittances, crypto, check cashing)
mortgage — Home mortgage loans (origination, servicing, escrow, modification, foreclosure)
student_loan — Federal or private student loans (servicing, repayment plans, forgiveness, collections on student debt)

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: money_transfer

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`money_transfer`	correct	55.0%	0.797	`money_transfer` 55.0% `bank_account` 45.0% `credit_card` 0.0% `debt_collection` 0.0% `student_loan` 0.0% `credit_reporting` 0.0%
System One Scorer	`money_transfer`	correct	30.8%	0.693	`money_transfer` 30.8% `debt_collection` 27.5% `bank_account` 21.9% `credit_card` 7.2% `mortgage` 4.5% `credit_reporting` 4.5%
decider-2b	`bank_account`	wrong	3.3%	0.074	`bank_account` 95.8% `money_transfer` 3.3% `credit_card` 0.4% `credit_reporting` 0.2% `debt_collection` 0.1% `mortgage` 0.1%
OpenJev	`money_transfer`	correct	61.7%	0.855	`money_transfer` 61.7% `bank_account` 37.9% `credit_card` 0.4% `mortgage` 0.0% `debt_collection` 0.0% `student_loan` 0.0%
JevFish	`bank_account`	wrong	4.5%	0.092	`bank_account` 95.2% `money_transfer` 4.5% `credit_card` 0.2% `mortgage` 0.0% `credit_reporting` 0.0% `student_loan` 0.0%
NanoJev	`money_transfer`	correct	32.8%	0.719	`money_transfer` 32.8% `debt_collection` 21.3% `bank_account` 18.5% `credit_reporting` 14.4% `credit_card` 9.7% `student_loan` 2.2%

Sample 5 of 6 · workflow-control:cfpb-d4eb63f50372 · 158 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "complaint": "Submitted completed borrower assistance form, and requested documentation to company to receive assistance in avoiding foreclosure as I was affected by XXXX. The company has continued their efforts to foreclose ignoring any communication from me. Ive emailed them everything requested to the email provided by the mortgage company representative via chat, as well as the fax number provided. To date, I continue receiving court foreclosure notices and filings- but absolutely NO communication regarding assistance. \n\n\nThey have ignored my continued repeated efforts to avoid foreclosure, and instead fabricate narratives in order to foreclose on my home. All my attempts have gone ignored. I have received no workout plan, trial payment plan, modification etc. \n\nConsumers have rights, and they have violated those rights for their own financial …"
}

Question product_queue · choice · primary (ranked)

7 options

bank_account — Checking or savings account, or bank account services (deposits, overdrafts, holds, closures, ATM/debit issues)
credit_card — Credit card or prepaid card (charges, rewards, card fees, card fraud, card billing disputes)
credit_reporting — Credit reporting agencies, credit repair, or other consumer reports (incorrect report data, disputes with bureaus, identity-theft entries)
debt_collection — A third-party or first-party debt collector's conduct or a debt the consumer says is not theirs
money_transfer — Money transfer, virtual currency, or money services (wires, P2P apps, remittances, crypto, check cashing)
mortgage — Home mortgage loans (origination, servicing, escrow, modification, foreclosure)
student_loan — Federal or private student loans (servicing, repayment plans, forgiveness, collections on student debt)

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: mortgage

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`mortgage`	correct	100.0%	1.000	`mortgage` 100.0% `debt_collection` 0.0% `credit_reporting` 0.0% `credit_card` 0.0% `money_transfer` 0.0% `bank_account` 0.0%
System One Scorer	`mortgage`	correct	90.7%	0.994	`mortgage` 90.7% `debt_collection` 5.7% `student_loan` 1.1% `money_transfer` 0.9% `bank_account` 0.6% `credit_reporting` 0.5%
decider-2b	`mortgage`	correct	99.9%	1.000	`mortgage` 99.9% `student_loan` 0.0% `credit_reporting` 0.0% `credit_card` 0.0% `debt_collection` 0.0% `bank_account` 0.0%
OpenJev	`mortgage`	correct	99.8%	1.000	`mortgage` 99.8% `student_loan` 0.1% `credit_card` 0.0% `bank_account` 0.0% `debt_collection` 0.0% `money_transfer` 0.0%
JevFish	`mortgage`	correct	100.0%	1.000	`mortgage` 100.0% `credit_reporting` 0.0% `student_loan` 0.0% `debt_collection` 0.0% `bank_account` 0.0% `credit_card` 0.0%
NanoJev	`mortgage`	correct	45.9%	0.776	`mortgage` 45.9% `debt_collection` 38.6% `credit_reporting` 4.7% `money_transfer` 3.7% `bank_account` 3.2% `student_loan` 2.2%

Sample 6 of 6 · workflow-control:cfpb-ee48fb465bc8 · 89 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "complaint": "When you go to midland credit management website they still have listed that I owe XXXX on this debt.when in reality as of this date XX/XX/2023, it is XXXXi have for the past XXXX months trying to get them to correct this on their site.i have contacted them XXXX  times about this but get the runaround.this is providing false information and needs to be corrected."
}

Question product_queue · choice · primary (ranked)

7 options

bank_account — Checking or savings account, or bank account services (deposits, overdrafts, holds, closures, ATM/debit issues)
credit_card — Credit card or prepaid card (charges, rewards, card fees, card fraud, card billing disputes)
credit_reporting — Credit reporting agencies, credit repair, or other consumer reports (incorrect report data, disputes with bureaus, identity-theft entries)
debt_collection — A third-party or first-party debt collector's conduct or a debt the consumer says is not theirs
money_transfer — Money transfer, virtual currency, or money services (wires, P2P apps, remittances, crypto, check cashing)
mortgage — Home mortgage loans (origination, servicing, escrow, modification, foreclosure)
student_loan — Federal or private student loans (servicing, repayment plans, forgiveness, collections on student debt)

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: debt_collection

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`debt_collection`	correct	93.0%	0.995	`debt_collection` 93.0% `credit_reporting` 7.0% `bank_account` 0.0% `credit_card` 0.0% `money_transfer` 0.0% `mortgage` 0.0%
System One Scorer	`debt_collection`	correct	81.1%	0.972	`debt_collection` 81.1% `credit_reporting` 14.3% `student_loan` 1.4% `mortgage` 1.0% `money_transfer` 0.9% `credit_card` 0.7%
decider-2b	`debt_collection`	correct	84.9%	0.984	`debt_collection` 84.9% `credit_card` 8.5% `credit_reporting` 3.6% `mortgage` 1.4% `student_loan` 1.2% `bank_account` 0.2%
OpenJev	`debt_collection`	correct	99.9%	1.000	`debt_collection` 99.9% `credit_card` 0.1% `credit_reporting` 0.0% `bank_account` 0.0% `money_transfer` 0.0% `mortgage` 0.0%
JevFish	`debt_collection`	correct	70.4%	0.936	`debt_collection` 70.4% `credit_card` 15.8% `credit_reporting` 12.5% `mortgage` 0.6% `student_loan` 0.5% `bank_account` 0.1%
NanoJev	`credit_reporting`	wrong	31.1%	0.669	`credit_reporting` 36.2% `debt_collection` 31.1% `credit_card` 22.7% `money_transfer` 4.2% `student_loan` 2.9% `bank_account` 2.3%