Agents in Financial Services

01Adoption has outrun understanding

Financial services adopted machine learning earlier than most sectors and is now adopting language-model systems faster than it can explain them. The most recent supervisory survey puts numbers on both halves of that sentence.

The 2024 Bank of England and FCA survey

In November 2024, the Bank of England and the Financial Conduct Authority published their third survey of AI in UK financial services, covering 118 firms across banking, insurance, investment, and market infrastructure. 75% of firms were already using AI and a further 10% planned to within three years, up from 58% using machine learning in the 2022 survey. Adoption was highest in insurance, at 95% of firms, and international banks, at 94%. Most adopters ran few systems: 56% reported ten or fewer use cases, and 10% more than fifty.

Understanding lags

The same survey asked firms how well they understood the AI they had deployed. 34% reported complete understanding and 46% partial understanding. The top perceived benefits were data and analytical insight, anti-money-laundering and fraud, and cybersecurity. The top perceived risk was cybersecurity, with third-party dependencies, model complexity, and embedded models named as the risks growing fastest. A firm that runs a system it partly understands, sourced from a third party, and embedded in a vendor product has three of the survey's top risks in one deployment.

02The customer-facing surface

The deployment that reaches the public first is the chatbot. It is also the one where a wrong answer has a named customer, a written record, and, as one tribunal has now confirmed, a liable firm.

How many people the chatbots reach

The US Consumer Financial Protection Bureau's June 2023 issue spotlight found that all ten of the largest US commercial banks had deployed chatbots, and that by 2022 about 37% of the US population, more than 98 million people, had interacted with a bank's chatbot. The figure was projected to reach 110.9 million by 2026. The CFPB's concerns were specific: chatbots that give inaccurate information, that fail to recognise a dispute, and that block access to a human, any of which can amount to a violation of consumer financial law.

Moffatt v. Air Canada

In November 2022, Jake Moffatt asked Air Canada's website chatbot about bereavement fares after a grandmother's death. The chatbot said a reduced fare could be applied for retroactively within 90 days. The airline's actual policy did not allow retroactive applications, and Air Canada refused the refund. In February 2024, the British Columbia Civil Resolution Tribunal found Air Canada liable for negligent misrepresentation. The airline had argued that the chatbot was a separate legal entity responsible for its own actions; the tribunal called that a remarkable submission and rejected it. Moffatt was awarded $650.88 in damages, the difference between the fare paid and the bereavement fare, plus interest and fees.

Air Canada argues it cannot be held liable for information provided by one of its agents, servants, or representatives, including a chatbot. It does not explain why it believes that is the case.

Moffatt v. Air Canada, 2024 BCCRT 149

The sum is small. The principle is large: a firm is responsible for what its chatbot says, the chatbot's statement is treated like any other representation on the firm's website, and the customer is entitled to rely on it. For a bank, the equivalent misstatements are a fee, a limit, a rate, or an eligibility rule, each of which is also a conduct matter.

03What the models get wrong

Chatbots fail in the way the underlying models fail. Two benchmark studies, one on financial filings and one on legal questions, measured those failure rates on questions with checkable answers.

FinanceBench

FinanceBench, from Islam and colleagues, is a set of 10,231 questions about publicly traded companies, each with an answer and the supporting passage from the company's filings. The questions were written to be clear-cut, as a minimum standard. On a manually reviewed sample of 150 cases across 16 model configurations, GPT-4-Turbo used with a retrieval system incorrectly answered or refused to answer 81% of questions. All models examined showed weaknesses including hallucination. These are the questions a support or research assistant in a bank would be asked about a client's own documents.

Legal hallucinations

Dahl, Magesh, Suzgun, and Ho, in the Journal of Legal Analysis, asked public models specific, verifiable questions about random US federal court cases. Hallucination rates ran from 58% with ChatGPT 4 to 88% with Llama 2. Two further findings weigh more for a regulated deployment: the models often failed to correct an incorrect legal premise built into the question, and they could not reliably predict when they were hallucinating, so their own confidence was not a usable signal of error.

What the two studies mean for escalation

Most chatbot designs route a conversation to a person when the model's confidence drops. Both studies undercut that design: the models answer confidently and wrongly, accept false premises, and cannot predict their own errors. Escalation should be triggered by the case, not by the model's self-report.

04The rulebook is already written

Firms sometimes treat language-model systems as a new category awaiting regulation. In the two largest jurisdictions the relevant obligations already exist, and both reach the systems described in this report.

The EU AI Act

Annex III of the EU AI Act lists the uses classified as high-risk. Point 5(b) covers AI systems used to evaluate the creditworthiness of natural persons or establish their credit score, with an exception for fraud detection. Point 5(c) covers risk assessment and pricing in life and health insurance. High-risk status brings conformity assessment, technical documentation, logging, human oversight, and accuracy and robustness requirements, and deployers of these systems must carry out a fundamental rights impact assessment before use. A language-model component that informs a lending or pricing decision is inside that perimeter.

SR 11-7

In the United States, the Federal Reserve and the OCC's Supervisory Guidance on Model Risk Management, SR 11-7, has applied since 2011. It requires that models be validated by parties independent of their developers, that validation cover conceptual soundness, ongoing monitoring, and outcomes analysis, and that the firm keep an inventory and documentation sufficient for a third party to understand how each model works and where it is used. Nothing in the guidance limits it to statistical models. A language-model system that produces outputs a firm relies on for a business decision is a model under SR 11-7's definition.

05Controls a risk function can sign off

None of the evidence argues against deploying these systems. It argues for treating them as what the rules already say they are: models, with validated inputs, documented behaviour, and monitored outputs, whose customer-facing statements bind the firm.

A checklist for the first customer-facing deployment

1. Inventory and validation. The system is listed as a model, with an owner, a validator independent of the build team, and a monitoring plan.
2. Grounded statements. Any fee, rate, limit, eligibility rule, or deadline is quoted from a retrievable policy source, or the system says it cannot answer.
3. Case-driven escalation. Monetary actions, unverifiable customer claims, complaint language, and ungrounded policy questions route to a person regardless of confidence.
4. Disclosure and access. The customer is told they are speaking to a system, and a route to a person is available at every turn.
5. Own-document error rate. Accuracy is measured on the firm's own product documents and filings, with method and sample size recorded.
6. Regime mapping. Each use is mapped to Annex III, SR 11-7, and consumer law before launch, and the mapping is kept with the model documentation.
7. Incident record. Every misstatement found in production is logged with the conversation, the source that should have been quoted, and the fix.

The surveys and studies above cover particular firms, models, and question sets, and the two benchmarks measure earlier models than a firm would deploy today. A firm's own error rate, on its own documents, is the number its supervisor will ask for.

Sources

1. Bank of England and Financial Conduct Authority (2024). Artificial intelligence in UK financial services – 2024. 21 November 2024.
2. Consumer Financial Protection Bureau (2023). Chatbots in consumer finance. Issue spotlight, June 2023.
3. Moffatt v. Air Canada, 2024 BCCRT 149 (British Columbia Civil Resolution Tribunal, 14 February 2024).
4. Islam, P., Kannappan, A., Kiela, D., Qian, R., Scherrer, N., Vidgen, B. (2023). FinanceBench: A New Benchmark for Financial Question Answering. arXiv:2311.11944.
5. Dahl, M., Magesh, V., Suzgun, M., Ho, D. E. (2024). Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. Journal of Legal Analysis 16(1), 64–93.
6. European Union (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act), Annex III.
7. Board of Governors of the Federal Reserve System and OCC (2011). Supervisory Guidance on Model Risk Management, SR 11-7 / OCC 2011-12.