AI agents fail in the data room, so deal work stays with people
In GDPx, a new agent test by the research firm SemiAnalysis published on 2 November, the best model scores 12 per cent; no model fully solved a finance or legal task.
Measured in real working environments rather than on single questions, AI agents still fall well short of what is promised. The tasks that cost banks and law firms the most were exactly those no model completed.
What happened
GDPx tests agents across eleven domains, including finance, law, chip design, biology, cyber-security and software engineering. Unlike earlier tests that put all information in the prompt, the models must work in interactive environments with time limits of up to eight hours. For the finance section SemiAnalysis and its partner Living Context rebuilt a virtual data room from a real transaction, with about 4,000 anonymised files and 24 tasks seen from the parties' perspectives. Under strict grading, counting only fully solved tasks, every model scored zero in finance and law. On partial credit, Claude Opus 5.5 averaged 44 per cent on a legal task, reviewing a draft agreement for a funding round, and GPT-6 Astra 25 per cent. Overall Opus 5.5 scored 12.0 per cent, Gemini 4 Carbon 10.9 and GPT-6 Astra 9.6. The authors note that safety filters held back some models on cyber and biology tasks. Assessment: the test is new and has not been checked by third parties. SemiAnalysis also criticises other tests; by its own review almost half the tasks in a well-known coding benchmark are faulty.
What follows
The finding cuts across the reasons companies currently give for cutting jobs. SUSE, for example, writes that AI has greatly reduced the need for some roles. GDPx shows that agents complete complex client work only in part. Assessment: for banks, advisers and law firms, agents can take on parts of the work, such as sifting documents or marking up drafts, but people must check the result. Those who shrink teams on the strength of marketing claims carry the risk of incomplete work. Yet partial scores on legal tasks were already approaching half; the gap may narrow with the next generation of models.
What changes in practice
- Banks
- In a rebuilt data room from a real transaction no agent fully solved a task; due-diligence work still needs human review.
- Law firms
- Reviewing a funding agreement, the best model averaged 44 per cent of the points: useful partial work, not a finished result.
- Managers
- Anyone justifying job cuts with AI should know the gap between partial and full completion; GDPx measures exactly that.
What to watch
- Whether other labs or universities reproduce the GDPx results
- Progress of new models on the finance and legal tasks
- How banks and law firms govern the use of agents in transactions
Sources
- SemiAnalysis: GDPx, Measuring Frontier Model Capabilities Across the Most Valuable Domains
- Winzheng: GDPx Benchmark Debuts, Strongest Model Scores Only 12%, Finance and Legal Tasks All Zero
- Xenospectrum: SemiAnalysis Launches GDPx, Claude Opus 5.5 Leads with 12.0
- AI Weekly: SemiAnalysis launches GDPx, best frontier model hits 12%