🔍 Read the full analysis: Z.ai’s Legal AI Benchmark: How Its Ranking Compares With Its Pass Rate on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s version-one Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models. The all-pass score means it met every rubric criterion on about one in five tasks, although Vals also reported a 92.5% pass rate for individual criteria. The ranking and scores are benchmark results, not proof that the model can safely complete legal work without professional review.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh of 81 models on Vals AI’s archived leaderboard for Harvey’s Legal Agent Benchmark, with a 19.17% all-criteria pass rate. The score means the model satisfied every rubric item on roughly one in five benchmark tasks; on individual criteria, Vals reported a much higher 92.5% pass rate.
The leaderboard result was recorded by Vals AI, which says it independently evaluated the model using Harvey’s grading protocol. The archived version-one leaderboard, last updated October 18, lists GLM Legal at $0.15 per test. Z.ai’s October 13 announcement described the result as 19.2%, an improvement of 12.5 percentage points over the base GLM-5.3-Flash model.
That improvement matches the figures on the same Vals table: the legal model scored 19.17%, compared with 6.67% for GLM-5.3-Flash. The leaderboard places the legal model below Gemini 4 Argon at 19.58% and above the base model, which ranked 33rd. The 12.5-point difference is a comparison between those two listed models, not a 12.5% relative increase.
The benchmark’s task score uses an all-or-nothing rule: a task counts as passed only when every required criterion is met. Vals reports that GLM Legal passed 92.5% of individual criteria, with a criterion counted only when both of its judges passed it. A high rate across separate criteria can coexist with a far lower task pass rate when one missed item makes the whole task incomplete.
Why the Two Scores Differ
The difference between 19.17% of tasks and 92.5% of criteria is central to interpreting the result. A legal work product can meet many individual requirements yet still fail a task if even one required detail is wrong or missing. The task score therefore speaks to end-to-end completion under the benchmark’s strict rubric, while the criteria score indicates performance across component checks.
For law firms and legal professionals, the distinction matters when judging claims about productivity. The leaderboard supports a comparison under a particular test protocol; it does not establish that the system can deliver complete legal work reliably in practice. The source material also reports that Z.ai presents its outputs as drafts for legal professionals, making review and verification part of the intended workflow.
The listed $0.15 per test provides a cost figure for that leaderboard run, not a complete procurement estimate. Z.ai separately lists API pricing of $0.37 per million input tokens and $1.25 per million output tokens. Those figures do not account for all costs of deploying an agent, connecting it to legal sources, checking citations, or handling client information.
Products Worth Considering
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Hazelnut Taupe
Bose QuietComfort Wireless Noise-Cancelling Over-Ear Headphones, Black
Apple Watch Series 12 GPS 46mm Smartwatch
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. It contains 1,250 tasks across 24 legal practice areas. Each task combines an instruction, matter materials and a requested work product, which is assessed against expert-written binary criteria. Harvey said the benchmark’s criteria number more than 75,000 and can cover details such as facts, citations, recommendations, deadlines, amounts and formatting.
Harvey’s benchmark documentation says a task is complete only if every criterion passes. The public leaderboard results use a private holdout set that mirrors the public task mix. Vals says its holdout includes 120 tasks, internet access is disabled, and two language-model judges grade each submission. The reported task score is the average of the pass rates from those judges.
Z.ai’s documentation identifies the model as glm-5.3-flashx-legal and describes it as optimized for legal workflows including research, contract review and drafting. Z.ai says it was built on GLM-5.3-Flash and received legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of modeled legal task environments. The company recommends using it as an agent connected to services providing laws, cases and company data.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its all-pass grading rule
Limits of the Benchmark Result
The result is specific to Vals AI’s version-one leaderboard and its 120-task private holdout, judges and grading setup. The source material also references a later Vals leaderboard version updated October 30, but does not provide its GLM Legal entry or enough details to establish whether the ranking changed. The seventh-place figure should therefore be read as the archived version-one result, not necessarily a current ranking.
Z.ai calls the result state-of-the-art among open-source models, but the source material says the company did not identify the comparison set and reports no published weights for the legal model. That makes the meaning of “open-source” in this specific comparison unclear. The independent leaderboard result supports the recorded score and ranking; it does not, on its own, verify the breadth of Z.ai’s comparison claim.
The source article also describes a four-task spot check in which the author says GLM Legal cited rulings that could not be found. That small check is not a benchmark and does not establish a general error rate. It also reports that OpenAI and Claude models in the spot check did not invent citations, while two Claude models cited a real ruling for the opposite of its holding. Those observations are limited to the author’s sample, not general findings about the systems.
It remains unclear from the cited material how Z.ai handles client data in deployments, which legal or privacy rules apply to particular uses, and what safeguards firms must configure. The source says Z.ai’s pages describe draft work for legal professionals and did not include a “not legal advice” notice. Neither the leaderboard nor the product description resolves how firms should verify citations or assign responsibility for errors.
What Firms Should Verify
The next useful benchmark check is whether GLM Legal appears on the later Vals leaderboard and, if so, whether the test version, score and rank are comparable with the archived result. A change in leaderboard version or evaluation method could affect comparisons; the available source material does not establish a later GLM result.
For organizations evaluating the model now, the reported result points to practical checks rather than a purchasing conclusion: test it on representative matters, review every citation and material conclusion, and clarify how client documents and connected legal data are processed. Z.ai recommends connecting the model to services for laws, cases and company data, but the source material does not detail the security or governance terms for those services.
Any assessment should keep task completion separate from partial rubric performance and from API price. The benchmark offers a defined comparison, while a firm’s decision also depends on the quality of its own review process, data controls and deployment costs. Further leaderboard updates or disclosure from Z.ai could clarify the model’s standing and the scope of its “open-source” claim.
Key Questions
What does GLM-5.3-FlashX-Legal’s 19.17% score mean?
It is the model’s all-criteria task pass rate on the archived Vals AI leaderboard. A task counts as passed only when every required rubric criterion passes, so the score means it completed all criteria on roughly one in five benchmark tasks.
How can the model have a 92.5% criterion pass rate but a 19.17% task pass rate?
The measures count different things. Criterion pass rate tracks individual rubric items; task pass rate counts a task only if every item is satisfied. A single missed criterion can make an otherwise strong task fail the all-pass test.
Was Z.ai’s claimed 12.5-point improvement confirmed by the leaderboard?
The archived Vals table lists 19.17% for GLM Legal and 6.67% for GLM-5.3-Flash, a difference of 12.50 percentage points. This verifies the difference between those entries on that leaderboard version, not a result across other evaluations.
Does seventh place mean the model is ready to handle legal work independently?
No. The ranking describes performance on a specific benchmark and holdout set. It does not establish that the model can safely perform legal work without professional review. Z.ai describes its outputs as drafts for legal professionals.
What is not yet clear about the ranking?
The supplied material does not give GLM Legal’s result on the later leaderboard version updated October 30. It also does not establish the scope of Z.ai’s open-source comparison claim or provide full deployment details for client-data handling and connected legal services.
Source: ThorstenMeyerAI.com
