Goodbye Ratings, Hello Benchmarks: If Legal Tech Can Benchmark Attorneys, Shouldn’t Attorneys Be Able to Ethically Benchmark Back?
Attorney ratings are so 2010. Peer review sites and popularity contests like Martindale-Hubbell, Superlawyers or Avvo had their moment. But in an AI era, benchmarks are the new currency of legal marketing. In the past year, AI legal tech companies, law librarians, and independent organizations have released benchmark reports though that’s been mostly a legal tech story because the reports evaluated AI tools, and not lawyers.
Until now. Last week, Crosby.AI — an AI-native law firm, not a tech vendor — released the Redline Bench, a benchmark it designed itself to claim that its attorneys outperform AI in contract negotiations. When a law firm starts benchmarking itself and publishing the results as marketing, bar ethics rules on advertising, comparative claims, verification potentially kick in. So the legal profession needs to understand what benchmarks are, how they work, and whether existing ethics rules are equipped to handle them.
Benchmarks are structured evaluation datasets that measure how well an AI model performs on specific tasks. In the legal context, that’s harder than it sounds. Most legal work produces unstructured text like briefs, opinions, contracts that resists automated scoring. As a Stanford University study on benchmarking legal AI explains, legal tasks are often inherently subjective. A claim’s merits, a contract’s adequacy, a brief’s persuasiveness are all matters on which equally competent lawyers can reasonably disagree, which means there is no firm truth to measure a model’s output.
Accuracy notwithstanding, with hundreds of millions in venture capital flowing into legal AI, companies are under pressure to differentiate. As a result, benchmarks have become a primary battleground in the legal tech space. Harvey released its open-source Legal Agent Benchmark (LAB), 1,250 tasks across 24 practice areas structured to mirror actual law firm assignments, graded on an all-or-nothing basis because, as Harvey explains, a deal memo that catches eight of ten risks isn’t 80% useful but materially incomplete. Thomson Reuters built CoCoBench to evaluate its next-generation CoCounsel Legal, representing over 15,000 hours of practitioner and engineering work, measuring not just whether an AI produces an answer but whether the full sequence of work behind that answer meets what TR calls a “fiduciary-grade standard.”
So what does benchmarking legal tech products have to do with lawyer ratings? That depends on who’s doing the benchmarking.
When legal tech companies like Harvey or Thomson Reuters benchmark their own AI products, the ethics picture is relatively straightforward. They’re technology vendors, not law firms, and while they answer to the Federal Trade Commission (which endorses truthful comparative advertising), bar ethics rules don’t apply.
But when a hybrid AI-native firm like Crosby.AI enters the picture, the lines blur. Crosby, together with AI research company micro1, built the RedlineBench, a multi-turn simulated SaaS contract negotiation in which AI models were tested against rubrics authored by Crosby’s own attorneys. Those same attorneys set the standards and implicitly defined what “good” looks like. The actual scoring was done by a panel of three LLM judges evaluating model outputs against the attorney-authored rubrics. The benchmark showed that current AI models fall meaningfully short of redlining by Crosby attorneys.
But here’s the real question. When Crosby claims that its benchmark study shows that Crosby attorneys outperform AI in contract negotiations, is it acting as a legal tech company doing marketing or as a law firm making comparative advertising claims subject to Rule 7.1?
That difference matters under existing bar rules because performance assessments and comparative statements in attorney advertising are permissible only when substantiated by verifiable, objective data. New York Opinion 1005 said “unsurpassed litigation skills” violates Rule 7.1 because it “compares the skills of the lawyer with others without factual support” and went further, holding that even posting a disclaimer that “prior results do not guarantee a similar outcome” will not cure the ethical problem. Texas prohibits statements like “we are the toughest lawyers in town” or “we are the best law firm in Texas if you want a large recovery,” and requires that even factual claims like settlement records or damage awards be accompanied by prominent qualifications to avoid creating unjustified expectations. D.C. Opinion 249 expressed concern over a lawyer’s ad promising that “we can help you when others can’t,” calling it the kind of comparative claim that is “incapable of substantiation.”
Benchmark-based claims raise similar issues under existing rules. When Crosby.AI says its attorneys outperform AI in contract negotiations, that’s a comparative statement that runs afoul of Rule 7.1 analysis as much as a phrase as the “toughest lawyers in town.” The fact that it’s framed as a benchmark result rather than a superlative doesn’t automatically save it because as the Stanford study acknowledges, the assessments are inherently subjective. In jurisdictions like North Carolina and New Jersey, self-laudatory awards or comparative accolades must not only disclose the methodology behind the claim but also reflect “objective, verifiable, and consistently applied factors.” Objectivity is lacking when a law firm gives its lawyers top ranking on a test it created and graded itself.
On the other hand, Crosby’s self-administered benchmark may not be as problematic as it seems. Crosby has been transparent about its methodology, disclosing who created the benchmark and how. What’s more, Crosby’s target clients are sophisticated corporations making high-stakes decisions about when to use human lawyers and when to use technology. Those clients have in-house counsel and IT professionals who can evaluate the methodology themselves. And let’s be real: what state bar is going to take on a biglaw-adjacent, VC-backed law firm over violations of consumer marketing rules?
As always, the risk falls on solos and smalls who may seek to mimic Crosby’s benchmark approach to show clients where the value of a human lawyer (or a human lawyer empowered by AI) surpasses a machine alone. In this context, benchmarks could prove helpful since less sophisticated consumer clients and small businesses actually need guidance to decide whether to hire a lawyer or use an AI tool far more than Crosby’s corporate clients do. Benchmarks are far from perfect, but if fully transparent, they offer more meaningful insight to consumers than opaque ratings and rankings. Regulators that reflexively apply decades-old advertising standards to human-versus-AI benchmark comparisons may wind up hurting the very clients they claim to protect in an increasingly AI-saturated market.
