PALO ALTO, Calif., July 09, 2026 (GLOBE NEWSWIRE) — DevRev, maker of the agentic AI product Computer, today announced Enterprise-Bench, an open, vendor-neutral benchmark for evaluating whether AI agents can operate in production enterprise environments. The benchmark measures the conditions that make enterprise AI difficult, including fragmented data, siloed systems, and permission boundaries.
It was developed with Laude Institute, whose Harbor harness executed the evaluation, and validated by Alexandros Dimakis, UC Berkeley professor, co-founder of Bespoke Labs, and DevRev board member. The framework, dataset, methodology, results, and full traces are public, and any vendor, customer, or researcher can run the benchmark and submit to the leaderboard.
As enterprises accelerate AI deployments, they still lack a neutral standard for evaluating whether AI agents can operate in production. That gap matters at a time when vendor claims are everywhere, but independent evidence is not. According to McKinsey’s 2025 State of AI survey, nearly two-thirds of organizations have not yet begun scaling AI across the enterprise and in any given business function, no more than 10% are scaling AI agents.
The benchmark draws on a historical parallel. In the early 1990s, database vendors made competing performance claims with no neutral basis for comparison. The Transaction Processing Performance Council (TPC) introduced common, auditable standards that shifted the conversation from marketing to evidence. DevRev believes enterprise AI has reached the same inflection point and is openly releasing the dataset and methodology behind Enterprise-Bench so that any vendor can be held to the same standard. Unlike benchmarks that focus only on task complexity, Enterprise-Bench is designed to reflect organizational complexity through an evaluation environment that mimics how enterprise systems are actually accessed in production.
“What Enterprise-Bench adds to the benchmark community is a way to measure organizational complexity, not just task complexity,” said Ahmed Bashir, CTO at DevRev. “In real enterprises, AI systems have to work across fragmented data, siloed systems, and permission boundaries. By building those constraints into the evaluation framework, Enterprise-Bench reflects the conditions these systems actually face in production.”
The methodology behind the results
The framework progresses in the same spirit as levels of autonomous driving, describing a progression toward greater independence at each stage. Enterprise-Bench’s L1-L2 benchmark, spanning factual retrieval and complex multi-source queries, is the first release, with L3 and L4 planned for later this year and into next.
Laude Institute, whose researchers are affiliated with UC Berkeley and Stanford, developed the Harbor evaluation harness used to run and verify the benchmark. DevRev worked with the Institute and Professor Alexandros Dimakis, a researcher in LLM training and co-founder of Bespoke Labs, throughout development, and published results on Harbor Hub alongside Harbor’s independent leaderboard.
“Most agent benchmarks today test consumer-style tasks, like booking a flight, where the data is clean and the end state is binary,” said Professor Alexandros Dimakis of UC Berkeley. “What’s different here is that the difficulty scales with the data itself. The correct answer stays fixed while the surrounding noise approaches a more realistic enterprise volume. That is a much harder and more useful test of whether an agent’s accuracy actually holds up. I haven’t seen this approach applied to enterprise data before, and I think it’s a meaningful contribution to how the field measures production readiness.”
The results: same model, very different outcomes
Computer, by DevRev, was run head-to-head against Claude Code on identical L1-L2 tasks, using the same data, the same model, and the same independent LLM judge. The results show that performance differences come from system architecture and data retrieval design, not just model capability.
- Accuracy: On the XL enterprise dataset DevRev was 48% more accurate. Computer, by DevRev achieved 94.3% accuracy compared with 63.6% for Claude Code.
- Token efficiency: Computer used 4.4x fewer tokens per correct response: ~5,598 versus ~24,461 for Claude Code.
- Cost at scale: As the dataset grew, Computer’s total token usage stayed roughly flat while Claude Code’s climbed 29%: ~54M versus ~92M on the largest runs.
- Model independence: The same Opus 4.8 model family was used, showing that the difference is not the model but the data retrieval architecture underneath it.
The results illustrate the benchmark’s central finding: enterprise AI systems require a data context layer in order to implement AI across the complex systems commonly found in any enterprise. Computer, by DevRev, is designed for that layer. This benchmark highlights the challenges organizations will face in implementing AI across enterprise systems and complex tasks using today’s open protocols. It is intended to show how different vendors and AI labs stack up on those tasks over time.
Why this benchmark is different
The benchmark evaluates on three axes that existing benchmarks ignore: precision (correct answer, verifiable source, auditable path); efficiency (does cost scale with question complexity or data volume); and safety (permission boundaries respected, every action traceable). Results are verified by an independent LLM judge against published criteria, and all traces must be submitted alongside scores.
DevRev is publishing the full dataset on Harbor Hub, along with the benchmark queries, judging criteria, evaluation harness, and its own results and traces. Competing vendors, customers, and independent researchers are invited to run the benchmark and submit results to the public leaderboard.
“Benchmarks that are not public are not benchmarks. They are marketing,” said Jeff Smith, Office of the CTO at DevRev, who led the benchmark initiative. “We are publishing everything, including the data, the methodology, our own traces. That is how trust gets built in this industry, and that is how, as vendors, we build better systems.”
Future benchmark iterations will evaluate different models, agent providers, and introduce new tasks that represent even higher levels of autonomy. The dataset and query set will be extended over time through a public contribution process. Enterprise-Bench will evolve as other organizations participate and as enterprise AI use cases mature toward L3 and L4, which are strategic orchestration and autonomous operation.
Resources
Enterprise-Bench, including the full dataset, evaluation harness, queries, judging criteria, and initial results, is publicly available here.
About DevRev
DevRev is an enterprise AI company. Its product, Computer, builds a knowledge graph of each customer’s business, synced from their existing systems with permissions intact, and uses that graph to power enterprise search, employee service desks, and self service customer support, including voice. Founded in 2020 and backed by Khosla Ventures and Mayfield, DevRev is led by co-founder and CEO Dheeraj Pandey, former co-founder and CEO of Nutanix and an Adobe board member, alongside co-founder Manoj Agarwal, former SVP of Engineering at Nutanix. Headquartered in Palo Alto, DevRev operates globally across eight offices.
Media relations contact:
DevRevpr@watersagency.com
Nguồn: GLOBENEWSWIRE – Đơn vị phát hành hoàn toàn chịu trách nhiệm về nội dung thông báo này.
