Center for Digital Ag Announces New Generative AI Benchmarking Consortium

Recent advances in Generative AI capabilities for natural language understanding and text generation are driving the creation of automated question-answering services for production agriculture. Ensuring the reliability and accuracy of the advice from these systems is paramount because incorrect responses to crop production questions can lead to serious consequences for farm incomes, soil health, and the sustainability of family farming operations. As such, the Center for Digital Agriculture (CDA) at the University of Illinois Urbana-Champaign has announced a new consortium, AI AgriBench, to evaluate and build confidence in AI-driven question-answering systems supporting farmers, agronomists, industry, and the broader agriculture community.

The overarching goal of this effort is to provide farmers, policymakers, and the public with trustworthy mechanisms to evaluate and achieve high confidence in the new generation of AI-driven agronomy tools. We need to ensure these services deliver on their promise of valuable and accurate technical information to farmers and ag professionals worldwide.

“Trust and transparency in AI-driven agronomic advice are critical for the future of digital agriculture. By bringing together leading experts in AI and agronomy, this consortium will establish rigorous benchmarking standards to ensure these technologies deliver accurate, reliable, and actionable insights for farmers and agricultural professionals worldwide,” said John Reid, Executive Director, Center for Digital Agriculture, U. of I.

The new public consortium led by the CropWizard project within the Center for Digital Agriculture will create and oversee the benchmarking effort. In addition to the U. of I., the consortium’s founding members include Bayer Crop Science, KissanAI, and the Extension Foundation. The consortium will be open to any organizations interested in contributing their expertise to support the creation, monitoring, and long-term maintenance of this benchmarking effort.

“Generative AI has immense potential in agriculture, but its success depends on our collective commitment to accuracy and reliability. By working together as an industry to establish standards, test models, and share benchmark results, we can build trust and ensure that these tools are valuable to farmers and their advisors. We are excited to join the AI AgriBench Consortium,” said Tami Craig Schilling, VP, Agronomic Digital Innovation, Bayer Crop Science.

This consortium brings together deep technical expertise in both AI and agronomy across leading academic and industry groups to tackle the challenges these new systems bring. Recent efforts have led to open source systems like CropWizard and proprietary commercial systems, all of which are designed for answering technical questions about agronomy for farmers, agronomists, scientists, and other agriculture professionals. Extensive data sets and software tools within these organizations provide the foundation for building reliable, automated evaluation pipelines based on expert-validated ground truth answers.

“Evaluating the accuracy of AI-driven question-answering services for agriculture is as important as it is technically challenging. The accuracy of such services is critical because farmers’ livelihoods and farm environments can be compromised by inaccurate or ineffective advice on crop management questions. The CropWizard project within AIFARMS is developing innovative benchmarking methods for achieving high confidence in such services. We are excited to coordinate a public and open consortium that supports trustworthy AI services for the benefit of farmers and the broader agriculture community,” said Vikram Adve, Donald B. Gillies Professor of Computer Science, U of I, & AIFARMS PI.

Over the next few months, the consortium will convene members and maintain a leaderboard on a publicly accessible, secure website. This website will be available to any groups wishing to evaluate their models and publish their scores. For full transparency, a detailed description of the evaluation methodology, data sets, and governance process will also be publicly available. An independent oversight board administered by CDA will supervise and advise the consortium’s operations.