Skip to main navigation Skip to search Skip to main content

AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios

  • Yunjia Qi
  • , Hao Peng
  • , Xiaozhi Wang
  • , Amy Xin
  • , Youfeng Liu
  • , Bin Xu
  • , Lei Hou
  • , Juanzi Li
  • BNRist
  • Shenzhen International Graduate School
  • Zhipu AI
  • Tsinghua University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Large Language Models (LLMs) have demonstrated advanced capabilities in real-world agentic applications. Growing research efforts aim to develop LLM-based agents to address practical demands, introducing a new challenge: agentic scenarios often involve lengthy instructions with complex constraints, such as extended system prompts and detailed tool specifications. While adherence to such instructions is crucial for agentic applications, whether LLMs can reliably follow them remains underexplored. In this paper, we introduce AGENTIF, the first benchmark for systematically evaluating LLM instruction following ability in agentic scenarios. AGENTIF features three key characteristics: (1) Realistic, constructed from 50 real-world agentic applications. (2) Long, averaging 1, 723 words with a maximum of 15, 630 words. (3) Complex, averaging 11.9 constraints per instruction, covering diverse constraint types, such as tool specifications and condition constraints. To construct AGENTIF, we collect 707 human-annotated instructions across 50 agentic tasks from industrial application agents and open-source agentic systems. For each instruction, we annotate the associated constraints and corresponding evaluation metrics, including code-based evaluation, LLM-based evaluation, and hybrid code-LLM evaluation. We use AGENTIF to systematically evaluate existing advanced LLMs. We observe that current models generally perform poorly, especially in handling complex constraint structures and tool specifications. We further conduct error analysis and analytical experiments on instruction length and meta constraints, providing some findings about the failure modes of existing LLMs. We have released the code3 and data4 to facilitate future research.

Original languageEnglish
Title of host publication39th Conference on Neural Information Processing Systems, NeurIPS 2025
PublisherNeural information processing systems foundation
Pages62980-63008
Number of pages29
ISBN (Electronic)9798331338275
StatePublished - 2025
Externally publishedYes
Event39th Conference on Neural Information Processing Systems, NeurIPS 2025 - San Diego, United States
Duration: 2 Dec 2025 → 7 Dec 2025

Publication series

NameAdvances in Neural Information Processing Systems
Volume38
ISSN (Print)1049-5258

Conference

Conference39th Conference on Neural Information Processing Systems, NeurIPS 2025
Country/TerritoryUnited States
CitySan Diego
Period2/12/25 → 7/12/25

Fingerprint

Dive into the research topics of 'AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios'. Together they form a unique fingerprint.

Cite this