+ AgentLab is a comprehensive framework for developing and evaluating agents on a variety of + benchmarks supported by BrowserGym. + It provides essential building blocks for creating web agents, unified LLM APIs, + and extensive reproducibility features for rigorous research. +
++ The framework supports large-scale parallel agent experiments using Ray, includes various + agent architectures, and maintains a unified leaderboard across multiple benchmarks + including WebArena, WorkArena, VisualWebArena, AssistantBench, and more. +
++ A unified environment for web agent research across multiple benchmarks. + Provides standardized interfaces and evaluation metrics. +
+ ++ Enterprise-focused web agent benchmark with realistic workplace tasks. + Multiple difficulty levels (L1, L2, L3) with high seed diversity. +
+ ++ Advanced benchmark with 682 compositional planning and reasoning tasks + for evaluating autonomous agents in enterprise workflows. +
+ ++ Innovative approach for context trimming in web agents, enhancing efficiency and security. +
+ +| Benchmark | +Tasks | +Max Steps | +Multi-tab | +Status | +
|---|---|---|---|---|
| WebArena | +812 | +30 | +✓ | +Available | +
| WorkArena L1-L3 | +33-341 | +30-50 | +✗ | +Available | +
| VisualWebArena | +910 | +30 | +✓ | +Available | +
| AssistantBench | +214 | +30 | +✓ | +Available | +
| MiniWoB | +125 | +10 | +✗ | +Available | +
| OSWorld | +369 | +Variable | +✗ | +Available | +
pip install agentlab
+playwright installfrom agentlab.agents.generic_agent import AGENT_4o_MINI
+from agentlab.experiments.study import make_study
+
+study = make_study(
+ benchmark="miniwob",
+ agent_args=[AGENT_4o_MINI],
+ comment="My first study",
+)
+
+study.run(n_jobs=5)agentlab-assistant --start_url https://www.google.com+ The BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents, particularly those leveraging automation and Large Language Models (LLMs). Many existing benchmarks suffer from fragmentation and inconsistent evaluation methodologies, making it challenging to achieve reliable comparisons and reproducible results. In an earlier work, Drouin et al. (2024) introduced BrowserGym which aims to solve this by providing a unified, gym-like environment with well-defined observation and action spaces, facilitating standardized evaluation across diverse benchmarks. We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature and includes AgentLab, a complementary framework that aids in agent creation, testing, and analysis. Our proposed ecosystem offers flexibility for integrating new benchmarks while ensuring consistent evaluation and comprehensive experiment management. As a supporting evidence, we conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks made available in BrowserGym. Among other findings, our results highlight a large discrepancy between OpenAI and Anthropic's latests models, with Claude-3.5-Sonnet leading the way on almost all benchmarks, except on vision-related tasks where GPT-4o is superior. Despite these advancements, our results emphasize that building robust and efficient web agents remains a significant challenge, due to the inherent complexity of real-world web environments and the limitations of current models. +
++ Standardized Gymnasium-compatible interface across all supported benchmarks, + enabling seamless switching between different evaluation environments. +
++ Support for 8+ web agent benchmarks including WebArena, WorkArena, + VisualWebArena, AssistantBench, MiniWoB, and more. +
++ Consistent evaluation metrics and protocols across benchmarks, + enabling fair comparison of agent performance. +
++ Built-in reproducibility features, experiment tracking, + and standardized reporting for reliable research outcomes. +
+@misc{dechezelles2025browsergymecosystemwebagent,
+ title={The BrowserGym Ecosystem for Web Agent Research},
+ author={Thibault Le Sellier De Chezelles and Maxime Gasse and Alexandre Drouin and Massimo Caccia and Léo Boisvert and Megh Thakkar and Tom Marty and Rim Assouel and Sahar Omidi Shayegan and Lawrence Keunho Jang and Xing Han Lù and Ori Yoran and Dehan Kong and Frank F. Xu and Siva Reddy and Quentin Cappart and Graham Neubig and Ruslan Salakhutdinov and Nicolas Chapados and Alexandre Lacoste},
+ year={2025},
+ eprint={2412.05467},
+ archivePrefix={arXiv},
+ primaryClass={cs.LG},
+ url={https://arxiv.org/abs/2412.05467},
+}+ Web agents powered by large language models (LLMs) must process lengthy web page observations to complete + user goals; these pages often exceed tens of thousands of tokens. This saturates context limits and + increases computational cost processing; moreover, processing full pages exposes agents to security risks + such as prompt injection. +
++ Existing pruning strategies either discard relevant content or retain irrelevant context, leading to + suboptimal action prediction. We introduce FocusAgent, a simple yet effective approach that leverages + a lightweight LLM retriever to extract the most relevant lines from accessibility tree (AxTree) observations, + guided by task goals. +
++ By pruning noisy and irrelevant content, FocusAgent enables efficient reasoning while reducing vulnerability + to injection attacks. Experiments on WorkArena and WebArena benchmarks show that FocusAgent matches the + performance of strong baselines, while reducing observation size by over 50%. Furthermore, a variant of + FocusAgent significantly reduces the success rate of prompt-injection attacks, including banner and pop-up + attacks, while maintaining task success performance in attack-free settings. +
++ Intelligent pruning of web page observations using lightweight LLM retrievers, + reducing context size by over 50% while maintaining performance. +
++ Significant reduction in vulnerability to prompt injection attacks, + including banner and pop-up attacks, while preserving task performance. +
++ Reduced computational costs and faster processing by focusing on + relevant content extracted from accessibility tree observations. +
++ Task-aware content extraction that maintains relevance while + eliminating noise and irrelevant context from web page observations. +
++ FocusAgent employs a lightweight language model as a retriever to identify and extract + the most relevant lines from accessibility tree observations. This approach ensures + that critical information is preserved while eliminating redundant or irrelevant content. +
++ Unlike existing approaches that either discard relevant content or retain irrelevant context, + FocusAgent's pruning strategy is designed to maintain optimal action prediction capabilities + while significantly reducing observation size. +
++ By removing potentially malicious content and focusing on task-relevant information, + FocusAgent reduces exposure to prompt injection attacks while maintaining high + performance on legitimate tasks. +
+Context Reduction
+Significant reduction in observation size while maintaining performance
+Benchmark Evaluation
+Validated on WorkArena and WebArena benchmarks
++ FocusAgent provides a practical solution for deploying web agents in + real-world scenarios where computational efficiency and security are critical. +
++ Significant reduction in computational costs through efficient context processing, + making web agents more accessible and scalable. +
++ Enhanced security against prompt injection attacks, enabling safer + deployment of web agents in enterprise and production environments. +
++ The approach is simple yet effective, making it easily adaptable to + various web agent architectures and deployment scenarios. +
+@article{kerboua2025focusagent,
+ title={FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agents},
+ author={Kerboua, Imene and Shayegan, Sahar Omidi and Thakkar, Megh and L{\\`u}, Xing Han and Boisvert, L{\\'e}o and Caccia, Massimo and Espinas, J{\\'e}r{\\'e}my and Aussem, Alexandre and Gasse, Maxime and Chapados, Nicolas and others},
+ journal={arXiv preprint arXiv:2510.03204},
+ year={2025}
+}+ The ability of large language models (LLMs) to mimic human-like intelligence has led to a surge in + LLM-based autonomous agents. Though recent LLMs seem capable of planning and reasoning given user + instructions, their effectiveness in applying these capabilities for autonomous task solving remains + underexplored. This is especially true in enterprise settings, where automated agents hold the + promise of a high impact. +
++ To fill this gap, we propose WorkArena++, a novel benchmark consisting of 682 tasks corresponding + to realistic workflows routinely performed by knowledge workers. WorkArena++ is designed to evaluate + the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding + abilities of web agents. Our empirical studies across state-of-the-art LLMs and vision-language models + (VLMs), as well as human workers, reveal several challenges for such models to serve as useful assistants + in the workplace. +
++ In addition to the benchmark, we provide a mechanism to effortlessly generate thousands of ground-truth + observation/action traces, which can be used for fine-tuning existing models. Overall, we expect this + work to serve as a useful resource to help the community progress toward capable autonomous agents. +
++ 682 tasks designed to evaluate compositional planning and reasoning abilities, + going beyond simple navigation to complex multi-step workflows. +
++ Comprehensive evaluation across state-of-the-art LLMs and vision-language models, + with human baseline comparisons for realistic performance assessment. +
++ Built-in mechanism to generate thousands of ground-truth observation/action traces + for fine-tuning and training autonomous agents. +
++ Realistic workflows performed by knowledge workers, focusing on + planning, reasoning, retrieval, and contextual understanding capabilities. +
++ Tasks requiring multi-step planning, decomposition of complex problems, + and strategic thinking to achieve enterprise-level objectives across + ServiceNow platform workflows. +
++ Evaluation of mathematical reasoning, logical inference, and quantitative + analysis capabilities required for data-driven enterprise decision making + and process optimization. +
++ Assessment of agents' ability to retrieve relevant information from + large knowledge bases, understand contextual relationships, and + apply retrieved knowledge to solve complex tasks. +
+Total Tasks
+Comprehensive task coverage across multiple enterprise scenarios
+Core Capabilities
+Planning, Reasoning, Retrieval, Context, Problem-Solving
+@article{boisvert2024workarena++,
+ title={WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks},
+ author={Boisvert, L{\\'e}o and Thakkar, Megh and Gasse, Maxime and Caccia, Massimo and Le Sellier De Chezelles, Thibault and Cappart, Quentin and Chapados, Nicolas and Lacoste, Alexandre},
+ journal={arXiv preprint arXiv:2407.05291},
+ year={2024}
+}