Quantifying Web Agents - A Survey on Web Agent Performance and Efficiency

Web agents are systems capable of navigating and interacting autonomously with the World Wide Web. While this concept is almost as old as the web itself, the recent success of large language models (LLMs) and their accompanying capabilities has accelerated the development of web agents.As the adoption of web agents continues, the need arises to compare actual performance and energy trade-offs, providing potential users and developers with insights on which web agent to select based on their preferences. However, concrete performance ratings and details on computational efficiency remain vague due to existing web agents and their respective benchmarks, which offer non-standardized performance ratings only. Further, the inside mechanisms of the agents obfuscate computational efficiency evaluations. In this survey, we systematically assess 79 web agents and their benchmarks (N=27), breaking down benchmark ratings and evaluating the computational efficiency of their crucial components. Our survey demonstrates two research gaps: (1) a lack of comparability among web agent performance ratings due to non-standardized metrics and (2) a lack of awareness of computational efficiency in the design of web agents, which puts their sustainability into question.We contribute step-by-step guides on developing web agents and selecting appropriate benchmarks for assuring computational efficiency. Our work enables further research and development of web agents with sustainability is a key factor.