The Factorio Learning Environment (FLE) introduces a novel way to assess the capabilities of large language models (LLMs) by challenging them with tasks in the game of Factorio. The environment provides structured tasks in lab-play and open-ended challenges in open-play, requiring agents to showcase their abilities in areas such as planning, automation, and resource management. Results show that while LLMs excel in short-horizon skills, they struggle with spatial reasoning and complex automation. Factors like coding skill, technology investment, planning, and spatial reasoning play crucial roles in determining agent performance. The release of FLE as an open-source platform aims to encourage further research on agent capabilities in complex domains.
https://jackhopkins.github.io/factorio-learning-environment/