Introduction
The Qwen team, working with Taobao and Tmall Group, has released E-Commerce Bench, a benchmark that hands an AI agent ¥100,000 in simulated capital (CNY, the environment models a Chinese marketplace) and a full year to run online stores. The accompanying paper, E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation, was posted to arXiv on August 31, 2026, and the code is on GitHub under Apache 2.0.
The premise is that long-horizon agentic work is not just short tasks chained together. Over 365 days an agent has to research a market, source inventory, adjust to demand shocks, and stay solvent — decisions that only pay off, or fail, hundreds of steps later.
Inside the Simulation
Product and supplier data are derived from a real e-commerce platform rather than generated:
- 6,886 products across 60 categories
- 576 suppliers, of which 152 are fraudulent across five scam types
- 12 store types, of which an agent may open up to four
- 8 scheduled promotions and 10 market events — supply-chain shocks and natural disasters that reshape demand
- A budget of 600 simulated minutes per day and up to 4,000 turns per episode
Storage fees, returns and seller reputation all feed back into the outcome, and the working-day limit means an agent that burns time on low-value actions simply gets less done.
Both sides of the market are deterministic, which is the design choice that makes the numbers comparable. Customer purchases and returns follow a fixed demand model; a negotiation kernel decides supplier pricing, concessions and walk-aways, with a language model used only to verbalize them. Run-to-run differences therefore come from the agent, not from a shifting world.
How It Differs From Vending-Bench
Year-long store simulations are not new: Google reported that Gemini 3 Pro topped Vending-Bench 2 on longer-horizon planning in November 2025. The paper positions E-Commerce Bench against that line directly — Vending-Bench's sequel does couple negotiation with adversarial suppliers, but only through sampled LLM counterparties and without an open, real-data economy. The catalog drawn from a live platform and the fixed negotiation kernel are what the Qwen team claims as new.
Seven Dimensions, No Overall Winner
The 18 frontier models evaluated are scored on year-end total assets plus negotiation quality, fraud avoidance, cash flow and solvency, operational efficiency, execution reliability, and learning over the horizon. Each model ran five independent episodes against one fixed world, 90 in all.
GPT-5.6 Sol earned the most, growing the ¥100,000 stake into ¥1,431,425 — roughly 14× — but the paper reports it placed 16th of 18 on fraud avoidance and trailed Claude Fable 5 on operational efficiency, measured as profit per tool call: ¥479 for Fable 5 against ¥363. (The GitHub README's summary table labels the GPT-5.6 Sol figure as best efficiency; the paper's own ranking does not. Trust the paper.)
By the paper's grouping — ten of the eighteen entries are counted as open-weight — Qwen3.8-Max-Preview leads that set with ¥416,252, 38% above GLM 5.2 (high), and posts the strongest learning over the horizon by bargaining suppliers down progressively across repeated orders.
The spread is the more striking result: best and worst differ by a factor of over 1,200, and 10 of the 90 episodes ended in insolvency. Bankruptcy was not spread evenly — it hit four models, with GPT-5.5, Claude Opus 4.6 and Gemini 3.1 Pro going under in 2 runs of 5 each, and Qwen3.5-Plus in 4 of 5. Profit and prudence come apart: the model that makes the most money is also among the easiest to defraud.
Conclusion
E-Commerce Bench is a useful counterweight to single-session agent evaluations. Its deterministic market and real-data catalog make a year of business decisions reproducible, and its seven-dimension scoring makes clear that a single headline number hides a lot. Whether the deterministic negotiation kernel generalizes to messier real counterparties is the open question; the code is public, so that is now testable.