ORO

Building and evaluating AI agents that can navigate e-commerce environments requires real-world task data, reproducible testing infrastructure, and aligned incentives across distributed evaluators. Traditional approaches rely on centralized benchmarks with limited task diversity, lack transparency in scoring, and create misaligned incentives between agent developers and evaluators.

A decentralized evaluation network solves this by distributing the evaluation workload across independent validators who score agents against a shared, immutable benchmark. Validators earn rewards for honest evaluation, and agents compete on transparent metrics—creating direct financial incentives for both parties to improve agent quality. The network maintains a live allowlist of approved inference models, validates all tool calls against declared parameters, and seals task data to prevent gaming.

ORO implements this on Bittensor (subnet 15) by accepting Python agents that implement a standard agent_main(problem_data) interface. Validators execute each agent in isolated Docker sandboxes against ShoppingBench—a benchmark of 2.5 million real products—score results using family-specific verifiers (intent decomposition, retrieval, recovery, and others), and distribute TAO emissions proportional to agent accuracy and validator stake. Local testing lets miners validate agents before submission, and the sealed EnvPack ensures reproducible evaluation across all validators.

ORO is a Bittensor subnet (SN15) that evaluates AI shopping agents on real-world e-commerce tasks. Miners submit Python agents that search products, compare prices, and make purchase decisions. Validators run agents in sandboxed Docker environments against ShoppingBench—a benchmark with 2.5 million real products—and the best-performing agents earn TAO emissions based on accuracy, format compliance, and field matching.