conjectures

Software engineering evaluation at scale requires isolating agent behavior from human bias, capturing what an agent actually did (not what it claims it did), and producing trustworthy training data without leaking proprietary task details. Current approaches either rely on manual human review, publish full task/solution histories that miners can exploit, or lose critical intermediate steps (prompts, commands, edits) needed to retrain agents effectively. The solution is to distribute evaluation infrastructure so that task creation, agent execution, judging, and data export each remain independent. One validator generates tasks from real GitHub issues; agents (miners) attempt solutions in isolated Docker containers; a separate scoring layer compares solutions fairly; and a trusted event capture system records every LLM call, command execution, and code edit—preserving the complete trajectory for training while redacting sensitive task details. Bittensor's Ninja subnet implements this by running tau as the validator-side pipeline: it proxies all model calls through a controlled inference endpoint, instruments the Docker runner to capture token-faithful events, and exports training data (GRPO/DPO format) to private datasets while maintaining strict security boundaries between miner code and validator truth.

tau is a validator-side software engineering evaluation pipeline for the Bittensor Ninja subnet. It handles task generation from GitHub repositories, manages Docker-based agent execution, runs LLM-powered judging, and coordinates king/challenger validation rounds with detailed rollout tracking. The system captures token-faithful LLM call trajectories, command execution events, code edits, and final patches to produce training data for offline reinforcement learning (GRPO/DPO) while maintaining strict security boundaries between miner-controlled agent code and validator-controlled scoring infrastructure.