Question 1

What does tau actually capture when an AI agent attempts to solve a coding task?

Accepted Answer

tau records the complete trajectory: every request sent to the LLM (model, messages, token limits), every response received (completions, token usage, latency), every shell command executed (with exit code, stdout, stderr, duration), and every code edit made to the repository. It also captures miner-provided logs (treated as untrusted) and final patch output. This full event stream is stored locally as JSONL and optionally exported to private training datasets.

Question 2

Why does tau use Docker to run agent code instead of running it directly?

Accepted Answer

Docker provides isolation: agents cannot see other tasks, cannot access the network, and cannot persist state between runs. tau instruments the Docker container with a lightweight event capture script that intercepts all command calls and code edits at the runner level, so the event stream comes from the validator (tau), not from potentially deceptive miner logs.

Question 3

How does tau prevent miners from gaming the evaluation with fake or misleading logs?

Accepted Answer

tau marks all events with a source tag: events from the tau proxy (LLM calls) and tau runner (commands/edits) are tagged as trusted; any logs from the miner are labeled untrusted. Scoring and training data export use only the trusted events. Miner logs are retained for debugging but never used as ground truth for rewards or model training.

Question 4

Can tau's evaluation data be published publicly, or does it leak sensitive information?

Accepted Answer

tau supports configurable redaction: private datasets (internal/training use) retain full trajectories, LLM prompts, and judge rationales; public datasets can be stripped to show only final patches, success/failure labels, and aggregate scores. Hidden promotion tasks are never exported publicly. Operators can choose visibility tier per dataset and define which fields are redacted for public release.

Question 5

How does tau generate the coding tasks that agents try to solve?

Accepted Answer

tau mines open GitHub repositories and issues, clones them to isolated workspaces, and uses commit history to extract reference solutions. Each task includes a clean repo state, an issue description, and a reference patch that shows the intended fix. Validators can run duel rounds (king vs. challenger) where two agents attempt the same task and are scored against each other, eliminating the need for an absolute pass/fail judgment.

Question 6

What formats does tau export for training AI models?

Accepted Answer

tau exports three formats: JSONL rollouts (full trajectories with all events), GRPO rows (prompt + trajectory + scalar reward), and DPO rows (task prompt with chosen trajectory and rejected trajectory for preference learning). These can be pushed to HuggingFace datasets for private model fine-tuning or stored locally for offline training.

conjectures

conjectures

Hardware Requirements

Hardware Requirements

Miner Setup

Validator Setup

Participate

Participate

Hardware Requirements

Hardware Requirements

Miner Setup

Validator Setup