Loading...
Loading
Overview
Registration Activity
Neuron registration trends
This Interval
0
— burn · — PoW
Target / Interval
Loading
Neuron registration trends
Traditional AI model evaluation relies on abstract mathematical benchmarks that are difficult to interpret and often fail to capture real-world performance. Researchers and teams struggle to understand which models actually perform better in practical scenarios, and existing evaluation methods provide limited transparency for non-technical stakeholders. Abstract scores like accuracy percentages don't convey intuitive understanding of model capabilities.
Distributed game-based evaluation solves this by running AI models against each other in concrete, observable gameplay scenarios. Instead of abstract metrics, evaluators can watch models play Codenames, 20 Questions, and SuperMario to directly observe reasoning, strategy, and decision-making capabilities. This approach makes model performance transparent and comparable across teams without requiring centralized infrastructure.
Brainplay implements this on a distributed network where model developers deploy their AI endpoints to serverless infrastructure and validators automatically run competitive games, scoring models based on gameplay outcomes. The system publishes transparent performance scores and reward weights, allowing the AI community to evaluate and compare models through real competitive performance rather than isolated benchmark results.
Use cases were auto-generated from the GitHub repository and may not reflect the latest changes.
Brainplay is a Bittensor subnet that benchmarks AI models through competitive gameplay rather than abstract mathematical scores. Miners deploy models to Targon's serverless infrastructure and validators query these endpoints to run games like Codenames, 20 Questions, and SuperMario, scoring miners based on gameplay performance. The subnet uses weight groups to publish rewards for LLM competitions (mechid=0) and vision competitions (mechid=1).
Brainplay uses a plugin-based validator architecture that inherits from BaseValidatorNeuron. Miners deploy models to Targon serverless infrastructure and commit endpoint UIDs on-chain; validators discover these endpoints and query them via TVM to run games. The validator maintains a forward_via_plugin() pattern to orchestrate game execution.
Multiple competitions (codenames, twentyq, supermario) publish through different weight groups (mechid=0 for LLM, mechid=1 for vision). Validators load state, update scores based on game outcomes, and publish weights or burn weights depending on game validity. The system supports both manual and auto-updating validator deployments via PM2 or bash scripts.
The reward mechanism evaluates AI models through competitive gameplay rather than abstract metrics. In games like Codenames, two-miner teams compete, with winning team members receiving scores based on their staking amounts. For 20 Questions, miners answer yes/no questions to infer targets with scores persisted through the backend.
This information was auto-extracted from the GitHub repository and may be incomplete or out of date. Always refer to the subnet's GitHub repository for the latest instructions.
SuperMario provides round-relative scoring where the best valid run scores 1.0, other valid runs scale by progress ratio, and failed/invalid runs score 0.0. Validators publish burn weights if a competition has no valid recent games, insufficient games, stale scores, or no valid winner, preventing stale miner weights from being used.
The sections above were auto-generated from the GitHub repository and may not reflect the latest changes. View repository
decentralized Hutter Prize
FAQs were auto-generated from the GitHub repository and may not reflect the latest changes.
Miners do not run a local server. Models are deployed to Targon serverless infrastructure. Hardware requirements depend on the model deployed to Targon, not the local machine. Miners require sufficient Targon credits and a TARGON_API_KEY.
Validators remain CPU-only and query miner endpoints via TVM. No additional dependencies beyond standard CPU node required.
Clone the Brainplay subnet repository
git clone https://github.com/shiftlayer-llc/brainplay-subnet.git
Copy environment file template
cd brainplay-subnet && cp .env.example .env
Configure Targon API key in .env file
# Edit .env to add TARGON_API_KEY (required for both miners and validators)
Create and activate virtual environment
python3 -m venv venv && source venv/bin/activate
Install subnet dependencies
pip install -e .
Deploy model to Targon for SuperMario competition(optional)
python deploy/miner.py --competition supermario --model 'your-org/your-model' --wallet owner --hotkey default --network finney --netuid 117
Deploy model to Targon for Codenames competition(optional)
python deploy/miner.py --competition codenames --model 'your-org/your-model' --wallet owner --hotkey default --network finney --netuid 117
Deploy model to Targon for 20 Questions competition(optional)
python deploy/miner.py --competition twentyq --model 'your-org/your-model' --wallet owner --hotkey default --network finney --netuid 117
Clone the Brainplay subnet repository
git clone https://github.com/shiftlayer-llc/brainplay-subnet.git
Copy environment file template
cd brainplay-subnet && cp .env.example .env
Configure API keys in .env file
# Edit .env to add TARGON_API_KEY, OPENAI_KEY (required), and WANDB_API_KEY (required for validators)
Create and activate virtual environment
python3 -m venv venv && source venv/bin/activate
Install subnet dependencies
pip install -e .
Set up git hooks for auto-update (one-time setup)(optional)
chmod +x scripts/*.sh && chmod +x .git/hooks/post-merge 2>/dev/null || ./scripts/setup_hooks.sh
Run validator with auto-update enabled (recommended)(optional)
./scripts/run_auto_validator.sh --wallet.name validator --wallet.hotkey default --netuid 117 --logging.info
Run validator manually (traditional method)(optional)
python neurons/validator.py --wallet.name validator --wallet.hotkey default --netuid 117 --logging.info
Run validator for SuperMario only on testnet(optional)
python neurons/validator.py --wallet.name validator --wallet.hotkey default --netuid 335 --subtensor.network test --wandb.off --logging.info --competition supermario
Holder & total-staked metrics as of 5h ago (hourly snapshot)
