Loading...
Loading
Overview
Registration Activity
Neuron registration trends
This Interval
0
— burn · — PoW
Target / Interval
Loading
Neuron registration trends
Organizations conducting market research, trend analysis, and predictive modeling require large-scale social media datasets, but centralized scraping services are expensive, proprietary, and often restrict access. Building and maintaining in-house scraping infrastructure demands significant engineering effort and ongoing maintenance.
A decentralized network of incentivized contributors—miners who scrape data and validators who verify quality—can distribute both the work and the cost. Validators sample data to confirm authenticity, penalize false claims, and reward high-quality contributions. This creates a trustless system where no single entity controls the data or the validation process.
Data Universe implements this on Bittensor by organizing participants into miners (who scrape X, Reddit, and YouTube) and validators (who verify and score). Miners configure what data to collect via JSON scraping profiles and compete on data freshness, relevance (via Dynamic Desirability voting), uniqueness, and credibility. Validators periodically re-scrape sample data to verify claims, maintain a credibility score for each miner, and set on-chain weights reflecting performance. Incentives are distributed proportionally to miner scores.
Use cases were auto-generated from the GitHub repository and may not reflect the latest changes.
Data Universe (Subnet 13) is a Bittensor subnet that collects, stores, and provides access to large-scale social media data from X, Reddit, and YouTube. Miners scrape data from these sources and store it locally, while validators verify data quality and assign incentives based on data freshness, desirability, uniqueness, and miner credibility. The subnet enables low-cost data scraping and sentiment analysis through products like Gravity, providing datasets for market research, predictive analytics, and other applications.
Data Universe operates on a decentralized scraping and validation architecture. Miners run scrapers configured via scraping_config.json to collect data from X, Reddit, and YouTube, storing it in a local SQLite database organized into DataEntityBuckets (identified by source, time, and label). Miners expose their MinerIndex (summary of stored data) and data via an Axon (local server) that responds to validator requests.
Validators maintain a MetagraphSyncer for real-time network state, enumerate all miners, request their MinerIndexes, randomly sample data buckets, verify data authenticity by re-scraping from sources, and calculate credibility scores. Validators also retrieve aggregated Dynamic Desirability weights (set by validator voting via the Gravity GitHub repo) every 24 hours to update scoring. Miners optionally upload data to S3 via presigned URLs for public access and validation.
Validators periodically set weights on-chain reflecting miner scores, with the weight calculation incorporating data value, credibility, and S3 validation results. The system uses pm2 for process management and wandb for logging/monitoring.
This information was auto-extracted from the GitHub repository and may be incomplete or out of date. Always refer to the subnet's GitHub repository for the latest instructions.
The incentive mechanism has multiple dimensions: (1) Data Freshness: Data older than 30 days or outside desirable job date ranges receives zero score. Scoring is weighted towards newer data. (2) Data Desirability: Validators collectively vote on which data labels (subreddits, hashtags) are desirable using the Dynamic Desirability system (Gravity). Desirable data receives scale factors from (default 0.3 + 1) up to 5.0; unspecified labels receive a default scale factor of 0.3. (3) Duplication Factor: Data stored by many miners is worth less than data stored by few miners, with value decreasing proportionally to storage count. (4) Miner Credibility: Validators periodically sample data from miners' claimed buckets and re-scrape to verify authenticity.
Credibility ranges 0-1, calculated as: new_credibility = 0.15 * validation_result + 0.85 * old_credibility. Credibility is applied as a multiplier (credibility^2.5) to the raw score. Raw score = data_type_scale_factor * time_scalar * scorable_bytes.
Final score = raw_score * credibility^2.5. S3 validation applies a binary gate: only miners passing all validation checks (filename format, record count, duplicate rate, scraper success, job content match) receive rewards. Miner reward = (miner_score / total_network_score) * total_reward_pool.
The sections above were auto-generated from the GitHub repository and may not reflect the latest changes. View repository
Scraping the world's social media data
FAQs were auto-generated from the GitHub repository and may not reflect the latest changes.
Holder & total-staked metrics as of 1h ago (hourly snapshot)
Miners do not require a GPU and can run on low-tier machines as long as they have adequate network bandwidth and disk space for the local SQLite database
Validators require at least 32 GB of RAM and a decent CPU (4+ cores). Do not require a GPU but need good network connectivity to query miners and validate data.
Clone the data-universe repository
git clone https://github.com/RusticLuftig/data-universe.git
Create and activate a Python virtual environment (or use Conda)
cd data-universe && python -m venv venv && source venv/bin/activate
Install the data-universe package and dependencies
python -m pip install -e .
Create a Bittensor wallet and register a hotkey (see Bittensor docs)
btcli wallet new_coldkey --wallet.name your-wallet && btcli wallet new_hotkey --wallet.name your-wallet --wallet.hotkey your-hotkey
Register your hotkey on Subnet 13
btcli subnets register --wallet.name your-wallet --wallet.hotkey your-hotkey --netuid 13
Set up optional .env file with API tokens (Apify for X/Reddit scraping, or Reddit OAuth credentials)(optional)
echo 'APIFY_API_TOKEN="your_token"' > .env
Start the miner using pm2 (install pm2 first if needed: npm install -g pm2)
pm2 start python -- ./neurons/miner.py --wallet.name your-wallet --wallet.hotkey your-hotkey
Verify the miner is actually scraping data (check logs for successful scrapes)
pm2 logs <process-name> | grep -i 'scrap\|status 403\|status 401'
Clone the data-universe repository
git clone https://github.com/RusticLuftig/data-universe.git
Create and activate a Python virtual environment (or use Conda)
cd data-universe && python -m venv venv && source venv/bin/activate
Install the data-universe package and dependencies
python -m pip install -e .
Create a Bittensor wallet and register a hotkey
btcli wallet new_coldkey --wallet.name your-wallet && btcli wallet new_hotkey --wallet.name your-wallet --wallet.hotkey your-hotkey
Register your hotkey on Subnet 13
btcli subnets register --wallet.name your-wallet --wallet.hotkey your-hotkey --netuid 13
Set up .env file with API tokens (Apify required for validators; Reddit OAuth optional)
echo 'APIFY_API_TOKEN="your_token" REDDIT_CLIENT_ID="..." REDDIT_CLIENT_SECRET="..."' > .env
Start the validator with auto-updates (recommended; requires pm2 and git)(optional)
pm2 start --name net13-vali-updater --interpreter python scripts/start_validator.py -- --pm2_name net13-vali --wallet.name your-wallet --wallet.hotkey your-hotkey
Alternatively, start the validator without auto-updates(optional)
pm2 start python -- ./neurons/validator.py --wallet.name your-wallet --wallet.hotkey your-hotkey
