iota

Training large language models requires enormous computational resources and infrastructure costs. A single organization cannot access sufficient GPU capacity, and the process typically demands months of specialized engineering to manage distributed training across heterogeneous hardware. Researchers, startups, and independent developers lack affordable pathways to pretrain foundation models at scale.

The IOTA framework distributes model pretraining across a network of independent participants running on commodity GPUs and diverse hardware. Participants operate as miners (compute contributors) or validators (quality checkers). The orchestrator assigns model layers to miners, automatically coordinates data flow between them, and ensures work is performed correctly through spot-checks. Miners compete to process training activations, earning rewards proportional to compute contributed. This eliminates the need for centralized infrastructure and makes large-scale pretraining accessible to distributed teams.

On the Bittensor SN9 network, IOTA currently pretrains a 1.5B parameter Llama-inspired model using data and pipeline parallelism. The system uses direct peer-to-peer activation transfer between miners (via iroh QUIC) to minimize latency and cloud costs, with cryptographic integrity verification at each layer boundary. Validators continuously spot-check miner work using S3-uploaded samples, ensuring training quality across the permissionless network.

IOTA is an incentivized framework for pretraining large language models (currently 1.5B Llama-inspired) across a distributed network of heterogeneous miners using data and pipeline parallelism. Miners compete to process activations while validators spot-check work; the system employs orchestrator-coordinated P2P activation transfer via iroh QUIC, S3 for weights/samples, and a Butterfly All-Reduce variant for weight synchronization.