iota

Large language model pretraining requires enormous computational resources and time. Centralized training on GPU clusters is expensive, slow, and inaccessible to most researchers and organizations. A distributed approach could leverage idle compute across many machines, but coordinating heterogeneous, unreliable, and mutually untrusting participants in a shared training run has been technically intractable—there is no standard way to verify that distributed workers performed their assigned computation correctly without trusting each worker's self-reports.

Iota addresses this by building a decentralized training network where miners compete to process training activations across model layers, submit their work to an orchestrator for coordination, and earn token rewards based on throughput and correctness. Validators independently spot-check miners by downloading and recomputing activations, using BLAKE3 hashes to verify integrity. The orchestrator routes activations between miners via peer-to-peer QUIC connections (eliminating S3 as a bottleneck), manages layer assignment, and coordinates weight merges via Butterfly All-Reduce.

IOTA (Incentivized Orchestrated Training Architecture) is a distributed framework for pretraining large language models across heterogeneous, permissionless, and token-incentivized machines. It uses data and pipeline parallelism to distribute model layers across miners, with validators spot-checking work. Miners compete to process activations efficiently, upload weights, and merge activations via Butterfly All-Reduce, while validators ensure correctness.