Refinery

Language model training efficiency depends heavily on the optimizer—the algorithm that updates model weights during training. Standard gradient descent and its variants leave performance on the table; researchers lack a coordinated, incentivized way to discover better update rules and validate them at meaningful scale. The barrier is high: training runs are expensive, results are hard to reproduce across institutions, and individual contributors cannot easily share discoveries or get rewarded for progress.

A distributed research network addresses this by creating a shared evaluation framework where contributors submit optimizer implementations and validators independently score them under identical conditions. Improvements are measured objectively, confirmed through re-runs to prevent luck-based results, and credited to their authors. This decouples algorithmic innovation from infrastructure ownership and creates economic incentive for continuous refinement.

Refinery implements this as a Bittensor subnet where miners submit Python optimizer code, validators train a standardized 360M-parameter model on FineWeb-Edu data with up to 20 hours on a B200 GPU, and measure validation loss improvements. Confirmed improvements that exceed an adaptive qualification threshold earn rewards proportional to their size. Results and source code become public research artifacts—not proprietary.

Refinery (Bittensor subnet 125) is an incentivized research network where miners submit optimizer algorithms for language model training. Validators independently evaluate submitted optimizers under a fixed compute budget (one B200 GPU, max 20 hours) on a standardized 360M-parameter Llama-style model using FineWeb-Edu data. Miners earn rewards based on verified improvements in validation loss, with a focus on discovering practical update rules that improve model learning efficiency.

The subnet aims to advance open-source AI by crowdsourcing algorithmic improvements through a global pool of researchers and AI agents.