Question 1

How is compression evaluated if it's not just token count reduction?

Accepted Answer

Compression is evaluated on real agent tasks with two equally important signals: (1) agent task performance—whether the agent still successfully completes the task with compression enabled, and (2) weighted token consumption—how many effective tokens are saved compared to an uncompressed baseline. Scoring rewards solutions that reduce tokens without causing previously successful tasks to fail.

Question 2

What does weighted token counting mean and why does it matter?

Accepted Answer

Weighted token counting assigns different costs to different token types: non-cached input tokens (weight 1.0), cached input tokens (weight 0.1), and output tokens (weight 3.0). This reflects the actual economic cost of inference, since cached tokens are cheaper and output tokens are more expensive. It allows evaluation to measure real impact rather than raw token reductions.

Question 3

Can I submit a compression solution if I'm not a machine learning researcher?

Accepted Answer

Yes. Miners submit Python functions with a standard interface. Solutions can be algorithmic (rule-based filtering), model-based (fine-tuned compression models), or hybrid. The only requirement is that your solution compresses text to a specified ratio while preserving the ability of an LLM to answer questions about the original content. Check the miner setup guide for submission steps.

Question 4

How often are miners evaluated and how are scores used?

Accepted Answer

Validators continuously execute agent tasks and report scores to the platform's scoring system. Scores are aggregated using a layered incentive mechanism: submissions compete across three benchmark types (swebench_verified at 50%, swe_explorer_explore and swe_explorer_edit at 25% each). Within each benchmark, winners are identified at different complexity levels (hard tasks, standard tasks, singles), and rewards are distributed based on the proportion of elements a miner wins.

Question 5

What happens if my compression causes task failure?

Accepted Answer

The scoring system penalizes quality regression. If a miner's compressed context causes an agent to fail tasks it previously succeeded on, the penalty floor applies (up to -2.0 on per-task score for standard tasks). Conversely, if compression maintains quality, the miner can earn bonus points proportional to token savings. This structure encourages solutions that preserve performance, not ones that simply strip context aggressively.

Question 6

How does SOMA handle hard versus standard tasks differently?

Accepted Answer

Hard tasks (where fewer than 2 baseline runs exist) use a bonus zone that awards extra credit for exceeding baseline performance on limited data. Standard tasks (2 or more baseline runs) apply a penalty zone if performance drops below 80% of the baseline run count, maintaining zone if performance is stable, and bonus zone if the miner outperforms. This encourages consistent performance on well-tested tasks while rewarding innovation on novel problems.

SOMA

SOMA

Hardware Requirements

Hardware Requirements

Miner Setup

Validator Setup

Participate

Participate

Hardware Requirements

Hardware Requirements

Miner Setup

Validator Setup