What is a Trusted Execution Environment and why do confidential AI workloads need one?

A TEE is a hardware-isolated region of a CPU where memory is encrypted and code execution is attested cryptographically. Without a TEE, model weights, prompts, and results exist unencrypted in RAM where the hypervisor, kernel, and cloud provider can inspect them. With a TEE (Intel TDX, AMD SEV-SNP, or NVIDIA CC), the CPU guarantees encryption at rest in memory and issues a cryptographic attestation report proving the exact code and measurement running inside. KubeTEE uses Kata Containers to run each AI workload in its own virtualized TEE, so data is protected in-use and customers can verify via attestation that no unintended code is running on their data.

How does KubeTEE ensure that miners don't cheat or stop serving jobs?

Every miner posts a 100 TAO deposit as on-chain registration collateral on their own hotkey—not held by KubeTEE, never slashable as confiscation. If a miner stops being scored (fails attestation, stops serving jobs, or breaches SLA), validators withhold emissions and the deposit **freezes** on the miner's hotkey rather than being taken. The freeze reverses when the miner returns to compliance. Enforcement is reversible and capital-backed: miners have committed real TAO and lose ongoing earnings (emission) if they defect, not one-time capital loss. Additionally, validators benchmark miner pricing against Targon, Lium, and Chutes each epoch—a miner priced far above competitors scores low, so customers route elsewhere and the miner earns less.

How can I verify that KubeTEE infrastructure is actually running my workload in a TEE?

Every Kata+CoCo pod includes a CoCo Trustee attestation report embedded in the TDX quote. Customers can request the attestation evidence from the validator or the Armada job metadata after completion. The quote contains a hash of the exact container image, the kernel, the initdata, and proof that guest debug is off. Verifying the quote through Intel Trust Authority confirms the measurement matches the committed image and that the guest ran in a real hardware TEE. Customers never have to trust KubeTEE's claim—cryptography is the proof.

What does it cost to run a job on KubeTEE, and how is pricing determined?

KubeTEE prices compute at a resources-per-hour rate (Alpha or TAO) set by the validator each epoch. The validator reads competitor pricing from Targon (SN4), Lium (SN51), and Chutes (SN64) public APIs, converts them to a common unit ($/GPU-hour by GPU type), and benchmarks SN90 miner compensation and customer pricing against that band. The price is **discovered from the market, not set by decree**. Miners priced at or below the target are scored fully; miners priced far above compete poorly. Payment is accepted in Subnet 90 Alpha (earned via trading), Alpha from other subnets, or TAO (both Finney TAO and Base-bridged TAO via Chainlink CCIP, live 2026-08-21). Jobs are billed at the resources price per hour, with pricing visible before submission so customers know the cost upfront.

Can KubeTEE run my existing inference or training code, or do I need to rewrite it?

Existing code runs unmodified. KubeTEE uses standard Kubernetes container images and the LiteLLM OpenAI-compatible API, so any client library or script that calls OpenAI already works—just point at llm.kubetee.ai instead. For batch jobs (fine-tuning, eval, pipelines), code submits to Armada queues as Kubernetes Job or Pod manifests, with no KubeTEE-specific SDK. The TEE boundary is transparent: the same container image runs confidentially, attestation happens inside the guest, and secrets (NGC API keys, HuggingFace tokens) are injected from CoCo Trustee, not Git or Kubernetes Secrets. No rewrite needed.

What happens if a miner node fails or goes offline?

Miner clusters run with high availability: 8 nodes minimum per cluster with 3+ dedicated GPU workers per GPU type (e.g., 3 H200 + 3 B200) so a single node failure does not drop a GPU class. Armada batch jobs run with gang scheduling so all replicas of a job are scheduled together or not at all—partial failures are retried. If a node is down long enough, the validator's Rancher readiness check will mark the cluster unhealthy and reduce emissions. If the entire cluster is down, the miner receives zero score that epoch. The 100 TAO deposit does not compensate for extended downtime, but the validator's grace and recovery windows (3-cycle grace before withholding) give miners time to recover without losing a single cycle to a transient outage. Customers see job queuing and latency increase when capacity is reduced, not jobs failing.

Is KubeTEE FIPS-validated or compliant with government security standards?

RKE2 is FIPS-140-2 validated today and has a FIPS-140-3 roadmap. TEE hardware components (AMD PSP, Intel CSME, NVIDIA security modules) hold partial FIPS validations for specific cryptographic functions, but no complete end-to-end TEE FIPS validation exists because the industry standard does not yet cover all TEE operations. KubeTEE uses FIPS-approved algorithms (AES-XTS for memory encryption, ECDSA P-384 for attestation signatures, SHA-384 for measurements) and documents where validation gaps exist. Organizations requiring FedRAMP High or DoD IL4+ can achieve authorization through documented risk acceptance and defense-in-depth (the FIPS baseline + hardware attestation + validator scoring provides multiple independent assurance layers).

How are secrets like API keys or model weights protected in KubeTEE?

Secrets never live in Git, Kubernetes Secrets, or host-visible storage. Instead, CoCo Trustee (Key Broker Service) holds the material and releases it only to a guest that has proven attestation and requested it with the correct initdata role. On pod startup, an init container calls Trustee with the hardware attestation report. If debug is off and measurements match policy, Trustee returns the secret (NGC API key, HuggingFace token, or TLS certificate). If attestation fails, Trustee returns nothing and the pod fails to start. Encrypted container images use the same flow: the pod cannot decrypt its layers without proving attestation first. Secrets that are accidentally leaked from a pod are useless outside the attested boundary—they cannot be reused on non-TEE infrastructure or by an attacker's container.

KubeTEE (SN90) — Confidential AI Compute

TAO$247.36

BTC0.0031

MCap$2.8B

Vol$266.1M

KubeTEE (SN90) — Confidential AI Compute | Bittensor.ai

Miner Setup

Miner onboarding is currently performed by KubeTEE. Supply your cluster hardware (minimum 8 nodes, Intel TDX-capable, 8-GPU workers) and register with the subnet owner. KubeTEE applies the kubetee.ai/hotkey binding label linking the cluster to the miner's registered hotkey.

# Contact KubeTEE to initiate onboarding (self-service not yet available in Phase 0)

Install RKE2 (FIPS-140-2 validated Kubernetes distribution) on all cluster nodes with FIPS mode enabled. FIPS must be enabled at OS installation time (fips=1 kernel parameter).

# Install RKE2 with FIPS mode enabled
echo 'fips: true
profile: cis-1.6
selinux: true
secrets-encryption: true' > /etc/rancher/rke2/config.yaml
systemctl enable rke2-server && systemctl start rke2-server

Deploy Kata Containers 4.1.0 with the kata-qemu-nvidia-gpu-tdx-runtime-rs and kata-qemu-tdx-runtime-rs runtime classes for TEE workloads. Apply KubeTEE overlays (ovmf-tdx-bump-202605, NVSwitch fix19, CSI fixes).

# Deploy Kata Containers 4.1.0 via kata-deploy
kubectl apply -f kata-rbac/base/kata-rbac.yaml
kubectl apply -k kata-deploy/overlays/rke2
kubectl apply -f kata-deploy/base/kata-deploy.yaml

Deploy the Confidential Containers operator and CoCo Trustee/KBS for remote attestation. Trustee verifies TDX quotes (Intel DCAP, PCCS collateral) and releases secrets only to attested guests with guest debug off.

# Deploy Confidential Containers operator and CoCo Trustee
kubectl apply -f confidential-containers-operator.yaml
# Configure CoCo Trustee (KBS) for attestation - provided by KubeTEE Fleet GitOps

Configure NVIDIA GPU Operator with Confidential Computing mode. Hopper GPUs require PPCIE mode with NVSwitch passthrough; Blackwell GPUs use cc.mode=on. 8 GPUs per worker node must be passed through to TEE pods.

# Enable NVIDIA GPU Operator and configure GPU passthrough for TEE
# Ensure BIOS TDX/SGX is enabled and kernel Intel TDX is loaded
# For Hopper (H100/H200): cc.mode=ppcie, nvidia.com/nvswitch: '4'
# For Blackwell (B200/B300): cc.mode=on

KubeTEE applies the kubetee.ai/hotkey label and enrolls the cluster into Rancher Fleet for GitOps-based infrastructure deployment. The cluster name follows the convention: <continent>-<country>-<city>-<miner-uid>.

# Register with Rancher Fleet for GitOps management (performed by KubeTEE)
# KubeTEE applies the hotkey binding label:
# kubectl label cluster <cluster-name> kubetee.ai/hotkey=<YOUR_HOTKEY_SS58>

Post the required 100 TAO deposit as on-chain registration collateral on the mining hotkey. Check current standing with: btcli query miner-collateral --netuid 90 --hotkey <hotkey> --coldkey <coldkey> --json

btcli tx add-collateral --netuid 90 --amount-alpha <amount> -w my_coldkey

Validator Setup

Clone the KubeTEE subnet repository and navigate to the validator directory.

git clone https://github.com/KubeTEE-AI/kubetee-subnet && cd kubetee-subnet/validator

Install Bittensor SDK (minimum 11.0.1 required for collateral support; 11.0.0 has no collateral support).

pip install bittensor>=11.0.1

Configure validator environment variables for collateral enforcement, pricing, and Hippius S3 publishing. The Taostats price feed and Targon payout feed are required for weight computation.

# Configure environment variables
export KUBETEE_COLLATERAL_REQUIRED_TAO=0       # Set to 100 for enforcement
export KUBETEE_COLLATERAL_ENFORCE=false          # Set to true after measuring compliance
export KUBETEE_COLLATERAL_GRACE_CYCLES=3
export KUBETEE_COLLATERAL_RECOVERY_MARGIN=0.05
export KUBETEE_TARGON_PRICE_FLOOR_FRAC=0.75
export KUBETEE_HIPPIUS_SECRET_KEY=<key>          # For publishing price snapshots (owner only)

Configure Rancher v3 API access credentials. The validator authenticates by signing a challenge with its Bittensor hotkey; the auth mechanism issues a narrow cluster/node-read role.

# Set Rancher API credentials for cluster inventory
export RANCHER_URL=https://<rancher-endpoint>
export RANCHER_TOKEN=<rancher-api-token>

Run the validator as a container (12 Python modules, 35 tests). Sets weights once per epoch with weights_rate_limit cooldown. Optionally run inside a Kata+CoCo TEE pod on the control plane for attestation of the validator itself.

# Run the validator container
docker run -d --name kubetee-validator \
  -e RANCHER_URL=$RANCHER_URL \
  -e RANCHER_TOKEN=$RANCHER_TOKEN \
  -e KUBETEE_COLLATERAL_REQUIRED_TAO=$KUBETEE_COLLATERAL_REQUIRED_TAO \
  kubetee-ai/validator:latest

Monitor the KubeTEE Validator/Miners Dashboard published to Hippius S3 each cycle. Check miner collateral coverage via kubetee_miner_collateral_coverage and kubetee_miner_collateral_status Prometheus metrics.(optional)

# Monitor validator dashboard
# Published each cycle to: https://s3.hippius.com/kubetee-validator/index.html

KubeTEE

Registration Activity

KubeTEE

Subnet Configuration

Market & Economics

For Miners

For Validators & Stakers

Registration Activity

What does KubeTEE do?

Use Cases

About

Subnet History

Frequently Asked Questions

Development Activity

Recent Activity

Hardware Requirements

What does KubeTEE do?

Use Cases

About

Subnet History

Frequently Asked Questions

Subnet Configuration

Market & Economics

For Miners

For Validators & Stakers

Hardware Requirements

Miner Setup

Validator Setup

Overview

Registration Activity

Participate

Overview

Subnet Configuration

Market & Economics

For Miners

For Validators & Stakers

Registration Activity

What does KubeTEE do?

Use Cases

About

Subnet History

Frequently Asked Questions

Participate

Development Activity

Recent Activity

Hardware Requirements

What does KubeTEE do?

Use Cases

About

Subnet History

Frequently Asked Questions

Subnet Configuration

Market & Economics

For Miners

For Validators & Stakers

Hardware Requirements

Miner Setup

Validator Setup