Rate limits

Every model has per-model throughput limits: a tokens-per-minute (TPM) limit and a requests-per-minute (RPM) limit, synced from the provider catalog. When your traffic exceeds a limit, Omnia returns 429.

Catalog numbers are the upstream provider’s advertised capacity, not a throughput guarantee on the shared gateway. In our own sustained-load measurements, real end-to-end throughput tops out well below the largest catalog figures (on the order of ~300 requests/minute sustained per workspace, as of August 2026); latency, upstream queuing, and shared capacity all bind first. Size your integration against what you measure, not the catalog column. For sustained high volume, use a dedicated endpoint: the GPU is reserved for you, so its ceiling is yours.

Limits are per model, not a single global ceiling. Spreading load across models also adds headroom.

Handling 429s

When you receive a 429, back off and retry with exponential backoff and jitter. Jitter spreads retries out over time so a fleet of clients doesn’t all retry in lockstep and stampede the limit again.

Catch the 429

Detect the 429 status (or a RateLimitError from your client library).

Wait, backing off exponentially

Sleep for an interval that doubles each attempt, capped at a ceiling (e.g. 30s), and add a small random jitter on top.

Retry, then give up

Retry up to a fixed number of attempts. If you’re still limited after the last attempt, surface the error rather than looping forever.

import time
import random

def with_backoff(fn, retries=5):
    for attempt in range(retries):
        try:
            return fn()
        except RateLimitError:  # your client's 429 exception
            if attempt == retries - 1:
                raise
            # Exponential backoff capped at 30s, plus jitter.
            time.sleep(min(30, 2 ** attempt) + random.random())

Because completions are only billed for tokens actually produced, a retried request that was rate-limited before it ran costs you nothing. Retrying 429s is safe on the wallet.

Reducing pressure

If you’re regularly hitting 429, reduce the load you put on a model’s limits rather than just retrying harder.

Batch embeddings

Send an array of inputs in a single embeddings request instead of one request per input. Fewer requests means less RPM pressure for the same work.

Cap output

Set max_tokens to what you actually need. Shorter outputs consume less of the TPM budget.

Limit concurrency

Cap how many requests you fire in parallel from a single workspace so you stay under the RPM limit instead of bursting past it.

Dedicated capacity

For sustained high volume, a dedicated endpoint gives you reserved throughput instead of sharing a shared model’s limits.

Checking headroom

You don’t have to wait for a 429 to know you’re approaching a limit. Rate-limit headroom, how close your traffic is running to a model’s limits, is visible in your observability. Each request’s detail in the request log shows its headroom, so you can watch how much of a model’s TPM and RPM budget you’re consuming and scale back (or move to a dedicated endpoint) before you start hitting limits.

Get started

Capture

Observe

Grade & Judge

Evaluate

Improve

Run at scale

Platform

Handling 429s

Reducing pressure

Batch embeddings

Cap output

Limit concurrency

Dedicated capacity

Checking headroom

​Handling 429s

​Reducing pressure

Batch embeddings

Cap output

Limit concurrency

Dedicated capacity

​Checking headroom

Handling 429s

Reducing pressure

Checking headroom