AIR

Capacity tracking

AIR can track token usage against a limit you configure per provider, and estimate whether a provider has enough remaining capacity before sending it a request. AIR never invents a provider's real remaining quota; it only works from usage the provider actually reports, or from nothing at all if you do not configure a limit.

Configuring a limit

python
air.add_provider(
    gemini,
    priority=1,
    limits={"tokens": 100_000, "reserve": 5_000},
)

limits is an optional dict with:

  • tokens: the total token budget you are giving this provider.
  • reserve (default 0): an amount to hold back so automatic routing never spends the very last bit of configured capacity. Usable capacity is tokens - reserve.

If you do not pass limits, AIR has no configured budget for that provider and capacity is always unknown for it; it is never skipped for capacity reasons.

Usage accounting

air.usage_of(provider_name) returns UsageTotals(prompt_tokens, completion_tokens, total_tokens, exact). After each successful call, AIR adds Usage.prompt_tokens, completion_tokens, and total_tokens from the ChatResponse (if the provider reported them) to that provider's running totals. If a response does not include usage data, exact is set to False and stays False from then on for that provider, meaning further capacity calculations for it are only an estimate, not a precise count.

Capacity kinds

air.routing.capacity.CapacityKind is a string enum:

  • unknown: no limits configured for this provider.
  • known: limits configured and every successful call so far reported usage (exact is still True).
  • estimated: limits configured but at least one successful call did not report usage (exact became False).

Token estimation

estimate_message_tokens and estimate_tokens (in air.routing.capacity) estimate a request's size as ceil((total characters / 4) * 1.2), plus ChatRequest.max_tokens if set. This is a conservative character based approximation, not a real tokenizer for any specific model.

Preflight skip

Before calling a provider with limits configured, AIR computes its known or estimated remaining capacity (tokens - reserve - total_tokens used so far) and compares it to the estimated size of the current request. If remaining capacity is lower, AIR skips that provider without ever calling it, emits a capacity_skipped structured event, and moves to the next provider in the ordered list (or fails with ProvidersExhaustedError if it was the last one). See structured-events.md.

Usage threshold events

python
from air.config import AIRConfig
air = AIR(config=AIRConfig(usage_thresholds=[0.5, 0.8, 0.95]))

AIRConfig.usage_thresholds is a list of fractions (default [0.8]). Each threshold fires at most once per provider: the first time that provider's total_tokens / limits.tokens ratio reaches or passes a configured threshold, AIR emits a usage_threshold structured event for it and will not emit that same threshold again for that provider.

What this does not do

AIR does not query any provider for its actual remaining quota; none of the four built in providers expose such an endpoint. If you never configure limits for a provider, none of this section applies to it, and it is never skipped for capacity reasons.