AIR

Fallback, health, and recovery

Which errors trigger fallback

air.routing.fallback.FALLBACK_ERRORS is the tuple of exception types that AIR catches and falls back from:

python
FALLBACK_ERRORS = (QuotaExceededError, RateLimitError, ProviderTimeoutError, ProviderAPIError)

If a provider's chat(...) raises one of these during priority or intelligent routing, AIR records the failure, emits a provider_switch event (if another provider is next in line), and tries the next provider in the ordered list.

AuthenticationError and ProviderResponseError are not in this tuple. If a provider raises either of these, or any other exception, AIR does not try another provider. The exception propagates directly to the caller, on the reasoning that these usually indicate a configuration or integration problem rather than a transient provider outage.

This same rule applies to the explicit provider selection path, except there is no next provider to fall back to regardless: any exception from the directly named provider propagates unchanged.

Health states

air.routing.health.HealthStatus is a string enum:

  • available - the provider's most recent call succeeded.
  • rate_limited - the provider's most recent call raised RateLimitError.
  • exhausted - the provider's most recent call raised QuotaExceededError.
  • temporarily_unavailable - the provider's most recent call raised ProviderTimeoutError or ProviderAPIError.
  • unknown - the provider has never been called yet (its initial state).

Read a provider's current health with air.health_of(name), which returns a ProviderHealth(status, reason, retry_after). reason is the string form of the exception that produced the current status, or None if the status is available or unknown. retry_after is always None in the current implementation; no provider adapter currently reports a retry-after time, so AIR has no automatic cooldown timer.

What happens on failure

Each fallback eligible failure updates that provider's health to the corresponding status above. A health status change (compared to the previous status) also emits a structured event: provider_exhausted for a transition into exhausted, or a generic health_change typed event for the other transitions. See features/structured-events.md.

What happens on success

A successful call always resets that provider's health to available. If the previous status was not already available, this emits a provider_recovered event, and if the previous status was specifically exhausted, it additionally emits a capacity_restored event.

Recovery behavior, and its limitation

rate_limited and temporarily_unavailable providers are not skipped ahead of time. AIR will simply try them again on the next request that reaches them in the routing order, and a success immediately restores available status.

exhausted is different: AIR actively skips a provider with exhausted status before ever calling it again, in both priority and intelligent routing (see core/routing.md). Because retry_after is never set, AIR has no way to know when an exhausted provider's quota resets, and will keep skipping it indefinitely in automatic routing.

The only way to recover an exhausted provider is to call it directly, bypassing the skip: set provider= on a ChatRequest passed to air.chat(...), or pass provider= to Session.chat(...). A successful direct call resets that provider's health to available (and emits provider_recovered / capacity_restored), after which normal automatic routing will consider it again.

All providers exhausted

If every provider in the ordered list either fails with a fallback eligible error or is skipped, AIR raises ProvidersExhaustedError and emits an all_exhausted event. ProvidersExhaustedError.attempts is a list of (provider_name, exception) tuples for every provider that was tried or skipped during that call, in order.