Design for switching models, not for choosing one

What it solves

• One key instead of an account per vendor
• Switching models is a config value
• `:free` variants cost nothing, subject to daily rate limits
• Fallback to another model during an outage is straightforward

The minimum design

Do not hard-code the model name. Put it in configuration and switching becomes a deploy. Go one step further and assign models per task: cheap models for classification and extraction, the expensive one only for final generation. That alone often halves the bill.

The downsides

• A small amount of added latency
• If the router is down, every model is down
• New models can arrive later than at the source
• Provider-specific features may be unavailable

The second is the real production concern. Route through the gateway by default but keep a direct path to your primary provider as a fallback.

FAQ