> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/portkey-AI/gateway/llms.txt
> Use this file to discover all available pages before exploring further.
# Automatic Retries
> Automatically retry failed requests with exponential backoff
## Overview
Automatic retries increase reliability by re-attempting failed requests without manual intervention. The Gateway implements intelligent retry logic with exponential backoff to handle transient errors gracefully.
## How It Works
When a request fails with a retryable status code, the Gateway automatically:
1. Waits for a calculated backoff period
2. Re-attempts the request
3. Increases backoff time exponentially after each failure
4. Returns the response once successful or after exhausting retry attempts
Retries use exponential backoff to prevent overwhelming providers during outages. The backoff strategy spaces out retry attempts intelligently.
## Configuration
### Basic Retry
Retry up to 3 times on default error codes:
```json theme={null}
{
"retry": {
"attempts": 3
}
}
```
Default retryable status codes: `[429, 500, 502, 503, 504]`
### Custom Status Codes
Retry on specific status codes only:
```json theme={null}
{
"retry": {
"attempts": 5,
"on_status_codes": [429, 503]
}
}
```
### Provider Retry Headers
Respect provider-specified retry delays:
```json theme={null}
{
"retry": {
"attempts": 3,
"use_retry_after_header": true
}
}
```
When enabled, the Gateway respects `retry-after`, `retry-after-ms`, and `x-ms-retry-after-ms` headers from providers.
The maximum retry timeout is 60 seconds. If a provider requests a longer delay, the Gateway will skip retries and return the error.
## Usage Examples
```python Python theme={null}
from portkey_ai import Portkey
client = Portkey(
api_key="PORTKEY_API_KEY",
provider="openai",
Authorization="sk-***",
config={
"retry": {
"attempts": 5,
"on_status_codes": [429, 500, 502, 503, 504]
}
}
)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "What's the weather like?"}]
)
```
```javascript JavaScript theme={null}
import { Portkey } from 'portkey-ai';
const client = new Portkey({
apiKey: 'PORTKEY_API_KEY',
provider: 'openai',
Authorization: 'sk-***',
config: {
retry: {
attempts: 5,
on_status_codes: [429, 500, 502, 503, 504]
}
}
});
const response = await client.chat.completions.create({
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: "What's the weather like?" }]
});
```
```bash cURL theme={null}
curl https://api.portkey.ai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-provider: openai" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "x-portkey-config: '{\"retry\":{\"attempts\":3,\"on_status_codes\":[429,500,502,503,504]}}' " \
-d '{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "What is AI?"}]
}'
```
## Implementation Details
### Exponential Backoff
The Gateway implements exponential backoff using the `async-retry` library:
```typescript theme={null}
// From src/handlers/retryHandler.ts
await retry(
async (bail, attempt) => {
const response = await fetch(url, options);
if (statusCodesToRetry.includes(response.status)) {
throw new Error('Retry needed');
}
return response;
},
{
retries: retryCount,
randomize: false
}
);
```
Backoff timing increases exponentially with each attempt, preventing server overload during outages.
### Provider Retry-After Headers
When `use_retry_after_header` is enabled, the Gateway checks for:
* `retry-after` (seconds)
* `retry-after-ms` (milliseconds)
* `x-ms-retry-after-ms` (milliseconds, Azure-specific)
If the specified delay exceeds 60 seconds or remaining retry budget, retries are skipped.
### Retry Limits
* **Maximum attempts**: 5
* **Maximum retry window**: 60 seconds total
* **Timeout per request**: Configurable via `request_timeout`
If a request times out (408 status), it counts toward the retry attempts. Configure both retries and timeouts appropriately.
## Response Headers
Track retry behavior through response headers:
```
x-portkey-retry-attempt-count: 2
```
This header indicates the number of retry attempts made before the request succeeded (0 means first attempt succeeded).
## Advanced Patterns
### Retries with Fallbacks
Combine retries with fallback providers:
```json theme={null}
{
"retry": {
"attempts": 3,
"on_status_codes": [429, 500, 502, 503, 504]
},
"strategy": {
"mode": "fallback"
},
"targets": [
{
"provider": "openai",
"api_key": "sk-***"
},
{
"provider": "anthropic",
"api_key": "sk-ant-***"
}
]
}
```
Behavior:
1. Try OpenAI
2. Retry up to 3 times with OpenAI
3. If all retries fail, fallback to Anthropic
4. Retry up to 3 times with Anthropic
### Per-Target Retry Configuration
Different retry strategies for different providers:
```json theme={null}
{
"strategy": { "mode": "fallback" },
"targets": [
{
"provider": "openai",
"api_key": "sk-***",
"retry": {
"attempts": 5,
"on_status_codes": [429]
}
},
{
"provider": "anthropic",
"api_key": "sk-ant-***",
"retry": {
"attempts": 2,
"on_status_codes": [503]
}
}
]
}
```
### Rate Limit Handling
Special configuration for rate limits:
```json theme={null}
{
"retry": {
"attempts": 5,
"on_status_codes": [429],
"use_retry_after_header": true
}
}
```
This configuration:
* Retries only on 429 (rate limit)
* Respects provider's `retry-after` header
* Optimal for handling rate limits gracefully
## Status Codes
### Default Retryable Codes
| Code | Meaning | Reason |
| ---- | --------------------- | ------------------------ |
| 429 | Too Many Requests | Rate limit exceeded |
| 500 | Internal Server Error | Temporary server issue |
| 502 | Bad Gateway | Upstream server error |
| 503 | Service Unavailable | Temporary unavailability |
| 504 | Gateway Timeout | Request timeout upstream |
### Non-Retryable Codes
| Code | Meaning | Why Not Retry |
| ---- | --------------- | ------------------------------ |
| 400 | Bad Request | Invalid request format |
| 401 | Unauthorized | Invalid credentials |
| 403 | Forbidden | No permission |
| 404 | Not Found | Resource doesn't exist |
| 408 | Request Timeout | Gateway timeout (configurable) |
408 (Request Timeout) is thrown by the Gateway when `request_timeout` is exceeded. This is already in OpenAI format and won't be retried by default.
## Best Practices
Balance reliability with latency. More retries increase success rate but add latency. For user-facing applications, 2-3 attempts is typically sufficient.
Retry only on codes that indicate transient errors. Don't retry on 400-level errors (except 429) as they indicate client errors that won't resolve with retries.
When dealing with rate limits, enable `use_retry_after_header` to respect provider retry guidance and avoid unnecessary retries.
Always set `request_timeout` when using retries to prevent indefinite waiting on slow requests.
Track retry counts in your logs to identify reliability issues with providers. High retry rates may indicate capacity problems.
## Related Features
Switch to backup providers when primary fails
Set maximum request duration
Distribute load across providers
Complete config reference