> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/portkey-AI/gateway/llms.txt
> Use this file to discover all available pages before exploring further.
# Anyscale
> Access open-source models including Llama, Mixtral, and Mistral through Anyscale Endpoints
## Overview
Anyscale Endpoints provides serverless access to popular open-source models built on Ray, offering fast inference, competitive pricing, and easy scaling. Perfect for production deployments of Llama, Mixtral, and other open models.
**Base URL:** `https://api.endpoints.anyscale.com/v1`
## Supported Features
* ✅ Chat Completions
* ✅ Completions
* ✅ Streaming
* ✅ Embeddings
* ✅ Function Calling (select models)
* ❌ Vision
* ❌ Image Generation
* ❌ Fine-tuning
## Quick Start
### Chat Completions
```python Python theme={null}
from portkey_ai import Portkey
client = Portkey(
provider="anyscale",
Authorization="***" # Your Anyscale API key
)
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-70B-Instruct",
messages=[
{"role": "user", "content": "Explain Anyscale Endpoints"}
]
)
print(response.choices[0].message.content)
```
```javascript JavaScript theme={null}
import Portkey from 'portkey-ai';
const client = new Portkey({
provider: "anyscale",
Authorization: "***" // Your Anyscale API key
});
const response = await client.chat.completions.create({
model: "meta-llama/Meta-Llama-3.1-70B-Instruct",
messages: [
{role: "user", content: "Explain Anyscale Endpoints"}
]
});
console.log(response.choices[0].message.content);
```
```bash cURL theme={null}
curl http://localhost:8787/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-portkey-provider: anyscale" \
-H "Authorization: Bearer ***" \
-d '{
"model": "meta-llama/Meta-Llama-3.1-70B-Instruct",
"messages": [
{"role": "user", "content": "Explain Anyscale Endpoints"}
]
}'
```
### Streaming
```python theme={null}
stream = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-70B-Instruct",
messages=[{"role": "user", "content": "Tell me a story"}],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
## Available Models
### Meta Llama
| Model | Context | Description |
| ------------------------------------------ | ------- | ------------------ |
| `meta-llama/Meta-Llama-3.1-405B-Instruct` | 128K | Largest Llama 3.1 |
| `meta-llama/Meta-Llama-3.1-70B-Instruct` | 128K | Efficient, capable |
| `meta-llama/Meta-Llama-3.1-8B-Instruct` | 128K | Fast, compact |
| `meta-llama/Llama-3.2-90B-Vision-Instruct` | 128K | Vision-enabled |
| `meta-llama/Llama-3.2-11B-Vision-Instruct` | 128K | Smaller vision |
### Mistral AI
| Model | Context | Description |
| --------------------------------------- | ------- | ------------- |
| `mistralai/Mixtral-8x22B-Instruct-v0.1` | 64K | Large MoE |
| `mistralai/Mixtral-8x7B-Instruct-v0.1` | 32K | Efficient MoE |
| `mistralai/Mistral-7B-Instruct-v0.1` | 32K | Compact |
### Google Gemma
| Model | Context | Description |
| ----------------------- | ------- | ------------ |
| `google/gemma-2-27b-it` | 8K | Latest Gemma |
| `google/gemma-2-9b-it` | 8K | Efficient |
### Qwen
| Model | Context | Description |
| --------------------------- | ------- | ----------- |
| `Qwen/Qwen2.5-72B-Instruct` | 32K | Latest Qwen |
| `Qwen/Qwen2.5-7B-Instruct` | 32K | Compact |
### Embeddings
| Model | Dimensions | Description |
| ------------------------ | ---------- | ----------------------- |
| `thenlper/gte-large` | 1024 | High-quality embeddings |
| `BAAI/bge-large-en-v1.5` | 1024 | Popular choice |
**Anyscale excels at:**
* **Production-ready** - Built for scale on Ray
* **Fast inference** - Optimized serving
* **Cost-effective** - Competitive pricing
* **Open models** - Popular OSS models
* **Easy scaling** - Serverless architecture
## Configuration Options
```python theme={null}
client = Portkey(
provider="anyscale",
Authorization="***" # Bearer token
)
```
| Header | Description | Required |
| --------------- | ---------------- | -------- |
| `Authorization` | Anyscale API key | Yes |
## Advanced Features
### System Messages
```python theme={null}
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-70B-Instruct",
messages=[
{
"role": "system",
"content": "You are a helpful AI assistant specialized in Python."
},
{
"role": "user",
"content": "How do I use async/await?"
}
]
)
```
### Temperature Control
```python theme={null}
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-70B-Instruct",
messages=[{"role": "user", "content": "Write creative content"}],
temperature=0.9, # Higher for creativity
max_tokens=500
)
```
### Embeddings
```python theme={null}
response = client.embeddings.create(
model="thenlper/gte-large",
input="Anyscale provides serverless inference for open models"
)
embedding = response.data[0].embedding
print(f"Dimensions: {len(embedding)}")
```
Batch embeddings:
```python theme={null}
response = client.embeddings.create(
model="thenlper/gte-large",
input=[
"First document",
"Second document",
"Third document"
]
)
for i, item in enumerate(response.data):
print(f"Document {i}: {len(item.embedding)} dimensions")
```
### Completions API
```python theme={null}
response = client.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
prompt="The future of AI is",
max_tokens=100
)
print(response.choices[0].text)
```
## Fallback Configuration
Fallback to Together AI:
```python theme={null}
config = {
"strategy": {"mode": "fallback"},
"targets": [
{
"provider": "anyscale",
"api_key": "***",
"override_params": {"model": "meta-llama/Meta-Llama-3.1-70B-Instruct"}
},
{
"provider": "together-ai",
"api_key": "***",
"override_params": {"model": "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo"}
}
]
}
client = Portkey().with_options(config=config)
```
## Load Balancing
Balance across different models:
```python theme={null}
config = {
"strategy": {"mode": "loadbalance"},
"targets": [
{
"provider": "anyscale",
"api_key": "***",
"override_params": {"model": "meta-llama/Meta-Llama-3.1-70B-Instruct"},
"weight": 0.7
},
{
"provider": "anyscale",
"api_key": "***",
"override_params": {"model": "meta-llama/Meta-Llama-3.1-8B-Instruct"},
"weight": 0.3
}
]
}
client = Portkey().with_options(config=config)
```
## Error Handling
```python theme={null}
from portkey_ai.exceptions import (
RateLimitError,
APIError,
AuthenticationError
)
try:
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-70B-Instruct",
messages=[{"role": "user", "content": "Hello"}]
)
except RateLimitError as e:
print(f"Rate limit: {e}")
except AuthenticationError as e:
print(f"Invalid API key: {e}")
except APIError as e:
print(f"API error: {e}")
```
## Best Practices
1. **Start with 70B** - Best balance of speed and quality
2. **Use 8B for volume** - Cost-effective for simple tasks
3. **Enable streaming** - Better user experience
4. **Set appropriate max\_tokens** - Control costs and latency
5. **Use system prompts** - Guide model behavior
6. **Implement retry logic** - Handle transient failures
7. **Monitor usage** - Track costs and performance
8. **Cache responses** - Reduce redundant calls
## Ray Integration
Anyscale Endpoints is built on Ray, providing:
* **Automatic scaling** based on demand
* **Efficient resource utilization** across clusters
* **Fast cold starts** with model caching
* **High availability** with redundancy
## Pricing
Anyscale offers competitive pricing for open models:
View detailed pricing for all Anyscale models
## Getting Started
1. Sign up at [Anyscale Endpoints](https://app.endpoints.anyscale.com/)
2. Get your API key
3. Start making requests
## Related Resources
Alternative open models platform
Ultra-fast inference
Balance across providers
Fallback configurations