Kubernetes Deployment

Deploy Portkey AI Gateway to Kubernetes clusters

Deploy the Portkey AI Gateway to Kubernetes for production-grade orchestration, scaling, and high availability.

Quick Deployment

Deploy using the official Kubernetes manifest:

1. Download the manifest

   Download the deployment manifest from the repository:

   wget https://raw.githubusercontent.com/Portkey-AI/gateway/main/deployment.yaml
   
2. Apply the manifest

   Deploy to your Kubernetes cluster:

   kubectl apply -f deployment.yaml
   
3. Verify deployment

   Check the deployment status:

   kubectl get pods -n portkeyai
   kubectl get svc -n portkeyai
   

Kubernetes Manifest

The complete deployment configuration:

deployment.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: portkeyai
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: portkeyai
  namespace: portkeyai
spec:
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: portkeyai
      version: v1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: portkeyai
        version: v1
    spec:
      containers:
      - image: portkeyai/gateway
        imagePullPolicy: IfNotPresent
        name: portkeyai
        ports:
        - containerPort: 8787
          protocol: TCP
        resources: {}
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
---
apiVersion: v1
kind: Service
metadata:
  name: portkeyai
  namespace: portkeyai
spec:
  ports:
  - port: 8787
    protocol: TCP
    targetPort: 8787
  selector:
    app: portkeyai
    version: v1
  sessionAffinity: None
  type: NodePort

Production Configuration

Resource Limits

Add resource requests and limits for production:

spec:
  template:
    spec:
      containers:
      - name: portkeyai
        image: portkeyai/gateway
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"

Health Checks

Add liveness and readiness probes:

spec:
  template:
    spec:
      containers:
      - name: portkeyai
        image: portkeyai/gateway
        livenessProbe:
          httpGet:
            path: /health
            port: 8787
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /health
            port: 8787
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3

Environment Variables

Add configuration via environment variables:

spec:
  template:
    spec:
      containers:
      - name: portkeyai
        image: portkeyai/gateway
        env:
        - name: LOG_LEVEL
          value: "info"
        - name: NODE_ENV
          value: "production"

Secrets Management

Store sensitive data in Kubernetes secrets:

# Create a secret
kubectl create secret generic portkey-secrets \
  --from-literal=api-key=your-api-key \
  -n portkeyai

Reference in deployment:

spec:
  template:
    spec:
      containers:
      - name: portkeyai
        image: portkeyai/gateway
        env:
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: portkey-secrets
              key: api-key

Scaling

Horizontal Pod Autoscaling

Automatically scale based on CPU/memory:

hpa.yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: portkeyai-hpa
  namespace: portkeyai
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: portkeyai
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Apply the HPA:

kubectl apply -f hpa.yaml

Manual Scaling

Scale replicas manually:

# Scale to 3 replicas
kubectl scale deployment portkeyai -n portkeyai --replicas=3

# Verify scaling
kubectl get pods -n portkeyai

Ingress Configuration

Nginx Ingress

Expose the gateway via Nginx Ingress:

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: portkeyai-ingress
  namespace: portkeyai
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - gateway.yourdomain.com
    secretName: portkeyai-tls
  rules:
  - host: gateway.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: portkeyai
            port:
              number: 8787

Traefik Ingress

For Traefik users:

ingress-traefik.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: portkeyai-ingress
  namespace: portkeyai
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  ingressClassName: traefik
  rules:
  - host: gateway.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: portkeyai
            port:
              number: 8787

Storage

ConfigMap for Configuration

Store configuration files:

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: portkeyai-config
  namespace: portkeyai
data:
  conf.json: |
    {
      "logLevel": "info",
      "cors": {
        "enabled": true,
        "origins": ["*"]
      }
    }

Mount in deployment:

spec:
  template:
    spec:
      containers:
      - name: portkeyai
        image: portkeyai/gateway
        volumeMounts:
        - name: config
          mountPath: /app/conf.json
          subPath: conf.json
      volumes:
      - name: config
        configMap:
          name: portkeyai-config

Persistent Storage

Add persistent volume for logs:

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: portkeyai-logs
  namespace: portkeyai
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Monitoring

Prometheus Metrics

Add Prometheus annotations:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8787"
    prometheus.io/path: "/metrics"

Service Monitor

For Prometheus Operator:

servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: portkeyai
  namespace: portkeyai
spec:
  selector:
    matchLabels:
      app: portkeyai
  endpoints:
  - port: http
    interval: 30s
    path: /metrics

Helm Chart

Create a Helm chart for easier management:

values.yaml

values.yaml

replicaCount: 2

image:
  repository: portkeyai/gateway
  pullPolicy: IfNotPresent
  tag: "latest"

service:
  type: ClusterIP
  port: 8787

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: gateway.yourdomain.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: portkeyai-tls
      hosts:
        - gateway.yourdomain.com

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80

Cluster Management

View Logs

# View logs from all pods
kubectl logs -n portkeyai -l app=portkeyai --tail=100 -f

# View logs from specific pod
kubectl logs -n portkeyai <pod-name> -f

Execute Commands

# Get a shell in a pod
kubectl exec -it -n portkeyai <pod-name> -- /bin/sh

# Run a command
kubectl exec -n portkeyai <pod-name> -- node --version

Port Forwarding

Access the gateway locally:

kubectl port-forward -n portkeyai svc/portkeyai 8787:8787

Updates and Rollbacks

Update Deployment

# Update image
kubectl set image deployment/portkeyai -n portkeyai \
  portkeyai=portkeyai/gateway:v2.0.0

# Watch rollout
kubectl rollout status deployment/portkeyai -n portkeyai

Rollback

# View rollout history
kubectl rollout history deployment/portkeyai -n portkeyai

# Rollback to previous version
kubectl rollout undo deployment/portkeyai -n portkeyai

# Rollback to specific revision
kubectl rollout undo deployment/portkeyai -n portkeyai --to-revision=2

Cleanup

Remove the deployment:

# Delete all resources
kubectl delete -f deployment.yaml

# Or delete namespace
kubectl delete namespace portkeyai

Next Steps

Helm Charts

Learn about Kubernetes Helm charts

Monitoring

Set up monitoring and observability