Cloud Run for GenAI APIs

Deploy serverless containers for scalable AI inference endpoints

What is Cloud Run?

Cloud Run is Google Cloud's fully managed serverless platform for deploying containerized applications. It automatically scales from zero to thousands of instances based on traffic, making it ideal for GenAI inference APIs that have variable load patterns—you only pay for actual request processing time.

Key Innovation: Cloud Run combines the flexibility of containers with serverless simplicity. Deploy any Docker image, get HTTPS endpoint automatically, and scale to zero when idle—perfect for cost-effective AI APIs.

Scale to Zero

Pay only when used

Auto HTTPS

TLS certificates free

1000+ instances

Auto-scaling

Any Language

Docker containers

How Cloud Run Works

1. Build

Create Docker image with your GenAI API

2. Deploy

Push to Cloud Run, get HTTPS URL

3. Scale

Automatic scaling based on requests

Deploy to Cloud Run

Quick Deploy (from Dockerfile)

# Build and deploy in one command
gcloud run deploy genai-api \
  --source . \
  --region us-central1 \
  --allow-unauthenticated

# Deploy from existing image
gcloud run deploy genai-api \
  --image gcr.io/my-project/genai-api:v1.0 \
  --region us-central1 \
  --memory 2Gi \
  --cpu 2 \
  --min-instances 1 \
  --max-instances 100 \
  --timeout 300 \
  --set-env-vars "MODEL_NAME=gpt-4,MAX_TOKENS=4096" \
  --set-secrets "OPENAI_API_KEY=openai-key:latest"

Deploy with Traffic Splitting (Canary)

# Deploy new revision without traffic
gcloud run deploy genai-api \
  --image gcr.io/my-project/genai-api:v2.0 \
  --region us-central1 \
  --tag canary \
  --no-traffic

# Send 10% traffic to canary
gcloud run services update-traffic genai-api \
  --region us-central1 \
  --to-tags canary=10

# Full rollout after validation
gcloud run services update-traffic genai-api \
  --region us-central1 \
  --to-latest

Configuration for GenAI Workloads

Memory & CPU

• Memory: 128Mi to 32Gi
• CPU: 1, 2, 4, 6, or 8 vCPUs
• GenAI APIs: Start with 2Gi/2 vCPU

Timeout & Concurrency

• Timeout: Up to 60 min (gen2)
• Concurrency: 1-1000 per instance
• LLM APIs: 80-100 concurrent

Scaling

• min-instances: Keep warm (cost)
• max-instances: Limit scaling
• Startup CPU boost: Faster cold start

Secrets

• Use Secret Manager for API keys
• Mount as env vars or files
• Auto-rotation supported

Deploy with Terraform

# cloud_run.tf
resource "google_cloud_run_v2_service" "genai_api" {
  name     = "genai-api"
  location = "us-central1"

  template {
    containers {
      image = "gcr.io/${var.project_id}/genai-api:${var.image_tag}"
      
      resources {
        limits = {
          cpu    = "2"
          memory = "4Gi"
        }
        cpu_idle = true  # Scale down CPU when idle
        startup_cpu_boost = true  # Faster cold starts
      }
      
      ports {
        container_port = 8080
      }
      
      env {
        name  = "MODEL_NAME"
        value = "gpt-4"
      }
      
      env {
        name = "OPENAI_API_KEY"
        value_source {
          secret_key_ref {
            secret  = google_secret_manager_secret.openai_key.secret_id
            version = "latest"
          }
        }
      }
    }
    
    scaling {
      min_instance_count = 1  # Keep warm
      max_instance_count = 100
    }
    
    timeout = "300s"
    max_instance_request_concurrency = 80
  }

  traffic {
    percent = 100
    type    = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
  }
}

# Make service publicly accessible
resource "google_cloud_run_v2_service_iam_member" "public" {
  location = google_cloud_run_v2_service.genai_api.location
  name     = google_cloud_run_v2_service.genai_api.name
  role     = "roles/run.invoker"
  member   = "allUsers"
}

output "service_url" {
  value = google_cloud_run_v2_service.genai_api.uri
}

Best Practices for GenAI APIs

Tip: Set min-instances=1 for production LLM APIs to avoid cold start latency on the first request.

Cold start optimization: Use min-instances, CPU boost, smaller images
Timeout: Set 60-300s for LLM calls that may take longer
Memory: 2-4Gi for typical inference, 8Gi+ for local models
Concurrency: 80-100 for I/O-bound LLM API calls
Secrets: Use Secret Manager, never env vars in image
Health checks: Implement /health endpoint for readiness
Logging: Structured JSON logs for Cloud Logging integration
Custom domain: Map your domain via Cloud Run domain mapping

Pricing Model

Cost Tip: Cloud Run charges per 100ms of CPU/memory usage. Scale to zero when not in use. Use cpu_idle=true to reduce costs when waiting for I/O.

Resource	Free Tier (monthly)	Price After
CPU	180,000 vCPU-seconds	$0.00002400/vCPU-second
Memory	360,000 GiB-seconds	$0.00000250/GiB-second
Requests	2 million	$0.40/million

Cloud Run vs Alternatives

Feature	Cloud Run	GKE	Cloud Functions
Containers	✅ Any Docker	✅ Any Docker	❌ Specific runtimes
Scale to Zero	✅ Yes	❌ Nodes always on	✅ Yes
Max Timeout	60 min (gen2)	Unlimited	9 min (gen1) / 60 min (gen2)
GPU Support	❌ Not yet	✅ Yes	❌ No
Best For	APIs, microservices	Complex workloads	Event-driven code

Learn More

Essential Resources

Test Your Knowledge

Score 8/10 or higher to pass