The Platform Engineer's Path #2intermediate

AI CI/CD, Observability & Cost Optimization

AI apps break differently than traditional software. Learn prompt version control, LLM evaluation pipelines, tracing with LangSmith and Langfuse, hallucination monitoring, and cost optimization through caching and model routing. Ship an AI ops dashboard.

What You Build

A complete AI ops stack: prompt registry, evaluation CI pipeline, tracing dashboard, cost optimizer, and production monitoring

Sections

Lessons

Estimated

Concepts

Prerequisites

LLM Orchestration & RAG Systems

cicdobservabilitycost-optimizationllm-opsmonitoringtracingevaluation

Dr. Maya Chen

lecture · formal

First 5 lessons are free (38% of the course). No login required.

AI Operations Foundations

Why AI systems need different ops. Cost anatomy and the tool landscape.

Why AI Systems Break Differently

25 min Build in Public

Tweet: Tweet: 'Traditional CI/CD catches code bugs. AI CI/CD catches prompt regressions, hallucinations, and cost spikes. Totally different game.'

The AI Ops Landscape: Tools, Platforms, Patterns

25 min

Cost Anatomy of an LLM Call

30 min Ship

Ship: Spreadsheet model breaking down cost per query across providers

CI/CD for AI

Prompt versioning, LLM evaluation pipelines, and integration testing.

Prompt Version Control & Testing

30 min Ship

Ship: Prompt registry with version history and A/B test framework

LLM Evaluation Pipelines: Automated Quality Gates

35 min Ship

Ship: CI pipeline that runs LLM evaluations on every PR

Preview — content blurs halfway through

Integration Testing for AI Features

30 min

Unlock the rest of this course

$5 for full course or $25/mo for everything

See pricing

Observability & Monitoring

Tracing, hallucination detection, drift monitoring, and dashboards.

Tracing LLM Calls: LangSmith, Langfuse, Phoenix

35 min

Monitoring Hallucinations & Drift in Production

30 min

Dashboards: Latency, Token Usage, Error Rates

30 min

Cost Optimization

Token budgets, caching, model routing, and provider redundancy.

Token Budget Management & Caching Strategies

30 min

Model Routing: Cheap Models for Easy Tasks

30 min

Rate Limiting, Fallbacks & Provider Redundancy

25 min

Final Ship

Build and deploy a complete AI operations dashboard.

Ship an AI Ops Dashboard

50 min

Next Course

DevOps & Engineering Methodologies

From Docker containers to Kubernetes orchestration, GitHub Actions CI/CD to canary deployments. Plus the engineering practices that make teams fast: trunk-based development, feature flags, SLOs, and blameless post-mortems.

AI CI/CD, Observability & Cost Optimization

What You Build

A complete AI ops stack: prompt registry, evaluation CI pipeline, tracing dashboard, cost optimizer, and production monitoring