AI Operations & RAG Engineering — Phase 6
A source-linked, hands-on guide to grounded AI, retrieval, inference boundaries and safe automation. Source review: 11 October 2026. The lessons describe checked-in implementations and tests; they do not claim current public-cloud uptime or a comprehensive model-safety audit.
How to use this guide
Read the design, inspect the linked code, perform the offline exercise, and record which conclusions are implemented, tested with fakes, historically demonstrated, or merely proposed. Start with the system boundary before tuning models.
| Sequence | Lesson | What you should be able to demonstrate |
|---|---|---|
| 01 | System boundaries | Distinguish rules, retrieval, LLM generation, and execution |
| 02 | Ingestion and chunking | Explain ingestion, indexing and document identity |
| 03 | Hybrid retrieval and reranking | Trace the actual ranking stages and thresholds |
| 04 | Evidence and abstention | Identify unsupported answers and fail closed |
| 05 | Bedrock and cloud identity | Separate task IAM, network and application permissions |
| 06 | LangGraph and MCP | Follow tool routing without granting write access |
| 07 | Prompt injection and safety | Protect instructions, citations, tools and secrets |
| 08 | Evaluation and regression gates | Measure retrieval quality with reproducible cases |
| 09 | Telemetry, latency and cost | Define useful, privacy-safe AI telemetry |
| 10 | Safe automation case studies | Compare PlatformPilot, CloudOps, OpsPilot and résumé scoring |
| 11 | Capstone and interview review | Design and defend a bounded operational assistant |
Source repositories
- RAG Ops Assistant: FastAPI, LangGraph, MCP, hybrid retrieval, reranking, Bedrock, and ECS/Fargate artifacts.
- PlatformPilot: deterministic Kubernetes advice and authenticated export boundaries.
- CloudOps Command Center: deterministic decision traces, operational evidence and approval-gated state changes.
- OpsPilot SaaS: rule-based classification with optional schema-constrained model classification.
- AI Resume Analyzer: deterministic résumé evidence/score and separate Bedrock workflow.
Validation discipline
The accompanying validate-guide.py checks documentation links and fences; ai-labs/examples/evaluate_sample.py checks a synthetic, offline retrieval fixture. These are not substitutes for the RAG repository's integration tests, live Bedrock checks, abuse testing, or production observability.
Back to Chapter 10 — AI Operations or the handbook home.