As organisations rush to embed AI into their products and operations, a critical question has emerged: what does it actually take to build reliable, production-grade AI systems on top of foundation models? AI Engineering: Building Applications with Foundation Models by Chip Huyen answers this question with rigour, clarity, and hard-won pragmatism. Published by O'Reilly, it is one of the most practically grounded books yet written on the discipline of building AI-powered applications — not training models from scratch, but engineering systems that sit on top of them.
This is not a book about prompt tricks or AI hype. It is a systems-level treatment of what Huyen calls "AI engineering" — a distinct discipline from traditional ML engineering — aimed squarely at practitioners who need to ship working AI products into production, maintain them, and improve them over time.
Huyen's central argument is that the rise of foundation models has created an entirely new engineering discipline. Where traditional machine learning required deep expertise in model training, data pipelines, and statistical theory, AI engineering asks different questions: How do you evaluate model outputs at scale? How do you construct context effectively? How do you make informed decisions about when to fine-tune versus prompt versus retrieve?
The book frames AI engineering around three core challenges: evaluation, context construction, and deployment. Of these, evaluation is positioned as the most important and most underserved. Huyen argues — convincingly — that the teams who will win are not those with the best models, but those who build the best feedback loops between production behaviour and improvement. Without rigorous evaluation, you cannot know whether your system is improving, degrading, or simply behaving unpredictably.
Throughout, the book maintains a clear-eyed view of the tradeoffs involved. Building on foundation models introduces new failure modes — hallucination, latency, cost sensitivity, and non-determinism — that traditional software engineering practices do not adequately address. Huyen's contribution is to name these challenges precisely and give practitioners a coherent framework for reasoning about them.
Huyen devotes substantial attention to the problem of evaluating AI outputs — and rightly so. Unlike traditional software where correctness can often be asserted programmatically, AI system quality is inherently probabilistic and context-dependent. The book introduces a layered evaluation strategy: from automated metrics and model-based evaluation through to human review and A/B testing in production.
The key insight is that good evals are not a nice-to-have; they are the engineering foundation everything else depends on. Without them, you cannot safely iterate, you cannot confidently deploy, and you cannot demonstrate improvement to stakeholders.
Before writing a single line of application code, define your evaluation strategy. Build a representative test set, agree on what "good" looks like across multiple dimensions (correctness, tone, safety, latency), and automate evaluation in your CI/CD pipeline. Treat eval infrastructure as a first-class engineering investment.
Retrieval-Augmented Generation (RAG) and prompt engineering are not afterthoughts — they are the primary levers available to most AI engineering teams. Huyen provides a rigorous treatment of context construction: how to select, structure, and inject the right information into a model's context window to improve response quality.
The book distinguishes between different retrieval strategies, chunking approaches, embedding models, and reranking techniques, while being honest about the complexity involved. Getting RAG right is an engineering problem in its own right, requiring careful attention to data quality, retrieval relevance, and context window management.
Invest in your data foundations before your model integrations. High-quality, well-structured internal knowledge is the raw material for effective RAG. Map your data sources, assess their freshness and accuracy, and build retrieval infrastructure that can evolve as your context requirements grow.
One of the book's most practically valuable contributions is a clear framework for deciding when to fine-tune a model versus rely on prompting or retrieval. Fine-tuning is expensive, requires labelled data, and creates a model you now own and must maintain. Prompting and RAG are faster and cheaper but have limits — especially for tasks requiring deep domain adaptation or consistent behavioural constraints.
Huyen's heuristic is to start with prompting, add retrieval when context volume demands it, and consider fine-tuning only when you have a clear evidence base showing it will solve a specific, measurable problem that other approaches cannot.
Create a decision log for model adaptation choices. For each use case, document what you tried, what you measured, and why you progressed (or didn't) to fine-tuning. This disciplines the team to justify the investment and builds institutional knowledge about what actually works in your context.
The non-determinism of LLM outputs demands a rethink of observability and incident management. Traditional logging and alerting assume that given the same inputs, you get the same outputs. AI systems break this assumption, which means you need new patterns: logging full traces including prompts and responses, tracking behavioural drift over time, and building dashboards that surface quality degradation rather than just infrastructure failures.
Huyen also addresses the economics of AI operations — model inference costs, latency budgets, and the impact of context window size on cost — making the case that operational efficiency is a product concern, not just an infrastructure one.
Instrument your AI systems from day one with structured logging of every model interaction. Capture inputs, outputs, latency, token counts, and user feedback signals. Build dashboards that track quality trends, not just uptime. Treat a degradation in output quality as seriously as you would a service outage.
The book's treatment of agentic AI — systems where models take sequences of actions, invoke tools, and operate with greater autonomy — is measured and appropriately cautious. Huyen acknowledges the excitement around agents while being clear about the engineering challenges: error compounding across steps, difficulty of evaluation, and the risks of autonomous action in production systems.
The key principle is to match the level of autonomy to the maturity of your evaluation and oversight infrastructure. Agentic systems are not a shortcut to capability — they are a multiplier of both value and risk.
Before building agentic systems, ensure you have robust evaluation for each individual capability the agent will use. Build human-in-the-loop checkpoints for high-stakes actions, and introduce autonomy incrementally as confidence in system behaviour grows. Avoid the temptation to grant full autonomy before you have the observability to justify it.
AI Engineering sits naturally alongside Accelerate and the DORA research tradition. The emphasis on feedback loops, evaluation-driven iteration, and deployment as a continuous practice maps directly onto the principles of high-performing software delivery teams. Huyen's insistence on measuring quality in production echoes the DORA finding that elite teams invest in observability and fast feedback — the medium has changed, but the discipline has not.
The book also connects strongly to the AI & Data Foundations capability area, particularly the standard around internal data being structured, governed, and accessible for AI use. Huyen makes clear that context quality is bounded by data quality: no amount of prompt engineering compensates for fragmented, undocumented, or unreliable internal knowledge. Teams investing in data governance are, in effect, investing in the ceiling of their AI capability.
AI Engineering is essential reading for software engineers, platform engineers, and engineering leaders who are building or planning to build AI-powered products on top of foundation models. It assumes a baseline of software engineering competence but does not require deep ML expertise — making it unusually accessible for generalist engineers entering the AI space. Technical architects evaluating AI adoption strategies will find the decision frameworks immediately applicable. It is less suited to data scientists focused on model training, or to executives looking for a strategic overview rather than a technical foundation.
AI Engineering is the book the industry needed. At a moment when most AI content oscillates between uncritical enthusiasm and vague caution, Chip Huyen has written something genuinely useful: a rigorous, practitioner-grade treatment of what it takes to build AI systems that work reliably in production. The writing is precise, the frameworks are actionable, and the honest acknowledgement of complexity and uncertainty throughout is refreshing.
Its primary limitation is one of timing — the field is moving quickly, and some specifics around tooling and model capabilities will date. But the underlying engineering principles — evaluate rigorously, construct context carefully, observe production behaviour, and increase autonomy incrementally — are durable. This belongs on the shelf of every engineering team seriously engaging with foundation models.