AI Engineering: Build, Measure and Debug LLM Systems

Build retrieval, tools and agents, then prove they work with evals, traces and a failure catalog

What you’ll learn
Parse any model response as typed content blocks and branch on stop_reason rather than on the text
Request structured output against a schema and describe a tool with a strict input schema, then name the limits that schema cannot cross
Plan a call as a context budget across four line items and name the item that breaks first
Order a prompt so the cache hits, then prove the saving by reading the usage fields of a real response
Chunk documents by structure and detect chunks whose tail was silently truncated by the encoder
Predict which queries an embedding will never find, and merge lexical and semantic rankings with reciprocal rank fusion by hand
Build a labelled query set from real user questions and compute recall at k and nDCG on it before the generator runs
Diagnose a retrieval failure as a miss, a wrong order, or bad ingestion, from three numbers
Implement a tool loop with step, spend and time budgets, and state what happens when each ceiling is reached
Recognise a non progressing agent run from its call trace and install a repetition detector that breaks it
Compute the reliability of a chain of steps and justify choosing a single pass over an agent loop
Assemble an eval set from production traces, choose a grader family, and write a rubric two judges apply the same way
Calibrate an LLM judge against human labels and decide, from the agreement number, whether it can be trusted
Report a pass rate with its spread against a do nothing baseline and refuse a win when intervals overlap
Define a regression gate as a number and a threshold and say which changes it catches and which it lets through
Read a failed API response for its error class, next step and request id, and tell a throughput refusal from a spend cap refusal
Compute cost per successful task with failed attempts kept in the numerator, and pick a build on that number
Build a trace of an agent run and name where the time went and where the money went

Requirements
Comfortable writing and reading Python: functions, dictionaries, JSON, and a virtual environment
Comfortable reading API documentation and making HTTP requests from code
An API key for a hosted language model provider and a small budget for calls, a few dollars is enough
A terminal, an editor, and the ability to install packages
A corpus of your own to retrieve from: documentation, tickets, contracts, notes, anything real, even a few dozen files
No machine learning background, no linear algebra, no GPU, no framework experience required

Description

This course contains the use of artificial intelligence.
You already ship software. Now a feature in your product calls a language model, and the ordinary tools of your craft stop helping. The thing answers well on Monday and badly on Thursday, the bill moves without explanation, and when a user complains you have nothing to point at except a screenshot. This course closes that gap. By the end of it you can build an LLM feature that retrieves from your own documents, calls your own tools in a bounded loop, and returns an answer you are willing to put a name on. More importantly, you can prove it works: a labelled query set that scores retrieval before the generator ever runs, an eval set built from real traces, a judge you calibrated against human labels, a do nothing baseline that tells you whether your clever version earned its keep, and a pass rate reported with its spread rather than as a single flattering number. You finish with a capstone you built on your own material, not on a toy dataset: hybrid retrieval with a labelled query set, a tool loop with step and spend budgets, an eval suite with a calibrated judge and a baseline, tracing across the whole run, and a written failure catalog holding the verbatim error texts your system actually produced. That artifact is the thing you show in a review, in an incident postmortem, or in an interview.

This course is for a working engineer who already writes Python comfortably, reads API documentation without flinching, and has just been handed responsibility for an LLM feature that has to survive contact with real users. You do not need machine learning background, linear algebra, or a GPU. You need a terminal, an API key, and a real corpus of your own, even a small one. It is a good fit if you have a prototype that works in a notebook and no idea whether it works anywhere else, if your retrieval returns plausible but wrong pages and you cannot say why, if your agent loops and you cannot say where, or if someone asked you what the feature costs per successful task and you could only quote the price per call. It is a poor fit if you are looking for a tour of frameworks, if you want to be walked through eight portfolio applications without stopping to measure any of them, if you are hoping to train or fine tune models, or if you do not code and are looking for a conceptual overview. This course spends its time on the parts that fail quietly, and that requires you to run things and read numbers rather than watch.

Here is what makes this different from the rest of the category. Every mainstream course in this space teaches the happy path and stops at the moment the notebook prints something plausible. Retrieval is taught as load, chunk, embed, retrieve, with no measurement of whether the retrieval is any good, no comparison of chunking strategies, and no answer to the question of why the wrong passage came back. Agents are taught as a loop that works in the demo, with no budget, no repetition detector, and no reliability arithmetic. Evaluation, when it appears at all, is a slide. The only place where evaluation is treated as a first class engineering subject charges four figures and runs as a live cohort, which puts it out of reach for most of the people who need it. This course puts that material on demand, at a price you can expense. Every feature here ships with the thing the category leaves out. Retrieval ships with recall at k and nDCG computed on a labelled set before the generator is involved. Judges ship with a calibration step against human labels, because an uncalibrated judge is a random number generator with good manners. Results ship with a do nothing baseline and a spread, because a pass rate without a baseline is a number that means nothing. And every module ends with a diagnostic lesson that hands you a broken artifact and asks you to name the failure from the evidence.

The core runs six modules.Module one makes the model call itself trustworthy: content blocks instead of strings, branching on stop_reason instead of on text, structured outputs and strict tool schemas, streaming assembly, refusal and truncation as two distinct silent killers, and pricing a single request from its own usage fields. Module two treats context as a budget you plan rather than a wall you hit: context rot, ordering a prompt so the cache actually hits, proving the saving from usage fields, deciding what belongs in the system prompt and what belongs in retrieval or a tool, context editing, compaction, and a diagnostic on finding the exact token that broke your prefix.Module three is retrieval you can measure: structural chunking, encoder truncation, contextual retrieval, what an embedding keeps and throws away, lexical search for exact identifiers, hybrid merging with reciprocal rank fusion computed by hand, reranking depth, recall and nDCG on a labelled set you build yourself, retrieval triage, a timed drill on twelve failing cases, and programmatic citation verification. Module four is tools and the agent loop: the loop itself, feeding errors back so the model self corrects, writing tool descriptions the model reads correctly, step and spend and time budgets, repetition detection, idempotency for retried side effects, the tool server protocol and its limits, the context cost of your tool menu, the reliability arithmetic that tells you when not to use an agent at all, sub agents, and a drill on three broken loops. Module five is evaluation that predicts production: error analysis on real traces, building an eval set, choosing graders, writing a rubric a judge can apply, calibrating that judge, baselines, variance, regression gates, the disagreement between offline and online metrics, eval triage, and an end to end integration. Module six is diagnosis and operations: reading API errors and logging request ids, telling two kinds of rate limit apart and retrying correctly, cost per successful task, tracing with generative AI spans, the lethal trifecta and how to break one of its three legs, screening untrusted tool output, blast radius limits that hold when the model is wrong, and a full integration.

After the capstone there is an extra module on the frontier of the field, four lessons on material that no one needs in order to finish the course but that keeps coming up in real teams. Harness economics asks why the loop around the model costs more than the model, and compares two harnesses on cost per successful task. Stateless tool servers walks through what the protocol removed and how to migrate a server that assumed a session. Retrieval as navigation treats a corpus like a filesystem and gives you the criterion for choosing between an index and navigation. Multi agent pathologies names three failure shapes, turf wars, error amplification, and collapse, and asks you to spot them from a system description. These lessons are marked as bound to a moving target on purpose: they are taught as principles with current practice as the example, so that when the surface changes the reasoning still holds.

Last, what this course does not contain, so you can decide before you buy.There is no fine tuning and no QLoRA: that is a separate skill with separate hardware, and pretending otherwise would waste your time. There is no training or serving of your own models. There is no operation of distributed vector databases at scale, which is an infrastructure specialty rather than an application engineering one. There is no prompt engineering as a craft beyond what a working feature actually needs, because the returns fall off fast and the measurement matters more. There is no framework tour: the material is deliberately close to the underlying interfaces, so it transfers when your team changes its stack. And there is no claim that this covers the whole field. It covers the part that decides whether your feature survives production, and it covers it with numbers.

Who this course is for
Backend and full stack engineers who have just been made responsible for an LLM feature in a real product
Engineers with a working prototype who cannot yet say whether it works, how often, or at what cost
Data and platform engineers building retrieval over internal documents who need retrieval quality as a number
Engineers whose agent loops, budgets or tool calls fail in ways they cannot reproduce or explain
Tech leads who have to review, gate or sign off on someone else’s LLM feature and want a vocabulary for that review
Engineers preparing for interviews who want one measured artifact instead of eight unmeasured demos
And who this is NOT for: people who do not code and want a conceptual overview; learners who want a guided tour of frameworks and eight portfolio apps without measuring any of them; anyone hoping to fine tune, train or serve their own models; teams looking for vector database infrastructure operations at scale

ANGERFRLAIUDFCRAMDRFEGRDERBUDGE

you must be registered member to see linkes Register Now