AI System Design: LLM Gateways, RAG & Agent Platforms

AI System Design: LLM Gateways, RAG & Agent Platforms

Design, prove and defend production LLM platforms: gateways, retrieval, agents, evals, security, reliability and cost

What you’ll learn
Decide whether a use case needs an LLM, and turn it into requirements with quality, latency, cost and data targets
Estimate tokens, throughput and cost per outcome for an AI feature, with every number traced to an assumption
Design a multi-tenant AI gateway with quotas, policy-based routing, retry budgets, fallback and tenant-safe caching
Run retrieval as a data pipeline: versioned embeddings, freshness SLOs, deletes that propagate, permission-aware search
Design an agent platform with a governed tool registry, delegated authority, durable execution, approvals and audit
Gate every prompt, model, index and policy change behind evaluations with a calibrated judge
Threat-model an LLM system, map the OWASP risks to controls, and design residency for the EU, the Gulf and India
Keep an AI feature within its SLOs and cost cap through a provider outage and a tenfold traffic spike
Build a cost model per outcome, attribute spend per tenant, and cut cost while proving quality held
Defend AI designs in reviews and interviews, with numbers, rejected options and decision records

Requirements
Several years of professional software engineering
Distributed-systems basics: caching, queues, load balancing, consistency and partitioning
Able to read Python code
Basic cloud and container concepts
No GPU and no paid AI API are needed: every lab runs on the Polaris simulator on your laptop

Description
AI Systems Architecture: Design, Scale, Secure, and Optimize LLM Platforms

Imagine you’re the architect, and your design review is tomorrow morning. Your diagram has a box labelled“LLM” — and no numbers.

This course is about everything that box hides.

The model is the least stable component in your system. Design around it.
Model latency, quality, pricing, and availability can all change. Your architecture must remain correct, safe, fast, and affordable despite those changes — and you must be able to prove that it works.

Build an AI Platform You Can Actually Test

Throughout this course, you’ll work onPolaris, a fictional enterprise’s shared AI platform.

Polaris runs on your laptop, with no GPU or paid API required. A deterministic model simulator stands in for real AI providers, making the results reproducible and the exercises free to run.

You won’t just discuss architectural principles. You’ll verify them usingfitness functions — automated tests that run the platform against defined workloads and fault scenarios, then fail when the architecture violates its requirements.

The goal is simple: move beyond diagrams and assumptions to measurable, testable AI systems.

What You’ll Design, Build, and Prove

1. Requirements, Capacity Planning, and Economics

Decide when you need an LLM — and when you don’t.

Define measurable quality requirements, latency budgets, and cost per successful outcome.

Estimate capacity, traffic, concurrency, and operating costs before committing to an architecture.

2. The Enterprise AI Reference Architecture

Design the complete request path from the edge to models, retrieval systems, and tools.

Integrate AI workloads with existing enterprise message brokers and event streams.

Understand the responsibilities and failure boundaries of each architectural component.

3. Multi-Tenant AI Gateway — Project 1

Implement identity, tenant isolation, quotas, and policy enforcement.

Route requests using cost, latency, data classification, and geographic region.

Design retry budgets, fallbacks, circuit breakers, and caching strategies.

Prove that caches and shared infrastructure never leak data across tenants.

4. Retrieval Engineering at Scale — Project 2

Treat retrieval-augmented generation (RAG) as a production data pipeline.

Manage versioned embeddings, document freshness, and deletion propagation.

Measure hybrid search, reranking, and retrieval quality.

Enforce permissions throughout multi-tenant retrieval, including across indexes and caches.

5. Enterprise Agent Platform

Design tool registries and Model Context Protocol (MCP) gateways.

Implement delegated authority and least-privilege tool access.

Handle durable execution, retries, human approval, and recovery.

Build a tamper-evident audit trail for agent actions.

6. Evaluation Engineering and Release Gates

Build golden evaluation datasets and calibrate model-as-judge evaluations.

Version models, prompts, policies, tools, and evaluation configurations as release units.

Implement tracing and quality monitoring.

Block releases automatically when evaluations detect a meaningful regression.

7. AI Security, Governance, and Data Residency

Build threat models for LLM applications, retrieval systems, and autonomous agents.

Map relevant OWASP guidance to concrete architectural controls.

Define policies for personally identifiable information (PII), logging, and sensitive data.

Compare data-residency architectures for the EU, the Gulf region, and India, using dated regulatory references and without presenting the material as legal advice.

8. Reliability Engineering — Project 3

Design graceful degradation, failover, circuit breakers, and load shedding.

Define service-level objectives (SLOs) that include AI quality, not just uptime and latency.

Handle provider outages without allowing retries to amplify the failure.

Test whether your platform can survive a provider outage during a tenfold traffic spike.

9. Cost Architecture and Optimization

Calculate unit economics and attribute costs to individual tenants.

Design model cascades, batching, and intelligent routing.

Determine when self-hosting becomes economically viable.

Walk through a cost optimization that targets a 40% reduction while verifying that quality remains within defined acceptance thresholds.

Apply the Architecture to 10 Enterprise Case Studies

You’ll work through ten realistic scenarios, from requirements and architecture to cost modelling, testing, and trade-off analysis

Enterprise copilot

Customer support agent that takes actions

Insurance claims automation

Product catalogue search

Event-driven airline operations assistant

Banking platform with data-residency requirements

Telecom triage agent

Multi-tenant SaaS AI platform

Coding-agent platform

Contact-centre voice assistant

Each case study follows the same engineering discipline

Define requirements and measurable success criteria.

Design the architecture and estimate its cost.

Test failure scenarios and scale limits.

Optimize the system and measure the results.

Examine rejected alternatives and understand why they were not selected.

You’ll also completetwo mock AI system design interviews and use a structured evaluation rubric to practise explaining and defending your architectural decisions.

Capstone: Design, Prove, and Defend Your Own AI Platform

The course culminates in a capstone project where you’ll design an AI platform of your own.

You’ll need to justify the architecture, quantify its performance and economics, test its behaviour under failure, and defend your decisions using evidence rather than assumptions.

The emphasis is on demonstrating engineering judgement — not simply drawing a convincing diagram.

What This Course Is — and Isn’t

This is not a vendor tour, a prompt-engineering course, or a course about training foundation models.

It’s about the architecture surrounding AI models: the gateways, retrieval pipelines, agents, evaluations, security controls, reliability mechanisms, and cost models that turn model capabilities into dependable enterprise systems.

Vendor-specific material will be clearly labelled, dated, and refreshed where appropriate. The core architectural principles are designed to remain useful as providers and models evolve.

Time Commitment

Plan for approximatelytwice the total video runtime if you complete the labs, three major projects, case studies, and capstone.

You’ll finish with more than a collection of architecture patterns. You’ll have a practical framework for designing AI systems, testing their limits, measuring their economics, and defending your decisions in real-world architecture reviews and AI system design interviews.

Who this course is for
Senior and staff engineers, solution and enterprise architects, and platform engineers
Tech leads and engineering managers who design, review or own production systems that call large language models
Engineers preparing for AI system design interviews
Architects in enterprises, banks, airlines, telecom operators and services firms who must meet residency and audit rules
Not for model training, ML mathematics, prompt-engineering tips or a single-framework tutorial

AYRTEMTYHELEMRITGATRFEGRWYERFRAGEFR

you must be registered member to see linkes Register Now