Self-Hosting LLMs Securely: Hardening Ollama and vLLM
Published 10/2026
Created by NEXUS ACADEMY
MP4 | Video: h264, 1280×720 | Audio: AAC, 44.1 KHz, 2 Ch
Level: Intermediate | Genre: eLearning | Language: English | Duration: 109 Lectures ( 10h 33m ) | Size: 2.2 GB
Harden an Ollama and vLLM inference host: bind, gateway, isolate, verify models, monitor and patch, all shown running
What you’ll learn
Draw the threat model of a self-hosted inference host: assets, trust boundaries and what the default install exposes
Measure a default Ollama and vLLM install before changing anything, and keep a baseline scoreboard of what fails
Lock Ollama to loopback, restrict CORS, run it as a dedicated user and sandbox it with systemd
Run Ollama and vLLM in containers as non-root, read-only, capability-free and published on 127.0.0.1 only
Use vLLM’s own flags correctly, including the real scope of –api-key, and close the endpoints it leaves open
Put an authenticating gateway in front of the model with TLS, per-client keys, an endpoint allowlist and rate limits
Deny ingress and egress with nftables and keep the model server off the internet
Verify model files before serving: formats, scanning, pinned revisions, digests, signatures and a model manifest
Log, monitor and alert on the model host with Prometheus, Grafana and Falco, and detect unexpected model changes
Patch on a rhythm, run incident drills and assemble an evidence pack mapped to NIST AI RMF and ISO/IEC 42001
Requirements
Comfort with the Linux shell, systemd and Docker basics. No machine-learning background is needed
A Linux machine or VM that can run Docker. No GPU is needed: every lab runs on CPU with small, openly licensed models
Description
“This course contains the use of artificial intelligence.”
Running an open-weight model on your own hardware keeps prompts and data in house, but it also makes you the operator of a new kind of server. This course treats a model server as a server: you measure what the default install exposes, harden it layer by layer, and prove every change with a check that runs.
You start by looking, not fixing. Ollama and a CPU build of vLLM are installed the documented way, their API surfaces are mapped endpoint by endpoint, and a verification script records every check that fails. That baseline scoreboard is what the rest of the course turns green.
Ollama is hardened on the host first: bind address, CORS, a dedicated service user, systemd sandboxing and resource limits, with the documentation’s own statement that it does not require authentication taken at face value. Then the same service goes into a container that runs as non-root, read-only, without capabilities and published on loopback only, with the image pinned by digest and scanned.
vLLM gets its own section. You read its security guidance line by line, set the bind address explicitly, and measure exactly which endpoints –api-key protects and which still answer without it. What the flags cannot close, the gateway does: nginx with TLS from a private CA, one key per client, an endpoint allowlist, rate and size limits, and an LLM proxy layer for per-client budgets.
The host is then isolated with a default-deny firewall and no route out, and models reach it only through a staging step. Model files are treated as code-adjacent: you compare pickle, safetensors and GGUF, scan files, pin Hugging Face downloads to a full commit, verify digests, sign and verify a model, and serve only what a model manifest lists.
The last sections are about running it: secrets out of compose files, what the servers log about prompts, structured gateway logs, private metrics, alert rules, Falco on the model container, advisory tracking and controlled upgrades with rollback. Two incident drills and a full rebuild from one repository close the course, with a final evidence pack mapped to NIST AI RMF, ISO/IEC 42001, MITRE ATLAS and the EU AI Act milestones.
This is a defensive course. Everything runs on your own lab host; nothing is pointed at systems you do not own and there is no internet scanning. GPU hosts and multi-node vLLM are covered conceptually and marked as such, because the recorded bench is CPU only. Twelve downloadable documents come with it, including a threat-model worksheet, hardening checklists for Ollama and vLLM, a gateway guide, a model intake policy, monitoring and incident runbooks and an auditor evidence index.
Who this course is for
Platform, DevOps and security engineers asked to run open-weight models on their own infrastructure
Security teams reviewing an internal Ollama or vLLM deployment before it goes into production
Homelab and small-team operators who want a local model server that is not reachable by accident
SEALEYTRTAHOSTERFRGSECUTERFRHARDE
you must be registered member to see linkes Register Now