Running Llama, Mistral and DeepSeek on your own GPU server

Open weights have closed the gap with the closed frontier models. Llama, Mistral and DeepSeek deliver results on most business tasks - summarization, extraction, classification, RAG, coding assistance - that are hard to tell apart from expensive APIs in practice. The difference: you can run them on your own hardware.

Why self-host instead of calling an API

Three reasons. Sovereignty: your prompts never leave your jurisdiction. Cost: an owned server is flat and predictable instead of metered per token. Control: no rate limits, no silent model swaps, no deprecation of the version your pipeline depends on.

Matching the model to VRAM

VRAM is the number that matters. A rough rule: a model in 16-bit precision needs about 2 GB of VRAM per billion parameters, about 1 GB in 8-bit, and about 0.5 GB in 4-bit - plus headroom for the KV cache, which grows with context length.

• 7-9B models (Mistral 7B, Llama 8B): run comfortably on a single 24 GB GPU, ideal for high throughput.
• 32-70B models (Llama 70B, DeepSeek): need 2-4 GPUs or aggressive quantization - the sweet spot for quality.
• Mixture-of-experts models: high parameter count but only part is active per token - lots of VRAM, high speed.

The serving stack

For single-seat use and experiments, Ollama is running in minutes. For production load with many concurrent requests, vLLM is the standard: continuous batching and PagedAttention extract several times the throughput from the same GPU. Text Generation Inference (TGI) is a solid alternative with an OpenAI-compatible API - handy when you just want to point existing code at your own endpoint.

Throughput, quantization and context

Quantizing to 4 or 8 bits halves or quarters the memory footprint with usually small quality loss - which is how a 70B model fits on hardware that otherwise wouldn't hold it. Continuous batching is the biggest throughput lever: instead of handling requests one at a time, the server keeps the GPU fully fed. And context isn't free - a long KV cache eats VRAM that would otherwise go to the model.

The right question isn't how big a model you can load - it's how many concurrent users you can serve at acceptable latency.

Where to run it

An 8-GPU server runs several mid-sized models in parallel, or a single large one with room to spare for context and concurrency. Hosted in an EU datacenter, you get redundant power, cooling and connectivity - without the data ever leaving the jurisdiction. You buy the server, we run it, and you connect over an encrypted link with full root access.

Open models democratized capability. Owning the hardware democratizes control. Together they're the most practical route to sovereign, predictable AI.