Building an AI stack when data cannot leave your network
Organisations that cannot send data to a vendor can assemble the whole pipeline from open-source parts. The licences are free; the operations are not.
AI Tools Hub Editorial · published · updated
Five layers
| Layer | Role | Open-source options |
|---|---|---|
| Model | Inference | Open models on your own GPUs, or a licensed domestic model |
| Retrieval | Vector search | Qdrant, Weaviate, Chroma, pgvector |
| Preprocessing | Parsing and chunking | Unstructured OSS, or your own |
| Application | RAG and chat interface | Dify Community Edition |
| Observability | Tracing and evaluation | Self-hosted Langfuse |
Check whether the constraint is really total
Before committing to full self-hosting, find out what is actually prohibited. These middle positions satisfy most real requirements:
- Model hosted externally, documents and index internal — excerpts still travel in the prompt
- External API under a contract that excludes training use
- A managed service pinned to a domestic region
- Masking or anonymising before anything leaves
Full self-hosting is expensive. Confirm the boundary with legal and IT before designing around the strictest possible reading, because a system nobody can operate is worse than one that sends redacted text to a vendor.
What you take on
- Updates, including security patches
- Backups and a tested restore procedure
- First-line incident response
- GPU drivers and runtimes, if you host models
- Capacity monitoring
The commonly missed requirement is having more than one person who understands the setup. A stack only one engineer can touch means that engineer cannot take leave, and when they move teams nobody can change anything.
Sequence it
- Prove the value on managed services with dummy data
- Establish the real constraints with legal and IT
- Move only the layers that breach them
- Go fully self-hosted only if the middle positions fail
FAQ
What does it cost?
The order of magnitude depends entirely on GPUs. Retrieval, preprocessing and the application layer run on an ordinary server. Hosting models yourself adds hardware, power and operations, and frequently exceeds the API spend it replaces.