Open-source and open-weight AI models give businesses an alternative to sending every prompt and document to a public AI service. A company can run a model on its own workstation, private server, data centre, or isolated cloud environment and connect it to internal documents through retrieval-augmented generation (RAG).
However, downloading a popular model does not automatically create a private or reliable AI system. The model licence, hardware, document pipeline, access controls, logging, backups, and network design all matter. A smaller model that answers your own test questions accurately can be a better business choice than a much larger model that is costly and difficult to operate.
This guide compares major open model families and provides a practical selection method for small businesses and organizations that need a private RAG system.
Terminology note: "Open source" is commonly used for downloadable AI models, but many are more accurately described as open-weight. Their trained weights are available, while their training data, complete training process, or licence may not meet every formal open-source definition. Always review the specific model's licence before commercial deployment.
In this guide
Quick comparison of popular open AI model families
Model releases change quickly. The table below focuses on representative, practically deployable families available as of August 2026 rather than declaring one permanent winner.
| Model family | Representative options | Licence | Practical strengths | Good business uses | Main limitations |
|---|---|---|---|---|---|
| OpenAI gpt-oss | gpt-oss-20b, gpt-oss-120b | Apache 2.0 | Reasoning, structured output, tool use and agent workflows | Complex document analysis, internal assistants, tool-using agents | Text-only; 120b needs enterprise-class hardware; mostly English training data |
| Mistral | Mistral 3 (3B/8B/14B), Mistral Small 4, Mistral Large 3 | Apache 2.0 | Efficient deployment, multilingual, general chat, coding and agents | Private knowledge assistants, multilingual support, workflow automation | Model sizes and serving requirements vary considerably across the family |
| Qwen | Qwen3 dense and MoE; Coder, VL and embedding variants | Apache 2.0 (verify per card) | Broad size range, multilingual, reasoning, coding and specialist models | Multilingual RAG, technical support, coding assistants, extraction | Not every Qwen service or derivative has the same licence; choose the exact checkpoint carefully |
| Google Gemma | Gemma 4 12B; smaller Gemma variants | Gemma terms | Efficient local operation, multimodal input, multilingual and edge deployment | Local assistants, document and image understanding, transcription workflows | Uses Google model-specific terms rather than Apache/MIT; validate with your legal requirements |
| Meta Llama | Llama 4 Scout & Maverick; smaller earlier Llama variants | Meta Community License | Large ecosystem, multimodal capability, broad deployment support | General assistants, document Q&A, multimodal search and prototypes | Community licence includes conditions and is not equivalent to Apache 2.0; large variants require substantial hardware |
| Microsoft Phi | Phi-4 and Phi-4 reasoning / multimodal variants | Open-weight (verify per model) | Strong capability in smaller sizes, reasoning and resource-conscious deployment | Classification, structured extraction, local copilots, narrow RAG tasks | Small models have less general knowledge and may be more sensitive to prompt and retrieval quality |
| DeepSeek | DeepSeek-R1 / V3 family and distilled models | MIT (verify per checkpoint) | Reasoning, coding and availability of smaller distilled variants | Technical research, analysis, coding and reasoning-heavy assistants | Full models are very large; data-governance concerns differ between self-hosting weights and using a hosted API |
| Kimi (Moonshot) | Kimi K2 and specialist coding / reasoning variants | Modified / open model terms | Agentic tool use, coding and long multi-step tasks | Engineering assistants and sophisticated agents | Full MoE models are not realistic for most small-business self-hosting; verify exact release terms |
Key model notes
- gpt-oss-20b can operate with approximately 16 GB of memory; gpt-oss-120b fits within 80 GB in its released quantization. Both use a 128K context window and Apache 2.0 licensing.
- Mistral 3 introduced dense 3B, 8B and 14B models plus a larger MoE model under Apache 2.0. Mistral Small 4 added multimodal, reasoning, coding and agent capabilities under the same licence.
- Qwen3 can switch between thinking and non-thinking behaviour and offers broad ecosystem coverage from general chat to coding and multilingual retrieval.
- Gemma 4 12B was demonstrated on everyday laptops with 16 GB of RAM. That does not guarantee production-level concurrency, but it makes Gemma worth testing for local pilots and offline tools.
- Llama 4 Scout and Maverick are multimodal MoE models with 17 billion active parameters. Meta says quantized Scout fits on one H100 GPU — more suitable for a data centre than a typical office server.
- DeepSeek-R1 and its distilled models were released under the MIT licence. The distilled variants are more realistic for smaller deployments than the full-scale model.
- Kimi K2 uses a MoE architecture with one trillion total parameters and 32 billion active parameters — relevant for advanced agent comparisons, but impractical for most small-business self-hosting.
Which model is best for each use case?
There is no universal best model. Select a model against a defined workload.
| Use case | Models worth testing first | Why |
|---|---|---|
| Internal policy, procedure and handbook RAG | Mistral 8B/14B-class, Qwen mid-sized, Gemma 4 12B, gpt-oss-20b | Practical balance of instruction following, local deployment and answer quality |
| Low-cost office assistant on one workstation | Gemma 4 12B, smaller Mistral, Phi or smaller Qwen variant | More realistic memory, power and latency requirements |
| Multilingual knowledge base | Qwen, Mistral or Gemma | Strong multilingual model families; test the exact languages and business vocabulary you use |
| Complex reasoning over retrieved documents | gpt-oss-20b / 120b, a Qwen reasoning model or DeepSeek distilled reasoning model | Reasoning modes improve multi-step analysis, though they add latency and can still hallucinate |
| Coding and IT knowledge assistant | Qwen Coder, Mistral coding model, gpt-oss or a DeepSeek coding / reasoning variant | Stronger fit for code, logs, scripts and tool calls than a generic small chat model |
| Image, diagram or scanned-document understanding | Gemma multimodal, Llama multimodal, Mistral Small 4 or Qwen-VL | Text-only models cannot directly interpret page images; OCR may still be needed for dependable document ingestion |
| High-volume classification and data extraction | Phi, smaller Mistral, Gemma or Qwen | A small model can be faster and cheaper if output is constrained and validated |
| Complex autonomous agents | gpt-oss, Mistral Small 4, capable Qwen or Kimi model | Tool use and instruction adherence matter, but agents require strong permissions, isolation and audit controls |
What a private RAG system actually includes
A model is only one component. A production private RAG system normally contains all of the following layers working together.
Approved sources such as SharePoint, Google Drive, file shares, databases or controlled uploads.
Extracts text, tables and metadata from PDF, Word, spreadsheet, image and scanned files.
Divides documents into retrievable sections without destroying important context.
Converts chunks and questions into vectors for semantic retrieval.
Stores embeddings and metadata. Common self-hosted options include PostgreSQL with pgvector, Qdrant, Milvus and OpenSearch.
Combines semantic search with keyword matching, filters and document permissions.
Reorders retrieved passages so the most relevant evidence reaches the language model.
Produces an answer using the retrieved evidence and system instructions.
Provides chat, search, feedback, citations, administration and integrations.
Identity, role-based access, encryption, secrets management, monitoring, backup, patching and audit logs.
A simple private RAG flow:
Three practical privacy levels
"Private AI" can mean different things. Define the boundary before selecting the model or platform.
Before you say "everything must remain private," clarify what that means
- No prompts or documents sent to an external model API
- No internet access from the AI environment
- Data must remain in Canada or another specified jurisdiction
- Only authorized employees can access particular documents
- Administrators must not see business content
- Logs must exclude prompts and retrieved passages
- Model files and software updates require security approval
- Backups must use the same encryption, location and retention controls
If all processing must stay inside the boundary, every component must be local or privately hosted — not only the language model. External OCR, embedding, reranking, telemetry, content moderation or analytics services can otherwise send data outside the intended environment.
A practical model-selection method
Step 1 — Define the workload
Collect 50–200 representative questions and the approved answers or source passages. Include easy queries, ambiguous questions, no-answer cases, permission-sensitive documents, tables, scanned pages, abbreviations and multiple languages.
Step 2 — Establish the privacy and licence requirements
Document where model weights, prompts, files, indexes, logs and backups may reside. Review the licence of the exact model checkpoint — not just the model family — and confirm commercial, redistribution and modification requirements.
Step 3 — Set realistic infrastructure limits
Record available RAM, GPU memory, storage, required concurrent users, target response time and monthly operating budget. Quantization can reduce memory use, but it may affect quality and runtime compatibility.
| Deployment tier | Reasonable starting point | Typical purpose |
|---|---|---|
| Laptop or compact workstation | Quantized small or mid-sized model | Individual assistant, evaluation, extraction and light RAG |
| Single-GPU server | Mid-sized model optimized for available GPU memory | Departmental RAG with controlled concurrency |
| Multi-GPU server | Larger dense model or mixture-of-experts model | Higher quality, concurrency or complex reasoning |
| GPU cluster | Very large models and high availability | Enterprise workloads that justify specialist operations |
Step 4 — Test a shortlist, not the whole market
Select three candidates:
- One small, inexpensive baseline
- One balanced mid-sized model
- One larger or reasoning-focused model
Use the same retrieval results, prompts and generation settings where possible. This isolates model quality from weaknesses in the search pipeline.
Step 5 — Score the complete system
| Criterion | Suggested weight |
|---|---|
| Answer correctness and support from sources | 30% |
| Retrieval quality | 20% |
| Citation accuracy | 15% |
| Privacy and licence fit | 15% |
| Latency and concurrent-user capacity | 10% |
| Infrastructure and operational cost | 10% |
Score answers with human reviewers from the relevant department. Automated model-as-a-judge tests can help with scale, but they should not be the only approval method.
Step 6 — Test failure behaviour
A useful private assistant must know when evidence is missing. Test whether it:
- Says it cannot find an answer rather than inventing one
- Cites the correct document and page or section
- Respects document-level permissions
- Rejects prompt injection contained inside retrieved documents
- Avoids exposing one user's chat or documents to another
- Handles deleted, expired and superseded documents correctly
Step 7 — Pilot with a narrow business process
Start with one department, one controlled document collection and a measurable task — for example, finding approved procedures or answering product-support questions. Review failed queries weekly before expanding the system.
Recommended starting choices
Small business with ordinary internal documents
Start by testing a capable 8B–20B-class model, such as an appropriate Mistral, Qwen, Gemma or gpt-oss variant. Pair it with hybrid retrieval, a strong embedding model and citations. This tier is more likely to fit one workstation or GPU server and is easier to operate than a frontier-scale model.
Multilingual business
Shortlist Qwen, Mistral and Gemma, then evaluate the languages actually used by employees and customers. Do not rely on a vendor's language count alone. Test names, local vocabulary, technical terms, mixed-language questions and the quality of citations.
Complex engineering or policy analysis
Compare gpt-oss, a suitable Qwen reasoning model, a DeepSeek distilled reasoning model and a larger Mistral option. Reasoning can help with multi-document synthesis, but the system should still cite the evidence used and separate retrieved facts from model inference.
Fully isolated private RAG system
Prefer models with clear downloadable weights and a licence approved by counsel. Host the model, embedding model, reranker, OCR, vector database, application, identity integration and monitoring within the controlled environment. Disable outbound telemetry and network egress unless explicitly approved.
Common mistakes to avoid
Final recommendation
For most small businesses, begin with a mid-sized self-hosted model and a carefully designed RAG pipeline. Evaluate at least three models using the company's own documents and questions. Choose the smallest model that consistently meets the required accuracy, latency, privacy and licence standards.
Organizations with strict confidentiality should treat private AI as an infrastructure and governance project — not a model download. A truly private system keeps inference, embeddings, retrieval, document processing, logs and backups inside the approved security boundary and applies the same identity and access rules as the source documents.
Ready to build a private AI system your organization can trust?
Nala Networks designs and implements private and cloud-based RAG systems using open models, secure retrieval and business-specific integrations.
Talk to Nala Networks about your requirements