xonecas.io
60255
Zn

Zoea-Nova / files

Zoea-Nova / documentation / guides / OLLAMA_MODELS.md

7.4 KB
1# Ollama Models for Zoea Nova
2
3Hardware: **NVIDIA GeForce RTX 3080 Ti (12GB VRAM)**
4
5This document catalogs Ollama models suitable for our setup, focusing on tool calling support (required for MCP integration) and thinking/reasoning capabilities.
6
7---
8
9## TL;DR Recommendations
10
11| Use Case | Model | Size | Pull Command |
12|----------|-------|------|--------------|
13| **Fast iteration/testing** | Qwen 3 4B | 2.6GB | `ollama pull qwen3:4b` |
14| **Balanced (recommended)** | Qwen 3 8B | 4.9GB | `ollama pull qwen3:8b` |
15| **Best quality** | Qwen 3 14B | 9.0GB | `ollama pull qwen3:14b` |
16| **Reasoning specialist** | DeepSeek R1 Tool Calling 14B | 9.0GB | `ollama pull MFDoom/deepseek-r1-tool-calling:14b` |
17
18---
19
20## Smallest Models with Tool Support
21
22### Llama 3.2 (Meta)
23
24| Variant | Size | Context | Tool Support | Notes |
25|---------|------|---------|--------------|-------|
26| 1B | 1.3GB | 128K | Limited | Edge/mobile optimized |
27| **3B** | 2.0GB | 128K | ✅ Yes | Outperforms Gemma 2 2.6B on tool use |
28
29```bash
30ollama pull llama3.2:3b
31```
32
33**Pros:** Extremely lightweight, fast inference, good for simple tool chains.
34**Cons:** Limited reasoning depth, may struggle with complex multi-step tool sequences.
35
36---
37
38### Qwen 3 (Alibaba) ⭐ Recommended
39
40Best balance of size, tool calling, and thinking capabilities.
41
42| Variant | Size | Context | Tool Support | Thinking | Notes |
43|---------|------|---------|--------------|----------|-------|
44| 0.6B | 0.4GB | 32K | ✅ Yes | ✅ Yes | Minimal, edge only |
45| 1.7B | 1.1GB | 32K | ✅ Yes | ✅ Yes | Lightweight |
46| **4B** | 2.6GB | 32K | ✅ Yes | ✅ Yes | Fast iteration |
47| **8B** | 4.9GB | 32K | ✅ Yes | ✅ Yes | Sweet spot |
48| **14B** | 9.0GB | 32K | ✅ Yes | ✅ Yes | Best quality for 12GB |
49| 30B MoE | 18GB | 32K | ✅ Yes | ✅ Yes | Won't fit in VRAM |
50
51```bash
52# Recommended for development
53ollama pull qwen3:4b
54
55# Recommended for production
56ollama pull qwen3:8b
57
58# Best quality (fits in 12GB)
59ollama pull qwen3:14b
60```
61
62**Thinking mode control:**
63- Enable: append `/think` to prompt
64- Disable: append `/nothink` to prompt
65- Set default in system prompt
66
67**Pros:** Native tool calling + thinking in same model, excellent structured output, streaming tool calls.
68**Cons:** Thinking mode can be verbose (no token limit control yet).
69
70---
71
72### Qwen 2.5
73
74Predecessor to Qwen 3, still solid for tool calling without thinking features.
75
76| Variant | Size | Context | Tool Support | Notes |
77|---------|------|---------|--------------|-------|
78| 7B | 4.7GB | 32K | ✅ Yes | Good JSON output |
79| 14B | 9.0GB | 32K | ✅ Yes | Excellent structured data |
80
81```bash
82ollama pull qwen2.5:7b
83ollama pull qwen2.5:14b
84```
85
86---
87
88### Mistral 7B v0.3
89
90| Variant | Size | Context | Tool Support | Notes |
91|---------|------|---------|--------------|-------|
92| 7B | 4.1GB | 32K | ✅ Yes | Uses raw mode |
93
94```bash
95ollama pull mistral:7b
96```
97
98**Pros:** Well-established, good general performance.
99**Cons:** Tool calling uses older raw mode format, no native thinking.
100
101---
102
103## Thinking/Reasoning Models with Tool Support
104
105### LFM2.5-Thinking (Liquid AI)
106
107Ultra-compact reasoning model. 1.2B params but matches Qwen3-1.7B on benchmarks.
108
109| Variant | Size | Context | Tool Support | Thinking | Notes |
110|---------|------|---------|--------------|----------|-------|
111| **1.2B** (Q4_K_M) | 731MB | 32K | ⚠️ No native | ✅ Yes | On-device capable |
112| 1.2B (Q8_0) | 1.2GB | 32K | ⚠️ No native | ✅ Yes | Higher precision |
113
114```bash
115ollama pull lfm2.5-thinking
116```
117
118**⚠️ Tool Calling Limitation:**
119LFM2.5-Thinking scores 56.97 on BFCLv3 (Berkeley Function Calling Leaderboard), meaning it *understands* function calling concepts. However, it does **NOT support Ollama's native `tools` API parameter**. Tool definitions must be embedded in the prompt and output parsed manually.
120
121**Not recommended for Zoea Nova** — use Qwen 3 instead for native tool support.
122
123**Benchmarks vs non-thinking variant:**
124- Math (MATH-500): 63 → **88**
125- Instruction following: 61 → **69**
126
127**Pros:** Extremely small (fits in 731MB), thinking traces built-in.
128**Cons:** No native Ollama tool API support. Requires manual prompt engineering for tools.
129
130---
131
132### Qwen 3 (see above)
133
134All Qwen 3 variants support both thinking AND tool calling natively. This is the recommended choice.
135
136---
137
138### DeepSeek R1 Distill (with Tool Calling)
139
140Community-maintained models that add tool calling to DeepSeek's reasoning models.
141
142| Variant | Size | Quantization | Tool Support | Notes |
143|---------|------|--------------|--------------|-------|
144| 7B Qwen | 4.7GB | Q4_K_M | ✅ Yes | Qwen architecture |
145| **8B Llama** | 4.9GB | Q4_K_M | ✅ Yes | Llama architecture |
146| **14B Qwen** | 9.0GB | Q4_K_M | ✅ Yes | Best reasoning |
147
148```bash
149# 8B variant
150ollama pull MFDoom/deepseek-r1-tool-calling:8b
151
152# 14B variant (recommended for reasoning tasks)
153ollama pull MFDoom/deepseek-r1-tool-calling:14b
154```
155
156**Pros:** Strong reasoning comparable to OpenAI o1, MIT licensed.
157**Cons:** Community-maintained (not official), tool calling added via custom templates.
158
159---
160
161## VRAM Usage Guidelines
162
163With 12GB VRAM and Q4_K_M quantization:
164
165| Model Size | VRAM Usage | Performance | Recommendation |
166|------------|------------|-------------|----------------|
167| ≤4B | ~3GB | 80+ tok/s | ✅ Fast iteration |
168| 7-8B | ~5GB | 50-60 tok/s | ✅ Balanced |
169| 14B | ~9GB | 40+ tok/s | ✅ Best quality |
170| 30B+ | >12GB | Won't fit | ❌ Avoid |
171
172**Note:** Context length affects VRAM. Long conversations may require smaller models or context pruning.
173
174---
175
176## Configuration for Zoea Nova
177
178Suggested `config.toml` provider settings:
179
180```toml
181[providers.ollama]
182endpoint = "http://localhost:11434"
183
184# For fast testing
185model = "qwen3:4b"
186
187# For production Myses
188# model = "qwen3:8b"
189
190# For complex reasoning tasks
191# model = "qwen3:14b"
192```
193
194### Thinking Mode Strategy
195
196For game Myses that need to plan:
1971. Use Qwen 3 with thinking enabled for strategic decisions
1982. Disable thinking (`/nothink`) for rapid tool execution
1993. Consider separate models: fast (4B) for actions, larger (14B) for planning
200
201---
202
203## Quick Start
204
205```bash
206# Install recommended models
207ollama pull qwen3:4b    # 2.6GB - fast testing
208ollama pull qwen3:8b    # 4.9GB - balanced
209
210# Verify tool support
211ollama run qwen3:8b
212>>> /show info
213```
214
215Test tool calling:
216```bash
217curl http://localhost:11434/api/chat -d '{
218  "model": "qwen3:8b",
219  "messages": [{"role": "user", "content": "What is 2+2?"}],
220  "tools": [{
221    "type": "function",
222    "function": {
223      "name": "calculate",
224      "description": "Perform arithmetic",
225      "parameters": {
226        "type": "object",
227        "properties": {
228          "expression": {"type": "string"}
229        },
230        "required": ["expression"]
231      }
232    }
233  }]
234}'
235```
236
237---
238
239## Currently Installed
240
241```
242llama3.1:8b           4.9 GB    ✅ native tool support
243qwen3:4b              2.5 GB    ✅ recommended (default)
244qwen3:8b              5.2 GB    ✅ recommended
245lfm2.5-thinking       731 MB    ⚠️ no native tool API
246```
247
248---
249
250## References
251
252- [Ollama Tool Calling Docs](https://docs.ollama.com/capabilities/tool-calling)
253- [Ollama Thinking Docs](https://docs.ollama.com/capabilities/thinking)
254- [Qwen 3 on Ollama](https://ollama.com/library/qwen3)
255- [LFM2.5-Thinking on Ollama](https://ollama.com/library/lfm2.5-thinking)
256- [LFM2.5-Thinking Blog Post](https://www.liquid.ai/blog/lfm2-5-1-2b-thinking-on-device-reasoning-under-1gb)
257- [DeepSeek R1 Tool Calling](https://ollama.com/MFDoom/deepseek-r1-tool-calling)