6119
My
mysis / files
mysis / documentation / guides / OLLAMA_MODELS.md
7.4 KB
1# Ollama Models for Zoea Nova 2 3Hardware: **NVIDIA GeForce RTX 3080 Ti (12GB VRAM)** 4 5This document catalogs Ollama models suitable for our setup, focusing on tool calling support (required for MCP integration) and thinking/reasoning capabilities. 6 7--- 8 9## TL;DR Recommendations 10 11| Use Case | Model | Size | Pull Command | 12|----------|-------|------|--------------| 13| **Fast iteration/testing** | Qwen 3 4B | 2.6GB | `ollama pull qwen3:4b` | 14| **Balanced (recommended)** | Qwen 3 8B | 4.9GB | `ollama pull qwen3:8b` | 15| **Best quality** | Qwen 3 14B | 9.0GB | `ollama pull qwen3:14b` | 16| **Reasoning specialist** | DeepSeek R1 Tool Calling 14B | 9.0GB | `ollama pull MFDoom/deepseek-r1-tool-calling:14b` | 17 18--- 19 20## Smallest Models with Tool Support 21 22### Llama 3.2 (Meta) 23 24| Variant | Size | Context | Tool Support | Notes | 25|---------|------|---------|--------------|-------| 26| 1B | 1.3GB | 128K | Limited | Edge/mobile optimized | 27| **3B** | 2.0GB | 128K | ✅ Yes | Outperforms Gemma 2 2.6B on tool use | 28 29```bash 30ollama pull llama3.2:3b 31``` 32 33**Pros:** Extremely lightweight, fast inference, good for simple tool chains. 34**Cons:** Limited reasoning depth, may struggle with complex multi-step tool sequences. 35 36--- 37 38### Qwen 3 (Alibaba) ⭐ Recommended 39 40Best balance of size, tool calling, and thinking capabilities. 41 42| Variant | Size | Context | Tool Support | Thinking | Notes | 43|---------|------|---------|--------------|----------|-------| 44| 0.6B | 0.4GB | 32K | ✅ Yes | ✅ Yes | Minimal, edge only | 45| 1.7B | 1.1GB | 32K | ✅ Yes | ✅ Yes | Lightweight | 46| **4B** | 2.6GB | 32K | ✅ Yes | ✅ Yes | Fast iteration | 47| **8B** | 4.9GB | 32K | ✅ Yes | ✅ Yes | Sweet spot | 48| **14B** | 9.0GB | 32K | ✅ Yes | ✅ Yes | Best quality for 12GB | 49| 30B MoE | 18GB | 32K | ✅ Yes | ✅ Yes | Won't fit in VRAM | 50 51```bash 52# Recommended for development 53ollama pull qwen3:4b 54 55# Recommended for production 56ollama pull qwen3:8b 57 58# Best quality (fits in 12GB) 59ollama pull qwen3:14b 60``` 61 62**Thinking mode control:** 63- Enable: append `/think` to prompt 64- Disable: append `/nothink` to prompt 65- Set default in system prompt 66 67**Pros:** Native tool calling + thinking in same model, excellent structured output, streaming tool calls. 68**Cons:** Thinking mode can be verbose (no token limit control yet). 69 70--- 71 72### Qwen 2.5 73 74Predecessor to Qwen 3, still solid for tool calling without thinking features. 75 76| Variant | Size | Context | Tool Support | Notes | 77|---------|------|---------|--------------|-------| 78| 7B | 4.7GB | 32K | ✅ Yes | Good JSON output | 79| 14B | 9.0GB | 32K | ✅ Yes | Excellent structured data | 80 81```bash 82ollama pull qwen2.5:7b 83ollama pull qwen2.5:14b 84``` 85 86--- 87 88### Mistral 7B v0.3 89 90| Variant | Size | Context | Tool Support | Notes | 91|---------|------|---------|--------------|-------| 92| 7B | 4.1GB | 32K | ✅ Yes | Uses raw mode | 93 94```bash 95ollama pull mistral:7b 96``` 97 98**Pros:** Well-established, good general performance. 99**Cons:** Tool calling uses older raw mode format, no native thinking. 100 101--- 102 103## Thinking/Reasoning Models with Tool Support 104 105### LFM2.5-Thinking (Liquid AI) 106 107Ultra-compact reasoning model. 1.2B params but matches Qwen3-1.7B on benchmarks. 108 109| Variant | Size | Context | Tool Support | Thinking | Notes | 110|---------|------|---------|--------------|----------|-------| 111| **1.2B** (Q4_K_M) | 731MB | 32K | ⚠️ No native | ✅ Yes | On-device capable | 112| 1.2B (Q8_0) | 1.2GB | 32K | ⚠️ No native | ✅ Yes | Higher precision | 113 114```bash 115ollama pull lfm2.5-thinking 116``` 117 118**⚠️ Tool Calling Limitation:** 119LFM2.5-Thinking scores 56.97 on BFCLv3 (Berkeley Function Calling Leaderboard), meaning it *understands* function calling concepts. However, it does **NOT support Ollama's native `tools` API parameter**. Tool definitions must be embedded in the prompt and output parsed manually. 120 121**Not recommended for Zoea Nova** — use Qwen 3 instead for native tool support. 122 123**Benchmarks vs non-thinking variant:** 124- Math (MATH-500): 63 → **88** 125- Instruction following: 61 → **69** 126 127**Pros:** Extremely small (fits in 731MB), thinking traces built-in. 128**Cons:** No native Ollama tool API support. Requires manual prompt engineering for tools. 129 130--- 131 132### Qwen 3 (see above) 133 134All Qwen 3 variants support both thinking AND tool calling natively. This is the recommended choice. 135 136--- 137 138### DeepSeek R1 Distill (with Tool Calling) 139 140Community-maintained models that add tool calling to DeepSeek's reasoning models. 141 142| Variant | Size | Quantization | Tool Support | Notes | 143|---------|------|--------------|--------------|-------| 144| 7B Qwen | 4.7GB | Q4_K_M | ✅ Yes | Qwen architecture | 145| **8B Llama** | 4.9GB | Q4_K_M | ✅ Yes | Llama architecture | 146| **14B Qwen** | 9.0GB | Q4_K_M | ✅ Yes | Best reasoning | 147 148```bash 149# 8B variant 150ollama pull MFDoom/deepseek-r1-tool-calling:8b 151 152# 14B variant (recommended for reasoning tasks) 153ollama pull MFDoom/deepseek-r1-tool-calling:14b 154``` 155 156**Pros:** Strong reasoning comparable to OpenAI o1, MIT licensed. 157**Cons:** Community-maintained (not official), tool calling added via custom templates. 158 159--- 160 161## VRAM Usage Guidelines 162 163With 12GB VRAM and Q4_K_M quantization: 164 165| Model Size | VRAM Usage | Performance | Recommendation | 166|------------|------------|-------------|----------------| 167| ≤4B | ~3GB | 80+ tok/s | ✅ Fast iteration | 168| 7-8B | ~5GB | 50-60 tok/s | ✅ Balanced | 169| 14B | ~9GB | 40+ tok/s | ✅ Best quality | 170| 30B+ | >12GB | Won't fit | ❌ Avoid | 171 172**Note:** Context length affects VRAM. Long conversations may require smaller models or context pruning. 173 174--- 175 176## Configuration for Zoea Nova 177 178Suggested `config.toml` provider settings: 179 180```toml 181[providers.ollama] 182endpoint = "http://localhost:11434" 183 184# For fast testing 185model = "qwen3:4b" 186 187# For production Myses 188# model = "qwen3:8b" 189 190# For complex reasoning tasks 191# model = "qwen3:14b" 192``` 193 194### Thinking Mode Strategy 195 196For game Myses that need to plan: 1971. Use Qwen 3 with thinking enabled for strategic decisions 1982. Disable thinking (`/nothink`) for rapid tool execution 1993. Consider separate models: fast (4B) for actions, larger (14B) for planning 200 201--- 202 203## Quick Start 204 205```bash 206# Install recommended models 207ollama pull qwen3:4b # 2.6GB - fast testing 208ollama pull qwen3:8b # 4.9GB - balanced 209 210# Verify tool support 211ollama run qwen3:8b 212>>> /show info 213``` 214 215Test tool calling: 216```bash 217curl http://localhost:11434/api/chat -d '{ 218 "model": "qwen3:8b", 219 "messages": [{"role": "user", "content": "What is 2+2?"}], 220 "tools": [{ 221 "type": "function", 222 "function": { 223 "name": "calculate", 224 "description": "Perform arithmetic", 225 "parameters": { 226 "type": "object", 227 "properties": { 228 "expression": {"type": "string"} 229 }, 230 "required": ["expression"] 231 } 232 } 233 }] 234}' 235``` 236 237--- 238 239## Currently Installed 240 241``` 242llama3.1:8b 4.9 GB ✅ native tool support 243qwen3:4b 2.5 GB ✅ recommended (default) 244qwen3:8b 5.2 GB ✅ recommended 245lfm2.5-thinking 731 MB ⚠️ no native tool API 246``` 247 248--- 249 250## References 251 252- [Ollama Tool Calling Docs](https://docs.ollama.com/capabilities/tool-calling) 253- [Ollama Thinking Docs](https://docs.ollama.com/capabilities/thinking) 254- [Qwen 3 on Ollama](https://ollama.com/library/qwen3) 255- [LFM2.5-Thinking on Ollama](https://ollama.com/library/lfm2.5-thinking) 256- [LFM2.5-Thinking Blog Post](https://www.liquid.ai/blog/lfm2-5-1-2b-thinking-on-device-reasoning-under-1gb) 257- [DeepSeek R1 Tool Calling](https://ollama.com/MFDoom/deepseek-r1-tool-calling)