How-To Guide
Run Hermes Locally with Ollama — Complete Privacy
Configure Hermes Agent with Ollama’s local OpenAI-compatible endpoint, exact model discovery, context checks, and tool-call troubleshooting.
Quick answer
Start Ollama, then run hermes model, choose a custom endpoint, and use http://127.0.0.1:11434/v1 with no API key. Confirm the exact detected model and test a real tool call. Use the broader Hermes local LLM guide to compare Ollama with LM Studio, vLLM, SGLang, and llama.cpp.
Ollama runs open-source LLMs entirely on your machine. No API keys, no cloud dependency, no data leaves your computer. Perfect for privacy-sensitive work, offline environments, or when you want to avoid API costs entirely.
Managed cloud · API costs included · Skill library · Cancel anytime
Before you start:
- ☑Hermes Agent installed
- ☑Ollama installed
- ☑Sufficient RAM/VRAM for your chosen model (8GB minimum, 24GB+ recommended)
Steps
- 1
Install Ollama
Download and install from ollama.com
- 2
Pull a model
ollama pull qwen3.5:35b or ollama pull gemma4:27b
- 3
Start Ollama server
ollama serve (runs on localhost:11434)
- 4
Configure Hermes
Set model: provider: ollama and model: base_url: http://localhost:11434/v1
- 5
Choose your model
Set model: default: qwen3.5:35b (or your pulled model)
- 6
Translate localhost across a Docker boundary
For containerized Hermes, replace host
127.0.0.1withhost.docker.internalon macOS/Windows or theollamaservice name on a shared Compose network. Keep the/v1suffix.
Pro Tips
- 💡Carnice 35B A3B is specifically tuned for Hermes tool calling — most reliable local model
- 💡Qwen 3.5 35B and Gemma 4 27B are popular community choices
- 💡Use Q4_K_M quantization for balance of quality and memory usage
- 💡Set context_length explicitly if auto-detection is wrong
- 💡Run the
/v1/modelsrequest from the same runtime as Hermes. Host-shell connectivity is not proof that the container can reach Ollama.
Troubleshooting
❌ Model doesn't use tools
✅ Many local models don't support function calling. Use Carnice, Qwen 3.5, or another Tier 1 model from the community recommendations.
❌ Out of memory
✅ Try a smaller model or lower quantization. 7B models need ~8GB, 35B models need ~24GB VRAM.
❌ Slow responses
✅ Local inference is CPU/GPU bound. Use a smaller model, enable GPU acceleration in Ollama, or accept the speed tradeoff for privacy.
❌ Connection refused
✅ Ensure ollama serve is running. Check that the base_url matches Ollama's actual address (default localhost:11434).
❌ Docker cannot connect to Ollama
✅ Use the Hermes Docker Compose guide, choose the host or sidecar endpoint, and verify /v1/models from inside the Hermes container.
❌ The Ollama model repeats tool calls at low context usage
✅ Confirm the effective Ollama context and Hermes context_length, then replay one observable tool task. Compare the identical task on a known-good hosted model; a route-specific failure is model/tool-use compatibility, not proof of memory corruption.
FAQ
How do I point Hermes at a local Ollama model?
Configure the Ollama provider with its local base URL and the model name. Make sure the model is started with at least 64K context (-c 65536).
Why does Hermes reject my local model?
Context size. Hermes requires at least 64,000 tokens and rejects smaller windows at startup. Start the Ollama model with -c 65536.
What's the benefit of the local Ollama provider?
Zero per-token cost, full privacy, and offline capability — inference runs on your hardware so no data leaves your machine.
Can containerized Hermes use Ollama on my Mac?
Yes. Use http://host.docker.internal:11434/v1, verify the endpoint from inside Hermes, and select the exact model ID returned by Ollama.
Can an Ollama model chat normally but still fail Hermes tools?
Yes. General chat quality and function-calling reliability are different. Verify the exact model ID, effective context, and one real tool call before choosing it for unattended Hermes work.