Feature

Run Hermes Agent with a Local LLM

Quick answer

Yes. Hermes Agent can use Ollama, LM Studio, vLLM, SGLang, llama.cpp, or another OpenAI-compatible local endpoint. The simplest current path is `ollama launch hermes`; the manual Ollama endpoint is `http://127.0.0.1:11434/v1`. Choose a model that supports tool calls, give Hermes a large enough context window, and prove a real file/terminal task—not just chat—before making it your default.

Key Points

✓Fastest path: run `ollama launch hermes`, choose a tool-capable local model, and let Ollama configure the endpoint
✓Manual Ollama endpoint: `http://127.0.0.1:11434/v1`; LM Studio commonly serves `http://127.0.0.1:1234/v1`
✓vLLM and SGLang are better fits for shared GPU servers and concurrent agent workloads
✓Local inference removes per-token API billing, but shifts cost to RAM, VRAM, storage, electricity, and operator time
✓A chat response is not a sufficient test: verify structured tool calls, file reads, terminal use, and long-context behavior
✓“Local model” does not mean every Hermes tool is offline; web search, browser automation, remote memory, and cloud speech can still send data out
✓Use a hosted fallback only for tasks allowed to leave the machine, and test that fallback before depending on it

How It Works

1Pick the serving layer: Ollama for the simplest local setup, LM Studio for desktop model management, or vLLM/SGLang for a shared GPU endpoint
2Start the server and verify its OpenAI-compatible `/v1/models` or chat-completions endpoint before changing Hermes
3Run `hermes model`, select the local/custom endpoint, confirm the detected model, and leave context length blank only when detection is reliable
4Run a tool-use smoke test, inspect context and latency, then add a tested hosted fallback only if the privacy policy allows it

Real-World Use Cases

Private local agent for code and files

Run Hermes on a workstation with Ollama or LM Studio so sensitive project files, memory entries, and tool outputs stay on your machine instead of being sent to a hosted model provider.

LM Studio as the local model browser

Use LM Studio to download and compare quantized models, then point Hermes at LM Studio's local OpenAI-compatible API once you know which model handles your tasks well.

OpenRouter fallback for hard or rate-limited work

Keep a local model as the default, but configure a hosted provider for tasks that need a larger context window, better instruction following, or a temporary escape hatch when local inference is too slow.

Team GPU server with vLLM

Run vLLM on shared GPU hardware and let multiple Hermes agents use the same OpenAI-compatible endpoint for high-throughput internal automation without per-token vendor bills.

Choose the local backend first

Use Ollama when you want the shortest setup and automatic local model discovery. Use LM Studio when a desktop GUI for downloads, quantization, and GPU settings matters. Use vLLM or SGLang when several Hermes agents need one high-throughput GPU API. llama.cpp is the lightweight CPU, Apple Metal, and edge-server route.

Hardware: fit the model before chasing benchmarks

Small models can run in 8 GB of RAM but often struggle with reliable tool calls. The current Hermes Ollama guide recommends 32 GB or more for larger agent-capable models; current Ollama guidance lists local options around 16–24 GB VRAM. CPU-only inference works, but long tool loops can take minutes. Pick the smallest model that completes your actual Hermes workflow reliably.

Context and tool calls are the two acceptance gates

Hermes agent turns include the system prompt, tool schemas, file context, observations, and conversation history. A model can answer chat correctly and still emit invalid tool JSON or run out of context. Verify the model endpoint, run a file-read plus terminal task, inspect whether tools actually executed, and test a longer second turn before enabling cron or a gateway.

Local inference is not the same as fully offline Hermes

The model prompt can stay local while a web-search tool, browser session, cloud speech service, remote MCP server, or hosted memory provider still uses the network. For an air-gapped deployment, remove cloud provider keys, disable network-dependent tools, pre-download models and dependencies, and test with the network disconnected.

Use hybrid routing deliberately

A local-first setup can keep routine and sensitive work on your hardware while a hosted fallback handles complex tasks. That fallback is a data boundary, not only a reliability setting. Allow it only for workloads permitted to leave the machine, document the provider/model, and test one forced failure before trusting an unattended job.

Under the Hood

Ollama now provides a direct `ollama launch hermes` path that can install Hermes, let you choose a local or Ollama Cloud model, configure `http://127.0.0.1:11434/v1`, and optionally connect a messaging gateway. The manual Hermes path remains useful when you need explicit control: start the model server, run `hermes model`, choose a custom endpoint, verify model discovery, and then start a new session.

For slow CPU-only inference, set `HERMES_API_TIMEOUT=1800` in `~/.hermes/.env`; it is an environment variable, not a `config.yaml` key. For Ollama models with too-small context, create or run a variant with at least a 64K context window. Do not reduce the context until the model fits if the result is that Hermes loses tool schemas or working history.

The real benchmark is operational: can the model read a file, call the terminal, preserve the requested format, recover from a failed tool, and complete a second turn without drifting? Run that test before moving a Telegram bot, cron publisher, or background monitor to local inference.

Local does not automatically mean cheap or private. Hardware depreciation, power, queue time, updates, remote access, and debugging belong in the cost model. Network tools and hosted fallbacks belong in the privacy model. If you want the Hermes workflow without managing GPU serving, provider routes, gateway uptime, and remote access, FlyHermes is the managed alternative.

Frequently asked questions

Can Hermes Agent run with Ollama?

Yes. The shortest current path is `ollama launch hermes`. For manual setup, run `hermes model`, choose a custom endpoint, and use `http://127.0.0.1:11434/v1` with the exact model name shown by Ollama.

Which local LLM backend should I use with Hermes Agent?

Use Ollama for the easiest setup, LM Studio for desktop model management, vLLM or SGLang for a shared GPU server, and llama.cpp for lightweight CPU or Apple Metal serving. Model tool-call reliability matters more than a chat benchmark.

How much hardware does a local Hermes Agent need?

A small model can run with about 8 GB RAM, but dependable agentic work usually benefits from 32 GB RAM or roughly 16–24 GB VRAM for stronger local models. CPU-only works, but long tool loops are slower.

Why does my local model chat but not use Hermes tools?

Many small or conversational models do not emit reliable structured tool calls. Test a concrete file-and-terminal task, confirm the model advertises tool support, use a stronger instruction model, and verify the context window is large enough for Hermes tool schemas.

Is Hermes fully private when the LLM runs locally?

Local inference keeps model prompts on your hardware, but web search, browser automation, remote MCP servers, hosted memory, cloud speech, and fallback providers can still send data over the network. Disable or isolate those services for a fully offline setup.

Can Hermes fall back from a local LLM to a cloud model?

Yes. Configure `fallback_providers` or use `hermes fallback`, but treat the fallback as a privacy boundary: only permit tasks that may leave the machine, and test the route before using it for unattended work.

Related Features

model agnostic open source reinforcement learning multi agent

Try Hermes Free → Deploy in 60 seconds