Hermes Agent Cron Jobs Not Running: Diagnosis and Safe Recovery

A Hermes cron job can stop before inference, finish its work but fail delivery, or survive a restart with an uncertain outcome. Those are different incidents. Recreating the schedule or repeatedly triggering the job can spend more money and duplicate work without fixing the cause.

hermes --version
hermes cron list
hermes cron status
hermes cron doctor
hermes cron runs --limit 20
hermes cron incidents
hermes gateway status

Diagnose the failure layer first#

Keep these records separate:

• Schedule and lifecycle: is the job enabled, recurring, and due? A one-shot or exhausted repeat count can legitimately be completed.
• Dispatch: did the scheduled fire reach a worker? Inspect last_fire_error and dispatch timing.
• Execution: did an attempt start and finish? Inspect hermes cron runs and the recorded error.
• Delivery: did the result reach the exact target? Inspect last_delivery_error and the destination itself.
• Outcome: is the requested report, file, record, or live page correct? A successful model response is not an acceptance test.

Record the job ID, profile, scheduled instant, attempt ID, and expected artifact before changing settings. Use the Hermes monitoring guide for broader trigger-to-delivery observability rather than treating every failure as a model outage.

Fix blocked_config without disabling the safety check#

Pre-dispatch validation checks whether the run has usable provider credentials, ready attached skills, known delivery configuration, and the explicitly requested MCP tools. When validation fails, current Hermes records blocked_config, alerts once, and avoids starting inference.

First compare the profile and HERMES_HOME named in the error with the environment of the working interactive shell. A service or container can read a different .env and auth.json. A successful chat in your personal profile does not establish that the scheduler profile has credentials.

Then repair the named requirement:

1. Authenticate the provider in the owning profile without copying secrets into a job prompt.
2. Make required skill commands and credential files available to the service user.
3. Confirm the destination exists and its gateway credentials are configured.
4. Check that a named MCP server belongs to this profile and can resolve tools.
5. Confirm the absolute workdir and script still exist.

A previously connected MCP server briefly reconnecting is handled differently from a server that never connected or has permanently revoked credentials. Read the actual error instead of disabling preflight to force a run. The profile isolation guide explains why identical profile names on different machines are not proof of identical state.

Cron worker exits before the task starts#

An error such as cron external worker exited before ownership acknowledgement, followed by ModuleNotFoundError, points to worker startup. Changing the prompt, bot token or model will not install the missing dependency. First save the attempt ID and stderr from that attempt. A bare exit code is not enough to choose a repair.

The official cron worker documentation describes the managed systemd path: the worker uses the gateway's Python interpreter and the gateway's checkout on its import path. Other launch modes can run in-process. Do not assume every installation starts an external worker, or that an older release has the current launch behavior.

Identify the Python that actually failed#

Record the gateway's service command, interpreter, source checkout or image digest, profile and error timestamp. Use the service definition and logs on the machine running the gateway, not the Python selected in your laptop's shell. If the whole gateway lives in a container, its interpreter path belongs to that container.

Current Hermes package management separates application updates from dependency repair. Start with these diagnostics on the affected installation, if its release supports them:

hermes --version
hermes pm status
hermes pm doctor

Do not guess a venv/bin/python path. PM-managed installations can select a different Python generation, and an old process retains its imported modules until it restarts. A developer checkout and a packaged Desktop installation can also have different update owners.

For a narrower check, save this as environment-probe.py. It prints runtime paths and whether Python can locate the module named in the error; it does not print environment variables, install packages or start a job. Only probe a trusted dependency: locating a dotted module can import its parent package.

import importlib.util
import json
import os
import sys

module = sys.argv[1] if len(sys.argv) > 1 else "ruamel.yaml"
try:
    spec = importlib.util.find_spec(module)
except ModuleNotFoundError:
    spec = None
print(json.dumps({
    "executable": sys.executable,
    "prefix": sys.prefix,
    "base_prefix": sys.base_prefix,
    "cwd": os.getcwd(),
    "module": module,
    "found": spec is not None,
    "origin": spec.origin if spec else None,
}, indent=2))

Run it with the interpreter identified from the failed service. Replace the example path and module; do not copy a path from someone else's installation:

WORKER_PYTHON="/absolute/path/from-your-service/python"
"$WORKER_PYTHON" environment-probe.py ruamel.yaml

Compare that result with the working shell's interpreter. found: false in one environment and true in the other explains why a shell import is insufficient evidence. found: true only proves discovery, not that the dependency and its native libraries can load successfully. A probe in a new shell also does not reproduce every environment setting inherited by the service.

Repair the owner of the dependency#

For damaged dependencies in a PM-managed installation, the current documented repair is hermes pm repair, followed by restarting the affected Hermes process. Read the diagnostic result first. PM repair is not an application updater, and it cannot repair missing application source or a broken Python executable. Use the installation owner's update or reinstall procedure for those failures. Docker bakes its runtime into the image; do not patch a disposable container with an arbitrary global pip install and call the deployment repaired.

Installing a package through an agent's Docker or SSH terminal may change the task environment without changing the gateway worker at all. The terminal backend boundary guide explains which process owns each environment.

Before retrying the production task, check the previous attempt and external output even if the latest worker failed early. Preserve any completed work. After the runtime repair, an authorized harmless test should prove worker startup, one read-only task and the expected local artifact. Verify channel delivery separately through the gateway recovery checks. A startup fix is not proof of a successful customer message or publication.

We checked the probe in two disposable Python environments, with ruamel.yaml installed in only one. It distinguished the environments, and a separate import succeeded only where the package was installed. This was a dependency-isolation test, not a reproduction of a Hermes scheduler failure or evidence that a particular release fixes every import error.

Check which model a scheduled job will actually use#

Current official documentation gives this resolution order:

1. A per-job model/provider pin.
2. The fleet default, cron.model and cron.model_provider.
3. The main agent model when the job fires.

An unpinned job can follow a changed default. Do not rely on older advice that every job automatically retains a creation-time model snapshot or that a model-drift guard always blocks a changed route. Inspect the installed version and the job's actual pin.

For an operator intentionally changing an existing job, the documented CLI controls are:

hermes cron edit <job_id> --pin
hermes cron edit <job_id> --unpin

--pin locks in the current main model; --unpin releases that lock. Choosing an explicit model/provider or a fleet default is an operator decision, not something an unattended job should silently do. A pinned job does not use the normal fallback chain. That improves route predictability but means an unavailable pinned provider can block the job even when interactive fallback works.

Keep provider spending alerts and bounded retries outside the prompt. Consult the provider cost and rate-limit runbook before adding another paid route. A pin is a routing control, not a monetary spending cap.

Distinguish a quota hold from a failed API call#

Some recurring jobs can be held when provider resolution reports an exhausted subscription window, supplies a retry-after duration, and the fallback chain is unavailable. The current documentation specifically describes the OpenAI Codex usage probe. Inspect quota_hold_until and next_run_at rather than assuming the scheduler forgot the job.

This is not every HTTP 429. A model API rate limit returned after execution begins follows the normal run behavior, not necessarily the pre-run quota-hold path. Dense schedules, intervals, manual runs, and sparse cron schedules also have different recovery rules. Do not promise an immediate catch-up merely because a provider window reopened.

For transient network failures before any model call, current Hermes can automatically retry recurring jobs. That is different from retrying a job that already edited files, posted messages, or called an external API. Determine which case you have before manually triggering another attempt.

Recover delivery_failed without repeating the work#

When execution succeeds but delivery lacks positive send evidence, current Hermes records last_status: delivery_failed with last_delivery_error. A delivery failure does not increment the execution failure streak. A quiet failure streak therefore does not prove that reports reached their readers.

Check whether the output artifact already exists. Then check the configured channel, chat, or thread, membership and permissions, platform errors, and active gateway profile. A message in the parent chat is not proof of delivery into the requested topic. Use the gateway troubleshooting guide when execution works but the channel stays silent.

For work with side effects, recover or redeliver the existing artifact where supported instead of regenerating it automatically. Reconcile first: a timeout can leave uncertainty about whether a send or external write succeeded. Never turn “I did not receive the report” into an unconditional instruction to repeat a payment, publication, or customer message.

Reconcile unknown attempts after a restart#

Hermes records attempts in the profile-local cron/executions.db. Attempts progress through claimed and running states to completed, failed, or unknown. An abandoned attempt is marked unknown when its recorded owner is proven gone; it is an audit record, not an automatic rerun request.

Before rerunning:

1. Inspect the attempt history and its timing.
2. Read back the exact external target: file, message, commit, deployment, or record.
3. Determine which steps completed and which remain uncertain.
4. Use an idempotent recovery action or continue only the missing step.
5. Verify the result before allowing the recurring schedule to continue.

Retained ledger identities can prevent some replay after restoring an old jobs file, but this is not exactly-once execution. Pruned history, legacy rows, interrupted attempts, an unavailable ledger, or restoring an older ledger can remove the evidence needed to identify a duplicate.

On systemd hosts, restart-safe workers require an appropriate user scope. Current Hermes can degrade to an external process without that scope; a gateway restart may then kill the worker. Operators who require fail-closed behavior can review cron.require_restart_safe_scope in the official docs. Do not assume Docker, a VPS, or a running gateway alone proves restart safety.

Interpret cron doctor and incidents correctly#

hermes cron doctor is read-only. It reports run failures, delivery failures, dispatch errors, missing scripts or workdirs, and suspicious next-run timing. Its exit code can remain nonzero after a successful catch-up because historical lateness clears at the next on-time dispatch.

That distinction matters for monitoring: an alert that says “job still failing” solely because doctor returned a finding can be misleading. Read the finding and distinguish a recovered-but-late run from a new failure.

hermes cron incidents tracks durable failure signatures. Repeated identical failures can be quiet during the reminder cooldown even while run history and failure streaks continue growing. Acknowledging an incident silences that signature; it does not repair the job. Inspect recent attempts rather than using absence of alerts as your health check.

Keep new scheduled jobs small and testable#

Use an agent job for judgment, research, writing, or browser work. Use script-only mode when a deterministic script can produce the final result without an LLM. A pre-run script can also decide whether an agent needs to wake. These choices change inference cost, not the need for working infrastructure and delivery.

Include this contract in a scheduled task:

Inputs: exact files, URLs, and allowed sources.
Workdir: absolute project directory on the runner.
Output: one named artifact and the intended destination.
Limits: bounded retries, allowed tools, no unapproved writes.
Completion: read back the artifact and verify its contents.
Failure: report the failing layer and preserve partial work.
Recovery: check for prior side effects before repeating a step.

The step-by-step scheduling tutorial owns initial setup. Cron expressions use the scheduler host's local timezone, so compare its clock with next_run_at. A sleeping or stopped host cannot execute a local schedule. Test one harmless job in the same profile, runtime, workdir, and destination that production will use; a manual trigger only queues execution and is not proof of completion.

Self-hosted control or managed operations?#

Self-hosting lets you choose providers, scripts, filesystem access, and gateway routing. It also makes someone responsible for uptime, updates, credential renewal, service supervision, backups, delivery incidents, and recovery after restarts. Name that owner before a customer or team depends on the schedule.

The self-hosted versus hosted responsibility comparison helps decide whether that control is worth the work. FlyHermes is the managed route for users who want browser/mobile access and connected channels without a VPS and gateway maintenance project. Check the plan's actual capabilities and model billing separately; managed hosting is not an exactly-once guarantee or proof that a particular external workflow succeeded.

FAQ#

Why does a cron job work manually but fail on schedule?#

Compare the profile, service environment, workdir, provider pin, tools, and delivery target. The interactive shell may have credentials or commands that the scheduler cannot read.

Does an unpinned job keep its original model?#

Not according to current official documentation. It follows the fleet cron default when configured, otherwise the main model at fire time. Inspect the installed release and set a deliberate pin when stable routing matters.

Should I rerun an unknown attempt?#

Not before checking the external artifact. Unknown means the recorded attempt lost its owner, not that every side effect failed. Reconcile first and repeat only what is safe.

Why does cron doctor still warn after recovery?#

A successful catch-up does not clear the recorded lateness finding immediately. The next on-time dispatch clears that warning; inspect the specific finding instead of assuming the latest run failed.

Does delivery_failed mean the agent did nothing?#

No. The work may have completed while the send failed. Inspect the artifact and delivery error before regenerating the work.

Keep unattended approvals separate from dashboard login#

Signing into the Hermes Dashboard does not grant a scheduled worker blanket permission to run dangerous commands. The documented approvals.cron_mode: deny setting fails closed when unattended execution reaches an approval requirement. Inspect the blocked run before retrying; do not switch all jobs to auto-approve merely because no human was present. Narrow the task, use a reviewed deterministic operation where appropriate, or arrange an attended run with the required decision.

Review the server security acceptance checklist for the separate dashboard, messaging and command-approval boundaries. A job can be authorized to start and still be denied a particular operation; record that as a blocked outcome, not a delivery failure or a reason to rerun already-completed side effects.

For a concrete input-to-output example, the apartment-alert workflow turns saved-search emails into a draft shortlist. It shows why mailbox access, deduplication and delivery checks need separate acceptance tests before scheduling.