What's Wrong with Hermes Function Calling in Goose?

Today I read an interesting post: Can MCP Help a Small Local Model Shine? Hands-on with Goose.

The author tried running two small local models in Goose: Hermes 2 Pro Mistral 7B and Gemma 4 E4B. Surprisingly, Hermes failed to complete the tool calls, while Gemma succeeded. In short, her conclusions were:

• Tool calling is not simply about model size. Reliability depends on whether the model's call format fits the host runtime.
• Tool failures can be quiet. A model may produce a confident answer even though no tool was executed, so users should inspect the tool trace and verify important values.
• Small local models can be good orchestrators when tools do the heavy work.

I agree with the overall direction. Putting deterministic or computationally heavy logic into MCP servers can make a small local model much more capable as an agent.

But the result also raised a few questions for me:

1. Hermes 2 Pro was fine-tuned for function calling. Why did it perform worse here?
2. The author suggested that the problem was a mismatch between the model's call format and Goose. Was that actually demonstrated, or inferred from the final failure?
3. She also mentioned that Hermes still failed after native tool calling was enabled. What does "native" mean here, and what does it really guarantee?

The post showed that Hermes did not execute the tool, but not the rendered prompt, raw model output, parser result, or MCP dispatch record. So I decided to walk through the pipeline and reproduce the problem myself.

A Complete Tool-Use Journey in Goose

Because Goose is open source, we can split one tool call into several stages instead of treating it as a single magical model ability.

The process looks roughly like this:

1. The user sends a prompt to Goose.
2. Goose collects tool metadata from active MCP extensions, including names and JSON Schemas.
3. Goose passes the conversation and tool definitions into a chat template.
4. The rendered prompt is sent to the local model through llama.cpp.
5. The model generates raw text or tool-call syntax.
6. On the native path, llama.cpp parses the model-specific syntax, and Goose turns the streamed result into a ToolRequest.
7. Goose dispatches the request to the correct MCP server.
8. The MCP server returns a ToolResult.
9. Goose adds the tool request and result to the conversation and renders another model turn.
10. The model writes its final answer using the real result.

Once the flow is separated, a failed tool call has several possible causes:

• the schema never entered the rendered prompt;
• the model never generated a call;
• the model generated a call format that the parser did not understand;
• the call was parsed but never dispatched;
• the result came back, but the model failed to use it.

The first one is especially important. If the model never sees the tool schema, there is little point arguing about its tool-selection ability.

What Is a Chat Template?

A model does not literally receive a JavaScript object containing system, user, assistant, and tools. It receives token IDs.

A chat template is the rendering program that converts structured conversation data into the textual protocol expected by the model. It decides how roles are separated, where tool definitions go, what a tool call should look like, and how tool results are represented in later turns.

Many templates use special tokens such as <|im_start|> and <|im_end|>. A special token is a tokenizer vocabulary entry used to mark structure rather than ordinary prose. If it is registered as a special token, it maps to one token ID instead of being split into several pieces.

The exact token IDs vary by tokenizer. The important part is consistency: during fine-tuning, the model learns that certain boundaries mean "user message," "assistant message," or "tool call." At inference time, the host should speak a compatible protocol.

It is also worth noting that not every XML-looking string is a special token. A model can produce <tool-result> as ordinary text even if the tokenizer does not contain a dedicated token for it.

Why GGUF Matters

Goose uses llama.cpp for local inference, and local models are commonly distributed as quantized GGUF files.

GGUF is not just a compressed weight file. It can also include tokenizer metadata, special-token definitions, architecture information, quantization metadata, and a chat template. In other words, the GGUF is part of the runtime contract.

In Goose v1.40.0, the featured model registry included:

FeaturedModel {
    spec: "bartowski/Hermes-2-Pro-Mistral-7B-GGUF:Q4_K_M",
    native_tool_calling: false,
    mmproj: None,
},
FeaturedModel {
    spec: "unsloth/gemma-4-E4B-it-GGUF:Q4_K_M",
    native_tool_calling: true,
    mmproj: Some(MmprojSpec {
        repo: "unsloth/gemma-4-E4B-it-GGUF",
        filename: "mmproj-BF16.gguf",
    }),
},

The Hermes entry points to a community-converted GGUF rather than loading files from the original Hermes repository at runtime. The mmproj difference is unrelated to text tool calling; it is for multimodal projection weights.

The native_tool_calling flag also needs careful interpretation. It is not a verdict on the entire model family. In this Goose version, it affects the default routing mode for that featured artifact.

Working Backward from the Failure

I ran Hermes and Gemma against the same deterministic MCP tools and captured the actual applied_prompt sent to each model.

The Hermes prompt contained the Goose system message and the user's request:

<|im_start|>system
...
## node
...
<|im_end|>
<|im_start|>user
You must call the lookup_metric tool exactly once with project=alpha.
Do not guess the value. After the tool returns, answer with the exact marker.
<|im_end|>
<|im_start|>assistant

But it did not contain the definition of lookup_metric, its parameters, or the required call format.

The Gemma prompt did:

<|tool>declaration:node__compare_metrics{...}<tool|>
<|tool>declaration:node__lookup_metric{...}<tool|>

At this point the first concrete failure was visible: Hermes had been asked to call a tool it had never been told how to call.

Where Did the Schema Go?

The relevant GGUF metadata key is tokenizer.chat_template. In my Chinese draft I initially called it tokenizer.chat_platform, which was simply the wrong name.

The embedded template in the Hermes GGUF was only a small, ordinary ChatML template:

{% for message in messages %}
{{ '<|im_start|>' + message['role'] + '\n'
   + message['content'] + '<|im_end|>\n' }}
{% endfor %}
{% if add_generation_prompt %}
{{ '<|im_start|>assistant\n' }}
{% endif %}

It never referenced tools, so passing tools_json into the renderer did not make the tools appear in the prompt.

Gemma's embedded template was much larger and contained explicit tool-handling logic:

{%- if tools -%}
  {%- for tool in tools %}
    {{- '<|tool>' -}}
    {{- format_function_declaration(tool) | trim -}}
    {{- '<tool|>' -}}
  {%- endfor %}
{%- endif -%}

Goose supports embedded, built-in, and custom inline templates. With the default embedded option, it asks llama.cpp to load the template from the GGUF:

ChatTemplate::Embedded => LoadedChatTemplates {
    default: model.chat_template(None).ok(),
    tool_use: model.chat_template(Some("tool_use")).ok(),
    force_default: false,
}

Goose does not automatically fetch the original model's Hugging Face tokenizer configuration. It treats the local GGUF as the model package, which is sensible for offline and reproducible inference.

However, the official Hermes tokenizer configuration contains both a normal default template and a dedicated tool_use template. The latter serializes tool definitions and tells the model to emit JSON inside <tool_call>...</tool_call>.

So this was not evidence that Hermes had never learned a function-calling protocol. The particular GGUF and selected template did not carry that protocol into the Goose prompt.

To confirm this, I replaced the GGUF's embedded template with the upstream Hermes tool_use template while keeping the same quantized weights. Tool calling then worked end to end: the schema appeared in the prompt, Hermes generated <tool_call> JSON, the parser emitted a ToolRequest, the MCP fixture recorded the call, and the final answer used the real marker.

That was the missing counterfactual. It turned a plausible explanation into a much stronger one.

What Does Native Tool Calling Mean?

The original post noted that Hermes still failed after native tool calling was enabled. At first glance, "native" sounds like a feature that should make tool use more stable. But it is better understood as a routing and protocol choice.

On Goose's llama.cpp native path, Goose supplies OpenAI-compatible messages and tool definitions:

OpenAIChatTemplateParams {
    messages_json,
    tools_json: tools,
    use_jinja: true,
    parse_tool_calls: true,
    ...
}

The selected template must still serialize those tools into the model's expected format. The model must generate the matching call syntax, and the llama.cpp parser must recognize it.

ForceNative forces that route, but it does not rewrite a broken or incomplete template. If the Jinja template never reads tools, the schema silently disappears during rendering.

So native tool calling is not a magic stability switch. It is a compatibility contract among the model's learned protocol, tokenizer, template, renderer, parser, and host runtime.

I also found a subtler issue in Auto mode. Goose's dry-run capability check considered this Hermes path native-compatible because llama.cpp reported parser capability, even though the rendered prompt still omitted the schemas. The check validated part of the pipeline, not the property we actually cared about.

What About Emulated Tool Calling?

Goose also has an emulated path for models that cannot use a working native protocol.

Instead of relying on the model's own tool-call format, Goose injects an application-level protocol into the system prompt. In the code mode I tested, the model is asked to generate an execute_typescript fenced block:

```execute_typescript
async function run() {
  const result = await Node.lookupMetric({ project: "alpha" });
  return result;
}
```

Goose's emulator parser watches for that exact fence and converts the code into a request to its code-execution layer.

This gives models without a usable native template another chance to call tools. But it does not remove the need for protocol alignment; it simply introduces a different protocol.

Hermes, for example, produced:

```typescript
await Namespace.lookup_metric({ project: "alpha" });
```

Please provide the exact instruction to proceed.

It used the wrong fence, invented the wrong namespace, and asked for confirmation instead of executing.

Gemma failed in a different way:

I need to find the lookup_metric tool.
<|tool_call>call: list_functions{}<tool_call|>

It chose a sensible discovery function but fell back to its native-looking syntax while Goose was waiting for the emulation fence. The parser ignored it.

I initially described these failures too broadly as weak instruction following. A more accurate description is protocol-generation mismatch: the model produced something reasonable-looking, but not the dialect the active adapter could parse.

The More Interesting Failure: Hallucinated Results

Some failed runs were more revealing than the successful ones.

Without a tool schema, Hermes sometimes narrated an imaginary tool execution:

Calling lookup_metric tool with project=alpha...

<tool-result>marker: 735</tool-result>

The exact marker from the tool result is: `marker: 735`

The strongest evidence that this was fabricated was not merely that the MCP fixture was configured to return 518. There was no parsed ToolRequest and no entry in the fixture's independent call log. The value first appeared in the model's raw generation.

The <tool-result> string was also not part of the active Goose protocol. It was unrecognized tag-like text, not necessarily a tokenizer special token.

I later ran a small follow-up comparing how Hermes and Gemma behaved when neither model had any tools at all. Goose explicitly showed an empty tool list, while the user asked for a fictional audit_marker result.

Gemma consistently said the tool was unavailable. Hermes variously repeated the placeholder as if it were a result, promised that it was about to call the tool, and once generated a concrete value:

AUDIT_EXECUTED::alpha::5f37594b904f408106016c242a536209

It is tempting to call this an honesty test, but that wording is stronger than the evidence. What I actually observed was runtime-state fidelity: in this setup, Gemma respected the fact that no tool was available, while Hermes was more likely to complete the user's requested success story without execution evidence.

This was a small diagnostic experiment, not a general Hermes-versus-Gemma personality test. Still, it is a useful reminder that tool-use reliability also depends on how a model responds when the infrastructure cannot satisfy its request.

Conclusion

After going through the whole pipeline, I came back to the original author's conclusion with more respect for it:

Reliability came from the fit between a model's call format and the host, not from parameter count.

But I would now state the Hermes failure more precisely:

In Goose v1.40.0, the selected Hermes GGUF supplied an embedded chat template that did not serialize tool definitions. Forcing the native route did not repair that template. Once the same quantized weights were paired with the upstream Hermes tool_use template, tool calling worked end to end.

So the experiment did not show that "Hermes 7B cannot call tools." It showed that tool calling belongs to the entire serving stack:

weights + GGUF metadata + chat template + renderer + parser + host routing

I do think Goose could make this failure much easier to understand. If a user forces native mode with a template that ignores tools, the runtime should warn loudly. It could also make template overrides more visible, or optionally support importing a trusted upstream template.

At the same time, the GGUF artifact itself should preserve the model's intended tool-use contract whenever possible. This is a boundary problem, not something that can be blamed entirely on either the model or the infrastructure.

I also still agree with the broader architecture: local models become far more useful when deterministic and computationally heavy work is delegated to MCP tools. But orchestration is not trivial. The model still has to choose the right tool, construct valid arguments, notice when no evidence exists, and explain failure instead of inventing success.

The final answer was fairly simple. The interesting part was learning how to get there: inspect the rendered prompt, raw generation, parser output, MCP log, and second model turn separately. Once those stages are visible, a vague "the model failed" becomes a concrete engineering problem.

The reproducible experiment package, including locked artifacts and raw evidence, is available in hermes-tool-calling-lab.