Telemetry & Tracing

Trace what your agents did

Agents now act on staging and production unattended, and off the shelf harnesses leave no record you can hold them to.

One session, one trace

Grove is an OpenTelemetry gateway in front of your agents, so each turn arrives as one trace in LangFuse or any OTLP backend.

• Every trace says what ran, where, at what cost, and where the time went between model wait and tool work.
• A harness exporting its own spans gives you a flat pile that cannot say which repository it ran in. Grove stamps identity on every span and rebuilds the parent and child shape a raw export throws away.
• Names and attributes follow the OpenTelemetry GenAI semantic conventions, so a Claude session and a Codex session are comparable rather than merely adjacent.
• Once traces land, the usual questions become queries. Which models the team runs, what a change cost, which tool calls eat the time.

What Grove puts on a span

A raw harness span knows the model and the token counts. It does not know the workspace, the branch or the ticket, because the harness was never told.

• Grove composes that identity into OTEL_RESOURCE_ATTRIBUTES when it launches the agent, so every span the process emits carries it.
• Grove's spans and the agent's come from two processes that share no context. Both carry the same langfuse.session.id, so LangFuse files them under one session rather than two unrelated traces.
• The same identity is written flat as metadata, grove_workspace_id, grove_repo, grove_branch and the rest, so a cohort can be scoped to one repo or split by harness.

Shell calls, in one shape you can grade

A shell command is where an agent touches the machine, so it is the tool call most worth reviewing, and no two harnesses spell it the same way. Grove normalizes every one into a single observation shape, a shipping label that reads the same whatever box it came in.

• input carries a stable command string plus the harness's own argv and arguments. output carries content, a truncated flag and exit_code when one was recorded.
• The facts you filter on are flat metadata. grove_tool_category is always shell, grove_shell_outcome is succeeded, failed, pending or unknown, and grove_tool_source says whether Grove replayed the transcript or re spelled the harness's own span.
• To grade them, create an LLM judge evaluator in LangFuse, filter on grove_tool_category equal to shell, and map its variables to input and output. Grove records facts and runs no evaluation of its own.
• The shape refuses to guess. Success is claimed only from a recorded exit code of zero, a cancelled call is not distinguishable from a failed one, a background call's result is its launch handle, and a payload over 4000 characters says it is truncated.

Turn it on

Telemetry ships in the telemetry extra, which grove-crew[all] includes. Grove reads variable names from config and values from the environment, so no secret is ever written to a config file.

{
  "telemetry": {
    "enabled": true,
    "host_env": "LANGFUSE_HOST",
    "public_key_env": "LANGFUSE_PUBLIC_KEY",
    "secret_key_env": "LANGFUSE_SECRET_KEY"
  }
}

• telemetry.env_file loads those variables from a file, or telemetry.env_command from a secrets manager. A daemon reads its environment once, so a file is what survives a key rotation.
• Grove reserves the OTLP exporter variables for every agent it launches, listed in telemetry.reserved_env, so a stray OTEL_EXPORTER_OTLP_ENDPOINT in your shell cannot split one session into two traces. A displaced value logs one warning naming the variable.

Receiving the harness's own traces

Claude Code and Codex can export OTLP directly, and Grove can be the collector they export to, rewriting what arrives before it lands.

{
  "telemetry": {
    "receiver": { "enabled": true, "path": "/otlp" }
  }
}

• The daemon serves POST /v1/traces under that path, off by default and bound to loopback. A burst from a twenty agent fleet cannot stall the dashboard, and a full queue sheds through OTLP's own partial_success.
• The Claude Code transform finds the turn root, reparents the sub agent tree onto it, flags errored tool calls and rewrites names into Grove's vocabulary. Anything unclaimed passes through untouched.

Backfilling history

grove usage backfill --telemetry replays sessions that predate telemetry.

Exporting old sessions writes off this machine, so it needs the --telemetry flag, an explicit --yes, and the profiles named in telemetry.backfill.profiles. --dry-run plans without writing, and without --telemetry the command only rebuilds the local projection.

See also

• Agent activity and sessions: the live half of the same signal.
• Configuration reference: every telemetry key.
• Agents: declaring an agent's kind.