🔭 OpenClaw Benchgen Observability Plugin

Stream OpenClaw agent traces to
Benchgen for observability and training-data capture,
and let Benchgen chat with your agent.

What is new in 0.8.5

A batch of spans that fails to reach Benchgen no longer disappears without a trace. OpenTelemetry reports a failed or timed-out export only to its own diagnostics logger, which nothing read, so a dropped batch left traces with just the spans sent before it (a cron run showing only its context, for example). Those errors now go to the gateway log as benchgen: OpenTelemetry error: ..., and one export may take 30 seconds instead of the SDK's 5, which a batch closing a long run can need.

A tool span closed by the idle reaper keeps the end time it reported, so a one-second command in a run that never completes no longer shows as minutes long.

What is new in 0.8.1

The shell environment of a Benchgen chat turn now also carries who is asking: BENCHGEN_USER_ID (the platform account, as the relay's headers name it) and BENCHGEN_USER_NAME, next to BENCHGEN_API_URL and BENCHGEN_API_TOKEN. A skill that reads data of several users (harness-improve over the agent's own traces) scopes itself by them; they never pass through the model, so a prompt cannot rename the caller.

What is new in 0.8.0

Questions with options become buttons. When the agent calls OpenClaw's ask_user tool in a Benchgen chat session, the plugin sends the question and its options to Benchgen as a choices frame ({questions: [{id, header, question, options: [{label, description?}], multiSelect}]}), and Benchgen's chat draws them as buttons under the reply; a click sends the option's label as the user's next message. The tool call itself is still blocked (the interactive control would hang the turn), and the block reason now tells the model that the options are on screen, so it ends its reply with a one-line question that names the options and stops. A Benchgen without button support ignores the frame; the model's one-line question still arrives. Frames of the ask_user call are not step frames.

What is new in 0.7.2

Step frames on every OpenClaw version. Benchgen shows what the agent is doing ("Launching the benchmark run") from tool.start frames, but they relied on the host's onToolStart reply option, which OpenClaw 2026.9 does not call, so no step ever arrived. Tool starts are now reported from the before_tool_call hook, which fires on every version; a host that also calls onToolStart is de-duplicated. A step frame now carries only a shell tool's command, never the whole parameter object: the exec tool's parameters get the user's API credential injected as environment, and other tools' arguments have no reason to leave the gateway.

What is new in 0.7.1

Turn heartbeat. Benchgen drops a chat turn after 5 minutes without a frame ("The agent went quiet mid-turn"), and a healthy turn can be silent for longer: one long tool call, or the model writing a large file, emits nothing until it completes. The plugin now sends a turn.progress frame once a minute between turn.started and turn.done. It carries no text and needs no change on the Benchgen side. The relay's hard cap of 15 minutes per turn still applies.

What is new in 0.7.0

Private data guard. One gateway can serve the public BenchGen platform chat and a private team channel while internal files sit on the same disk. List the private locations in privateData.paths (or BENCHGEN_PRIVATE_PATHS) and the sessions that may read them in privateData.allowSessions (or BENCHGEN_PRIVATE_ALLOW_SESSIONS): fragments of the session key, such as a Telegram group id or agent:main:main. Every other session, platform chat included, gets the tool call refused, and the log names the session key. It is deny by default and it is a guard rail, not a sandbox: it matches the literal path in the tool parameters, so it stops ordinary requests, not a determined attacker. Hostile users need a separate gateway.

What is new in 0.6.0

Trace mirror. A gateway can send a copy of every trace to a second Benchgen project, on top of its own: set mirror.publicKey and mirror.secretKey (or BENCHGEN_MIRROR_PUBLIC_KEY / BENCHGEN_MIRROR_SECRET_KEY). The mirror inherits the primary endpoint unless mirror.baseUrl says otherwise, and mirror.environment (for example dev) is stamped on the mirrored copy only, so those traces stay recognisable inside the mirror project. Chat is not mirrored. A mirror that points at the primary project is refused, and openclaw benchgen show prints the effective mirror.

What is new in 0.5.3

Per-turn usage for Benchgen's accounting. LiteLLM knows what the gateway spent but not for whom (one key for every user), so the plugin now sums the model.usage events of a turn (usage.js, fed by the tracer) and sends a turn.usage frame on the relay a moment after turn.done (usageSettleMs, 1500 ms, so the last usage event of the turn is counted): {conversationId, messageId, sessionKey, agentId, status, sender, usage: {calls, inputTokens, outputTokens, totalTokens, cacheReadTokens, costUsd, model}, toolCalls, durationMs}. Benchgen stores one row per turn and shows turns and tokens per platform user on Administration > Usage. The HTTP endpoint (SSE) does not get the frame, its response is over by then. Token numbers need the model entry to report usage (compat.supportsUsageInStreaming: true for a hand-declared LiteLLM model, see 0.5.0).
The user's platform API credential for the agent's skills. Benchgen's relay may send auth: { token, apiBase } with a message frame (platform agents only): the token of the user behind the turn and the API root it is good for. The chat bridge keeps it per session key (authOf) and a before_tool_call hook adds it to every exec/bash tool call of that session as environment variables, BENCHGEN_API_URL and BENCHGEN_API_TOKEN, so a skill does curl -H "Authorization: Token $BENCHGEN_API_TOKEN" "$BENCHGEN_API_URL/submissions/?mine=true". It is never prompt text: no session transcript, trace or model request carries it, and a value the model itself puts into env is overridden. A turn without the field clears the previous credential; hosts without api.on run without it. Needs the same hooks.allowConversationAccess entry as the context block (0.5.0 below).
Fix: the context block never reached the prompt. OpenClaw 2026.7.2 registers a plugin once per instance in the same gateway process (gateway + pre-warmed agent runtime), and before_prompt_build fires in the runtime instance, which has no chat bridge. The per-session blocks now live in a process-global store (Symbol.for key) written by the turn runner and read by the hook through contextForSession.

What is new in 0.5.0

The platform's per-user context block. Benchgen's relay may send context with a message frame (platform agents only): a short text block with the user's name, credits and their recent datasets, benchmark runs, training jobs and agents. The chat bridge keeps it per session key (contextOf) and a before_prompt_build hook puts it into the system context of that turn as prependSystemContext, so it never lands in the session history or in the user message the model router routes on. Hosts without api.on (before 2026.7.2) run without the block. Capped at 8000 characters; a turn without the field clears the previous block. Operators: OpenClaw builds that gate typed hooks (the npm 2026.7.2-beta.* and 2026.8.* lines) block before_prompt_build for a non-bundled plugin and log typed hook "before_prompt_build" blocked unless the plugin entry allows it:
```
"plugins": { "entries": { "benchgen": {
  "hooks": { "allowConversationAccess": true, "allowPromptInjection": true }
} } }
```
Token usage on generations comes from OpenClaw's model.usage event, which the host emits only when the provider returned usage. For a model declared by hand under an OpenAI-compatible provider (LiteLLM, a router) set compat.supportsUsageInStreaming: true on the entry, otherwise OpenClaw sends no stream_options.include_usage, the stream carries no usage, and the trace shows a generation with usageReported: false and zero tokens.

What is new in 0.4.0

Chat turns that arrive through the BenchGen relay now carry the person on the trace: Langfuse userId is the BenchGen account (the relay sends the platform email as the sender id), sessionId is the BenchGen chat, and the display name sits in the trace metadata as user_name. Runs without a person (cron, heartbeat, CLI) keep the OpenClaw agent id in userId, as before. Nothing to configure.

Why This Plugin

The plugin runs inside the OpenClaw Gateway process. It subscribes to the diagnostics event stream and turns each conversation turn into one nested trace (model calls, tool executions, and retrieval steps) grouped by session. Those traces flow to your Benchgen project, where they can be viewed and exported for analysis, SFT, and RL fine-tuning.

@benchgen/benchgen-openclaw adds native Benchgen tracing for OpenClaw runs:

LLM request/response spans (with token usage and cost)
Tool call spans with inputs, outputs, and errors
Retrieval / search spans
Run-level metadata, grouped by OpenClaw session

Prompt, response, and tool I/O text are recovered best-effort from the per-session trajectory transcript, since message content is not delivered over the public diagnostics bus. A trace's response is what the user got in that turn: the texts the agent posted with the message tool (how group chats are usually answered), then its final text unless that is NO_REPLY or repeats a sent text. A turn that has not replied yet has no response, never the previous turn's. One trace is one run: runs OpenClaw starts on its own (cron jobs, background command completions) share the gateway's trace context, and each still gets a trace of its own.

The same plugin also connects the agent to Benchgen for chat: Benchgen can send messages to the agent and receive its replies (streamed), through an outbound WebSocket the plugin keeps open (works behind NAT) or through a direct POST /benchgen/chat on the gateway. Chat turns are ordinary agent turns, so they are traced like everything else. See Chat with Benchgen.

If your gateway is remote, install and configure the plugin on that host.

Install and first run

Prerequisites:

OpenClaw >=2026.6.1 for traces; >=2026.7.2 for chat (the chat bridge needs api.runtime.channel.inbound.dispatch, which 2026.7.1 does not expose; the plugin logs this and keeps streaming traces)
Node.js >=22.12.0

1. Install the plugin in OpenClaw

openclaw plugins install @benchgen/benchgen-openclaw

If the Gateway is already running, restart it after install.

2. Configure the plugin

openclaw benchgen configure

Paste the agent's public key and secret key from the Observability card on its Benchgen page. The wizard verifies the keys against the ingest endpoint, asks whether Benchgen may chat with the agent (default: yes) and for the chat relay URL (the card's BENCHGEN_CHAT_URL; Enter keeps the built-in default, which is the production relay), and writes it all into your OpenClaw config, no manual JSON editing. Restart the gateway to apply.

3. Check effective settings

openclaw benchgen status

4. Send a test message

openclaw gateway run
openclaw message send "hello from openclaw"

Then confirm traces in your Benchgen project.

Configuration

Recommended config shape

{
  "plugins": {
    "entries": {
      "benchgen": {
        "enabled": true,
        "config": {
          "enabled": true,
          "publicKey": "pk-your-public-key",
          "secretKey": "sk-your-secret-key",
          "baseUrl": "https://traces.benchgen.com",
          "chat": {
            "enabled": true,
            "sessionScope": "conversation"
          },
          "mirror": {
            "publicKey": "pk-second-project",
            "secretKey": "sk-second-project",
            "environment": "dev"
          }
        }
      }
    }
  }
}

mirror is optional: leave it out and traces go to the primary project only.

Chat keys (all optional, under config.chat):

Key	Description	Default
`enabled`	Chat bridge on/off (relay + gateway endpoint)	`true`
`relay`	Keep an outbound WebSocket to Benchgen so it can reach this gateway behind NAT	`true`
`url`	Relay WebSocket URL; the agent page's Observability card shows the one for your BenchGen environment (`BENCHGEN_CHAT_URL`); `configure` asks for it	`wss://benchgen.com/api/public/openclaw/chat`
`httpEndpoint`	Serve `POST /benchgen/chat` on the gateway port	`true`
`sessionScope`	`"conversation"`: one agent session per Benchgen conversation; `"main"`: the agent's main session	`"conversation"`
`agentId`	Agent that answers when the message names none	routed/default agent

Mirror keys (all optional, under config.mirror):

Key	Description	Default
`enabled`	Mirror on/off	`true` when both keys are set
`publicKey`	Public key of the second project (`BENCHGEN_MIRROR_PUBLIC_KEY`)	none (no mirror)
`secretKey`	Secret key of the second project (`BENCHGEN_MIRROR_SECRET_KEY`)	none (no mirror)
`baseUrl`	Ingest endpoint of the second project (`BENCHGEN_MIRROR_BASE_URL`)	the primary `baseUrl`
`environment`	Label stamped on the mirrored copy only (`BENCHGEN_MIRROR_ENVIRONMENT`)	none

Private data keys (all optional, under config.privateData):

Key	Description	Default
`paths`	Path prefixes only allowed sessions may touch (`BENCHGEN_PRIVATE_PATHS`, comma separated)	none (guard off)
`allowSessions`	Session key fragments that may touch them (`BENCHGEN_PRIVATE_ALLOW_SESSIONS`, comma separated)	none (everyone refused)

Environment fallbacks

Set these before starting the gateway; they are used as fallbacks when no config keys are present:

Variable	Description	Default
`BENCHGEN_PUBLIC_KEY`	Project public key	none (required)
`BENCHGEN_SECRET_KEY`	Project secret key	none (required)
`BENCHGEN_BASE_URL`	Ingest endpoint	`https://traces.benchgen.com`
`BENCHGEN_CHAT_URL`	Chat relay WebSocket URL	`wss://benchgen.com/api/public/openclaw/chat`
`BENCHGEN_CHAT_ENABLED`	`false` turns the chat bridge off	`true`
`BENCHGEN_MIRROR_PUBLIC_KEY`	Mirror project public key	none (no mirror)
`BENCHGEN_MIRROR_SECRET_KEY`	Mirror project secret key	none (no mirror)
`BENCHGEN_MIRROR_BASE_URL`	Mirror ingest endpoint	the primary endpoint
`BENCHGEN_MIRROR_ENVIRONMENT`	Environment label on mirrored traces	none
`BENCHGEN_PRIVATE_PATHS`	Comma separated private path prefixes	none (guard off)
`BENCHGEN_PRIVATE_ALLOW_SESSIONS`	Comma separated session key fragments allowed to read them	none

Config precedence: plugins.entries.benchgen.config → environment variable → built-in default.

Event mapping

OpenClaw event	Benchgen entity	Notes
`run.started`	trace start	starts the nested run trace for a session
`context.assembled`	span	records assembled context for the turn
`model.usage`	generation span	writes model output, usage, and cost
`model.call.error`	generation span (error)	closes the span with error details
`tool.execution.started`	tool/retriever span start	captures tool name + input
`tool.execution.completed`	tool/retriever span end	captures output + duration
`tool.execution.error`	tool/retriever span end (error)	captures error + duration
`tool.execution.blocked`	tool/retriever span end (blocked)	captures blocked reason
`run.completed`	trace finalize	closes pending spans and the trace

Chat with Benchgen

What it does

Benchgen sends a message → the plugin runs it as a normal turn of your OpenClaw agent (same system prompt, tools, session store and permissions as any channel) → the reply streams back to Benchgen. Two transports, same behavior:

Transport	Direction	When to use
Relay (default on)	plugin → Benchgen, outbound WebSocket, kept alive with reconnect + pings	Always works: the gateway connects out, so it can sit behind NAT / docker / a firewall.
HTTP endpoint (default on)	Benchgen → gateway, `POST /benchgen/chat` on the gateway port	When Benchgen can reach the gateway directly; also handy for local testing with `curl`.

Chat turns run under channel benchgen. With sessionScope: "conversation" each Benchgen conversation gets its own agent session, agent:<agentId>:benchgen:direct:<conversationId>, so parallel chats never share context and every turn is traced with that session key and the tag benchgen-chat; Benchgen matches a chat to its trace by session key.

The keys that authorize trace ingest are also what authorize chat: the plugin authenticates to the relay with Authorization: Basic base64(publicKey:secretKey), and the HTTP endpoint accepts the same header (or Bearer <secretKey>). Anyone holding the project keys can talk to the agent; set chat.enabled: false if that is not wanted.

Quick check with curl (HTTP endpoint)

curl -sS http://127.0.0.1:18789/benchgen/chat \
  -u "pk-your-public-key:sk-your-secret-key" \
  -H 'Content-Type: application/json' \
  -d '{"conversationId":"demo-1","text":"hello from benchgen"}'

{
  "ok": true,
  "conversationId": "demo-1",
  "messageId": "…",
  "status": "ok",
  "sessionKey": "agent:main:benchgen:direct:demo-1",
  "agentId": "main",
  "text": "Hi! …",
  "replies": [{ "text": "Hi! …", "kind": "final" }]
}

Add -H 'Accept: text/event-stream' (or "stream": true in the body) to get the reply as it is produced; the SSE events are the frames described below. GET /benchgen/chat (same auth) returns the bridge status, including whether the relay is currently connected.

Chat protocol (relay), version 1

The plugin opens chat.url with headers Authorization: Basic …, x-benchgen-plugin: benchgen-openclaw/<version> and x-benchgen-protocol: 1, and repeats the keys inside the first frame (hello.auth); relays that cannot read upgrade headers authenticate from there; a relay ignores every other frame until a valid hello has arrived. All frames are JSON text; every plugin→Benchgen frame carries ts (epoch ms).

Plugin → Benchgen

`type`	Fields	Meaning
`hello`	`protocol`, `auth{publicKey,secretKey}`, `plugin{name,version}`, `host{openclawVersion}`, `agents[{id,default}]`, `capabilities[]`	Sent first on every (re)connect; nothing else is sent before it.
`turn.started`	`conversationId`, `messageId`, `sessionKey`, `agentId`	The message was accepted and the agent is running.
`reply.partial`	`conversationId`, `messageId`, `text`, `delta?`, `replace?`	Streaming: `text` is the reply-in-progress so far.
`tool.start`	`conversationId`, `messageId`, `name`, `args?`	The agent started a tool call.
`choices`	`conversationId`, `messageId`, `questions[{id, header, question, options[{label, description?}], multiSelect}]`	The agent asked a question with options (`ask_user`); draw them as buttons and send the picked label as the next user message.
`reply`	`conversationId`, `messageId`, `text`, `kind` (`block` \| `final` \| `tool`), `mediaUrls?`	A delivered reply message. A turn may deliver several.
`turn.done`	`conversationId`, `messageId`, `status` (`ok` \| `error` \| `dropped` \| `aborted`), `error?`, `reason?`, `sessionKey`, `agentId`, `replies{tool,block,final}`	Always the last frame of a turn. `aborted`: Benchgen sent `abort` for it.
`pong`	none	Answer to a `ping`.

Benchgen → plugin

`type`	Fields	Meaning
`message`	`conversationId` (recommended), `messageId?`, `text` (required), `sender?{id,name}`, `agentId?`, `timestamp?`	Run one turn. Missing ids are generated and echoed back.
`ping`	none	Liveness; the plugin answers `pong`. The plugin also sends WebSocket ping frames itself.
`abort`	`conversationId`, `messageId`	The user stopped the turn in the chat. The plugin aborts the model call and the running tool and ends the turn with `turn.done` `aborted`; `turn.usage` still follows for what was spent. Announced in `hello.capabilities` as `abort`.
`hello.ack`	`protocol`, `agentId` (BenchGen agent id), `chat{connected,disabled}`	Informational answer to `hello`.
`chat.status`	`connected`, `skipped?`, `error?`	Informational: whether BenchGen (auto-)connected this agent for chat after the hello.

The relay closes with 4401 for unknown keys, 4403 for revoked keys, 4408 when no valid hello arrives, and 4409 when a newer socket with the same keys supersedes this one (the plugin reconnects with backoff in every case).

Unknown frame types are ignored. An invalid message gets a turn.done with status: "error" and the reason. Turns are serialized per conversationId and run in parallel across conversations. Frames produced while the relay is disconnected are queued (bounded) and flushed on reconnect.

How it works

The plugin runs its trace pipeline on a dedicated, isolated OpenTelemetry provider so it never interferes with OpenClaw's own telemetry. Traces are buffered and flushed in the background, and an idle reaper closes any observations orphaned by dropped events. On shutdown, in-flight traces are flushed before the provider is torn down.

Chat uses the host's public plugin runtime (api.runtime.channel.inbound.dispatch and friends, the same entry point OpenClaw's bundled channels use), so no OpenClaw core changes are needed and the turn is indistinguishable from one that arrived over Telegram or WebChat. On hosts whose SDK lacks that surface the plugin logs which piece is missing and keeps streaming traces.

Known limitations

No OpenClaw core changes are included in this repository and relies on native hooks within the OpenClaw ecosystem.

Chat: benchgen is not a registered OpenClaw channel plugin: it dispatches turns through the runtime but has no outbound adapter. So the agent answers Benchgen inside a turn, but cannot message Benchgen on its own (heartbeats, cron, the message tool), and openclaw channels status does not list it. Slash commands sent from Benchgen run without operator authority.

License

MIT

Name		Name	Last commit message	Last commit date
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
benchgen-screenshot.jpg		benchgen-screenshot.jpg
chat.js		chat.js
chat.test.js		chat.test.js
cli.js		cli.js
configure.js		configure.js
index.js		index.js
mapping.js		mapping.js
mirror.js		mirror.js
mirror.test.js		mirror.test.js
openclaw.plugin.json		openclaw.plugin.json
otel-log.js		otel-log.js
otel-log.test.js		otel-log.test.js
package-lock.json		package-lock.json
package.json		package.json
private.js		private.js
private.test.js		private.test.js
tech.md		tech.md
tracer.js		tracer.js
tracer.test.js		tracer.test.js
transcript.js		transcript.js
transcript.test.js		transcript.test.js
usage.js		usage.js

Navigation Menu

Repository navigation

About

Repository files navigation

🔭 OpenClaw Benchgen Observability Plugin

What is new in 0.8.5

What is new in 0.8.1

What is new in 0.8.0

What is new in 0.7.2

What is new in 0.7.1

What is new in 0.7.0

What is new in 0.6.0

What is new in 0.5.3

What is new in 0.5.0

What is new in 0.4.0

Why This Plugin

Install and first run

1. Install the plugin in OpenClaw

2. Configure the plugin

3. Check effective settings

4. Send a test message

Configuration

Recommended config shape

Environment fallbacks

Event mapping

Chat with Benchgen

What it does

Quick check with curl (HTTP endpoint)

Chat protocol (relay), version 1

How it works

Known limitations

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

🔭 OpenClaw Benchgen Observability Plugin

What is new in 0.8.5

What is new in 0.8.1

What is new in 0.8.0

What is new in 0.7.2

What is new in 0.7.1

What is new in 0.7.0

What is new in 0.6.0

What is new in 0.5.3

What is new in 0.5.0

What is new in 0.4.0

Why This Plugin

Install and first run

1. Install the plugin in OpenClaw

2. Configure the plugin

3. Check effective settings

4. Send a test message

Configuration

Recommended config shape

Environment fallbacks

Event mapping

Chat with Benchgen

What it does

Quick check with curl (HTTP endpoint)

Chat protocol (relay), version 1

How it works

Known limitations

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages