---
title: Telemetry
description: "Trace A2 with OpenTelemetry, and the spans it reports."
---

```ts server/orders.ts
import { createServer } from 'experimental-a2/server'
import { otel } from 'experimental-a2/otel'
import { orders } from '@/contracts'

export const ordersServer = createServer({
  contract: orders,
  telemetry: otel(),
  // ...store, scheduler, and handlers
})
```

With any OpenTelemetry setup registered (on Vercel, `@vercel/otel`), one trace
shows the request, `a2.append`, the inline `a2.drain`, an `a2.event` span per
handler, and the appends those handlers make. Failures mark their `a2.event`
span, and `a2.event.outcome = dead_lettered` is the attribute to alert on.
Agent servers add `a2.ai.*` spans.

## `experimental-a2/otel`

```ts
otel(options?: { tracer?: Tracer; tracerName?: string }): A2Telemetry
```

OpenTelemetry instrumentation for a server (`@opentelemetry/api` is a peer
dependency). Pass it as `telemetry`. Spans nest through the active context, so
the happy path lands in one trace. `A2Telemetry` is a one-method interface:
anything with `span(name, attributes, fn)` works, and `otel()` is the adapter
A2 ships.

## The span catalogue

This catalogue is a contract: renames and additions are breaking for
dashboards and are called out in the package's `CHANGELOG.md`. All spans carry
`a2.contract` and `a2.session_id`. Core spans describe the durable runtime;
`a2.ai.*` spans describe the built-in AI handlers.

| Layer | Span | Wraps |
| ----- | ---- | ----- |
| Core | `a2.append` | validation + the store write (one batch) |
| Core | `a2.drain` | one drain pass, including concurrent handlers |
| Core | `a2.event` | one claimed dispatch, including its handler and completion |
| Core | `a2.claim` | one `claimAvailable` store operation |
| Core | `a2.commit` | one `completeAttempt` or `failAttempt` store operation |
| Core | `a2.state` | one `state()` read: snapshot-plus-tail load + reducer fold |
| Core | `a2.snapshot` | one background batch of checkpoint writes |
| Core | `a2.redrive` | one `session.redrive()`: the store operation and starting the session's wakeup |
| AI | `a2.ai.control` | applying an input/work report/session-close decision and preparing its returned events |
| AI | `a2.ai.generate` | one generation handler, including Core calls and model/SDK work |
| AI | `a2.ai.context` | constructing model messages, including tool prompt reconstruction when uncached |
| AI | `a2.ai.compaction` | automatic summarization or the custom compaction callback |
| AI | `a2.ai.tool` | one durable tool handler, including application execution |
| AI | `a2.ai.ack` | waiting for a persisted work report's canonical events and control receipt |

Durations include nested work: `a2.event` contains its AI handler and later
commit, AI handlers call Core, and `a2.state` includes reducer code.
Acknowledgements overlap control processing, claims can cover several events,
and snapshots run in the background, so summing spans, or subtracting summed
children, does not yield exclusive Core or AI overhead. Model and application
I/O is included; its child spans have the detail.

Attributes arrive in two waves: **at start** (passed to `span()`) and
**mid-span** (via `handle.setAttribute`, for outcomes and counts known only
after the work ran). A recorder must merge both; the ones you alert on are all
mid-span.

| Attribute             | Span        | When  | Values                                                |
| --------------------- | ----------- | ----- | ----------------------------------------------------- |
| `a2.append.source`    | `a2.append` | start | `external` \| `handler`                                |
| `a2.append.mode`      | `a2.append` | start | `inline` \| `dispatch`                                |
| `a2.append.types`     | `a2.append` | start | comma-joined event types                              |
| `a2.append.count`     | `a2.append` | start | batch size                                            |
| `a2.append.armed`     | `a2.append` | mid   | `false` when the scheduler arm failed and this append degraded to append-driven healing |
| `a2.drain.outcome`    | `a2.drain`  | mid   | `settled` \| `busy` \| `stalled` \| `failed`          |
| `a2.drain.processed`  | `a2.drain`  | mid   | events processed this pass                            |
| `a2.claim.outcome` | `a2.claim` | mid | `claimed` \| `busy` \| `settled` |
| `a2.claim.count` | `a2.claim` | mid | number of claimed events, or zero |
| `a2.commit.operation` | `a2.commit` | start | `complete` \| `fail` |
| `a2.commit.index` | `a2.commit` | start | event log position |
| `a2.commit.attempt` | `a2.commit` | start | durable 1-based attempt ordinal |
| `a2.commit.outcome` | `a2.commit` | mid | `completed` \| `failed` \| `dead_lettered` \| `superseded` |
| `a2.event.type`       | `a2.event`  | start | the event's type                                      |
| `a2.event.index`      | `a2.event`  | start | log position                                          |
| `a2.event.id`         | `a2.event`  | start | event id                                              |
| `a2.event.attempt`    | `a2.event`  | start | same durable 1-based ordinal as `ctx.attempt`          |
| `a2.event.lane`       | `a2.event`  | start | stored lane value; absent for concurrent unlaned work  |
| `a2.event.handled`    | `a2.event`  | start | normally `true`; `false` when a custom-adapter row has no handler |
| `a2.event.outcome`    | `a2.event`  | mid   | `processed` \| `failed` \| `dead_lettered` \| `superseded` \| `surrendered` |
| `a2.event.aborted`    | `a2.event`  | mid   | `true` when `abortOn` fired during the run            |
| `a2.state.reducer`    | `a2.state`  | start | the reducer's name                                    |
| `a2.state.snapshot`   | `a2.state`  | mid   | `hit` \| `miss` \| `rejected` (schema guard discarded it) \| `memory` (served from the server's in-memory state) |
| `a2.state.folded`     | `a2.state`  | mid   | events folded past the snapshot                       |
| `a2.state.index`      | `a2.state`  | mid   | the frontier the returned state reflects              |
| `a2.snapshot.reducer` | `a2.snapshot` | start | the reducer's name                                  |
| `a2.snapshot.index`   | `a2.snapshot` | start | greatest checkpoint index in the write batch       |
| `a2.snapshot.count`   | `a2.snapshot` | start | number of checkpoints in the write batch           |
| `a2.redrive.index` | `a2.redrive` | start | log position of the event being redriven |
| `a2.redrive.action` | `a2.redrive` | start | `retry` \| `skip` |
| `a2.redrive.redriven` | `a2.redrive` | mid | `true` when the event was dead-lettered and is now resolved |
| `a2.redrive.armed` | `a2.redrive` | mid | `false` when the scheduler arm failed and the wakeup degraded to append-driven healing |
| `a2.ai.control.type` | `a2.ai.control` | start | `ai.control.requested` \| `ai.work.reported` \| `ai.session.closed` |
| `a2.ai.control.id` | `a2.ai.control` | start | triggering event id |
| `a2.ai.control.attempt` | `a2.ai.control` | start | durable 1-based attempt ordinal |
| `a2.ai.generate.request_id` | `a2.ai.generate` | start | generation request event id |
| `a2.ai.generate.attempt` | `a2.ai.generate` | start | durable 1-based attempt ordinal |
| `a2.ai.context.request_id` | `a2.ai.context` | start | generation request event id |
| `a2.ai.context.phase` | `a2.ai.context` | start | `generation` \| `refresh` \| `tool` |
| `a2.ai.compaction.request_id` | `a2.ai.compaction` | start | generation request event id |
| `a2.ai.compaction.generation_id` | `a2.ai.compaction` | start | generation attempt id |
| `a2.ai.compaction.policy` | `a2.ai.compaction` | start | `automatic` \| `custom` |
| `a2.ai.tool.request_id` | `a2.ai.tool` | start | originating generation request event id |
| `a2.ai.tool.call_id` | `a2.ai.tool` | start | tool call id |
| `a2.ai.tool.name` | `a2.ai.tool` | start | tool name |
| `a2.ai.tool.attempt` | `a2.ai.tool` | start | durable 1-based attempt ordinal |
| `a2.ai.ack.request_id` | `a2.ai.ack` | start | reporting worker event id |
| `a2.ai.ack.report_id` | `a2.ai.ack` | start | work report event id, matching the control span's id |

- Drain outcomes: `settled` means nothing actionable is left; `busy` means live
  per-event claims or a failed event's retry hold remain; `stalled` means a
  caught failure in this drain needs a later retry. `failed` appears only on
  scheduler deliveries: the drain threw after arming its successor watchdog,
  the span records the error, and that successor retries the session.
- A failing handler marks its `a2.event` span with the exception and error
  status. Alert on `a2.event.outcome = dead_lettered`. A dead-lettered event
  blocks only later events in its lane, until `session.redrive()` resolves it;
  other lanes stay eligible.
- `superseded` means a stale completion or failure did not change the log.