Observability

Production is the real evaluation. Every run emits a structured trace so you can answer "why did it do that?" long after the run has ended.

Trace contents#

• The plan, every revision to it, and the reason for each revision.
• Every tool call: arguments hash, latency, retries, result size, policy decision.
• Every observation and how it changed the plan.
• Model routing decisions per step.
• Approval requests, who decided, and how long it took.
• Budget consumption over the run timeline.

Metrics worth alerting on#

export async function POST(req: Request) {
  const event = await verifyWebhook(req);

  if (event.type === class="tok-str">"policy.denied") {
    metrics.increment(class="tok-str">"pimsy.policy.denied", { tool: event.tool, rule: event.rule });
    if (event.rule === class="tok-str">"codebase.dependency_add") await pager.page(class="tok-str">"agent-policy-denial");
  }

  return new Response(null, { status: class="tok-num">204 });
}