EBOEngineering Behavior Observatory

Every finding traced to the native line behind it.

The Engineering Behavior Observatory captures how an engineering agent did the work: native sessions, tool calls, hooks and workspace changes, across eight harnesses. Each behavioral assessment cites the records it rests on, and each published number recomputes from the study's own data.

Watch the tour 3:50 Quickstart
npm install -g engineering-behavior-observatory
Instrument specification
Capture8 harnesses, each through its own official boundary
Qualificationbefore normalization; unqualified bundles rejected
Schemaebo.uniform-event/v1, 10 families × 4 phases
Judgmentcited evidence, an alternative explanation, may abstain
IntegritySHA-256 per cited native line, re-hashed on validation
Missing dataexplicit; never false, never zero
Sharinginternal, partner and restricted packets; export fails closed
The EBO Atlas viewer: an embedding cloud of agent actions coloured by study arm with one unit's fields open, facet panels on the right, and per-trial swimlanes of tool calls and errors below.

Plate I The Behavior Atlas over a three-arm Claude Opus 5.5 study: direct, Seatbelt and Seatbelt 1M.

  1. Study 15,894 action units, embedded with Qwen3-Embedding-8B.
  2. Cloud one point per action, coloured by arm and labelled by facet.
  3. Unit every field of one action: task, arm, tool, command head, check kind.
  4. Facets counts per task and condition; a filter applies everywhere.
  5. Lanes each trial's actions in order, with cost, duration and errors.
  6. Record a lane bar's kind, time and cluster; a click opens the native record.

Numbers computed in the viewer are exploratory. Certified tallies come from EBO's aggregation reports.

Signal path

Execution completion, capture quality and task quality are different facts, so EBO keeps them apart. Native evidence stays authoritative: every later stage is derived from it and never replaces it.

  1. Frozen task and harness

    An admitted, digest-pinned task packet or Harbor task, run by one selected native harness. Observational tasks need no reference solution.

  2. Capture

    Native messages, tool activity, exposed lifecycle hooks, usage, OTLP receipt and workspace state, written continuously. Failed and interrupted attempts stay valid partial bundles.

  3. Qualification

    Each bundle's evidence is checked before anything is derived from it. An unqualified bundle is rejected before normalization.

  4. Normalization

    Native records project into one schema of observation categories, not a shared protocol.

    Codex · commandExecution
    tool
    Pi · bash
    tool
    Agent SDK hook · PreCompact
    context · before

  5. Evaluation

    Structural extractors report exact facts. An LLM judge reads an explicit evidence selection, optionally blinded to the evaluated model, weighs an alternative explanation and may abstain.

    constructiveadversemixedcontext-dependent

  6. Claims

    A study's claims are authored, never generated, and validated: each number recomputes, each supporting assessment exists, each cited line re-hashes.

  7. Atlas and packets

    Explore in the Atlas; share an evidence packet that a partner verifies offline against its manifest.

Anatomy of a claim

What a partner receives for each finding. Validation fails if one number no longer recomputes or one cited line no longer hashes to its recorded SHA-256; partner and restricted packets are built only when every claim validates.

Atlas bundle contract

claimchecked at every build

  • text · cohort · caveatthe finding in the study's words, and its scope
  • numbers[]each with a label, a value and a kind
    • kindcertified report tally, judge count or exploratory lab figure
    • exprreport pointer, SQL or attempt audit; recomputed, must equal value
  • support[]the judge assessments behind the claim
    • outcomeconstructive, adverse, mixed or context-dependent
    • citations[]one per cited event
      • event_key<attempt>/<event>, attempt-scoped
      • native.path · locatorfile and line of the native record
      • native.sha256re-hashed from the record on every validation
  • views[]figures whose query is recomputed and hashed into a receipt

The Atlas

One coordinated view over a study. Brush the cloud and the panels follow; open any bar and read what the agent produced, verbatim.

  • inspect
  • edit / write
  • read
  • check
  • other shell
  • git / vcs
  • error
Fig. 2 Swimlanes, drawn the way the Atlas draws them. Illustrative sequence, not study data.
Clusters × arm
Observed against expected units per cluster and arm, with adjusted residuals.
Swimlanes
Trials in time or step order, with context per request, failure chains, compactions and an aligned diff.
Assessments
Judge outcomes by dimension and attempt, each opening its rationale and cited lines.
Claims
The claims ledger, each number's recomputation, and the attempt audits.
Figures
View specs rendered live, each with a receipt that its numbers match the build.

Harnesses

Each adapter keeps its harness's own methods, identities, lifecycle and completion semantics. Coverage is capability-specific: not every harness exposes the same hooks, histories or telemetry.

Prerequisites and known gaps

HarnessBoundaryNative evidence retained
Claude Agent SDKDirect TypeScript SDKebo agent-sdk runMessages, passive lifecycle hooks, session identity, usage, OTLP receipt state, workspace
CodexOwned app-server, JSON-RPC stdioebo codex runThread, turn and item records, history readback, usage, independently checked OTLP signals
Devin CLIOwned devin acp, JSON-RPC stdioebo devin runACP session, tool-call and permission records, stop reason and usage, agent-stop statistics
CursorDirect TypeScript SDKebo cursor runStream, callbacks, conversation, official JSONL store, terminal and separate billing evidence
PiTypeScript session and extension APIsebo pi runSession tree, tool and message lifecycle, passive observer, compaction, retries, usage
Factory DroidOfficial TypeScript SDK, owned droid exec stream-jsonrpcebo factory runBoth protocol directions recorded before SDK parsing: messages, tool calls and results, permission decisions, hooks, compaction, child sessions, turn and session usage
OpenHandsAgent Server REST and WebSocketlibrary APIStream and final event reconciliation, conversation state, exposed hooks, workspace
DeepSeek HarnessOfficial TypeScript JSON-RPC clientlibrary APIDurable session events, lifecycle notifications, selected runtime and plugin composition

Evidence packets

One folder a partner can open from disk: narrative, claims, evaluations, metrics and qualified evidence, cross-referenced, with the interactive viewer when the folder is served over HTTP.

Packet contract

VariantSharedRemoved or withheld
internalEverythingNothing
partnerAll data, under the export pipeline's sharing rulesSecrets, build environment values, local paths and users, hidden reasoning; a string the final scan still flags is withheld whole
restrictedNarrative, claims, evaluations, metrics and the viewerUnit text reduced to structural labels; commands, targets, error text and code removed; native lines, records, audits and tables withheld and listed with their SHA-256
 
ebo packet build <bundle-dir> <output-dir> --variant partner
ebo packet verify <packet-dir>    # or open verify.html

manifest.json (ebo.packet/v1) lists every file with its SHA-256, size, media type, layer and redaction count. ro-crate-metadata.json is an RO-Crate 1.2 description generated from it.

The tour

3 min 50 s, 1080p, narrated with burned-in captions. From capture to the native record, on real studies.

Operating constraints

The rules every adapter, extractor and export is reviewed against.

Native evidence is authoritative
Sessions, streams, events and hooks are semantic evidence; OpenTelemetry is timing and resources; the workspace is the outcome.
Nothing is synthesized
Missing or unsupported evidence is represented explicitly, never turned into false or zero.
Retries never replace
A retry is a new linked attempt; the failed or interrupted one stays a valid partial bundle.
Export fails closed
Unknown sharing classification blocks export. Secrets, environment values, local identifiers and hidden reasoning are excluded.
Judges can abstain
A semantic claim needs cited evidence and an alternative explanation. Human calibration is optional and recorded separately.
Exploratory is labeled
Only EBO's aggregation reports are certified; every other number says what it is.

First report

The bundled synthetic demo makes no model calls and needs no credentials. It shows evidence drilldown, review states, missing evidence and comparison caveats. Requires Node.js 24.19.0, Git and npm.

Quickstart CLI reference 
# from a source checkout
git clone https://github.com/trilogy-group/engineering-behavior-observatory.git
cd engineering-behavior-observatory
nvm use && npm ci && npm run build && npm link

# a synthetic trajectory report
node dist/test/atlas-fixture.js .ebo/first-look
ebo corpus query .ebo/first-look/index.jsonl
ebo atlas build .ebo/first-look/atlas.json .ebo/first-report
# open .ebo/first-report/index.html