Frozen document. Dates and status refer to its original source. Private resources and unreviewed images are unavailable.

Atlas agentic SDLC measurement and improvement guide

Status: working build guide, September 29, 2026. It describes what Atlas captures today, gaps found during the build, and the next capture contract. Planned fields are not presented as existing dashboard data. Use this record when starting a later agentic SDLC project, and revise it when Track Greptile and subagent review activity in Journey metrics or the final certification gate changes the implementation.

Why this record exists

Atlas has a useful history of decisions, issues, agents, tokens, and elapsed time. It also exposed a measurement trap: the Unit test row showed zero minutes even though unit-test commands ran. The row assigns an entire transcript turn to one primary SDLC phase; tests inside Build or Other turns do not receive a separate time allocation. PR #58 clarified that display without inventing a retrospective duration. Earlier slices ran tests, but a consistent red-before-green sequence was not recorded. PR #60 now makes test-first tracer bullets and their evidence the rule for future behavior.

The purpose is to make three questions answerable without guesswork: what work actually finished, how much calendar time and recorded agent work it consumed, and which tests and reviews support the result. A number is useful only with its source, unit, cutoff, and limitation.

Current capture path

flowchart LR

  CT[Codex transcript metadata] --> CM[capture_journey_metrics.py]

  GH[GitHub issues and lifecycle events] --> CI[capture_issue_journey.py]

  CM --> JS[Journey metrics JSON and dated ledger]

  CI --> IS[Issue JSON and dated ledger]

  JS --> UI[Journey dashboard]

  IS --> UI

  PR[GitHub PR checks and comments] -. issue 59 .-> RV[Review activity dataset]

  RV -. planned .-> UI

  EV[Test and demo evidence in issues and CI] --> JR[Dated journey milestone]

  JS --> JR

  IS --> JR

The Dashboard Refresh button and one-minute background check use the loopback Journey server. They recapture finished transcript turns and issue metadata in memory. Dated JSON/Markdown publication is a separate reviewed step described in the capture guide. A failed GitHub refresh leaves the saved issue snapshot visible and must not be reported as live issue data.

Definitions that prevent misleading totals

Each turn receives one curated primary pipeline stage and one primary SDLC phase. Mixed work is labeled Mixed rather than fractionally allocated without command evidence. The zero Unit test phase in a historical snapshot means zero turns had Unit test as their primary phase; it does not mean zero tests ran. Unit, system, E2E, and deployment command counts/times need a separate command-level event stream. Keep source coverage and unknown values visible beside every chart.

What we should have captured from the first slice

The missing record is a structured event at each meaningful gate, attached to a stable task/issue/PR/commit ID. Before work starts, record the approved public test seam, expected user behavior, owner, blockers, agent/harness/model/reasoning setting, and an explicit start event. At each red→green loop, record the test identifier, command, start/end or monotonic duration, exit code, result summary, relevant environment/version, and whether it was unit, contract, system, or E2E. Record a failed attempt and later success as separate events. At a review, record the PR and exact head SHA, reviewer or bot, check start/end, conclusion, finding IDs/severity, and resolution commit. At a deploy or demo gate, record version/seed manifest, resource peaks, the hosted or local origin, test result, and stop/pause/resume events with evidence.

Historical transcripts can establish turn-level time, tokens, models, and some visible test outcomes. They cannot reliably reconstruct every command's elapsed time, the first failed test before code, exact idle versus tool time, or a bot's model usage. Do not fill those gaps from estimates. A missing value should carry unavailable and a reason, while a partial count should say lower bound with its coverage window.

Efficient event collection for future runs

1. Use one append-only event per task transition or completed command, with a unique event ID, source ID, UTC timestamp, local display timezone, issue/PR/commit, agent/harness, and event type. Store only metadata and redacted result summaries; never command environment variables or raw customer data.
2. Wrap test/build/deploy commands once to capture monotonic duration, exit status, and a small result summary. Let CI upload the detailed logs as private artifacts. A wrapper can be incremental; label its coverage start date so older commands are not silently counted as zero.
3. Ingest provider usage once per completed turn/call and deduplicate by provider request or turn ID. Capture the exact model reported by telemetry rather than assuming a CLI flag selected it. Do not poll providers for tokens every minute.
4. Poll or receive GitHub review/check events at PR-head changes, review completion, and milestone capture. Key Greptile findings by comment ID and commit SHA; an updated review of the same PR is a new review run, not another PR.
5. Keep the minute refresh as a view cache. Publish immutable, reviewed snapshots at issue closure and wave convergence. Recompute summaries from source events, verify totals, and retain the cutoff and gaps.

For a new local test or build, run python3 scripts/record_sdlc_command.py --phase unit --issue 62 -- python3 -m unittest tests.test_record_sdlc_command -v. Choose unit, contract, system, e2e, build, or deploy for the command's actual purpose. The wrapper writes append-only JSONL to ignored local/metrics/commands.jsonl by default, records elapsed time with a monotonic clock, and preserves the command's exit status. It stores the executable basename and argument count, never raw arguments, paths, environment, or output. When running from an isolated worktree, pass --output pointing to the Dashboard server checkout's ignored local/metrics/commands.jsonl to include that event in its live view. The Stages table deduplicates recorded events and shows command counts/time beside turn totals. A dash means no instrumented command for that phase, not that no test ran. Record command-level SDLC timing evidence owns this opt-in capture; never reconstruct earlier command durations from turn totals.

Suggested quality checks: event IDs are unique; every command end has a start or an explicit missing-start reason; no duration is negative; timestamp units are stated; totals reconcile without double counting; review comments refer to a known PR/head; every model/reasoning value has a source; and secret/redaction checks pass. These are pipeline tests, distinct from Atlas product tests.

Test-first and review work

The agent contract now requires a public seam and a red→green cycle for each new behavior. The approved full-stack acceptance seam is the Mac loopback HTTPS route with real Keycloak, Camunda, PostgreSQL and ClamAV, pinned en-US/es-US Contentful releases, and only simulated payments/legacy systems. A narrower unit or contract seam must come from an accepted spec or owner-confirmed interface. Workers keep focused tests; the orchestrator runs integration at convergence. The first recorded test-first review fix is Resolve recovery PR review findings, whose recovery API regression failed before the fix and passed afterward; the live HTTPS gate for that PR remains separate.

Review metrics should answer: how many PR heads were reviewed, which review runs finished, how long checks took, how many actionable findings were opened and resolved, and how much our agents spent responding. Track Greptile and subagent review activity in Journey metrics owns the end-to-end capture and Dashboard view. It must keep Greptile elapsed time separate from subagent turn time/tokens, and show Greptile tokens/model as unavailable. A finding is fixed before merge only after the linked code change and current-head review are checked; a green check alone does not prove all comments were addressed.

Architecture improvement gate

Assess deep-module opportunities before product wave schedules the requested improve-codebase-architecture skill after private evidence and recovery platform work, before the next product wave. The skill reads the domain glossary and accepted ADRs, selects recent hotspots, applies the codebase-design module/interface/depth/seam vocabulary and deletion test, and produces a visual before/after HTML report in temporary storage. The owner chooses a candidate before any refactor issue is created. Track the assessment's calendar/run/token metrics as Design or Review work; count a selected refactor as a feature only if its own tracer-bullet issue delivers observable product behavior. The report's path is temporary, so its findings and owner decision need durable links in the issue/ADR before the temporary file disappears.

External agentic CLI experiment

Atlas first tested the authenticated native Grok CLI on a synthetic, tool-free task. An escalated model listing showed Grok 4.7, 4.7 Build Fast, 4.6, and 4.5. A one-turn grok-4.7 medium-reasoning call returned the correct JSON in about eight seconds. Its response telemetry identified grok-4.7-build, 16,720 input tokens, 1,152 cached input, 437 output, 409 reasoning, 18,309 total, and a reported cost of $0.01245692. The local Pi model list lacked 4.6/4.7 and its inspection unexpectedly installed global packages; Pi is not the current Atlas delegation path.

For Record command-level SDLC timing evidence, the owner explicitly approved a bounded requirements packet to Grok. Native grok-4.7 at medium reasoning returned source and test drafts without filesystem tools. The CLI's actual model was grok-4.7-build; it reported 18,683 input, 37,583 output including 30,126 reasoning, 56,266 total tokens, and $0.08937376 over about 6 minutes 16 seconds of wall time. Codex inspected the result, made the append lock fail closed, applied it through a red→green test cycle, and independently ran the focused tests and local Dashboard preview. This produced useful code, but the 18k-token input overhead and 30k reasoning output are too high for a small task. Future Grok packets should request one narrow artifact, cap output and turns, and compare quality plus rework as well as elapsed time.

For future Grok work, use a compact issue packet: stable blockers, exact accepted contracts, conceptual ownership, forbidden paths, tools/network permissions, focused tests, and the standard structured handoff. Pin a verified model and medium/high reasoning appropriate to risk. Capture the native CLI's actual model/usage/error telemetry and command elapsed time, then have the orchestrator inspect the diff and independently run focused tests. Do not send credentials or private evidence. Prefer Sol/Luna Codex workers for small tasks where Grok's startup tokens dominate. Compare harnesses only on comparable work, with quality and rework included, rather than tokens alone.

Next capture milestones

This guide is a contract for capturing evidence, not a claim that all listed metrics already exist. The capture guide gives the current commands; the dated Journey record links published artifacts and decisions.

October 1 hybrid review corrections

The private cloud bundle now supplies 13 recorded root intervals and 37 measured commands, with missing token/model and linked-worker coverage explicit. Greptile and two separate code-review axes found eight capture defects or gaps. Regression fixes protect cached history against unavailable or older exports, skip partial JSONL commands, preserve zero-activity dates, constrain host provenance, keep cloud attribution unclassified without evidence, and retain active-agent identity without partial usage. A sanitized tracked bootstrap makes fresh-controller recovery reproducible.

Native Grok is authenticated and completed a generic helper-and-test authoring task in 139.018 seconds, at requested medium reasoning. Telemetry reports grok-4.6-build and 28,482 total tokens. Its raw input, cache-read and output fields were 16,990, 128 and 11,364; the total reconciles only when cache-read tokens are included with the reported input. The normalizer accepts either provider inclusion convention when the total proves the decomposition. It rejects inconsistent telemetry and keeps missing usage unknown. Raw thought, prompts and source files are discarded. This arithmetic observation is not a claim about subscription billing.

The external review collector reads explicit task_type=review evidence and reports it separately from Codex and Greptile. External helper authoring has task_type=build, so it is retained as build evidence and does not inflate review counts. Generically generated code is inspected and independently regression-tested before acceptance. Native startup still contributes roughly 17,000 input tokens per one-turn invocation; batching useful bounded tasks is preferable to many tiny calls.