Skip to content

About this article

  • Audience: Operators certifying shadows for production handoff after soak gates and harness runs
  • Goal: After reading, you can score a soak, interpret letter grades, and know which checks block release
  • Type: How-to

Summary

Production soak grading turns telemetry, encoder compliance, harness results, and visual checkpoints into a 0–100 score and letter grade. Release requires ≥ 85 with zero critical failures. Run the automated scorer after orchestrator batches or standalone soak directories complete.

Prerequisites

  • Completed soak artifacts under [operator-artifacts] or cost-sentinel/logs/soak-*
  • Phantom stack healthy on both shadows (site-role-a, site-role-b)
  • API quick reference for proxy probes during manual cross-check
  • Soaks and automation for orchestrator flags

Letter grades

GradeScoreMeaning
A90–100Production-ready; safe for demo and fleet handoff
B80–89Acceptable with documented warnings
C70–79Degraded; fix before customer-facing use
D60–69Major gaps; re-run soak
F< 60Failed; block release

Gate rule: Overall score ≥ 85 and no critical-category failures.

Run the automated scorer

bash
cd [repo-root]

SOAK_DIR=[operator-artifacts]/soak-YYYYMMDD-HHMM \
  bash cost-sentinel/scripts/grade-production-soak.sh

The orchestrator can invoke the same scorer post-batch and emit a summary plus handoff snippet when --update-handoff is set.

Scoring breakdown (100 points)

1. Phantom Vision stack (10 pts)

  • Proxy /health returns HTTP 200 throughout soak (5 pts)
  • MQTT broker reachable on port 1883 (3 pts)
  • VMS proxy on port 8788 (2 pts)

2–3. Shadow transport and live path (20 pts each)

  • POST /pull-recording succeeds on at least 3 of 3 attempts (9 pts)
  • MP4 stream HEAD returns 200 (6 pts)
  • GET /stream/snapshot returns body > 1 KB (3 pts)
  • MQTT freshness below stale threshold in ≥ 80% of samples (2 pts)

Probe surfaces: API quick reference.

4. Edge encoding compliance (20 pts)

  • H.265 verification passes at expected 30 FPS on both shadows (15 pts)
  • Typical justified chunk size > 10 MB (3 pts)
  • Sidecar justified flag true (2 pts)
  • Optional playback calibration (+3 pts): set GATE_PLAYBACK_CALIBRATION=1 and confirm picture count matches playback duration

5. Soak gate — thermal and encoder (from soak_gate.sh)

  • 60 one-minute samples over the soak window
  • FAIL if more than 25% of samples exceed 78°C die temperature
  • FAIL if more than 25% of samples show no recent H.265 growth
  • Monitor via MCP get_recording_status and SOC temperature reads

6. Harness and Phantom modules

  • Live MCP harness at ≥ 80% on mass-transit scenarios (3 repeats)
  • Unit gates: device identity, Hermes mesh, MCP health (five FULL_KEYS), spatial fusion (layer3 depth contract)
  • Wired into orchestrator post-probe, Phantom readiness, and batch grade

7. Visual, E2E, and TAP (fidelity bonus)

  • Pre/during/post checkpoints: proxy snapshot plus /fleet/mqtt/latest overlay
  • Playwright soak-gate visual spec (full-page and API checks)
  • tap.log records every proxy, MQTT, and script call

Phase 2 and batch patterns

PatternExpected outcome
normal_short, recovery_after_stallPASS on healthy shadow after prune
encoder_stall_sim, disk_criticalFAIL or INCOMPLETE (self-heal exercised)
Matrix batchMatch scenario expect field per row

Strict promotion rule: Bump tracker tier only after fresh H.265 growth, encoder verify PASS, and harness ≥ 80%.

Recommended flags: --prune-first, --visual, --tap, --update-handoff, --batch --matrix. Re-probe after any config or storage change.

Manual cross-check

When automation yields INCOMPLETE (for example soak log ends without a summary line), use the watcher output and post-probe JSON:

bash
curl -s "$PHANTOM_PROXY/fleet/health"
curl -s "$PHANTOM_PROXY/fleet/mqtt/latest"

Confirm storage_health is not CRITICAL, recent_chunks is growing, and MQTT rate is near the ~10 Hz contract.

Next steps

Operator depth

Live fleet state and harness evidence live in private operator handoff (not published) (private).