Appearance
About this article
- Audience: Operators certifying shadows for production handoff after soak gates and harness runs
- Goal: After reading, you can score a soak, interpret letter grades, and know which checks block release
- Type: How-to
Summary
Production soak grading turns telemetry, encoder compliance, harness results, and visual checkpoints into a 0–100 score and letter grade. Release requires ≥ 85 with zero critical failures. Run the automated scorer after orchestrator batches or standalone soak directories complete.
Prerequisites
- Completed soak artifacts under
[operator-artifacts]orcost-sentinel/logs/soak-* - Phantom stack healthy on both shadows (
site-role-a,site-role-b) - API quick reference for proxy probes during manual cross-check
- Soaks and automation for orchestrator flags
Letter grades
| Grade | Score | Meaning |
|---|---|---|
| A | 90–100 | Production-ready; safe for demo and fleet handoff |
| B | 80–89 | Acceptable with documented warnings |
| C | 70–79 | Degraded; fix before customer-facing use |
| D | 60–69 | Major gaps; re-run soak |
| F | < 60 | Failed; block release |
Gate rule: Overall score ≥ 85 and no critical-category failures.
Run the automated scorer
bash
cd [repo-root]
SOAK_DIR=[operator-artifacts]/soak-YYYYMMDD-HHMM \
bash cost-sentinel/scripts/grade-production-soak.shThe orchestrator can invoke the same scorer post-batch and emit a summary plus handoff snippet when --update-handoff is set.
Scoring breakdown (100 points)
1. Phantom Vision stack (10 pts)
- Proxy
/healthreturns HTTP 200 throughout soak (5 pts) - MQTT broker reachable on port 1883 (3 pts)
- VMS proxy on port 8788 (2 pts)
2–3. Shadow transport and live path (20 pts each)
POST /pull-recordingsucceeds on at least 3 of 3 attempts (9 pts)- MP4 stream
HEADreturns 200 (6 pts) GET /stream/snapshotreturns body > 1 KB (3 pts)- MQTT freshness below stale threshold in ≥ 80% of samples (2 pts)
Probe surfaces: API quick reference.
4. Edge encoding compliance (20 pts)
- H.265 verification passes at expected 30 FPS on both shadows (15 pts)
- Typical justified chunk size > 10 MB (3 pts)
- Sidecar
justifiedflag true (2 pts) - Optional playback calibration (+3 pts): set
GATE_PLAYBACK_CALIBRATION=1and confirm picture count matches playback duration
5. Soak gate — thermal and encoder (from soak_gate.sh)
- 60 one-minute samples over the soak window
- FAIL if more than 25% of samples exceed 78°C die temperature
- FAIL if more than 25% of samples show no recent H.265 growth
- Monitor via MCP
get_recording_statusand SOC temperature reads
6. Harness and Phantom modules
- Live MCP harness at ≥ 80% on mass-transit scenarios (3 repeats)
- Unit gates: device identity, Hermes mesh, MCP health (five FULL_KEYS), spatial fusion (layer3 depth contract)
- Wired into orchestrator post-probe, Phantom readiness, and batch grade
7. Visual, E2E, and TAP (fidelity bonus)
- Pre/during/post checkpoints: proxy snapshot plus
/fleet/mqtt/latestoverlay - Playwright soak-gate visual spec (full-page and API checks)
tap.logrecords every proxy, MQTT, and script call
Phase 2 and batch patterns
| Pattern | Expected outcome |
|---|---|
normal_short, recovery_after_stall | PASS on healthy shadow after prune |
encoder_stall_sim, disk_critical | FAIL or INCOMPLETE (self-heal exercised) |
| Matrix batch | Match scenario expect field per row |
Strict promotion rule: Bump tracker tier only after fresh H.265 growth, encoder verify PASS, and harness ≥ 80%.
Recommended flags: --prune-first, --visual, --tap, --update-handoff, --batch --matrix. Re-probe after any config or storage change.
Manual cross-check
When automation yields INCOMPLETE (for example soak log ends without a summary line), use the watcher output and post-probe JSON:
bash
curl -s "$PHANTOM_PROXY/fleet/health"
curl -s "$PHANTOM_PROXY/fleet/mqtt/latest"Confirm storage_health is not CRITICAL, recent_chunks is growing, and MQTT rate is near the ~10 Hz contract.
Next steps
Operator depth
Live fleet state and harness evidence live in private operator handoff (not published) (private).