Appearance
About this article
- Audience: Operators interpreting orchestrator decisions, Phantom module rates, and fleet contract telemetry
- Goal: After reading, you understand what PASS/INCOMPLETE/FAIL means, which rates the contract enforces, and how storage pressure affects them
- Type: Concept
Summary
The v3 shadow orchestrator drives full validation cycles: apply a layer, verify contracts, soak with a watcher, probe via the phantom proxy, and optionally update handoff with TAP logs. Phantom readiness agents re-prove modules in five-minute interleaved loops. Storage health directly gates encoder throughput and MQTT freshness—prune-first is the recovery pattern.
Prerequisites
- Glossary: layer2 tracker, layer3 depth, TAP, Hermes mesh
- API quick reference for
/fleet/health,/fleet/mqtt/latest,/tools/call
Orchestrator cycle
One cycle chains mature scripts (soak_gate.sh, visual_soak_gate.py, verify_fleet_contract.py) behind tools/v3_shadow_orchestrator.py:
- Apply — push layer env (for example
layer2_trackerorlayer3_depth) - Verify — fleet contract checks (MQTT ~10 Hz, 30 FPS H.265, 60 s chunks)
- Soak — timed samples with thermal and encoder gates
- Decide — PASS, INCOMPLETE, or FAIL from gate thresholds
- Probe — proxy snapshot of die temp, chunks, storage, tracks
- Artifact — checkpoints,
tap.log,summary.json, optional handoff snippet
Key flags: --batch --matrix, --prune-first, --visual, --tap, --update-handoff, --hermes, --layer layer3_depth.
Scenario expectations
| Scenario | Typical expect | Why |
|---|---|---|
success, normal_short | PASS | Happy path after prune |
stall_sim, encoder_stall_sim | FAIL | Exercises self-heal |
disk_critical | FAIL / INCOMPLETE | Storage blocks encoder |
recovery_after_stall | PASS post-prune | Confirms recovery |
promote_gate | PASS | Layer promotion check |
Dry-run with --dry-run before live matrix batches. See Soaks and automation for CLI examples.
Phantom module rates
phantom_vms_readiness_agent.py runs default modules over ~300 s:
device_identity_fp, peer_sidecar, hermes_mesh, mcp_health, cross_a2a_federation, depth_fusion
Each module logs success/attempts and a percentage rate to TAP. Target ≥ 80% per module at steady state. Transient 0% early in a run is normal during bake, proxy restart, or disk pressure—re-run after recovery.
MCP health asserts five FULL_KEYS: peer_comm_status, hermes_mesh, listen_ports, overlay_mount, cert_fingerprint_present. Probe via API quick reference.
Fleet contract rates
| Signal | Contract | Probe surface |
|---|---|---|
| MQTT telemetry | ~9.8–10 Hz | /fleet/mqtt/latest |
| H.265 encoder | 30 FPS, 60 s chunks | get_recording_status |
| RF-DETR stride | ~10 Hz | Sidecar rfdetr_stride_fps |
| Depth (layer3) | 5–10 FPS monocular | Spatial fusion unit gate |
Healthy shadows show growing recent_chunks, die temp below gate threshold on most samples, and storage_health of OK or WARNING after prune.
Storage impact
When storage_health is CRITICAL (very low free space, thousands of stale chunks), expect:
- Dropping
recent_chunksand encoder fail streaks - Soak INCOMPLETE or FAIL before full gate emission
- Phantom module transient FAILs on justified-clip paths
Recovery pattern: --prune-first (delete oldest .h265, keep a bounded count), restart recorder if needed, re-probe with /fleet/health and get_recording_status. Post-prune WARNING with adequate free space restores contract rates.
HITL cross-check
Automation does not replace spot checks:
soak_gate.sh— one-hour thermal + encoder + MQTT samplevisual_soak_gate.py— annotated JPEGs with hz, die temp, tracks, storage- Playwright e2e — full-page and API telemetry assertions
TAP logs provide an audit trail of every proxy and MQTT call for post-run review in [operator-artifacts].
Next steps
Operator depth
Live fleet state and harness evidence live in private operator handoff (not published) (private).