About this article

• Audience: Operators running soak gates, batch scenarios, or Phantom readiness through the v3 orchestrator
• Goal: After reading, you can run single and batch soaks, interpret PASS/INCOMPLETE/FAIL, and collect TAP and visual artifacts
• Type: How-to

Summary

The v3 shadow orchestrator coordinates apply, contract verify, soak, visual checkpoints, and optional handoff updates in one cycle. It reuses mature scripts (soak_gate.sh, visual_soak_gate.py, tap_shadow_session.py, verify_fleet_contract.py) and delegates to the proxy at PHANTOM_PROXY for fleet probes. Use --prune-first when storage is tight.

Prerequisites

• Phantom stack up: export PHANTOM_PROXY=https://<phantom-proxy>:8788
• SSH to shadows with YOUR_DEVICE_PASSWORD
• Repo at [repo-root] with tools/v3_shadow_orchestrator.py
• API quick reference for proxy probes

1. Single soak cycle

Run one shadow through apply → verify → soak → decide:

cd [repo-root]
python3 tools/v3_shadow_orchestrator.py \
  --shadow <device-ip> \
  --layer layer2_tracker \
  --duration 60 \
  --visual --tap --update-handoff \
  --mqtt-host <phantom-proxy>

Pass criteria: H.265 chunks grow on most samples, die temp stays at or below 78°C on most samples, and storage_health is not blocking. More than 25% failures on thermal or encoder checks marks FAIL. Missing final soak markers can yield INCOMPLETE—the watcher and post-probe still decide.

2. Batch and matrix runs

Exercise multiple scenarios safely—start with --dry-run:

python3 tools/v3_shadow_orchestrator.py \
  --batch --matrix --dry-run \
  --scenarios success,stall_sim,disk_critical,recovery_after_stall,layer_variants \
  --visual --update-handoff

python3 tools/v3_shadow_orchestrator.py \
  --batch --matrix \
  --scenarios success,encoder_stall_sim,disk_critical,recovery_after_stall,promote_gate \
  --prune-first --visual --tap --update-handoff

Common scenario names: normal_short, success, stall_sim, encoder_stall_sim, disk_critical, recovery_after_stall, promote_gate, handoff_correctness, layer_variants, video_understanding.

Useful flags: --prune-first, --tap, --visual, --update-handoff, --smoke, --watcher, --promote, --hermes. For layer3, add --layer layer3_depth and promote_gate to --scenarios.

3. Phantom 5-minute modules

Run readiness modules interleaved with soak phases:

python3 tools/phantom_vms_readiness_agent.py \
  --proxy-url "$PHANTOM_PROXY" \
  --hermes-validate \
  --modules device_identity_fp,peer_sidecar,hermes_mesh,mcp_health,cross_a2a_federation,depth_fusion \
  --tap --report [operator-artifacts]/phantom-readiness.md

For video understanding, extend modules with gemini_video_understanding and video_intelligence_annotate. The orchestrator can wire run_phantom_unit_gate before and after probes. See Video understanding.

4. Visual gates and E2E

Annotated checkpoints capture proxy snapshots plus MQTT metrics at each phase boundary:

python3 tools/v3_shadow_orchestrator.py \
  --shadow <device-ip> \
  --scenarios normal_short \
  --visual --tap --dry-run

UI regression via Playwright:

cd [repo-root]/cost-sentinel
npx playwright test e2e/soak-gate-visual.spec.ts

Grading summary:

Result	Meaning
PASS	Encoder and thermal samples within thresholds; justified chunks present
INCOMPLETE	Soak ended without a final gate marker; probes may still pass
FAIL	Encoder silent or thermal/encode failure rate above 25%

Self-heal on stall or disk critical: prune, restart_app via proxy, re-verify chunks, re-soak. Hermes playbooks cover h265_stall and disk_critical.

5. Artifacts per run

Outputs land under test-results/v3-runs/<timestamp>_<scenario>/:

Artifact	Contents
index.md	Phase timeline and decisions
checkpoints/*.jpg + *.json	Snapshot + metrics at each boundary
soak.log / .pass	Raw soak output and pass marker
tap.log	Timestamped proxy, MQTT, and script calls
summary.json	PASS / INCOMPLETE / FAIL with temp, hz, storage
handoff_snippet.md	Handoff block for operator review

6. Contract probes during soak

Re-probe through the proxy at soak boundaries:

curl -s "$PHANTOM_PROXY/fleet/mqtt/latest" | jq .
curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_recording_status","arguments":{}}' | jq .
curl -s "$PHANTOM_PROXY/fleet/playback-ready?shadow=site-role-a&limit=3" | jq .

Healthy MQTT rate is roughly 10 Hz. Recorder sidecars should show justified: true and duration_s near 60 on continuous runs.

Troubleshooting

INCOMPLETE with OK thermals: Encoder may have stalled—prune, restart app, re-soak. See Gate interpretation.

Batch FAIL on disk scenarios: Run --prune-first or manual prune before the matrix.

Transient readiness FAILs post-layer: Wait for container stabilize, then re-run Phantom modules.

Next steps

• Host and Phantom stack — start proxy before orchestrator
• Updates, prune & storage — fix CRITICAL storage first
• Monitoring, observability & TAP — read tap.log and checkpoints
• Video understanding — cloud escalation on justified clips
• API quick reference — proxy tools and fleet endpoints

Operator depth

Live fleet state and harness evidence live in private operator handoff (not published) (private).