Phantom Vision VMS / Ghost Protocol

Appearance

Common Issues ​

About this article ​

  • Audience: Operators running soaks or daily fleet checks
  • Goal: After reading, you can diagnose and fix the four most frequent failures
  • Type: Troubleshooting

Summary ​

These four issues cover most soak gate failures and fleet health degradations on Shadow fleets. Probe through the proxy first (PHANTOM_PROXY:8788), then restart apps or prune recordings as needed.

Encoder stall ​

Symptom: recent_chunks stays at zero, encode_fail_streak climbs above 24, and soak samples stop advancing past ~50/60. get_recording_status may report "Recording appears stalled."

Cause: The H.265 FIFO reader or ghost_core encoder path stopped while the agent still runs. Storage pressure and partial restarts (pkill without app restart) are common triggers.

Fix: Prune recordings if storage is tight, then restart the edge app (oakctl app stop/start). Probe with get_recording_status until recent_chunks resumes. Re-run a short soak gate before a full 60-minute soak.

Disk / storage critical ​

Symptom: .h265_status or /fleet/health shows storage_health: CRITICAL — high used percentage, under 1 GB free, hundreds of chunks. Recorder may still report is_writing_data: true but chunk growth stalls.

Cause: Sixty-second H.265 chunks fill internal storage faster than rotation or pruning keeps up. Overlay lag after partial prunes can leave health CRITICAL even when writes resume.

Fix: Run the shadow prune script (keep ~30 newest chunks), confirm free space via get_recording_status, then restart the app. For sustained recording, attach external storage before boot. Re-probe /fleet/health until status is WARNING or OK.

Thermal soak fail ​

Symptom: Soak gate marks thermals FAIL when die temperature exceeds policy limits, or MQTT publishes THERMAL_CRITICAL. Short soaks may PASS while longer runs fail near sample 40–50.

Cause: Sustained 30 fps encode plus RF-DETR inference raises SoC die temperature. Enclosure placement, ambient heat, or missing renice/thermal mitigation worsen peaks above 70–85 °C.

Fix: Check die temp in /fleet/mqtt/latest during a soak. Improve airflow, confirm thermal watchers are active, and prune or reduce load if storage pressure adds I/O heat. Restart the app after mitigation, then re-run soak with read_soc_temp monitoring.

MQTT stale ​

Symptom: /fleet/mqtt/latest shows stale_sec above 120, telemetry Hz drops below ~8, or contract verify reports low MQTT rate. Detections and health topics stop updating while the device appears online.

Cause: Broker disconnect, wrong mqtt_host in device config, or the proxy ingest client lost its subscription after a stack restart. Network blips between shadow and host broker also produce transient staleness.

Fix: Confirm /fleet/mqtt/config broker host matches the device registry entry. Restart the proxy MQTT stack and the shadow edge app. Run contract verify for 20 s at expected ~10 Hz. If stale persists, reprovision MQTT host via /fleet/provision-shadow.

Next steps ​

Operator depth ​

Live fleet state and harness evidence live in private operator handoff (not published) (private).