Phantom Vision VMS / Ghost Protocol

Appearance

Recovery ​

About this article ​

  • Audience: Operators restoring a shadow after stall, container loss, or failed soak
  • Goal: After reading, you can choose the right recovery path and confirm the device is healthy again
  • Type: Troubleshooting

Summary ​

Recovery runs lightest-first: restart the app through the proxy, power-cycle when the container is gone, unlock when the camera proxy is stuck, and re-deploy for a durable bake from source. After any heavy step, verify registry, recording, MQTT, and H.265 output before returning the shadow to soak.

Prerequisites ​

  • Proxy at http://PHANTOM_PROXY:8788 — see API quick reference
  • SSH to the shadow with YOUR_DEVICE_PASSWORD
  • Repo checkout at [repo-root] with deploy scripts

Choose a recovery path ​

SituationStart here
Encoder stall or stale MQTT; container still in registryApp restart
Device online but no container in registryPower cycle
Camera proxy stuck; deploy blockedUnlock
Overlay drift or repeated restarts failFull re-deploy

App restart ​

Symptom: get_recording_status reports a stall, recent_chunks is flat, or MQTT is stale while the container remains in /fleet/registry.

Cause: The encoder or FIFO reader stopped while oak-agent stayed up. Killing ghost_core without an app restart often leaves the recorder half-dead.

Fix: Restart through the proxy — prefer restart_app over manual pkill:

bash
curl -s -X POST "http://PHANTOM_PROXY:8788/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"restart_app","arguments":{}}'

SSH fallback:

bash
sshpass -p 'YOUR_DEVICE_PASSWORD' ssh root@<device-ip> \
  'oakctl app stop <app-id>; oakctl app start <app-id>'

Re-probe get_recording_status until recent_chunks advances. Prune first if storage is critical — see Common issues.

Power cycle ​

Symptom: The shadow answers on the network but /fleet/registry shows no container after a host crash.

Cause: Unclean shutdown left oak-agent or runc in an inconsistent state that a soft restart cannot clear.

Fix: Power off 30–60 seconds, then power on. Wait for oak-agent and probe /fleet/registry and /fleet/health via API quick reference. If recording does not resume within a few minutes, continue to unlock or re-deploy.

Unlock (surgical reset) ​

Symptom: Deploy or start fails with camera proxy errors, orphaned runc containers, or stale xlink sockets.

Cause: A prior crash wedged oak-agent, runc, or the depthai-device-rvc4 proxy. A new container cannot attach while the proxy is stuck.

Fix: Run unlock from [repo-root] before any bake:

bash
cd [repo-root]
./scripts/unlock_camera.sh <device-ip> YOUR_DEVICE_PASSWORD

Unlock stops oak-agent, clears orphaned containers and IPC sockets, and restarts the agent. Do not kill -9 the device proxy during boot — let unlock handle the sequence.

Full re-deploy ​

Symptom: Hotpatches diverged from source, layers fail durably, or unlock plus restart does not restore quality.

Cause: Overlay writes do not survive reinstall. Production recovery needs a clean bake from source.

Fix:

bash
cd [repo-root]
./scripts/deploy_oakapp.sh <device-ip> YOUR_DEVICE_PASSWORD

The wrapper unlocks the camera, syncs source, builds, installs with --force-new, and starts the app. After deploy, prefer proxy and runc exec for diagnostics — avoid further oakctl churn.

Post-redeploy verification ​

Symptom: Deploy succeeded but soak gates, playback, or peer mesh still fail.

Cause: Install success does not guarantee encoder output, MQTT freshness, or Hermes readiness — each needs an explicit probe.

Fix: Before declaring recovery complete:

  1. Registry and identity — shadow ready in /fleet/registry; get_device_identity returns fingerprint and mesh status (API quick reference).
  2. Recording — get_recording_status shows is_writing_data: true and growing chunks; storage WARNING or OK post-prune.
  3. MQTT — /fleet/mqtt/latest fresh within ~120 s at ~10 Hz.
  4. H.265 — fresh chunk over 1 MB; shadow verifier at expected fps (typically 30) returns PASS.
  5. Soak — short gate (recovery_after_stall, normal_short) and live MCP harness before a full 60-minute soak.

Thermals under policy with mitigation active. On failure, retry from the lightest applicable step above.

Next steps ​

Operator depth ​

Live fleet state and harness evidence live in private operator handoff (not published) (private).