Phantom Vision VMS / Ghost Protocol

Appearance

About this article ​

  • Audience: Operators who run the Phantom Vision stack on demand for VMS work
  • Goal: After reading, you can start and stop the stack safely, unblock fleet actions, and recover when proxy or MQTT services fail
  • Type: Troubleshooting

Summary ​

The Phantom Vision stack stays down by default to limit host cost and background traffic. Bring it up only when you need fleet visibility, soak automation, or Phantom readiness. Most VMS failures trace to an idle stack, port conflicts, or MQTT not wired to the proxy after a partial restart.

Prerequisites ​

  • Repo checkout at [repo-root] with cost-sentinel installed
  • export PHANTOM_PROXY=https://<phantom-proxy>:8788
  • Host and Phantom stack for baseline start/stop commands
  • Recovery when a shadow needs app restart or re-deploy after stack recovery

UI shows stack down and blocks fleet actions ​

Symptom: The VMS OPS UI loads but reports stack down. Fleet registry, playback-ready, readiness, and MCP forwards are disabled or return connection errors.

Cause: Proxy (:8788) and MQTT (:1883) are not running. The UI can start independently on :5173 without the backend stack.

Fix: Start the stack and confirm health before any fleet work:

bash
cd [repo-root]/cost-sentinel
npm run stack:up
npm run stack:status

Expect proxy and MQTT both healthy. Re-open the UI section or refresh, then probe $PHANTOM_PROXY/health and /fleet/registry.

Stack will not start (ports in use) ​

Symptom: npm run stack:up fails, or stack:status shows one service down. lsof reports listeners on :8788 or :1883 from an old session.

Cause: A previous stack instance, manual broker, or dev server still holds the ports. Foreign Mosquitto installs on the host are a common conflict.

Fix: Tear down cleanly, then verify ports are free:

bash
cd [repo-root]/cost-sentinel
npm run stack:down
# optional: skip stopping Vite if you need the UI
bash scripts/phantom-vision-stack.sh down --no-vite
lsof -i :8788 -i :1883 2>/dev/null || echo "stack idle"

Stop any foreign broker on those ports, then run npm run stack:up again. If the proxy still fails, check cost-sentinel logs and restart Docker-backed MQTT if your install uses it.

Proxy up but MQTT stale or missing ​

Symptom: /health passes for the proxy, but /fleet/mqtt/latest shows high stale_sec, low telemetry Hz, or empty device topics. Soak and readiness modules fail on MQTT contract checks.

Cause: MQTT broker started without the expected fleet registry wiring, or the proxy ingest client lost its subscription after a restart. Device mqtt_host in registry may not match the running broker.

Fix: Restart the full stack (not proxy alone):

bash
cd [repo-root]/cost-sentinel
npm run stack:down && npm run stack:up

Confirm broker config and shadow publish host align in /fleet/mqtt/config. If staleness persists after stack restart, reprovision MQTT on the shadow via /fleet/provision-shadow and restart the edge app. See Common issues — MQTT stale.

Fleet work blocked after stack is up ​

Symptom: Stack status is healthy, but readiness returns critical FAIL, playback-ready never merges, or incident-triggered H.265 pulls do not run.

Cause: Shadows are not registered, SSH credentials are wrong, or storage/encoder health on the device blocks VMS paths. Readiness and playbooks expect at least one shadow (site-role-a / site-role-b) visible in registry.

Fix: Run the readiness gate after stack up:

bash
cd [repo-root]
python3 tools/phantom_vms_readiness_agent.py \
  --proxy-url "$PHANTOM_PROXY" \
  --hermes-validate \
  --tap

Proxy equivalents: GET /fleet/readiness and POST /fleet/readiness/remediate with an approved playbook such as h265_stall. If readiness stays critical, follow Recovery before re-running soaks. When finished, run npm run stack:down to return to idle posture.

Next steps ​

Operator depth ​

Live fleet state and harness evidence live in private operator handoff (not published) (private).