About this article

• Audience: Operators who need live fleet state without SSH on every shadow
• Goal: After reading, you can tap MQTT and proxy APIs, forward device MCP tools, and read orchestrator TAP logs during soaks
• Type: How-to

Summary

The Phantom proxy at PHANTOM_PROXY is the primary observability surface for Ghost Protocol shadows. It caches MQTT telemetry, forwards device MCP tools, buffers debug events, and exposes fleet health—so you can monitor die temp, detections, recorder status, and snapshots without direct device SSH for most workflows.

Prerequisites

• Phantom proxy: export PHANTOM_PROXY=https://<phantom-proxy>:8788
• At least one shadow registered (site-role-a / site-role-b)
• Optional: [repo-root]/oak-vms-firmware/scripts/tap_shadow_session.py for MQTT sidecar logs
• API quick reference for curl templates

1. Fleet snapshot through the proxy

Start with aggregated fleet views—these answer whether shadows are alive and writing data.

# Registry — shadows, roles, LAN IPs
curl -s "$PHANTOM_PROXY/fleet/registry" | jq .

# Health — recorder, storage, thermal grades per shadow
curl -s "$PHANTOM_PROXY/fleet/health" | jq .

# Latest MQTT — telemetry, detections, status (~10 Hz cache)
curl -s "$PHANTOM_PROXY/fleet/mqtt/latest" | jq .

Check each shadow for payload.telemetry (die_temp, fps), payload.detections (person count, track_id), and freshness fields (anomaly_alerts, stale_sec).

2. Forward MCP tools

Device MCP runs inside the container on port 8765. The proxy forwards calls so you do not need to reach the device directly.

# Recording health — storage, chunk inventory, stall hints
curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_recording_status","arguments":{}}' | jq .

# Latest RF-DETR detections — person count, track ID, die temp
curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_latest_dets","arguments":{}}' | jq .

# Device identity — serial, firmware, COTA labels
curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_device_identity","arguments":{}}' | jq .

get_device_health is also useful for thermal and sidecar status. Full tool catalog: API reference.

3. Live snapshots and debug ring

# JPEG snapshot (graceful fallback when live stream unavailable)
curl -s "$PHANTOM_PROXY/stream/snapshot?target=<device-ip>&w=320&q=3" -o snapshot.jpg

# Debug events — vmsDebug ring (live/warm_ready, snapshot_fail, recorder_status)
curl -s "$PHANTOM_PROXY/debug/events" | jq .

When a shadow is offline, the proxy may fall back to cached H.265 tails via SSH ffmpeg rather than returning a hard error.

4. MQTT bus tap

Shadows publish on devices/<shadow>/# (telemetry, dets, health, status, data_manifest). Two ways to observe:

1. Proxy cache — GET /fleet/mqtt/latest (preferred for scripts and orchestrator)
2. Direct subscribe — tap_shadow_session.py logs key events to a sidecar file for soak audits

cd [repo-root]/oak-vms-firmware
python3 scripts/tap_shadow_session.py --test-dump --shadow site-role-a --count 20

Run alongside soak_gate.sh or orchestrator --tap for a parallel audit trail. Topic layout: MQTT architecture.

5. Orchestrator TAP and visual checkpoints

The v3 shadow orchestrator records every proxy, MQTT, and script call when you pass --tap. It also writes annotated checkpoints when you pass --visual:

cd [repo-root]
python3 tools/v3_shadow_orchestrator.py \
  --shadow <device-ip> \
  --smoke --visual --tap --dry-run

Per-run artifacts land under test-results/v3-runs/<timestamp>_<scenario>/:

Artifact	Purpose
tap.log	Timestamped proxy GETs, MCP calls, MQTT subs
checkpoints/*.jpg + *.json	Proxy snapshot + metrics at each phase
summary.json	PASS / INCOMPLETE / FAIL with temp, hz, storage

Playwright soak-gate specs hit the same proxy endpoints for UI verification during E2E runs. See Soaks & automation.

Key metrics to watch

Metric	Healthy signal	Source
MQTT rate	~10 Hz telemetry	/fleet/mqtt/latest
Die temp	Below soak gate threshold (typically ≤78°C)	telemetry + get_latest_dets
Storage	storage_health not CRITICAL; recent chunks ≥ 1	get_recording_status
Track ID	Present in dets when scene has people	/fleet/mqtt/latest dets
Recorder	is_writing_data: true; justified latest chunk	get_recording_status

If storage is CRITICAL, prune and restart before re-soaking—see Updates, prune & storage.

TAP without direct device access

For routine monitoring, the proxy ingests MQTT, caches fleet state, and forwards MCP. /fleet/mqtt/latest, /debug/events, and /tools/call expose live session state; orchestrator --tap and tap_shadow_session.py add FIFO and manifest sidecar reads. Healthy surfaces return 200; snapshots degrade gracefully when a device is offline.

Quick repro checklist

curl -s "$PHANTOM_PROXY/fleet/mqtt/latest" | jq '.devices | keys'
curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_recording_status","arguments":{}}'
python3 oak-vms-firmware/scripts/tap_shadow_session.py --test-dump --shadow site-role-a --count 5

Next steps

• API quick reference — copy-paste curl catalog
• Fleet status — readiness grades and blockers
• Soaks & automation — orchestrator batch matrix and visual gates
• Updates, prune & storage — fix CRITICAL storage before soak

Operator depth

Live fleet state and harness evidence live in private operator handoff (not published) (private).