Phantom Vision VMS / Ghost Protocol

Appearance

About this article ​

  • Audience: Firmware contributors adding device plugins or reusing oak-examples pipeline patterns
  • Goal: After reading, you can structure a plugin, apply a depth layer, verify via MCP, and align with precomp-first release gates
  • Type: How-to

Summary ​

Ghost Protocol extends sovereign oakapp through the plugins/ tree, Edge AI v3 layer applies, and patterns borrowed from oak-examples. New work must preserve RF-DETR precomp authority, expose health via MCP/proxy, and pass plugin tests before soak promotion.

Prerequisites ​

  • Glossary — precomp, layer 3, sovereign
  • Edge AI architecture — pipeline mental model
  • SSH to root@<device-ip> with YOUR_DEVICE_PASSWORD
  • PHANTOM_PROXY=https://<phantom-proxy>:8788

Understand the plugin layout ​

Plugins live under [repo-root]/oak-vms-firmware/plugins/ (M8 bus/WiFi/controller, Axis VAPIX, test stubs). They load through oakapp bake or layer apply with configuration under device storage paths.

Extension points:

ComponentYour hook
ghost_coreNeuralNetwork nodes, encoder settings; ObjectTracker with host fallback when device tracker unavailable
ghost_agent_v2Fusion, Hermes, justified recorder sidecars
MCP serverTool definitions and A2A responses
entrypointFIFO and background service launch

Read M8 plugin architecture before hardware-specific plugins.

Apply a depth layer (example) ​

Monocular depth (layer 3) is the reference pattern for non-default NN extensions:

bash
cd [repo-root]/oak-vms-firmware

# Push models to device
./scripts/push_device_models.sh <device-ip>

# Apply layer (set APPLY_RESTART=1 to restart oakapp)
APPLY_RESTART=1 ./scripts/apply_depthai_v3_layer.sh layer3_depth \
  <device-ip> YOUR_DEVICE_PASSWORD

Verify:

bash
curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_device_health"}' | jq .

# Optional orchestrator layer probe
python3 tools/v3_shadow_orchestrator.py --shadow <device-ip> --layer

Expect spatial fusion fields (x, y, z) in detection payloads when the layer is active. Contract tests cover this shape.

Enable object tracker fallback ​

Set GHOST_ENABLE_OBJECT_TRACKER=1 in the device profile environment. When SNPE linking fails on RVC4, ghost_agent_v2 assigns tracks on the host until device ObjectTracker is available.

Test before merge ​

bash
# Plugin system integration (sovereign runc exec inside container)
pytest tests/test_plugin_system.py -q

# Spatial fusion contract
pytest tests/test_spatial_fusion.py -q

Add harness scenarios when behavior affects live MCP detection authority.

Reuse oak-examples ​

The oak-examples tree catalogs streaming, neural-network, integration, and tutorial projects. When authoring:

  1. Follow upstream EXAMPLE_AUTHORING.md conventions
  2. Target Shadow 1 / sm8550 pipelines
  3. Keep precomp RF-DETR authoritative—do not replace live dets with cloud inference
  4. Expose new capabilities via MCP port 8765 and Phantom proxy 8788
  5. Attach junit, soak gate, and optional visual checkpoints for release candidacy

Historical example inventories live in appendices/internal-plans/ (not public-synced).

Release alignment checklist ​

  • [ ] Plugin loads in bake or layer apply without breaking H.265 primary stream
  • [ ] get_device_health and recording status stable after restart
  • [ ] No cloud API keys added to edge image
  • [ ] Tests green; short orchestrator batch PASS with --tap
  • [ ] Docs updated if public MCP contracts change

Next steps ​

Operator depth ​

Layer apply evidence, tracker env snapshots, and fleet-specific plugin state live in private operator handoff (not published) (private).