About this article

• Audience: Operators deploying monocular depth on Ghost Protocol shadows and validating spatial fusion output
• Goal: After reading, you can apply layer3_depth, confirm the cumulative layer env on device, and verify enriched detections with x/y/z coordinates
• Type: How-to

Summary

Layer3 adds monocular depth inference on top of the encode, RF-DETR, and object-tracker stack (layers 0–2). Depth runs at roughly 5–10 FPS outside the Sync pipeline—do not enable NeuralDepth on single-lens Shadow 1 devices. The apply script ships a cumulative depthai_v3_layer.env; spatial fusion pairs tracklets with depth samples to produce optional x, y, z fields in detection payloads.

Prerequisites

• SSH to target shadow with YOUR_DEVICE_PASSWORD
• Phantom proxy: export PHANTOM_PROXY=https://<phantom-proxy>:8788
• Monocular depth model at /data/models/monocular_depth.dlc on device
• API quick reference for post-deploy probes

1. Understand cumulative layers

Each layer file includes all lower layers. Applying layer3_depth ships encode baseline plus RF-DETR, tracker, and depth variables:

Layer	Adds
layer0_baseline	Encode-only baseline
layer1_rfdetr	RF-DETR person detection
layer2_tracker	Object tracker and track_id
layer3_depth	Monocular depth + spatial fusion
layer4_thermal	Thermal governor
layer5_full	All layers

The apply script resolves layer3_depth → config/depthai_v3_layers/layer3_depth.env, merges layers 0–3, and rsyncs to the device config path.

2. Apply layer3 to a shadow

cd [repo-root]/oak-vms-firmware
APPLY_RESTART=1 ./scripts/apply_depthai_v3_layer.sh layer3_depth <device-ip>

When prompted, enter YOUR_DEVICE_PASSWORD. APPLY_RESTART=1 runs oakctl stop/start after deploy so ghost_core reloads the merged env.

Confirm the active layer on device:

ssh <device-ip> 'grep DEPTHAI_V3_LAYER /data/config/depthai_v3_layer.env'

3. Layer3 environment variables

The shipped layer3_depth.env sets detection, tracker, and depth tuning:

DEPTHAI_V3_LAYER=layer3_depth
RF_DETR_MODEL_PATH=/data/models/ghost_protocol_transit.dlc
RF_DETR_STRIDE_FPS=10
TRACK_CLASS_IDS=0,1,2,3
DEPTHAI_V3_ENABLE_TRACKER=1
MONO_DEPTH_MODEL_PATH=/data/models/monocular_depth.dlc
MONO_DEPTH_FPS=8

Push models before apply if the device lacks monocular_depth.dlc:

./scripts/push_device_models.sh <device-ip>

Replace placeholder depth weights with a production DepthAnything or MiDaS .dlc per your release plan—the env contract stays the same.

4. Validate spatial fusion output

After restart, probe detections through the proxy. Layer3 enrichments add numeric x, y, z when depth frames align with tracklets:

curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_latest_dets","arguments":{}}' | jq .

Healthy layer3 payloads include the base contract (c, p, b, track_id) plus non-null x, y, z when depth sampling succeeds. MQTT cache mirrors the same fields:

curl -s "$PHANTOM_PROXY/fleet/mqtt/latest" | jq '.devices["site-role-a"].payload.detections'

Run unit gates locally before fleet promote:

cd [repo-root]/oak-vms-firmware
python3 -m pytest tests/unit/test_spatial_fusion.py -q

5. Orchestrator and Phantom integration

Exercise layer3 in soak batches with prune-first and visual checkpoints:

cd [repo-root]
python3 tools/v3_shadow_orchestrator.py \
  --shadow <device-ip> \
  --layer layer3_depth \
  --scenarios normal_short,recovery_after_stall,promote_gate \
  --prune-first --visual --tap --update-handoff --hermes

Pair with Phantom readiness to grade depth_fusion alongside Hermes and MCP modules:

python3 tools/phantom_vms_readiness_agent.py \
  --proxy-url "$PHANTOM_PROXY" \
  --hermes-validate \
  --modules depth_fusion,mcp_health \
  --tap --report [operator-artifacts]/layer3-readiness.md

Promote gate expectations: normal_short and recovery_after_stall should PASS when storage is healthy and the encoder is writing justified chunks. Transient readiness FAILs during layer3 post reports are handled in harness; re-run readiness after stack stabilizes.

Troubleshooting

Symptom	Likely cause	Fix
No x/y/z in dets	Depth model missing or fps mismatch	Push monocular_depth.dlc; confirm MONO_DEPTH_FPS
Layer env not active	Overlay lag or stale entrypoint	Re-apply with APPLY_RESTART=1; rebuild image if entrypoint stale
Fusion nulls only	No depth frame at bbox center	Normal for edge tracklets; check mono stream health

Depth in Sync or NeuralDepth on single-lens hardware is unsupported—keep monocular depth on the layer3 path only.

Next steps

• API quick reference — get_latest_dets and MQTT probes
• Phantom Vision — readiness wizard with depth_fusion module
• Soaks & automation — --layer layer3_depth batch matrix
• Edge AI architecture — pipeline and precomp authority

Operator depth

Live fleet state and harness evidence live in private operator handoff (not published) (private).