About this article

• Audience: Operators and integrators who need the on-device vision pipeline before tuning layers or debugging detections
• Goal: Understand how ghost_core, ghost_agent_v2, and RF-DETR precomp metrics fit together on the edge
• Type: Concept

Summary

Ghost Protocol runs a dual-stream vision pipeline inside each Shadow edge app bundle: a 4K H.265 primary stream for archival and a 640×640 RF-DETR substream for on-device detection. The agent fuses precomp metrics first—person counts and weapon classes remain authoritative on the edge. Cloud video reasoning runs only after justified clips reach the proxy.

Prerequisites

• Architecture overview — fleet tiers and dataflow
• One shadow deployed with RF-DETR models on /data/models or bundled in the image
• Phantom proxy reachable — see API quick reference

Dual-stream pipeline

ghost_core (C++) owns the camera graph:

• Primary stream — 4K H.265 at ~30 fps, written to a video FIFO for the recorder
• Detection substream — 640×640 BGR frames fed to RF-DETR on the Hexagon DSP via SNPE
• NN output — length-prefixed JSON over a dedicated FIFO for the agent

ghost_agent_v2 (Python) consumes both FIFOs:

• Rotates justified 60-second H.265 chunks with sidecar metadata
• Reads RF-DETR telemetry at ~10 Hz stride (RF_DETR_STRIDE_FPS=10)
• Assigns track_id with a host-side IoU tracker when the device ObjectTracker is unavailable on SNPE
• Publishes health and detections over MQTT on a ~30-second cadence

The entrypoint script creates FIFOs, loads layer environment files, starts ghost_core in the background, and runs the agent in the foreground.

RF-DETR precomp authority

On-device RF-DETR is the immutable source for safety counts:

Aspect	Contract
Model	rfdetr_small_inference_model_fixed.dlc (~135 MB float16, 640×640 input)
Classes	23 labels including person (c=0) and weapon categories
Stride	~10 Hz via layer environment (RF_DETR_STRIDE_FPS=10)
Confidence	Default threshold 0.35 for person counts — see model card

Precomp metrics inject first into MCP tools, A2A queries, and Hermes enrichment. LLM layers never override on-device counts. Local on-device reasoning models are retired; temporal natural-language questions escalate through the proxy after precomp.

Outputs land in .latest_dets.json, MQTT detection topics, MCP get_latest_dets / get_rfdetr_status, and ONVIF person events (c=0). See ONVIF reference.

Layer3 monocular depth

Layer3 adds orthogonal spatial context—monocular depth at ~5–10 fps fused with tracklets in the agent. Depth enrichment does not alter precomp person counts. Enable via layer environment and the depth apply script; verify with spatial fusion unit tests before promotion.

Precomp versus cloud escalation

Edge steps are always sovereign—no cloud API keys in the container image, Dockerfile, or entrypoint. The proxy owns upload, auth, and fusion. Details: cloud video escalation.

Operational notes

• Do not kill ghost_core without restarting the app if the H.265 FIFO reader stalls—encoder soak gates may fail.
• Prefer durable edge app builds over hot-patching ghost_core on running containers.
• ONVIF reuses the existing H.265 encoder and RF-DETR events; no separate RTSP stack.

Next steps

• Justified recording contract — sidecar fields and playback verification
• Cloud video escalation — proxy-owned Gemini and Video Intelligence
• Model card — RF-DETR precomp parameters and gates
• RF-DETR ML Pipeline hub — training, custom classes, DLC specs, fleet model names
• Architecture diagrams — ports, protocols, and component relationships

Operator depth

Live fleet state and harness evidence live in private operator handoff (not published) (private).