Phantom Vision VMS / Ghost Protocol

Appearance

About this article ​

  • Audience: Operators and integrators who need the cloud escalation mental model before running video-understanding workflows
  • Goal: Understand how justified edge recordings reach Gemini and Video Intelligence through the proxy—without cloud keys on devices
  • Type: Concept

Summary ​

Ghost Protocol keeps the edge sovereign: RF-DETR precomp runs first on every shadow, and justified 60-second H.265 chunks with sidecar metadata are the only inputs for cloud video APIs. The Phantom proxy at PHANTOM_PROXY pulls or serves clips, owns every cloud credential, and fuses structured labels and natural-language summaries back into MQTT, debug events, and operator workflows.

Prerequisites ​

Edge readiness (no cloud keys) ​

The edge exposes readiness through existing surfaces—no new encoder or upload path:

SurfacePurpose
get_recording_statusRecorder health, storage, chunk inventory via MCP forward
/fleet/playback-readyJustified chunks with transcoded play URLs
/pull-recordingLatest H.265 chunk pull with SSH fallback
recent_manifest.jsonlRolling chunk list for multi-minute context assembly

Healthy sidecars report duration_s ~60, justified: true, rfdetr_stride_fps: 10, and continuous group labels. RF-DETR stride and fusion run before any cloud call. Cloud API keys never ship in the edge app image, entrypoint, or device overlay.

bash
curl -s "$PHANTOM_PROXY/fleet/playback-ready?shadow=site-role-a&limit=3"
curl -s "$PHANTOM_PROXY/pull-recording?target=<device-ip>&which=latest"

Proxy-owned cloud services ​

The proxy centralizes two complementary services after precomp:

Gemini video understanding — File API upload for justified MP4 clips; timestamped Q&A (MM:SS prompts); optional videoMetadata clipping (last two minutes) and custom fps for motion-heavy scenes; context cache for 5–10 minute manifest groups.

Google Cloud Video Intelligence — Structured LABEL_DETECTION, OBJECT_TRACKING, and optional shot-change features via videos:annotate on object-storage URIs; poll operations until complete.

Auth lives on the proxy host only. The proxy extends /tools/call (for example video_understand) or dedicated video routes, then fuses gemini_summary, temporal events, and intel_labels into TAP traces and MQTT topics.

Escalation flow ​

Orchestrator scenarios and Phantom modules (gemini_video_understanding, video_intelligence_annotate) automate pull → upload → fuse for soak and readiness gates.

Best practices ​

  • One video per Gemini prompt; reuse File API uploads across related questions.
  • Use custom fps (5–10) or clip offsets when default 1 fps misses rapid motion.
  • Prefer low mediaResolution for cost on long manifest groups.
  • Always assert precomp person counts before trusting cloud temporal summaries.
  • Poll Video Intelligence operations to completion before fusing labels.

Next steps ​

Operator depth ​

Live fleet state and harness evidence live in private operator handoff (not published) (private).