About this article

• Audience: Operators running cloud video analysis on edge-recorded H.265 chunks through the Phantom proxy
• Goal: After reading, you can pull justified clips, run Gemini Video and Cloud Video Intelligence analysis on the proxy host, and wire results into Phantom readiness gates
• Type: How-to

Summary

Edge devices record justified 60-second H.265 chunks with RF-DETR precomp as the authoritative detection layer—no cloud API keys ship in the edge app image. The Phantom proxy at PHANTOM_PROXY owns cloud escalation: it pulls recordings, transcodes to MP4, and calls Gemini Video or Cloud Video Intelligence. Use this path for temporal natural-language QA and structured label detection on transit footage after precomp gates pass.

Prerequisites

• Phantom proxy: export PHANTOM_PROXY=https://<phantom-proxy>:8788
• Justified chunks on at least one shadow (justified: true in recording status)
• Cloud credentials on the proxy host only (Gemini API key or gcloud ADC for Video Intelligence)—never on edge devices
• API quick reference for pull and playback-ready endpoints

1. Confirm justified clips are ready

Edge authority stays with RF-DETR precomp and the H.265 recorder. Verify a shadow is writing justified chunks before cloud analysis:

curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_recording_status","arguments":{}}' | jq '.justified_ok, .storage_health'

curl -s "$PHANTOM_PROXY/fleet/playback-ready?shadow=site-role-a&limit=3" | jq .

Playback-ready returns MP4 play URLs for justified segments. Sidecars include duration_s (~60), keyframe counts, and rfdetr_stride_fps.

2. Pull and transcode through the proxy

The proxy reuses /pull-recording (SSH SFTP + ffmpeg) and caches under recordings-cache/ and recordings-preview/:

curl -s "$PHANTOM_PROXY/pull-recording?target=<device-ip>&which=latest" \
  -o [operator-artifacts]/latest-justified.mp4

For scripted prep, prefer the playback-ready playUrl when available—it skips redundant transcode when cache is warm.

3. Run Gemini Video analysis (proxy host)

Run Gemini calls on the machine hosting vms-proxy, not on the shadow:

# On proxy host — GEMINI_API_KEY in environment
python3 - <<'PY'
from google import genai
client = genai.Client()
myfile = client.files.upload(file="[operator-artifacts]/latest-justified.mp4")
response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=[
        myfile,
        "Summarize this transit cabin video. Include timestamps (MM:SS) "
        "for people entering or exiting and other salient events."
    ],
)
print(response.text)
PY

Best practices for transit footage:

• Use File API for 60-second justified MP4s (reuse across prompts)
• Set custom videoMetadata fps (5–10) for fast motion; default 1 fps may miss brief events
• Clip with start_offset / end_offset for last-N-minute windows from recent_manifest.jsonl
• Place the text prompt after the video part in the contents array
• Use low mediaResolution for cost control on long or repeated analysis

4. Run Cloud Video Intelligence (proxy host)

For structured labels and object tracking, upload to GCS and annotate from the proxy host:

# On proxy host — gcloud ADC with Storage access
gcloud storage cp [operator-artifacts]/latest-justified.mp4 gs://<your-bucket>/recent_justified.mp4
gcloud ml video detect-labels gs://<your-bucket>/recent_justified.mp4

Poll the operation until done: true. Results include segmentLabelAnnotations with entity descriptions, confidence, and time offsets—fuse these with Gemini NL timestamps for operator dashboards.

5. Integrate with Phantom and orchestrator

Add video modules to the 5-minute readiness gate:

python3 tools/phantom_vms_readiness_agent.py \
  --proxy-url "$PHANTOM_PROXY" \
  --hermes-validate \
  --modules gemini_video_understanding,video_intelligence_annotate \
  --tap --report [operator-artifacts]/video-readiness.md

Run the dedicated orchestrator scenario for end-to-end TAP and checkpoints:

python3 tools/v3_shadow_orchestrator.py \
  --shadow <device-ip> \
  --scenarios video_understanding \
  --prune-first --visual --tap --update-handoff --hermes

Artifacts land under test-results/v3-runs/ with tap.log, timeline JSON in checkpoints, and fused Gemini + Intel summaries. Layer3 depth (spatial) and video understanding (temporal) complement each other—see Layer3 depth & fusion.

Troubleshooting

Symptom	Cause	Fix
Empty Gemini response	Clip too large for inline path	Switch to File API upload
Missed motion events	Default 1 fps sampling	Raise fps in videoMetadata
Auth errors on edge	Keys in container	Move all cloud auth to proxy host only
Intel operation stuck	Async poll not complete	Poll until annotationProgress reaches 100%

Next steps

• API quick reference — pull-recording and playback-ready
• Phantom Vision — 5-minute module wiring
• Mass transit scenarios — video_understanding in the batch matrix
• Cloud video escalation — architecture fit

Operator depth

Live fleet state and harness evidence live in private operator handoff (not published) (private).