About this article

• Audience: Operators and integrators running cloud video understanding on justified recorder clips
• Goal: After reading, you can prepare justified clips, choose prompts, and call proxy surfaces for Gemini and Video Intelligence
• Type: Reference

Summary

Justified 60-second H.265 chunks from the edge recorder are the canonical input for cloud video understanding. The phantom proxy transcodes clips to MP4, exposes playback URLs, and owns cloud API keys—edge devices stay sovereign. Use RF-DETR precomp metrics first; escalate to Gemini File API or Video Intelligence only on justified clips.

Prerequisites

• Phantom proxy at https://<phantom-proxy>:8788 with cloud credentials on the proxy host only
• At least one shadow producing justified chunks (justified: true in sidecar/manifest)
• API quick reference for /fleet/playback-ready, /pull-recording, and /tools/call

Justified clip contract

Field	Expected value
Duration	~60 s wall-time per chunk
Frame rate	~30 FPS (PTS deltas ~0.033 s)
Sidecar flags	justified: true, rfdetr_stride_fps: 10.0
Manifest	recent_manifest.jsonl with keyframe_count and continuous groups

Wall-clock time in the clip should match depicted events. Verify with ffprobe PTS deltas before upload.

Proxy surfaces

export PHANTOM_PROXY=https://<phantom-proxy>:8788

# Recording status via MCP forward
curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_recording_status","arguments":{}}'

# Justified chunks with cached MP4 play URLs
curl -s "$PHANTOM_PROXY/fleet/playback-ready?shadow=site-role-a&limit=5"

# Pull latest chunk and transcode to MP4 (for File API upload)
curl -s "$PHANTOM_PROXY/pull-recording?target=<device-ip>&which=latest"

See API quick reference for fleet health, MQTT cache, and snapshot probes.

Gemini File API prompts

Use the File API for clips over 20 MB, long duration, or reuse across prompts. Default fps=1; raise to fps=5 for rapid bus motion. One video per prompt; place text after the video part.

Event description with timestamps:

Describe the key events in this video. Include audio and visual details.
Provide timestamps in MM:SS for salient moments.

Summary and quiz:

Summarize this video. Then create a quiz with 3 questions based on transit
safety content (crowding, falls, unattended items). Use MM:SS timestamps.

Density and safety:

List all visible persons with rough counts/density over time. Note any safety
risks (unattended bags, falls, high density). Timestamps required.
Use precomputed RF-DETR context if provided.

For 5–10 minute continuous groups, use context cache across repeated prompts on the same assembled clip.

Video Intelligence prompts

Annotate via videos:annotate with features such as LABEL_DETECTION, OBJECT_TRACKING, and SHOT_CHANGE_DETECTION. Poll the operation until done: true. Results include segmentLabelAnnotations with entity labels, time segments, confidence, and category entities (for example transport, road, vehicle). Fuse labels back into TAP, MQTT, or A2A for conversational queries about the last clip.

Orchestrator and Phantom modules

cd [repo-root]

python3 tools/phantom_vms_readiness_agent.py \
  --proxy-url "$PHANTOM_PROXY" \
  --hermes-validate \
  --modules gemini_video_understanding,video_intelligence_annotate \
  --tap --report [operator-artifacts]/video-understanding.md

python3 tools/v3_shadow_orchestrator.py \
  --shadow <device-ip> \
  --scenarios video_understanding \
  --visual --tap --prune-first

Artifacts per run: checkpoints (proxy snapshot + MQTT overlay), tap.log, and a handoff snippet. Transient module FAILs during disk or connection pressure are expected; re-run after prune and recovery.

Gotchas

• Upload large justified MP4s through the File API, not inline base64.
• Use proxy + SSH df when overlay storage status lags live disk.
• Never bake cloud keys into edge app images—the proxy centralizes credentials.

Next steps

• Architecture: cloud video escalation
• Video understanding how-to
• API reference · MQTT reference

Operator depth

Live fleet state and harness evidence live in private operator handoff (not published) (private).