About this article

• Audience: Operators, integrators, partners, and agency reviewers with docs-hub access who need the full RF-DETR / TrOCR and Ghost Protocol ML Pipeline story—not only the short precomp contract
• Goal: After reading, you can explain RF-DETR and TrOCR architecture, licenses, classes, datasets, retrain paths, .dlc specs, deploy/eval on Shadow 1 / Shadow 2 / host VMS, and name every Ghost Protocol custom model artifact (including Gemma 4)
• Type: Concept + Reference (flagship)

Summary

Ghost Protocol runs RF-DETR (Receptive Field Detection Transformer) as the on-device vision backbone for Phantom Vision / COTA Shadow fleets. Detection is sovereign: inference runs on the Qualcomm Hexagon DSP inside each Shadow camera. Person and threat metrics from precomp are authoritative—no LLM, including our custom Gemma 4 reasoner, may override counts.

On Shadow 2, Ghost Protocol cascades signage detections into TrOCR (encoder + decoder QNN DLCs) for license-plate and sign text. TrOCR is a secondary cascade, not a precomp authority—boxes still come from RF-DETR.

This page is the hub entry for:

• High-level RF-DETR concept, Apache 2.0 licensing, and multi-vendor stack
• TrOCR requirements, classes (cascade triggers), DLC specs, and retrain via the ML Pipeline
• Exact .dlc / package specs for production bake and custom QCS8550 builds
• Class taxonomies (production precomp, Omni 15-class, weapons ontology)
• Datasets, snapshots/thumbnails, and training methods
• The Ghost Protocol ML Pipeline offering — two tracks: Track V (vision/edge) and Track L (language/host Specter)
• A full prep walkthrough and turnaround guidance for quick reuse of the same vendor platforms
• How models are tested, evaluated, and deployed to S1, S2, and the host VMS
• Exact artifact names per endpoint (edge DLCs + host Specter / Gemma options)

Models are USA-made, USA-trained, and USA-hosted under Ghost Protocol control (Texas HQ; US-region cloud jobs; on-premise edge inference). This docs hub is password-protected for authorized operators and partners—not an open public scrape surface.

For the short safety contract only (person c=0, conf ≥ 0.35), see the RF-DETR model card.

Prerequisites

• Architecture overview — edge, mesh, proxy tiers
• Edge AI and RF-DETR precomp — dual-stream pipeline
• Model card — production precomp lookup
• NIST AI RMF — compliance hooks
• Phantom proxy and Edge Builder reachable on the operator network when following deploy or train steps

---

USA made, trained, and hosted

Ghost Protocol treats model sovereignty as a product requirement, not a slogan.

Pillar	Practice
Made	Firmware, packaging, ontology, and fine-tune recipes authored by the Ghost Protocol team (Texas HQ) for Shadow / QCS8550 fleets
Trained	Fine-tunes run on Ghost Protocol–controlled Google Cloud Vertex AI / Colab Enterprise jobs with US regions preferred for GPU and data residency
Hosted (training data & weights)	Datasets and checkpoints live in Ghost Protocol–controlled GCS project storage (US); compiled DLCs staged in operator-controlled deploy trees
Hosted (inference)	Live detection runs on the camera (Hexagon DSP)—no foreign cloud required for person/threat counts
Escalation	Optional cloud video understanding is proxy-owned, justified-clip only, and never replaces precomp

Procurement-facing posture: edge video and detection stay under customer control; training and compilation for Ghost Protocol models are executed under US-jurisdiction cloud accounts when cloud GPUs are used. See also the sovereign RFP language on the marketing docs hub when writing RFPs.

---

What RF-DETR is

RF-DETR (Receptive Field Detection Transformer) is a transformer-based object detector designed for efficient edge execution. Compared with full global self-attention at every layer, RF-DETR uses receptive-field-aware attention so the model keeps strong detection quality while fitting Hexagon DSP / QNN memory and latency budgets on Qualcomm QCS8550 (Shadow 1 and Shadow 2).

In Ghost Protocol:

• RF-DETR is the vision backbone for bounding boxes, class IDs, and confidences
• Host-side tracking (IoU / OpenCV) may assign track_id when device ObjectTracker is unavailable on SNPE
• Downstream systems (MQTT, MCP, ONVIF, Hermes, A2A) consume precomp metrics first
• Natural-language layers (Gemma 4, or proxy cloud models) describe and reason—they do not invent counts

---

Licenses and compliance

Component	License / agreement	Notes
RF-DETR model code / rfdetr package	Apache License 2.0	Commercial modification and redistribution of the framework allowed under Apache terms
Edge AI pipeline	Apache License 2.0	Camera pipeline and oakapp integration
Qualcomm QNN / AI Hub / .dlc runtime	Qualcomm AI Hub Software License Agreement	Compiled Deep Learning Containers and QNN libraries; commercial use on authorized Snapdragon/QCS silicon
Ghost Protocol fine-tuned weights	Proprietary product weights	Trained by Ghost Protocol for customer fleets; distribution controlled by contract
Third-party datasets (Roboflow, Kaggle, HF, customer video)	Source-specific	Must be honored when ingesting external labels
AdaFace (optional S2 face gallery path)	MIT	Host FR path; see NIST compliance

Compliance hooks for Govern / Map / Measure / Manage: NIST AI RMF page. Precomp immutability is the primary Measure control for safety counts.

---

RF-DETR model specs (.dlc format)

Ghost Protocol ships detection as Qualcomm QNN Deep Learning Containers (.dlc), usually wrapped for fleet install. Two related packages appear in production and custom paths—do not assume they share the same class ID map.

Custom Ghost Protocol QCS8550 package (ML Pipeline / AI Hub)

Primary path for Ghost Protocol–trained Omni / weapons fine-tunes compiled for the fleet:

Spec	Value
Deploy package	rf_detr_oak4.tar (~104–109 MB typical staged size)
Inner model file	rf_detr_qcs8550.dlc (~104 MB typical)
QAI Hub graph name	rf_detr_qcs8550
Architecture family	RF-DETR Nano (RFDETRNano training entrypoint)
Target SoC	Qualcomm QCS8550 (Shadow)
Target runtime	QNN DLC (--target_runtime qnn_dlc)
Compile input_specs	images: [1, 3, 544, 544] (NCHW: batch × channels × height × width)
Context graphs option	context_enable_graphs=rf_detr_qcs8550
Edge backend config	"backend": "snpe", "runtime": "dsp", "performance_profile": "high_performance"
Tar contents	rf_detr_qcs8550.dlc (+ config.json when packaged with backend metadata)
Intermediate export	ONNX (best.onnx / inference_model.onnx) — not the fleet load format
Optional side artifact	.tflite may appear in staging trees; fleet deploy format is .dlc inside the tar

Production precomp bake (ghost_core contract)

Primary path for authoritative safety counts documented in the model card:

Spec	Value
Artifact	rfdetr_small_inference_model_fixed.dlc (~135 MB float16 cited)
Deploy alias	ghost_protocol_transit.dlc (pushed under device model dirs)
Runtime substream	640×640 NHWC BGR888i into the NN node
Class head (parser)	23-class lineage (NUM_CLS=23 parse path)
Person gate	Class c=0, confidence ≥ 0.35
Weapon class IDs (contract)	{10, 11, 12, 13} — gun, knife, rifle, shotgun (see model card)
Stride	~10 Hz (RF_DETR_STRIDE_FPS, layer env)
Backend	Hexagon DSP via SNPE

Resolution map (read carefully)

Path	Spatial size	Notes
QAI Hub compile (current ML Pipeline script)	544×544	Static input_specs for rf_detr_qcs8550
ghost_core production substream	640×640	Precomp bake / model-card path
Some Omni export recipes	672×672	Lifecycle training notes; only valid if export + compile match

Rule: train export shape, compile input_specs, and runtime preprocess must match the package you inject. Mismatched shapes cause silent accuracy loss or load failure.

Config snippet (package metadata)

{
  "model": "rf_detr_qcs8550.dlc",
  "backend": "snpe",
  "backend_properties": {
    "runtime": "dsp",
    "performance_profile": "high_performance"
  }
}

---

Architecture

Dual-stream edge (every Shadow)

Layer	Role
ghost_core	Owns camera graph, encoder, RF-DETR NN, FIFO handoff
ghost_agent_v2	Precomp fusion, recording, MQTT, status files
Precomp	Immutable safety metrics before any LLM or cloud
Phantom proxy	Host VMS BFF, tools, justified pull, cloud escalation
Hermes / A2A	Fleet mesh and conversational surfaces that read precomp

Full ML lifecycle (train → fleet)

S1 vs S2 responsibilities (same RF-DETR package)

	Shadow 1 (Macro Sentry)	Shadow 2 (Precision Sentry)
RF-DETR package	rf_detr_oak4.tar / rf_detr_qcs8550.dlc	Same
Typical AI focus	Crowd density, cleanliness, QR (host), broad counting	Face crop, plate/sign OCR cascade, finer behavior
TrOCR DLC	Not required on macro path	trocr_encoder_qcs8550.dlc + trocr_decoder_qcs8550.dlc
Stereo / nist_fr	No stereo depth path	Optional FATE-style depth PAD + host FR (see NIST page)
Gemma 4 artifact	Optional same filename	Optional same filename

---

Classes supported

Critical: two class ID spaces

Taxonomy	Person ID	Use when
Production precomp contract	0	ghost_core bake, model card, ONVIF person events, soak harness person gates
Omni-Detector 15-class (custom train)	8	Ghost Protocol Omni fine-tune / handoff taxonomy

Deploying Omni weights without remapping labels and agent gates will break person counting. Always version the class list with the DLC.

A) Production precomp (safety contract)

ID	Label (best-effort harness map)	Role
0	person	Authoritative count if p ≥ 0.35
1–9	bicycle, car, bus, train, truck, backpack, handbag, suitcase, cell_phone	Context / transit-adjacent
10–13	gun, knife, rifle, shotgun	Weapon set in model-card / harness
14–22	(additional head slots in 23-class parse)	Do not invent labels without bake notes

Full operational contract: model card.

B) Omni-Detector 15-class (Ghost Protocol custom taxonomy)

ID	Class name	Group
0	handgun	Weapons (CRITICAL_THREAT)
1	rifle	Weapons
2	knife	Weapons
3	car	Transit (TRANSIT_LOG)
4	bus	Transit
5	truck	Transit
6	motorcycle	Transit
7	bicycle	Transit
8	person	People (TRACKING)
9	face	People
10	license_plate	Signage → TrOCR on S2
11	sign	Signage → TrOCR on S2
12	qr_code	Signage → host QR on S1
13	trash_pile	Cleanliness
14	debris	Cleanliness

C) Weapons ontology builder (Casper / Edge Builder path)

Default fine-tune ontology used by the Phantom Vision builder when specializing weapons/bags:

ID	Class
0	handgun
1	rifle
2	knife
3	backpack

Customers add classes through the builder ontology compile step before training.

---

Training tech stack

Layer	Technology
Model family	RF-DETR, package RFDETRNano (Apache 2.0)
Training framework	PyTorch (Lightning where the rfdetr train stack uses it)
Accelerators	NVIDIA A100, L4, or T4 on Vertex; local CUDA; Apple MPS for light local runs
Dataset formats	YOLO detection labels, COCO (convertible), Roboflow YOLOv8 export
Ontology	ontology_config.json → compiled labels.json
Cloud training	Google Cloud Vertex AI / Colab Enterprise
Object storage	GCS under Ghost Protocol–controlled buckets (US)
Export	ONNX fixed spatial size matching compile target
Compile	Qualcomm AI Hub (qai_hub), QNN DLC, graph rf_detr_qcs8550
Edge runtime	Ghost Protocol Edge AI pipeline + SNPE/QNN on Hexagon DSP
Product UI / API	Phantom Vision Edge Builder (phantom-vision-builder + builder API)
Eval	Host ONNX soak, live MCP harness, orchestrator soaks, vision-verify

Default builder train knobs (API): epochs (e.g. 25), batch_size (e.g. 8), gpu_type (e.g. NVIDIA_TESLA_T4). Omni recipes may use higher epoch counts (e.g. 50) and larger batches on bigger GPUs.

---

Dataset types you can train on

Accepted formats

Format	How Ghost Protocol uses it
YOLO detection	Primary train path (.txt labels + images + data.yaml / class names)
COCO JSON	Supported via conversion into the unified YOLO-style train tree
Roboflow export	Common ingest (YOLOv8 download) with per-source class remaps
GCS trees	Builder gcs_dataset path for Vertex jobs
RGB + LWIR pairs	Optional corridor multimodal sets when available (lifecycle design)

Domains already mapped in Ghost Protocol recipes

• Weapons and threat objects
• Transit vehicles and corridor context
• People and faces
• Signage (plates, signs, QR)
• Cleanliness / maintenance (trash, debris)
• Customer custom classes via ontology (agency-specific equipment, PPE, etc.)

Quality bar (required for production-grade fine-tunes)

• Sequence-aware splits — frames from one continuous clip stay in the same train/val/test split (prevents leakage)
• Typical 80 / 10 / 10 train / val / test when building full corridors
• Stratify rare classes so val/test still see knives, plates, debris, etc.
• Clamp boxes to valid normalized coordinates
• Stable class names before remap into global ontology IDs
• Prefer high-resolution sources; the train pipeline resizes to the model input

Snapshots and thumbnails

Snapshots and thumbnails are first-class inputs for the ML Pipeline—not afterthoughts.

Asset	Best use
Fleet snapshots	Capture stills from live preview, justified clip frames, or periodic sampling on S1/S2
Thumbnails	Fast human review for class discovery, hard negatives, and false-positive audits
Justified ~60 s clips	Richer temporal context; export stills or short segments for labeling

Recommended loop

1. Pull stills or short clips via Phantom proxy (playback-ready / pull-recording) or host preview
2. Produce thumbnails for review boards (accept/reject class candidates)
3. Label accepted frames in YOLO/COCO (Roboflow or internal tool)
4. Upload to GCS → Edge Builder dataset compile → train

Snapshots excel at class discovery and hard-negative mining. For tracking-sensitive classes (loitering, multi-frame behavior), also keep short video sequences, not only isolated stills.

---

Training methods

Ghost Protocol fine-tunes RF-DETR rather than training giant detectors from scratch on every job.

Method	Description
Ontology-driven fine-tune	Compile class list → remap multi-source labels → train RFDETRNano
Multi-source merge	Pull weapons / traffic / construction / LPR-style datasets, remap into one taxonomy
Vertex job	Remote GPU train with GCS dataset URI, epochs, batch, GPU type
Local / MPS prototype	Short epoch smoke trains before paying for large Vertex runs
ONNX export	Fixed 1×3×H×W for AI Hub
QAI Hub compile	QCS8550 QNN DLC → package rf_detr_oak4.tar
Fleet inject	Overlay or /data/models deploy + layer env enable

---

Ghost Protocol ML Pipeline (how we offer further training)

The Ghost Protocol ML Pipeline is the productized path to train, package, and deploy Ghost Protocol–owned models. It is one offering with two tracks that share the same USA residency story (Vertex/Colab Enterprise, GCS) but different compile/deploy ends:

Track	Models	Train	Package	Deploy
V — Vision (edge)	RF-DETR, TrOCR	Vertex / Colab Enterprise	Qualcomm AI Hub → .dlc / tar	S1 / S2 DSP
L — Language (host)	Gemma 4 host / Specter option	Vertex / Colab + Unsloth QLoRA (when Path A)	Ollama / MLX / adapter	Phantom Vision VMS host Hermes/Specter

Quick turnaround value: operators already know the Track V walkthrough (data → Vertex → package → deploy → soak). Track L reuses the same operational muscle (US GCS, Vertex jobs, model cards, soak/eval gates) without inventing a second company process — it does not send language models through Qualcomm AI Hub.

Track V — Vision (RF-DETR / TrOCR) — production today

Edge Builder stages (operator UI) — Track V

Step	What you do	Backend surface
1 Ontology	Add/remove classes; compile labels	GET/POST/DELETE /api/ontology…, POST /api/ontology/compile
2 Dataset	Point at GCS dataset; compile	Dataset status + GCS path
3 Training	Epochs, batch, GPU; launch job	POST /api/pipeline/launch
4 Compile DLC	Submit Qualcomm compile	POST /api/pipeline/compile
5 Deploy	Mark/deploy to fleet	Deploy status + camera inject scripts

Builder clients: Phantom Vision Edge Builder panel (VMS Infrastructure) and the embeddable phantom-vision-builder package. Jobs poll until TRAINING_COMPLETE, COMPILE_COMPLETE, DEPLOYED, or FAILED.

Track L — Language (Specter / Gemma on VMS host)

Host Specter Agent experience uses Hermes CLI + skills + Hindsight + gbrain/wiki. Models are host-only (never Hermes/LLM on S1/S2). Precomp remains count authority.

Stage (mirrors Track V ops)	Track L action	In-repo surface
1 Curriculum / ontology	Product SOP domains (specter, precomp, guardrail, vms…)	Playbooks + skills allowlist
2 Dataset	Factory gather → clean → train/val/test/eval	ML Pipeline/gp-gemma4/scripts/dataset_factory.py
3 Train	Path B (default): no LoRA — stock gemma4:26b MoE + skills/memory. Path A: Unsloth QLoRA on Vertex/Colab when justified	Recipes under ML Pipeline/gp-gemma4/recipes/ (Path A)
4 Package	Ollama tag / adapter + model card	MODEL_CARD.md; not QAI Hub
5 Deploy	Hermes env on VMS host	HERMES_SPECTER_* on vms-proxy
6 Eval / promote	Live Specter bench + fleet-gate; default stays Grok until gate	ML Pipeline/gp-gemma4/eval/; docs/specter/PATH_B_IMPRESSIONS.md

Path B (impressions, near-term): skills + Hindsight + gbrain + LLM wiki + optional Hermes MOA + stock large Gemma MoE on host. Path A (company SFT): only after corpus release audit floors + measured gap vs stock Gemma. Do not train on un-audited dumps.

Do	Don't
Reuse Vertex/Colab US region + GCS prefixes for Path A	Send Gemma through Qualcomm AI Hub / .dlc
Serve Specter via Hermes on host	Install Hermes on cameras
Keep precomp as count authority	Let LLM invent person counts
Cite dataset release on model card	Mix tenant Plane A into product SFT

What Ghost Protocol offers customers

• New detection classes for agency-specific objects (Track V)
• Domain fine-tunes (transit corridor, depot, platform, weapons focus) (Track V)
• Compile for QCS8550 and package as rf_detr_oak4.tar (Track V)
• Deploy to Shadow 1, Shadow 2, and verify through host VMS (Track V)
• Optional cascade models (TrOCR on S2) when signage OCR is in scope (Track V)
• Evaluation harness and soak gates before production promote (both tracks)
• Specter Agent language path on the host VMS: product curriculum, optional company Gemma SFT, Hermes/MOA orchestration (Track L)
• Same USA-made / USA-trained / USA-hosted posture across vision and language training data

---

Prep walkthrough: getting ready to use the ML Pipeline

Use this as a readiness runbook before spending GPU or AI Hub minutes.

Phase 0 — Decide scope

1. Classes: Reuse Omni 15-class, weapons ontology, or a custom list?
2. Endpoints: S1 only, S2 only, both, or host-VMS evaluation first?
3. Success metrics: val mAP, person-count stability vs current precomp, weapon recall, plate crop rate, false-positive budget
4. Class ID strategy: If you replace the production precomp bake, plan agent/ONVIF gate updates before fleet promote

Phase 1 — Prerequisites checklist

• [ ] Final class list written down (names + IDs)
• [ ] Labeled data ready or labeling sprint scheduled
• [ ] US-region GCS + Vertex project access for Ghost Protocol (or customer) account
• [ ] Qualcomm AI Hub account/token for compile
• [ ] Edge Builder API reachable on operator network
• [ ] Target cameras unlocked; oak-agent healthy
• [ ] Free storage for model inject on device
• [ ] Post-deploy soak window booked
• [ ] Rollback artifact identified (previous tar or pure-encode layer)

Phase 2 — Data prep

1. Collect snapshots, thumbnails, and short clips from fleet or lab
2. Label in YOLO or COCO; remaps into global ontology
3. Apply sequence-aware train/val/test split
4. Upload to GCS; record the exact gcs_dataset URI for the builder
5. Spot-check thumbnails for mislabels and class imbalance

Phase 3 — Train

1. Set epochs / batch / GPU in Edge Builder or Vertex submitter
2. Launch job; poll status
3. On success, export ONNX and verify spatial size matches the compile target (e.g. 544 for current QAI script)
4. Archive checkpoint path + class list + date

Phase 4 — Compile and package

1. Submit ONNX to Qualcomm AI Hub (QCS8550, QNN DLC, graph rf_detr_qcs8550)
2. Download rf_detr_qcs8550.dlc
3. Package rf_detr_oak4.tar (+ config.json)
4. Tag the package (version, ontology hash, train job id)

Phase 5 — Evaluate

1. Host static / ONNX soak where available
2. Single-device lab deploy first (prefer non-production role)
3. Probe get_rfdetr_status / get_latest_dets via $PHANTOM_PROXY
4. Run mass-transit harness scenarios that exercise precomp authority
5. Run soak (thermals, storage, encode independence)
6. Compare against success metrics from Phase 0

Phase 6 — Deploy and promote

1. Inject package to S1 and/or S2
2. Confirm layer env: AI enabled (DEPTHAI_V3_ENCODE_ONLY=0), model paths set
3. Confirm host VMS surfaces (MQTT dets, proxy tools)
4. Keep pure-encode rollback path documented
5. Only then promote to full fleet

Operational considerations

Topic	Guidance
Thermal	Enabling DSP NN on both cameras raises die temp; schedule soaks and respect AI suspend thresholds
Pure encode vs AI layer	H.265 archival can run encode-only; AI is a layered profile—do not confuse the two
Class remap	Omni person 8 ≠ precomp person 0
Gemma 4 / Specter	Host NL path (Track L); never overrides RF-DETR counts; edge NPU Gemma is historical inventory only
Track reuse	Track L reuses Vertex/GCS/eval discipline from this walkthrough — not QAI Hub or oakapp inject
Privacy	Keep training media under US residency policies; strip unnecessary PII before cloud train
Storage	Model + TrOCR DLCs are large; prune overlays before inject

---

Turnaround

Guidance for planning—not a contractual SLA unless your SOW says otherwise.

Stage	Typical duration
Ontology + labeling plan	0.5–2 days (if classes already agreed)
Labeling / dataset build	1–15+ days (long pole when new video is required)
Snapshot/thumbnail review HITL	Parallel with labeling
Vertex fine-tune (25–50 epochs, modern GPU)	~4–24 hours wall-clock per run
Qualcomm AI Hub compile + package	~1–6 hours
Lab deploy + smoke probes	Same day once DLC ready
Soak / harness before promote	30 minutes–24 hours depending on gate
End-to-end when labeled data is ready	~3–10 business days typical
End-to-end when new field data + labeling required	2–6+ weeks typical

Faster iterations (class tweak, small data add) compress toward the low end. Full corridor ontologies with rare classes sit at the high end.

---

How models are tested, evaluated, and deployed

Test and evaluate

Gate	What it proves
Host static / ONNX	Parsers, tensor shapes, local latency without claiming device USB
Host soak	Sustained RF-DETR (+ optional TrOCR) latency on workstation path
Live MCP harness	Precomp authority, A2A/MCP tools, scenario library (mass transit safety)
Orchestrator / soak	Thermals, H.265 independence, storage health, encode fail streak
Vision-verify	Person-vs-nonperson quality gates complementing—not overriding—precomp
NIST FR path (S2)	Optional face quality / gallery—orthogonal to RF-DETR counts

Always assert: responses that touch safety counts set or honor rf_detr_precomputed / precomp-first fusion. LLMs may be offline; counts remain valid.

Deploy to Shadow 1 and Shadow 2

High-level operator flow (credentials and tunnels are environment-specific):

1. Stage rf_detr_oak4.tar (and S2 TrOCR DLCs if used)
2. Ensure AI layer env on device (model path vars, encode-only off)
3. Inject into oakapp writable overlay or durable /data/models per your deploy script
4. Restart / re-apply layer; confirm NN load in app logs
5. Probe via proxy:

curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_rfdetr_status","arguments":{}}'

curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_latest_dets","arguments":{}}'

Symmetric RF-DETR package for both cameras; S2-only extras are TrOCR DLCs and stereo-assisted paths.

Deploy / run on the host VMS

Mode	What runs where
Production VMS	Host Phantom proxy consumes MQTT/MCP precomp from cameras; operators never need the DLC on the laptop
Edge Builder	Ontology/train/compile orchestration on operator network
Host-in-loop cascade	Optional workstation path with staged rf_detr_oak4.tar + TrOCR DLCs for lab visualization (does not replace ghost_core archival)
Local ONNX eval	best.onnx / inference_model.onnx for offline metrics before compile

H.265 justified recording remains independent of AI when pure-encode profiles are used—see recording contract.

---

TrOCR cascade (signage OCR)

TrOCR (Transformer OCR, Microsoft trocr-small-stage1 lineage) turns RF-DETR signage crops into text on the Precision Sentry path. Ghost Protocol packages it as a two-DLC encoder/decoder pair for QCS8550, driven host-in-the-loop over the Edge AI XLink path (or host ONNX for lab).

Why TrOCR sits after RF-DETR

Concern	Owner
Where is the box?	RF-DETR only (precomp authority unchanged)
What does the text say?	TrOCR (S2) or QR host decode (S1)
May OCR override person counts?	Never

Classes / cascade triggers

Omni-Detector IDs that route to OCR paths:

Omni ID	Class	Cascade
10	license_plate	S2 TrOCR encoder → decoder
11	sign	S2 TrOCR encoder → decoder
12	qr_code	S1 host cv2.QRCodeDetector (not TrOCR)
other	weapons / transit / person / cleanliness	No TrOCR

On production precomp taxonomies without Omni IDs, treat plate/sign classes as signage routing only when the deployed labels include them—do not invent OCR for person/weapon IDs.

TrOCR model specs (.dlc format)

Spec	Encoder	Decoder
Exact artifact	trocr_encoder_qcs8550.dlc	trocr_decoder_qcs8550.dlc
ONNX intermediate	trocr_encoder.onnx (~88 MB staged)	trocr_decoder.onnx (~147 MB staged)
Staged DLC size (typical)	~88 MB	~146 MB
Target SoC	Qualcomm QCS8550 (Shadow)	Same
Runtime	QNN DLC / SNPE on Hexagon	Autoregressive steps with static KV cache
Base family	Microsoft TrOCR small stage1 (TrOCRProcessor.from_pretrained("microsoft/trocr-small-stage1") for host decode)	Same lineage
Crop preprocess	BGR → RGB → 384×384 float [0,1] NCHW batch pixel_values (not 320—breaks DeiT pos embeds)	—
Encoder input (host)	[1, 3, 384, 384] float32	—
Max sequence	—	MAX_SEQ_LEN = 20 tokens
Token IDs	—	START=2, EOS=2, PAD=1
Decoder layers	—	6 (NUM_DECODER_LAYERS)
Attn heads / dim	—	8 heads × 32 embeddings/head
Self-attn KV (static)	—	(1, 8, 19, 32) in; 20 out then shift-pad back to 19
Cross-attn KV (from encoder)	(1, 8, 578, 32) per layer key/val (×6)	Consumed each decode step
Compile toolchain	Qualcomm AI Hub + qai_hub_models.models.trocr, QAIRT 2.45.x, float QNN DLC, device QCS8550 (Proxy)	Split encoder/decoder jobs—not monolithic
Primary fleet role	Shadow 2 Precision Sentry (PHANTOM_DEVICE_ID contains shadow2)	Shadow 2
S1	Models may be staged; runtime plate/sign OCR is S2-gated (S1 uses host QR)	Same

Known hardware note: some Shadow DLC loads report TROCR_DLC_FW_CRASH when encoder/decoder graphs are not SNPE/container-compatible. Fleet soaks may mark device_xlink PARTIAL with RF-DETR OK and TrOCR steps disabled. Treat full-cascade PASS as a recompile/repackage gate, not a silent ignore.

Requirements to run TrOCR

Requirement	Detail
Detector	Working RF-DETR package producing plate/sign boxes
Hardware	Shadow 2 (S2) preferred for on-device cascade; host ONNX path for lab
Storage	~250 MB+ for both DLCs on device/staging
Env	TROCR_ENCODER_MODEL / TROCR_DECODER_MODEL (or defaults under model dir); PHANTOM_DEVICE_ID=shadow2 for S2 routing
AI layer	DEPTHAI_V3_ENCODE_ONLY=0 when sharing the camera with NN work
USB claim	Full XLink cascade needs exclusive device claim (or host ONNX offline)
Host decode	transformers TrOCRProcessor for final batch_decode (host only)

Retrain / recompile TrOCR via Ghost Protocol ML Pipeline

TrOCR is not the same train graph as RF-DETR Nano, but it rides the same pipeline stages Ghost Protocol uses for edge models: data → train/export ONNX → Qualcomm AI Hub → package → deploy → soak.

Stage	What Ghost Protocol does
1. Data	Harvest justified stills / thumbnails of license_plate and sign crops from S2 (RF-DETR boxes). Prefer multi-angle, night, glare, motion blur.
2. Ontology	Keep detector classes stable (license_plate, sign). OCR training labels are strings, not new detector IDs.
3. Train / source weights	Default path: pretrained Microsoft TrOCR small + QAI Hub Models trocr package. Optional domain fine-tune (agency plates/signs) on US-region GPU when partners supply labeled crops—still export at 384×384 / 20-token edge limits.
4. Export	Produce trocr_encoder.onnx + trocr_decoder.onnx (opset ~17) with fixed shapes: encoder (1,3,384,384); decoder self-KV (1,8,19,32); cross-KV (1,8,578,32) ×6.
5. Compile	Qualcomm AI Hub split jobs → trocr_encoder_qcs8550.dlc + trocr_decoder_qcs8550.dlc (qnn_dlc, float, --compute_unit npu, QCS8550 Proxy). Never compile as one fused graph for this fleet path.
6. Package / stage	Place beside rf_detr_oak4.tar in the deploy tree; set env paths on S2.
7. Evaluate	Host ONNX decode smoke → single-device XLink probe → dual-shadow soak. Fail closed on TROCR_DLC_FW_CRASH.
8. Promote	Only after RF-DETR precomp still authoritative and OCR quality meets partner metric (CER/WER on holdout plates).

Edge Builder note: the Phantom Vision Edge Builder today is optimized for RF-DETR detector ontology → train → DLC. TrOCR retrain uses the same vendor platforms and operator runbook (GCS/Vertex/QAI Hub/deploy); treat TrOCR as a second pipeline track in the builder until a dedicated TrOCR job type is exposed. Operators still request “retrain OCR” through Ghost Protocol ML Pipeline services.

TrOCR turnaround (typical)

Stage	Duration
Crop harvest + labeling	2–10 business days (data-dependent)
Fine-tune + ONNX export	~4–48 hours wall-clock
Dual DLC compile	~2–12 hours
Lab + soak (incl. FW crash gate)	1–3 days
End-to-end when labeled OCR data ready	~1–2 weeks typical

---

Exact Ghost Protocol custom model names

Use these exact filenames in runbooks, inventory, and change tickets.

RF-DETR and related vision packages

Exact name	Kind	Role
rf_detr_oak4.tar	Deploy tarball	Fleet package for S1 and S2 AI layer
rf_detr_qcs8550.dlc	QNN DLC	Inner model; graph name rf_detr_qcs8550
rfdetr_small_inference_model_fixed.dlc	QNN DLC	Production precomp bake (model card)
ghost_protocol_transit.dlc	QNN DLC	Deploy alias for transit precomp path
casper1_rfdetr_n	Train run name	Weapons/ontology fine-tune job family
casper1_rfdetr_n.dlc	QNN DLC	Casper1 compiled artifact name in staging
best.onnx / inference_model.onnx	ONNX	RF-DETR train export intermediate
trocr_encoder_qcs8550.dlc	QNN DLC	TrOCR encoder (S2 cascade)
trocr_decoder_qcs8550.dlc	QNN DLC	TrOCR decoder AR (S2 cascade)
trocr_encoder.onnx / trocr_decoder.onnx	ONNX	TrOCR export intermediate / host path
config.json	Metadata	SNPE DSP high_performance binding for package

Shadow 1 endpoints

Role	Exact name(s)
Primary detector package	rf_detr_oak4.tar → rf_detr_qcs8550.dlc
Common env keys	RF_DETR_MODEL_PATH, RF_DETR_OAK4_MODEL (device model dir)
Production precomp aliases (when on precomp bake)	rfdetr_small_inference_model_fixed.dlc, ghost_protocol_transit.dlc
Reasoning model (custom-trained)	ghost_protocol_gemma4_e2b.litertlm

Shadow 2 endpoints

Role	Exact name(s)
Primary detector package	Same rf_detr_oak4.tar / rf_detr_qcs8550.dlc
TrOCR encoder	trocr_encoder_qcs8550.dlc
TrOCR decoder	trocr_decoder_qcs8550.dlc
Production precomp aliases	rfdetr_small_inference_model_fixed.dlc, ghost_protocol_transit.dlc
Reasoning model (custom-trained)	ghost_protocol_gemma4_e2b.litertlm

Host Phantom Vision VMS

Role	Exact name / surface
Live detection source	Camera precomp via MQTT / MCP (get_rfdetr_status, get_latest_dets)
Staged cascade models	rf_detr_oak4.tar, trocr_encoder_qcs8550.dlc, trocr_decoder_qcs8550.dlc
Train export (Track V)	best.onnx, inference_model.onnx
Builder control plane	Edge Builder API (ontology + pipeline jobs)
Specter brain (Track L)	Hermes CLI + Kanban; HERMES_SPECTER_MODEL / HERMES_SPECTER_PROVIDER
Specter default model	grok-4.5 (xai-oauth)
Specter local MoE option	Ollama gemma4:26b (Gemma 4 26B-A4B class)
Company SFT option	gp-gemma4-26b-a4b-specter-rN after Path A train gate
Corpus factory	ML Pipeline/gp-gemma4/datasets/releases/
Operator Path B	Skill allowlist + Hindsight + gbrain + LLM wiki (no LoRA required)

Gemma 4 — reasoning models (host primary; edge historical)

Field	Exact value
Host role (current)	Specter / operator NL on Phantom Vision VMS host — consumer of precomp, never count authority
Default agent	Hermes + grok-4.5; skills + Hindsight always
Stock large host option	Gemma 4 26B-A4B MoE / 31B via Ollama (gemma4:26b / gemma4:31b)
Company SFT naming	gp-gemma4-{size}-{domain}-{rev} (e.g. gp-gemma4-26b-a4b-specter-r1)
Train path (Path A)	Unsloth QLoRA on Vertex/Colab Enterprise after dataset factory release audit — not Qualcomm AI Hub
Edge historical artifact	ghost_protocol_gemma4_e2b.litertlm (LiteRT-NPU) — keep in inventories; not production primary NL
Edge production posture	On-device NPU Gemma path retired/constrained; RF-DETR precomp stays live on cameras
Relationship to RF-DETR	Consumer of precomp only

See historical notes for the NPU primary-path pivot. For Specter impressions without company SFT, use the Path B runbook in the firmware monorepo (docs/specter/PATH_B_IMPRESSIONS.md).

---

References and citations

External

Resource	Why it matters
Qualcomm AI Hub	DLC compile for QCS8550
Qualcomm AI Hub Software License	QNN/DLC commercial terms
Google Cloud Vertex AI	GPU fine-tune jobs
Edge AI architecture	Shadow vision pipeline runtime
NIST AI RMF	Risk framework mapping
Apache License 2.0	RF-DETR and Edge AI pipeline licensing

Internal (this hub)

Page	Topic
Model card	Precomp person/weapon contract
Edge AI architecture	Dual-stream + FIFO
ONVIF	Person events from precomp
NIST compliance	RMF + FR hooks
Video understanding	Cloud after justified clips
Deploy guide	oakapp layers and models
Soaks and automation	Gates before promote
External refs map	Vendor link index

---

Next steps

• Model card — print the short safety contract for reviewers
• Firmware deploy — push models and layers to Shadows
• A2A / Hermes — conversational surfaces that read precomp
• NIST compliance — map this pipeline into RMF language
• Track V: retrain / recompile detection via Edge Builder + this prep walkthrough
• Track L: Specter host language path — monorepo ML Pipeline/gp-gemma4/ + docs/specter/PATH_B_IMPRESSIONS.md
• Contact Ghost Protocol for a custom-class SOW (ontology + data + turnaround) or Specter/host language SOW (curriculum + optional company Gemma SFT)

Operator depth

Live fleet CIDs, bake SHAs, soak tables, and session chronology live in private operator handoff (not published) (private). This page documents contracts and pipeline design—not live telemetry.