About this article

• Audience: Operators wiring multi-shadow fleets and external agent integrations
• Goal: After reading, you can verify MCP tool forwarding, A2A peer discovery, and Hermes mesh health across shadows through the Phantom proxy
• Type: How-to

Summary

Ghost Protocol shadows expose a local MCP server for tool calls, an HMAC-signed A2A query surface for conversational and peer workflows, and a UDP Hermes mesh for internal fleet gossip. External systems reach these surfaces through the Phantom proxy at PHANTOM_PROXY—not by joining the UDP mesh directly. This guide walks through registry checks, identity probes, A2A peer hello, and readiness validation.

Specter agent demos (kanban handoff + fleet hygiene): Specter Agent · Hermes Kanban launch.

Prerequisites

• Phantom proxy: export PHANTOM_PROXY=https://<phantom-proxy>:8788
• At least two shadows registered (site-role-a, site-role-b) on a shared bus
• Fleet HMAC secrets provisioned on each shadow (see Security)
• Optional: peer mTLS certs at /data/config/shadow_peer.crt and .key for direct sidecar traffic
• API quick reference for curl templates

1. Confirm fleet registry and bus membership

Start with the central registry. Both shadows should appear with role labels and shared bus membership.

curl -s "$PHANTOM_PROXY/fleet/registry" | jq .

Look for each shadow entry: roles, bus_id, and LAN address fields. Cross-shadow visibility flows through the proxy and MQTT—shadows do not need direct peer-to-peer HTTP for routine fleet awareness.

2. Query device identity and mesh status via MCP

Device MCP runs inside each container on port 8765. The proxy forwards calls so external agents never need direct device access.

# Full identity — serial, firmware, cert fingerprint, mesh peers
curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_device_identity","arguments":{}}' | jq .

# Hermes mesh status — active peers, heartbeat freshness
curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_hermes_mesh_status","arguments":{}}' | jq .

# Enriched health — cert_fingerprint_present, peer_comm_status, hermes_mesh
curl -s -X POST "$PHANTOM_PROXY/tools/call?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -d '{"tool":"get_device_health","arguments":{}}' | jq .

Healthy shadows return cert_fingerprint_present: true, hermes_mesh: "active", and a non-empty peer list when siblings are on the same subnet. Use get_recording_status and get_latest_dets for recorder and detection context—see the API quick reference.

3. Run A2A peer discovery through the proxy

The A2A surface accepts signed queries for conversational safety checks and peer handshakes. All POST requests require an X-A2A-Signature header computed from your fleet-provisioned HMAC secret.

# List available A2A routes (unsigned GET)
curl -s "$PHANTOM_PROXY/a2a/query"

# Peer hello — requires X-A2A-Signature header
curl -s -X POST "$PHANTOM_PROXY/a2a/query?target=<device-ip>" \
  -H "Content-Type: application/json" \
  -H "X-A2A-Signature: <computed-hmac>" \
  -d '{"q":"peer_hello"}' | jq .

A successful peer handshake returns an acknowledgment in the response payload. The proxy mediates cross-shadow A2A so device endpoints stay behind zero-trust boundaries. External SaaS agents should use MCP HTTP forwarding rather than joining the internal UDP mesh.

4. Understand Hermes mesh and peer sidecar ports

Three ports cooperate on each shadow:

Port	Protocol	Purpose
8765	HTTP (MCP)	Tool calls and A2A queries; proxied via PHANTOM_PROXY
11555	UDP broadcast	Hermes mesh heartbeats and fleet gossip (HMAC-signed)
8777	HTTP (mTLS)	Direct peer sidecar for authenticated device-to-device queries

Hermes mesh broadcasts signed heartbeats enriched with device identity; discovered peers appear in mesh_peers. The 8777 peer sidecar starts from the container entrypoint when certs and PEER_IP are set.

5. Validate with Phantom readiness

Run the readiness agent with Hermes validation to grade mesh, A2A, and MCP health across the fleet:

cd [repo-root]/oak-vms-firmware
python3 tools/phantom_vms_readiness_agent.py --hermes-validate --execute-safe

Or trigger via the proxy:

curl -s "$PHANTOM_PROXY/fleet/readiness" | jq .

The agent exercises get_device_identity, get_hermes_mesh_status, cross-shadow registry visibility, and A2A federation modules. Transient failures during stack startup are normal—re-run after MQTT and proxy services are up.

6. Self-heal with Hermes playbooks

Phantom playbooks automate fleet onboarding, federation, and incident response. Common playbooks for mesh workflows:

• fleet-onboarding.yaml — provision shadows and register bus membership
• connection-fleet.yaml / connection-proxy.yaml — wire proxy and MQTT paths
• federation.yaml — cross-shadow A2A and cert handoff
• fleet-health.yaml — periodic mesh and readiness checks Trigger playbooks through the proxy phantom agent API or the Hermes fleet controller:

python3 tools/hermes_fleet_controller.py --observe-only --device site-role-a

Use --observe-only first to diagnose without executing remediation. See Phantom Vision for the full playbook catalog and Cost Sentinel stack.

Security provisioning

Provision secrets per fleet before production—never rely on image defaults.

Secret	Provision path
A2A HMAC	/data/config/a2a_hmac_secret or A2A_HMAC_SECRET env
Peer shared secret	PEER_SHARED_SECRET env (entrypoint export)
Hermes mesh HMAC	Fleet key injected at bake or via operator config
Peer mTLS certs	scripts/generate_peer_certs.sh → /data/config/shadow_peer.{crt,key}

Rotate secrets and restrict cert file permissions (chmod 600 on keys) during fleet onboarding.

Troubleshooting

Symptom	Likely cause	Fix
hermes_mesh: "inactive"	Mesh thread not started or wrong subnet	Restart container; confirm UDP 11555 not blocked
Empty mesh_peers	Siblings offline or HMAC mismatch	Align fleet HMAC keys; verify both shadows on LAN
A2A 401 / signature error	Missing or wrong X-A2A-Signature	Recompute HMAC from provisioned secret
cert_fingerprint_present: false	Certs not deployed	Run generate_peer_certs.sh and redeploy to /data/config
Peer sidecar not listening on 8777	PEER_IP unset or cert guard failed	Check entrypoint logs at /app-storage/logs/peer_comm.log

Next steps

• API quick reference — copy-paste curl catalog for fleet, MQTT, and tools
• Monitoring, observability & TAP — debug ring, MQTT tap, orchestrator traces
• Phantom Vision — 5-minute readiness gate and playbook execution
• Architecture overview — Hermes peer mesh mental model

Operator depth

Live fleet state and harness evidence live in private operator handoff (not published) (private).