Agentic Video Search & Summarization

Turn raw video into searchable, summarized, queryable insight — locally.

Summary

Deploy NVIDIA's Video Search and Summarization (VSS) AI Blueprint on Dell deskside systems: a local vision-language model plus a locally-served LLM for natural-language search, timestamped summarization, and event detection over your own footage — no cloud dependency, no data-residency risk. Dell adds S/M/L sizing (GB10 · RTX PRO 6000 towers · GB300), a pinned LLM lineup on vLLM, recipes, and deployment guides.

Personas

Reference NVIDIA playbooks

Use cases

Upstream profiles: base (search, Q&A, clip retrieval), alert-verification (compliance), real-time-alerts (continuous ingest + alerts). What this solution is not: a hosted SaaS; a CCTV replacement (consumes existing cameras, doesn't manage them); a general-purpose agent runtime (see our OpenClaw on NemoClaw solution); face recognition (identity is a per-site policy call).

Architecture

Three upstream areas: real-time video intelligence (VIOS / VST extraction, embeddings → message broker); downstream analytics (trajectories, incidents, verified alerts); agentic & offline processing (search, Q&A, summarization, clip retrieval via MCP).

   User (Web UI, API, or agent)
             │
             ▼
   VSS Agent + VSS Agent UI (Docker Compose)
             │
             ├──► VIOS / VST (video ingest + embeddings)
             │        └──► Message broker + vector / graph DB
             │                 └──► Analytics (trajectories, incidents, alerts)
             │
             ├──► VLM (vision-language) — Nemotron 3 Nano Omni · vLLM
             │
             └──► LLM (summarization / Q&A) — Nemotron 3.5 Lightning · vLLM

   Dell deskside system (S: GB10 / T2 · M: Dell Pro Precision 9 T4 · L: Dell Pro Precision with GB300)

Key containers: VSS Agent + UI, VIOS / VST, LLM + VLM (vLLM), Phoenix. Mode: Standard VSS (hybrid) — both models served as local vLLM endpoints on one GPU (--use-remote-llm, --use-remote-vlm).

S / M / L sizing

Three sizing axes: LLM concurrency, VLM ingestion throughput, video storage; YOLO26 per-camera FPS adds a camera-count limit.

LLM (summarization, Q&A, alert reasoning)

VLM (all tiers)

Nemotron 3 Nano Omni 30B A3B on local vLLM at every tier (TP / multi-instance on M/L).

Object detection (YOLO26, measured)

All runs FP16 TensorRT; camera counts assume 15 FPS per camera.

All variants beat real-time (> 30 FPS); T2 ~2× faster than GB10 for pure vision, accuracy identical. Pick the smallest variant meeting the accuracy floor.

Video storage (second sizing axis)

GB/hour depends on your footage — measure early on actual footage. Planning bands: local disk for short-retention S tier; NAS for shared 30-day corpora and M tier; S3-compatible store for 90+ day / L-tier archives. Index size runs 5–15 % of raw video. Tier memory: S/M/L Sizing.

Model recommendation

• LLM default — Nemotron family, per tier. S: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (MoE, NVFP4, reasoning + tool calling — give clients max_tokens >= 512). M/L: Nemotron 3.5 Super VL.
• Alternatives (one per tier; recipes in Deployment recipes):
• S — nvidia/Qwen3.8-27B-NVFP4: dense 27B NVFP4, MTP speculative decode.
• M — nvidia/Qwen3.8-Flash-Next-NVFP4: MoE, MTP, CPU-offloaded n-gram table (≥128 GB host RAM).
• L — RedHatAI/GLM-5.3-Flash-NVFP4: 320B-total / 18B-active MoE; ~181 GiB NVFP4 fits one 252 GB GB300 with FP8 KV.
• VLM pairing (all tiers): Nemotron 3 Nano Omni 30B A3B (nvidia/nemotron-3-nano-omni-30b-a3b), local vLLM — vision + audio + language jointly. Inline commands in Developer Deploy Guide.

Pin per system for reproducibility; alternatives trade VRAM footprint against caption / summary quality.

Inference stack

vLLM-based serving. LLM and VLM run as containerized vLLM servers (public vllm/vllm-openai image); hybrid mode points VSS at them via --use-remote-llm / --use-remote-vlm. NVFP4 MoE models need the Marlin pin on public images (--moe-backend marlin + VLLM_NVFP4_GEMM_BACKEND=marlin), as the Lightning recipes bake. Domain fine-tunes via VLM_CUSTOM_WEIGHTS=<path>. See Inference Stack & Models.

Deployment recipes

Inside the VSS hybrid stack the LLM is co-located with the VLM at 0.35 utilization — the inline command in Developer Deploy Guide is that variant. Full flow in the developer guide.

Benchmarks & performance (validation)

LLM concurrency — S tier measured (prior reference models, 200 in / 1000 out)

Dell Pro Precision 9 T2 (Nemotron 3 Nano 30B A3B NVFP4) — interactive comfort c2–c4, large TTFT tail at c8:

Dell Pro Precision with GB10 (Nemotron Nano 9B v2 FP8) — comfort c2–c4, acceptable through c16 (~16 tok/s per session):

M / L (planning estimate, not benchmarked for VSS): scale the S envelope — M ~2×, L ~3–4× S-tier concurrency at similar TTFT — and confirm with the promotion gates below.

GPU memory co-location (S tier, measured on T2)

Peak co-located GPU memory 68.3 GB of 96 GB (71.4 %), both containers at --gpu-memory-utilization 0.35. VLM must start first — vLLM needs free GPU at startup.

Planning notes

Plan a video-ingest headroom buffer of 20–30 % on peak ingestion. Detection capacity is measured — YOLO26 table above.

Management environment

• Storage: corpus + vector / graph indices sized against retention; disk, NAS, and S3-compatible stores all supported upstream.
• Monitoring: Prometheus on vLLM /metrics (both containers); DCGM GPU telemetry; Phoenix for VSS ingestion; Grafana panels for VLM ingest (chunks / sec), summarization latency, storage growth.
• Identity: org gateway with AD/LDAP-backed auth in front of the VSS UI (port 7777) on multi-user tiers.
• Data handling: video at rest may be sensitive — document retention, encryption at rest, access control, and audit trail before rollout.

Promotion gates

Promotion gates (S → M → L) — promote only when all pass: healthy endpoint through the full sweep; error rate ≤ 1 %; p99 latency meets the use-case SLO; sustained tok/s and req/s on target; ≥ 2 consistent runs. A failed gate holds the tier and gets documented.

Dependencies & Risk factors

• Dependencies: upstream VSS Blueprint (Docker Compose; Docker CE version-sensitive); vLLM on GB10 / Precision / GB300; Nemotron LLM + Omni VLM availability; NGC + Hugging Face credentials; Container Toolkit.
• Risk factors: M/L sizing is scaled from S; storage sizing depends on your footage; NVFP4 MoE on public images requires the Marlin backend pin.

Related

• Developer Deploy Guide · IT-Admin Guide · Agentic Coding
• sizing · stack · personas · glossary · recipe catalog · repo home