Deployment Recipes

Self-contained, runnable shell scripts that serve a model behind an OpenAI-compatible API on NVIDIA hardware. Recipes are the deployment layer for the reference architectures and solution templates in this repo.

Layout

recipes/<platform>/<server>/docker-<model-slug>.sh   # Docker recipe (vLLM or NIM)
recipes/<platform>/<server>/serve-<model-slug>.sh    # llama.cpp native recipe

Platforms: gb10, precision, gb300 · Servers: vllm.

Conventions

• #!/usr/bin/env bash + set -euo pipefail.
• --port PORT (or -p) CLI override, env-var fallback declared at top.
• docker pull then docker run -d --restart unless-stopped (pre-clean the old container with docker rm -f); print the verify curl at the end.
• GB10 (DGX Spark, aarch64): --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864, mount ~/.cache/huggingface.

Readiness contract, health checks, monitoring (all recipes)

Every docker-*.sh now builds, monitors and waits before returning (shared code: lib/common.sh, lib/monitoring.sh):

1. pull image, replace the old container, docker run -d --restart unless-stopped;
2. start/refresh the shared tokenomics Prometheus + Grafana (see ../monitoring/);
3. block until ready — container running, /health 200, served name in /v1/models, and a tiny completion succeeds — then print one machine-readable line:

READY {"ready":true,"container":"lightning-3.5-gb10","port":8000,"base_url":"http://localhost:8000/v1",
       "served_name":"nvidia/nemotron-3.5-lightning","models":[...],"seconds_to_ready":212,
       "max_model_len":200000,"client_max_output_tokens":16384,"client_prompt_budget_tokens":182592,
       "prometheus":"http://127.0.0.1:9090","grafana":"http://127.0.0.1:3000/d/deskside-tokenomics"}

Common flags on every recipe: --no-wait, --wait-timeout SEC, --no-monitoring, --dry-run (prints the exact docker commands, tokens redacted, changes nothing — also the safe way for an agent to preview a rollout). Env equivalents: DESKSIDE_WAIT, DESKSIDE_WAIT_TIMEOUT, DESKSIDE_MONITORING, DESKSIDE_DRY_RUN.

Per-recipe health assessment (field debugging; read-only except one tiny completion): each recipe has a sibling health-<model>.sh — host/driver/runtime/memory/disk → container (state, restarts, OOM, restart policy) → endpoint (/health, /v1/models, smoke completion, optional --tool-test structured tool call) → load/KV-cache → log-signature scan with fixes → monitoring. Exit 1 on any FAIL; --json for automation; --bundle [DIR] writes a token-redacted debug tarball.

recipes/gb10/vllm/health-nemotron-3.5-lightning-nvfp4.sh --tool-test
recipes/precision/vllm/health-qwen3.8-flash-next-nvfp4.sh --json
recipes/lib/health.sh --container NAME --port PORT --served MODEL      # any vLLM container
recipes/lib/client-config.sh --port 8000 --served nvidia/nemotron-3.5-lightning   # window-aligned client configs

Container names, default ports and served names are unchanged (precision Lightning: lightning-3.5-precision-<port>; that recipe now also re-derives the name when --port is given). Tested by tests/test_recipe_contract.sh (mock server, ready/died/timeout/--no-wait paths) and tests/test_consistency.py (dry-run of all recipes); the health script was run against live containers (Lightning and Super VL on T6, Super VL on GB300, GLM-5.3-Flash on GB300).

Models

Each model carries its size designation — where a solution designates it. Agentic = the S/M/L designation in the Agentic Coding solution; further solution columns get added as other templates designate models.

There is no Nemotron L-class designation right now — 3.5 Super replaces 3 Ultra — so Super VL, the M/L default, also serves the L tier; Lightning multi-instance / DP is the measured fallback and GLM-5.3-Flash is the L alternative.

Agentic Coding — model × platform

Support rule: S-class models run on S/M/L hardware, M-class on M/L, L-class on L. Cells give quick perf numbers where measured (see notes); ✓ = in designation range, no published number yet; — = outside the designation range. Tests are run in-designation only.

1. Super VL is the M/L default (flags from the model card plus multi-user settings from the Nemotron-3-Super recipes); measured results are in note 4; Lightning remains the S-tier default. No Nemotron L class exists beyond it (3.5 Super replaces 3 Ultra).
2. Aggregate output tok/s on the agent-workflow shape (ISL 60k / OSL 1k) at concurrency 20 unless annotated otherwise; "chat" cells are aggregate chat-sweep peaks; GB300 per-session concurrency from the re-measured agent ladder in the solution README. Routed (multi-instance) figures are taken behind a LiteLLM router with --num_workers matched to the host's core allocation; launch-era figures taken through a single-worker router (which capped the endpoint near 2.4k tok/s) are retired as measurement artifacts, so those cells carry no published number yet.
3. Measured with the predecessor unsloth/ NVFP4 checkpoint on locked-clock GB10; the current nvidia/ ModelOpt pin keeps the same architecture (MTP preserved).
4. Super VL, aiperf 0.7.0, 64 requests per level: agent = ISL 60k / OSL 1k aggregate output tok/s at concurrency 32; chat = ISL 200 / OSL 200 at concurrency 64 (the ladder ended before the chat peak). T4 numbers are one TP=2 instance on two RTX PRO 6000 Max-Q GPUs of a T6; T6 numbers are two TP=2 replicas behind LiteLLM on four of its five GPUs. Raw ladders: tests/data/.

Recommended deployments

Exactly one recommended recipe per model × platform × use-case set, chosen by best measured performance (ties broken by fewer moving parts). Use-cases: chat = aggregate throughput at high concurrency (200-in / 200-out sweep); agent = agentic long-context shape (ISL 60k / OSL 1k) at concurrency 20. Routed variants always run LiteLLM with --num_workers matched to the host's core allocation.

Multi-instance topologies require the multi-GPU *_GPUS device pin (precision recipes; gb10 qwen recipe's QWEN38_27B_GPUS). TP > 1 on PCIe RTX PRO towers needs NEMOTRON_LIGHTNING_DISABLE_CUSTOM_AR=1 (see the recipe headers).

Add new recipes and models here and keep these tables in sync.

Sizing

Use the sizing rules in S/M/L Sizing to choose a model + quant that fits the target platform's memory budget.