Skip to content

Latest commit

 

History

History
301 lines (250 loc) · 19 KB

AGENTS.md

File metadata and controls

301 lines (250 loc) · 19 KB

AGENTS.md

TensorRT-LLM: open-source library for optimized LLM inference on NVIDIA GPUs. Python and C++ codebase with a PyTorch execution path.

If a CLAUDE.local.md file exists alongside this file, read and respect it — it contains developer-specific overrides that supplement this shared guidance.

Rules (Read First)

CRITICAL (YOU MUST):

  • Read and follow CODING_GUIDELINES.md for ALL code changes (C++ and Python)
  • NVIDIA copyright header on ALL new files (update year on modified files)
  • git commit -s (DCO sign-off required). Never attribute AI tools in sign-off line. Always rely on git to do the sign off instead of directly adding sign off in commit message.
  • Do not add co-authors to the git commit message unless explicitly instructed to do so by the user.
  • pre-commit hooks run on commit — if files are modified by hooks, re-stage and commit again
  • LLM args or nested-config changes must run python3 scripts/generate_llm_args_golden_manifest.py and commit tensorrt_llm/usage/llm_args_golden_manifest.json; new fields require telemetry/privacy CODEOWNER approval
  • Changes to serving YAML fields, aliases, defaults, types, or constraints (including nested, disaggregated, and VisualGen configs) must regenerate and inspect the editor JSON Schemas with python3 scripts/generate_trtllm_serve_schemas.py, using a TensorRT-LLM build matching the checkout. Update the generator's YAML-specific adaptations when configuration annotations alone do not capture the change, and add regression coverage for the affected accepted/rejected values. Run pytest -m cpu_only tests/unittest/llmapi/test_trtllm_serve_config_schemas.py. Commit the generated tensorrt_llm/schemas/*.json snapshots; never hand-edit them. The snapshots also include the package version in their $id and title; python3 scripts/bump_version.py <version> updates that metadata with python3 scripts/generate_trtllm_serve_schemas.py --sync-version, which does not import TensorRT-LLM or require a matching build. python3 scripts/generate_trtllm_serve_schemas.py --check and the CPU schema tests reject missing or stale snapshots. Wheels and HTML docs publish these same files.
  • Change the package version only with python3 scripts/bump_version.py <version>, which updates every file that records it. When a file starts recording the package version (for example a tensorrt_llm==<version> pin), add it to VERSION_PATTERNS in that script in the same change. The version-records-check pre-commit hook (bump_version.py --check) rejects records that do not match tensorrt_llm/version.py.
  • When adding or renaming a public model architecture identifier, update tensorrt_llm/usage/architecture_allowlist.py with its exact identifier. It must be publicly documented by the upstream model provider or in docs/source/models/supported-models.md; never add private, customer-specific, or arbitrary user-supplied names
  • PR title format: [JIRA/NVBUG/None][type] description (e.g., [TRTLLM-5516][perf] optimize cuda graph padding)
  • Put a change's rationale and history (why it was made, what it replaces, ticket/PR references) in the PR description, not in code comments. Code comments should explain the code as it stands for a future reader, not narrate how it got there.
  • Set LLM_MODELS_ROOT env var when running tests that need model weights

Common Commands

Task Command
Unit tests pytest tests/unittest/
C++ component tests cmake --build cpp/build_RelWithDebInfo --target common-tests (also runtime-tests, executor-tests, batch-manager-tests, kernel-tests)
C++ test scheduling ctest --test-dir cpp/build_RelWithDebInfo -L common --output-on-failure (build the selected tests first)
Specific test pytest tests/unittest/llmapi/test_llm_args.py
Pattern match pytest tests/unittest -k "test_llm_args"
Integration tests LLM_MODELS_ROOT=/path/to/models pytest tests/integration/defs/...
Serve model trtllm-serve <hf_model> --port 8000
Serve with config trtllm-serve <hf_model> --config config.yaml
Benchmark trtllm-bench --model <hf_model> throughput --dataset <path>
Find CI stage for test python scripts/test_to_stage_mapping.py --tests "test_name"

Installation & Build

Building TensorRT-LLM requires Docker and may involve compiling C++ components. See the Installation Guide for pre-built release containers and pip install, build from source for development builds, and Container Images for information about the container images.

Reference Configs

examples/configs/database/ contains pareto-optimized serving configurations across multiple models, GPUs, ISL/OSL combinations, and concurrency levels. Use these as starting points for deployment and benchmarking rather than hand-tuning parameters. See deployment guides for model-specific walkthroughs.

Architecture

See architecture diagram for the full Mermaid diagram.

Backends

Backend Status Entry Point Key Path
PyTorch Default TorchLlmArgs _torch/pyexecutor/ → PyExecutor → PyTorch Engine

Shared C++ Core (via Nanobind)

Both backends share these C++ components:

  • Scheduling pipeline: Scheduler → BatchManager (in-flight batching) → KV Cache Manager
  • Decoding pipeline: Decoder (token generation orchestration) → Sampling

Request Flow

HuggingFace Model → LLM API → PyTorch Executor
    → Scheduler → Model Forward → Decoder → Sampling → Generated Tokens

Serving

  • trtllm-serve: OpenAI-compatible REST + gRPC server, supports all backends
  • Disaggregated serving: separates prefill (context) and decode (generation) across GPUs
    • KV cache exchange via NIXL (default), UCX, or MPI

Multimodal request data

Read the multimodal developer guide before changing multimodal paths. Reuse MultimodalParams; request Params own cache references. Encoder and LLM forward Params are execution views.

Key Files

File Role
tensorrt_llm/llmapi/llm.py Main API entry point
tensorrt_llm/llmapi/llm_args.py Complete configuration schema (Pydantic)
tensorrt_llm/llmapi/llm_utils.py Model loading, model-specific default overrides
tensorrt_llm/models/modeling_utils.py Base classes for all models (PretrainedConfig, PretrainedModel)
tensorrt_llm/executor/executor.py Execution abstraction (GenerationExecutor)
tensorrt_llm/models/automodel.py Auto-discovery and model registry
tensorrt_llm/_torch/models/ PyTorch backend model implementations (distinct from the top-level models/ package)
tensorrt_llm/_torch/attention/ATTENTION_DEVELOPER_GUIDE.md Attention, MLA, backend families, sparse backends, metadata contracts, and KV-cache behavior - read before modifying anything under tensorrt_llm/_torch/attention/
tensorrt_llm/_torch/moe/fused_moe/MOE_DEVELOPER_GUIDE.md MoE architecture, backends, communication, development patterns — read before modifying MoE code
tensorrt_llm/_torch/pyexecutor/kv_cache/REGIONS_DEVELOPER_GUIDE.md KV cache regions: persistent state declared with a life cycle of its own and laid out as native regions — read before adding model- or backend-specific state to the KV cache managers
CODING_GUIDELINES.md C++ and Python coding standards (referenced throughout, must read before contributing)

Design Patterns

Pattern Key Points
Config hierarchy BaseLlmArgs → TorchLlmArgs, model-specific defaults override generics, Pydantic validation
Model architecture Each model: Config (inherits PretrainedConfig) + ForCausalLM (inherits PretrainedModel)
Model defaults Architecture-specific overrides in llm_utils.py (attention kernels, quant, spec decoding, cache)
Attention backends TorchLlmArgs.attn_backend selects kernel: TRTLLM (default), FlashInfer, FlashAttention
Distributed execution Tensor/pipeline parallelism via Mapping class, multiple backends (MPI, Ray, RPC)
Auto-discovery Models self-register via automodel.py, resolved by HF config architectures field

VisualGen

VisualGen is a vertical alongside LLM for Diffusion-Transformer (DiT)-based image/video generation (text-to-image, text-to-video, image-to-video). It is not an LLM backend — it has its own engine, args, params, and outputs — but shares ops and kernels with the PyTorch backend where it makes sense (attention, quantization, parallelism).

Key entry points:

  • Public Python API: from tensorrt_llm import VisualGen, VisualGenArgs, VisualGenParams.
  • Serving CLI: trtllm-serve <HF id> --visual_gen_args <YAML path>.

Key files:

  • tensorrt_llm/_torch/visual_gen/ENGINEERING_CRITERIA.md: Engineering criteria for any change under tensorrt_llm/visual_gen/ or tensorrt_llm/_torch/visual_gen/ — API discipline, feature/test/lossy-vs-lossless requirements, examples & docs rules. Read before modifying anything in those trees.
  • tensorrt_llm/visual_gen/: VisualGen public Python API. User-facing surface — before modifying anything here, pause and confirm with the user that a public API change is actually intended; do not infer it from the surrounding task.
  • tensorrt_llm/_torch/visual_gen/: VisualGen internal implementation. All non-user-facing code belongs here.

Anti-Patterns / Gotchas

  • Pre-commit modifies files in-place — if hooks fail, files are already modified. Re-stage (git add) and commit again.
  • Protected APIs exist — changes to LLM API signatures will fail tests/unittest/api_stability tests. Get code owner review.
  • Integration tests need GPUs + models — always set LLM_MODELS_ROOT and ensure GPU access. Unit tests don't.
  • Copyright year — update to current year when modifying existing files; add full header to new files.
  • Avoid broad exception handling — catch specific exceptions, not bare except: (see CODING_GUIDELINES.md).
  • One concern per PR — avoid scope creep. If a PR touches unrelated areas, split it.
  • User-facing configuration classes - when editing or defining any user-facing configuration classes (particularly BaseLlmArgs or any class used in its fields), you MUST follow the Pydantic guidelines in CODING_GUIDELINES.md.

Development Workflow

  1. Set up build environment (see installation docs)
  2. Make changes following CODING_GUIDELINES.md
  3. Test locally with pytest

Branching policy and PRs

  • The main repository (upstream) is located at https://github.com/NVIDIA/TensorRT-LLM/
  • Branches should always be pushed to the user-specified fork (usually origin)
  • If pushing fails to due pre-push pre-commits hooks getting updated, just re-push immediately
  • PRs should be opened on the main repository
    • Target main unless fixing a release branch bug
    • See CONTRIBUTING.md for full PR policies

GitHub CLI authentication (GH_CONFIG_DIR)

The gh CLI uses ~/.config/gh by default for authentication. Different GitHub hosts or forks may require a different config directory. Before running any gh command (e.g., gh pr create, gh api, gh pr comment):

  1. Check if the user has specified a custom GH_CONFIG_DIR (e.g., in CLAUDE.local.md or environment). If so, use it.
  2. If not explicitly set, default to ~/.config/gh; do not ask for confirmation.
  3. Prefix all gh commands with the resolved config dir: GH_CONFIG_DIR=<path> gh ...
  4. If the command fails due to missing authentication or the wrong GitHub host/account, report the failure and ask for the correct GH_CONFIG_DIR.

CI / Testing

See CI overview for full details.

Layer Location Notes
Unit tests tests/unittest/ Run in pre-merge CI; some tests require GPU
API stability tests/unittest/api_stability/ Protects committed API signatures
Integration tests tests/integration/defs/ Requires GPU + LLM_MODELS_ROOT
Test lists tests/integration/test_lists/test-db/ Per-GPU YAML files (l0_a10.yml, l0_h100.yml, etc.)
Test waives tests/integration/test_lists/waives.txt Skip known-failing tests with NVBug links
Performance See benchmarking guide trtllm-bench and trtllm-serve benchmarks

Pass product or test Python file paths to scripts/run_pyrefly.sh for a file-list static check; pre-commit supplies these paths with pass_filenames: true. If none are eligible, the script skips the check. --full-project requests a scan of all configured Python files under tensorrt_llm/ and tests/. For supplied files, diagnostics are filtered to added and modified lines when Git provides them and retained at file level otherwise. File-list checks print every remaining diagnostic; full-project checks save the report and print its absolute path. Jenkins runs a full-project check with path/module and symbol exclusions. Whole-project checks always return success; file-list checks can opt into --strict. Jenkins and pre-commit checks are advisory. The [tool.pyrefly] section in pyproject.toml defines product roots, diagnostics, and native exclusions. [tool.pyrefly-tests] in the same file defines test roots and helper search paths. The wrapper generates a temporary test configuration that inherits shared rules and exclusions, runs each scope separately, and combines the diagnostics into one report. Temporary configurations are removed after the check. Mixed file lists are selected from each configuration's project-includes; full-project mode runs both passes. [tool.pyrefly-exclusions] defines additional symbol exclusions for the wrapper. Pyrefly is pinned in requirements-dev.txt. Missing imports are reported except for PyTorch, generated bindings, and configured third-party libraries. In test file-list mode, missing imports of declared dependencies that are not installed are also filtered; full-project scans retain them. Release-Check installs pytest so that pytest.skip() remains recognizable as a terminating call. Tests use the same diagnostics as product code, including missing-argument, unexpected-keyword, and bad-argument-count. The wrapper reads literal __extra_import_path__ lists or tuples under tests/ without running test code and adds existing directories to the shared search path for all test files. Ancestors of the declaring file are skipped because fallback searches them per file. Product checks use only the project root, with fallback disabled. The broader helper search paths and fallback resolution apply only to the test pass. --no-path-module-filter disables exclusions in both passes while preserving their separate search paths and shared diagnostics.

New C++ unit tests should use add_tllm_gtest with explicit SOURCES, LINK_LIBRARIES, OWNER, and scheduling LABELS. Request FULL_STACK explicitly for integration tests and include gpu in LABELS for tests that need a GPU. CI controls test concurrency through parallel_run_ctest. google-tests continues to build all C++ tests; component aggregate targets build migrated tests. C++ integration fixtures build the collected tests with incremental native targets. cpp-tests-<directory> targets (for example cpp-tests-common or cpp-tests-executor-bounce) mirror recursive CTest directory coverage, including legacy and full-stack executables. OWNER aggregates are component subsets and are not interchangeable with these directory targets. The Linux Release source-build fixture reuses the installed CI library for the RNN formatter test. CI source and wheel use the same commit and image per architecture. Link/load errors fail the tests. Direct CMake builds use source libraries unless explicitly configured otherwise. CONFIG_LINUX_X86_64_VANILLA also builds google-tests and publishes the cpp_test_bundle CMake component as a separate cpp-tests-<main-tar-name> artifact. The main package does not include the test programs. Only -CPP- stages using this build configuration fetch and unpack the companion into src/cpp/build, including fat-image consumers. Build reuse requires both artifacts. Linux Release integration fixtures use this bundle directly, retaining CTest selection and retries. The two FMHA tests share a test-only library packaged next to their executables; other runtime libraries come from the matching installed wheel. Test artifacts keep symbols but omit BOLT's extra relocation records. CPP stages collect only tests/integration/defs/cpp and use the standard test environment setup. CTest discovers cases on the test node, so the build stage does not need a GPU. Without a bundle, or with an explicit TRTLLM_BUILD_ROOT, fixtures retain the incremental build path.

Vendor source tools and promotion

Use the tools under scripts/vendor/ to manage locked vendor sources, prepare lock-only promotions, and optionally monitor source-update PRs. Read Vendored Sources for commands, contributor metadata, and operator guidance before changing vendor state or publishing promotions.

Triggering CI

CI is triggered by posting comments on the PR. Basic commands:

  • /bot run — trigger the standard CI pipeline
  • /bot run --disable-fail-fast — run all stages even if earlier ones fail (only add when explicitly needed)

For a full list of up-to-date bot commands, post /bot help as a PR comment and check the bot's reply.

Trouble Shooting

  • Use TLLM_LOG_LEVEL_BY_MODULE to enable per-module log filtering (e.g., "debug:_torch,runtime;info:serve"); see Module-Level Logging for details.

Key Documentation

Topic Path
Coding guidelines CODING_GUIDELINES.md
Architecture overview docs/source/developer-guide/overview.md
PyTorch backend docs/source/torch/arch_overview.md
Adding a new model docs/source/torch/adding_new_model.md
ModelExpress (MX) checkpoint loading docs/source/features/model-express.md
Disaggregated serving docs/source/features/disagg-serving.md
Speculative decoding docs/source/features/speculative-decoding.md
Quantization docs/source/features/quantization.md
Parallelism strategies docs/source/features/parallel-strategy.md
KV cache docs/source/features/kvcache.md
API change guidelines docs/source/developer-guide/api-change.md
Feature compatibility matrix docs/source/features/feature-combination-matrix.md
Supported models docs/source/models/supported-models.md
Deployment guides docs/source/deployment-guide/
Examples & customization docs/source/examples/
Performance analysis docs/source/developer-guide/perf-analysis.md