TensorRT-LLM: open-source library for optimized LLM inference on NVIDIA GPUs. Python and C++ codebase with a PyTorch execution path.
If a
CLAUDE.local.mdfile exists alongside this file, read and respect it — it contains developer-specific overrides that supplement this shared guidance.
CRITICAL (YOU MUST):
- Read and follow
CODING_GUIDELINES.mdfor ALL code changes (C++ and Python) - NVIDIA copyright header on ALL new files (update year on modified files)
git commit -s(DCO sign-off required). Never attribute AI tools in sign-off line. Always rely ongitto do the sign off instead of directly adding sign off in commit message.- Do not add co-authors to the git commit message unless explicitly instructed to do so by the user.
pre-commithooks run on commit — if files are modified by hooks, re-stage and commit again- LLM args or nested-config changes must run
python3 scripts/generate_llm_args_golden_manifest.pyand committensorrt_llm/usage/llm_args_golden_manifest.json; new fields require telemetry/privacy CODEOWNER approval - Changes to serving YAML fields, aliases, defaults, types, or constraints (including nested,
disaggregated, and VisualGen configs) must regenerate and inspect the editor JSON Schemas with
python3 scripts/generate_trtllm_serve_schemas.py, using a TensorRT-LLM build matching the checkout. Update the generator's YAML-specific adaptations when configuration annotations alone do not capture the change, and add regression coverage for the affected accepted/rejected values. Runpytest -m cpu_only tests/unittest/llmapi/test_trtllm_serve_config_schemas.py. Commit the generatedtensorrt_llm/schemas/*.jsonsnapshots; never hand-edit them. The snapshots also include the package version in their$idandtitle;python3 scripts/bump_version.py <version>updates that metadata withpython3 scripts/generate_trtllm_serve_schemas.py --sync-version, which does not import TensorRT-LLM or require a matching build.python3 scripts/generate_trtllm_serve_schemas.py --checkand the CPU schema tests reject missing or stale snapshots. Wheels and HTML docs publish these same files. - Change the package version only with
python3 scripts/bump_version.py <version>, which updates every file that records it. When a file starts recording the package version (for example atensorrt_llm==<version>pin), add it toVERSION_PATTERNSin that script in the same change. Theversion-records-checkpre-commit hook (bump_version.py --check) rejects records that do not matchtensorrt_llm/version.py. - When adding or renaming a public model architecture identifier, update
tensorrt_llm/usage/architecture_allowlist.pywith its exact identifier. It must be publicly documented by the upstream model provider or indocs/source/models/supported-models.md; never add private, customer-specific, or arbitrary user-supplied names - PR title format:
[JIRA/NVBUG/None][type] description(e.g.,[TRTLLM-5516][perf] optimize cuda graph padding) - Put a change's rationale and history (why it was made, what it replaces, ticket/PR references) in the PR description, not in code comments. Code comments should explain the code as it stands for a future reader, not narrate how it got there.
- Set
LLM_MODELS_ROOTenv var when running tests that need model weights
| Task | Command |
|---|---|
| Unit tests | pytest tests/unittest/ |
| C++ component tests | cmake --build cpp/build_RelWithDebInfo --target common-tests (also runtime-tests, executor-tests, batch-manager-tests, kernel-tests) |
| C++ test scheduling | ctest --test-dir cpp/build_RelWithDebInfo -L common --output-on-failure (build the selected tests first) |
| Specific test | pytest tests/unittest/llmapi/test_llm_args.py |
| Pattern match | pytest tests/unittest -k "test_llm_args" |
| Integration tests | LLM_MODELS_ROOT=/path/to/models pytest tests/integration/defs/... |
| Serve model | trtllm-serve <hf_model> --port 8000 |
| Serve with config | trtllm-serve <hf_model> --config config.yaml |
| Benchmark | trtllm-bench --model <hf_model> throughput --dataset <path> |
| Find CI stage for test | python scripts/test_to_stage_mapping.py --tests "test_name" |
Building TensorRT-LLM requires Docker and may involve compiling C++ components. See the Installation Guide for pre-built release containers and pip install, build from source for development builds, and Container Images for information about the container images.
examples/configs/database/ contains pareto-optimized serving configurations
across multiple models, GPUs, ISL/OSL combinations, and concurrency levels.
Use these as starting points for deployment and benchmarking rather than hand-tuning parameters.
See deployment guides for model-specific walkthroughs.
See architecture diagram for the full Mermaid diagram.
| Backend | Status | Entry Point | Key Path |
|---|---|---|---|
| PyTorch | Default | TorchLlmArgs |
_torch/pyexecutor/ → PyExecutor → PyTorch Engine |
Both backends share these C++ components:
- Scheduling pipeline: Scheduler → BatchManager (in-flight batching) → KV Cache Manager
- Decoding pipeline: Decoder (token generation orchestration) → Sampling
HuggingFace Model → LLM API → PyTorch Executor
→ Scheduler → Model Forward → Decoder → Sampling → Generated Tokens
trtllm-serve: OpenAI-compatible REST + gRPC server, supports all backends- Disaggregated serving: separates prefill (context) and decode (generation) across GPUs
- KV cache exchange via NIXL (default), UCX, or MPI
Read the multimodal developer guide before changing multimodal paths.
Reuse MultimodalParams; request Params own cache references.
Encoder and LLM forward Params are execution views.
| File | Role |
|---|---|
tensorrt_llm/llmapi/llm.py |
Main API entry point |
tensorrt_llm/llmapi/llm_args.py |
Complete configuration schema (Pydantic) |
tensorrt_llm/llmapi/llm_utils.py |
Model loading, model-specific default overrides |
tensorrt_llm/models/modeling_utils.py |
Base classes for all models (PretrainedConfig, PretrainedModel) |
tensorrt_llm/executor/executor.py |
Execution abstraction (GenerationExecutor) |
tensorrt_llm/models/automodel.py |
Auto-discovery and model registry |
tensorrt_llm/_torch/models/ |
PyTorch backend model implementations (distinct from the top-level models/ package) |
tensorrt_llm/_torch/attention/ATTENTION_DEVELOPER_GUIDE.md |
Attention, MLA, backend families, sparse backends, metadata contracts, and KV-cache behavior - read before modifying anything under tensorrt_llm/_torch/attention/ |
tensorrt_llm/_torch/moe/fused_moe/MOE_DEVELOPER_GUIDE.md |
MoE architecture, backends, communication, development patterns — read before modifying MoE code |
tensorrt_llm/_torch/pyexecutor/kv_cache/REGIONS_DEVELOPER_GUIDE.md |
KV cache regions: persistent state declared with a life cycle of its own and laid out as native regions — read before adding model- or backend-specific state to the KV cache managers |
CODING_GUIDELINES.md |
C++ and Python coding standards (referenced throughout, must read before contributing) |
| Pattern | Key Points |
|---|---|
| Config hierarchy | BaseLlmArgs → TorchLlmArgs, model-specific defaults override generics, Pydantic validation |
| Model architecture | Each model: Config (inherits PretrainedConfig) + ForCausalLM (inherits PretrainedModel) |
| Model defaults | Architecture-specific overrides in llm_utils.py (attention kernels, quant, spec decoding, cache) |
| Attention backends | TorchLlmArgs.attn_backend selects kernel: TRTLLM (default), FlashInfer, FlashAttention |
| Distributed execution | Tensor/pipeline parallelism via Mapping class, multiple backends (MPI, Ray, RPC) |
| Auto-discovery | Models self-register via automodel.py, resolved by HF config architectures field |
VisualGen is a vertical alongside LLM for Diffusion-Transformer (DiT)-based image/video generation (text-to-image, text-to-video, image-to-video). It is not an LLM backend — it has its own engine, args, params, and outputs — but shares ops and kernels with the PyTorch backend where it makes sense (attention, quantization, parallelism).
Key entry points:
- Public Python API:
from tensorrt_llm import VisualGen, VisualGenArgs, VisualGenParams. - Serving CLI:
trtllm-serve <HF id> --visual_gen_args <YAML path>.
Key files:
tensorrt_llm/_torch/visual_gen/ENGINEERING_CRITERIA.md: Engineering criteria for any change undertensorrt_llm/visual_gen/ortensorrt_llm/_torch/visual_gen/— API discipline, feature/test/lossy-vs-lossless requirements, examples & docs rules. Read before modifying anything in those trees.tensorrt_llm/visual_gen/: VisualGen public Python API. User-facing surface — before modifying anything here, pause and confirm with the user that a public API change is actually intended; do not infer it from the surrounding task.tensorrt_llm/_torch/visual_gen/: VisualGen internal implementation. All non-user-facing code belongs here.
- Pre-commit modifies files in-place — if hooks fail, files are already modified. Re-stage (
git add) and commit again. - Protected APIs exist — changes to LLM API signatures will fail
tests/unittest/api_stabilitytests. Get code owner review. - Integration tests need GPUs + models — always set
LLM_MODELS_ROOTand ensure GPU access. Unit tests don't. - Copyright year — update to current year when modifying existing files; add full header to new files.
- Avoid broad exception handling — catch specific exceptions, not bare
except:(seeCODING_GUIDELINES.md). - One concern per PR — avoid scope creep. If a PR touches unrelated areas, split it.
- User-facing configuration classes - when editing or defining any user-facing configuration classes (particularly
BaseLlmArgsor any class used in its fields), you MUST follow the Pydantic guidelines inCODING_GUIDELINES.md.
- Set up build environment (see installation docs)
- Make changes following
CODING_GUIDELINES.md - Test locally with
pytest
- The main repository (
upstream) is located at https://github.com/NVIDIA/TensorRT-LLM/ - Branches should always be pushed to the user-specified fork (usually
origin) - If pushing fails to due pre-push pre-commits hooks getting updated, just re-push immediately
- PRs should be opened on the main repository
- Target
mainunless fixing a release branch bug - See
CONTRIBUTING.mdfor full PR policies
- Target
The gh CLI uses ~/.config/gh by default for authentication. Different GitHub hosts or forks may require a different config directory. Before running any gh command (e.g., gh pr create, gh api, gh pr comment):
- Check if the user has specified a custom
GH_CONFIG_DIR(e.g., inCLAUDE.local.mdor environment). If so, use it. - If not explicitly set, default to
~/.config/gh; do not ask for confirmation. - Prefix all
ghcommands with the resolved config dir:GH_CONFIG_DIR=<path> gh ... - If the command fails due to missing authentication or the wrong GitHub host/account, report the failure and ask for the correct
GH_CONFIG_DIR.
See CI overview for full details.
| Layer | Location | Notes |
|---|---|---|
| Unit tests | tests/unittest/ |
Run in pre-merge CI; some tests require GPU |
| API stability | tests/unittest/api_stability/ |
Protects committed API signatures |
| Integration tests | tests/integration/defs/ |
Requires GPU + LLM_MODELS_ROOT |
| Test lists | tests/integration/test_lists/test-db/ |
Per-GPU YAML files (l0_a10.yml, l0_h100.yml, etc.) |
| Test waives | tests/integration/test_lists/waives.txt |
Skip known-failing tests with NVBug links |
| Performance | See benchmarking guide | trtllm-bench and trtllm-serve benchmarks |
Pass product or test Python file paths to scripts/run_pyrefly.sh for a file-list static
check; pre-commit supplies these paths with pass_filenames: true. If none are
eligible, the script skips the check. --full-project requests a scan of all
configured Python files under tensorrt_llm/ and tests/. For supplied files,
diagnostics are filtered to added and modified lines when Git provides them and
retained at file level otherwise.
File-list checks print every remaining diagnostic; full-project checks save the
report and print its absolute path.
Jenkins runs a full-project check with path/module and symbol exclusions.
Whole-project checks always return success; file-list checks can opt into
--strict. Jenkins and pre-commit checks are advisory. The [tool.pyrefly]
section in pyproject.toml defines product roots, diagnostics, and native exclusions.
[tool.pyrefly-tests] in the same file defines test roots and helper search paths.
The wrapper generates a temporary test configuration
that inherits shared rules and exclusions, runs each scope separately, and
combines the diagnostics into one report. Temporary configurations are removed
after the check. Mixed file
lists are selected from each configuration's project-includes; full-project mode
runs both passes.
[tool.pyrefly-exclusions] defines additional symbol exclusions for the wrapper.
Pyrefly is pinned in requirements-dev.txt. Missing imports are reported except for
PyTorch, generated bindings, and configured third-party libraries. In test file-list
mode, missing imports of declared dependencies that are not installed are also
filtered; full-project scans retain them. Release-Check installs pytest so that
pytest.skip() remains recognizable as a terminating call.
Tests use the same diagnostics as product code, including missing-argument,
unexpected-keyword, and bad-argument-count. The wrapper reads literal
__extra_import_path__ lists or tuples under tests/ without running test code and adds
existing directories to the shared search path for all test files. Ancestors of
the declaring file are skipped because fallback searches them per file. Product
checks use only the project root, with fallback disabled. The broader helper search paths
and fallback resolution apply only to the test pass. --no-path-module-filter
disables exclusions in both passes while preserving their separate search paths
and shared diagnostics.
New C++ unit tests should use add_tllm_gtest with explicit SOURCES,
LINK_LIBRARIES, OWNER, and scheduling LABELS. Request FULL_STACK explicitly
for integration tests and include gpu in LABELS for tests that need a GPU.
CI controls test concurrency through parallel_run_ctest. google-tests
continues to build all C++ tests; component aggregate targets build migrated tests.
C++ integration fixtures build the collected tests with incremental native targets.
cpp-tests-<directory> targets (for example cpp-tests-common or
cpp-tests-executor-bounce) mirror recursive CTest directory coverage, including
legacy and full-stack executables. OWNER aggregates are component subsets and
are not interchangeable with these directory targets.
The Linux Release source-build fixture reuses the installed CI library for the
RNN formatter test. CI source and wheel use the same commit and image per
architecture. Link/load errors fail the tests. Direct CMake builds use source
libraries unless explicitly configured otherwise.
CONFIG_LINUX_X86_64_VANILLA also builds google-tests
and publishes the cpp_test_bundle CMake component as a separate
cpp-tests-<main-tar-name> artifact. The main package does not include the test
programs. Only -CPP- stages using this build configuration fetch and unpack
the companion into src/cpp/build, including fat-image consumers. Build reuse
requires both artifacts. Linux Release integration fixtures use this bundle
directly, retaining CTest selection and retries. The two FMHA tests share a
test-only library packaged next to their executables; other runtime libraries
come from the matching installed wheel. Test artifacts keep symbols but omit
BOLT's extra relocation records.
CPP stages collect only tests/integration/defs/cpp and use the standard test
environment setup. CTest discovers cases on the test node, so the build stage
does not need a GPU. Without a bundle, or with
an explicit TRTLLM_BUILD_ROOT, fixtures retain the incremental build path.
Use the tools under scripts/vendor/ to manage locked vendor sources, prepare
lock-only promotions, and optionally monitor source-update PRs. Read
Vendored Sources for commands, contributor metadata,
and operator guidance before changing vendor state or publishing promotions.
CI is triggered by posting comments on the PR. Basic commands:
/bot run— trigger the standard CI pipeline/bot run --disable-fail-fast— run all stages even if earlier ones fail (only add when explicitly needed)
For a full list of up-to-date bot commands, post /bot help as a PR comment and check the bot's reply.
- Use
TLLM_LOG_LEVEL_BY_MODULEto enable per-module log filtering (e.g.,"debug:_torch,runtime;info:serve"); see Module-Level Logging for details.
| Topic | Path |
|---|---|
| Coding guidelines | CODING_GUIDELINES.md |
| Architecture overview | docs/source/developer-guide/overview.md |
| PyTorch backend | docs/source/torch/arch_overview.md |
| Adding a new model | docs/source/torch/adding_new_model.md |
| ModelExpress (MX) checkpoint loading | docs/source/features/model-express.md |
| Disaggregated serving | docs/source/features/disagg-serving.md |
| Speculative decoding | docs/source/features/speculative-decoding.md |
| Quantization | docs/source/features/quantization.md |
| Parallelism strategies | docs/source/features/parallel-strategy.md |
| KV cache | docs/source/features/kvcache.md |
| API change guidelines | docs/source/developer-guide/api-change.md |
| Feature compatibility matrix | docs/source/features/feature-combination-matrix.md |
| Supported models | docs/source/models/supported-models.md |
| Deployment guides | docs/source/deployment-guide/ |
| Examples & customization | docs/source/examples/ |
| Performance analysis | docs/source/developer-guide/perf-analysis.md |