Skip to content

ci: stop leaving the TT_LLM_ENGINE token on CI machines - #5561

Open
vcankovicTT wants to merge 1 commit into
mainfrom
vcankovic/ci-no-saved-llm-engine-token
Open

vcankovicTT wants to merge 1 commit into
mainfrom
vcankovic/ci-no-saved-llm-engine-token

Conversation

@vcankovicTT

@vcankovicTT vcankovicTT commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Where the token is exposed today (example)

Take the job C++ Build & Unit Tests in test-gate.yml, which runs on pull requests that touch C++ code:

Step What happens to the TT_LLM_ENGINE token
1. Checkout repository Uses token: ${{ secrets.TT_LLM_ENGINE }}. Checkout saves the token in the repo's git settings file, .git/config, and keeps it there for the rest of the job.
2. Set up C++ CI dependencies → "Configure git for private submodules" Runs git config --local … and saves the token in .git/config a second time. Nothing ever removes this copy.
2. Set up C++ CI dependencies → "Init tt-llm-engine submodule" The only step that actually needs the token.
2. Set up C++ CI dependencies → "Cache Cargo registry", "Restore ccache", "Install C++ build dependencies" Run with the token sitting on disk. install_dependencies.sh downloads and builds third-party code (for example Drogon).
3–5. Restore cached tokenizer files, Build C++ server, Run C++ unit tests Run with the token sitting on disk. The build also downloads third-party libraries through CMake FetchContent.

After steps 1 and 2, the job's .git/config contains this (real token replaced with <TOKEN>):

[http "https://github.com/"]
	extraheader = AUTHORIZATION: basic <base64 of "x-access-token:<TOKEN>">
[url "https://x-access-token:<TOKEN>@github.com/"]
	insteadOf = https://github.com/

Anything that runs later in the job, including third-party code, can read the token with a simple cat .git/config.

The same happens in 9 more jobs:

  • test-gate.yml: C++ Lint & Format Check, RemoteKVManager Kafka E2E
  • cpp-heavy-checks.yml: C++ Build (artifact for benches), C++ Server Mock Pipeline Benchmark, C++ Server Prefill/Decode Split Test, C++ Server Gateway Split Test, C++ Server Prefill/Decode Smoke Tests, C++ ThreadSanitizer
  • nightly-cpp-benchmarks.yml: Full Benchmark Suite (only the checkout copy)

Four of the cpp-heavy-checks jobs run on tt-ubuntu-2204-large-stable runners. If such a runner is reused between jobs, the copy saved by git config --local stays on its disk after the job ends.

Why this is risky

  • Every later step in the job can read it, including third-party actions and the third-party code the build downloads. If any of them is ever compromised, it can copy the token.
  • On a runner that is reused, the git config --local copy stays on disk after the job.
  • If the work folder is ever copied somewhere (an artifact, a cache, a Docker build context), the token goes with it. Inside a published Docker image it is very hard to take back.

Anyone who gets this token can read the private tt-llm-engine repository. A token should be handed only to the command that needs it, and only while that command runs.

What this PR changes

  • .github/actions/setup-cpp-ci/action.yml: removes the "Configure git for private submodules" step, the one that saved the token. The next step already gives the token to the one git command that needs it (git -c … submodule update), and a -c value is never written to disk. The token now comes in through env: instead of being pasted into the script text.
  • test-gate.yml, cpp-heavy-checks.yml, nightly-cpp-benchmarks.yml: add persist-credentials: false to the 10 checkout steps that use the token, so checkout doesn't keep it either.
  • nightly-cpp-benchmarks.yml: its "Init tt-llm-engine submodule" step relied on the token saved by checkout. It now gets the token the same way as setup-cpp-ci.

After this PR, the token is used only inside the single "Init tt-llm-engine submodule" command and is never saved to .git/config.

Why nothing should break

  • The removed step wasn't doing any work. Git does not use a --local setting when it downloads a submodule; only the -c on the next line does. I checked this with throwaway local repos: with only -c, the submodule download works and nothing is saved; with only the saved --local setting, the download fails and the value is left in .git/config.
  • CI shows the same thing. cpp-heavy-checks, which uses -c, passed its submodule setup on 2026-11-07. nightly-cpp-benchmarks, which relies on the token saved by checkout, failed at its "Init tt-llm-engine submodule" step on 2026-11-07 and 2026-11-08. This PR should fix that step.
  • Nothing else needs the saved token. I found no other step in these jobs that runs git against a private repo.

Testing

  • actionlint: no new findings in the changed workflows.
  • test-gate runs on this PR, and its C++ jobs use setup-cpp-ci. Right now every workflow in this repo ends in startup_failure on all branches (unrelated to this PR), so CI will confirm this change once that is fixed.
  • cpp-heavy-checks and nightly-cpp-benchmarks don't run on PRs; they can be checked with a manual run on this branch.

The C++ jobs need the TT_LLM_ENGINE token to download the private
tt-llm-engine submodule. Today the token is also saved to .git/config and
left there:

- setup-cpp-ci wrote it with `git config --local` and nothing removed it.
- The actions/checkout steps that use it keep it in .git/config for the
  rest of the job (persist-credentials defaults to true).

Saved there, every later step in the job can read it, and on a reused
runner it stays on disk after the job.

Changes:
- setup-cpp-ci: remove the "Configure git for private submodules" step.
  The next step already gives the token to the one git command that needs
  it (`git -c ... submodule update`). Git does not use the saved --local
  setting for the submodule download anyway, so nothing else changes. The
  token now comes in through env: instead of being pasted into the script.
- test-gate, cpp-heavy-checks, nightly-cpp-benchmarks: set
  `persist-credentials: false` on the checkouts that use the token.
- nightly-cpp-benchmarks: its submodule step relied on the token saved by
  checkout and has been failing at that step, so it now gets the token the
  same way as setup-cpp-ci.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vcankovicTT
vcankovicTT marked this pull request as ready for review November 10, 2026 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants