Misc. bug: Unable to start llama-server as systemd service without specifying huggingface cache options

[ryant00000]

Name and Version

llama-cli --version
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 97252 MiB):
Device 0: NVIDIA RTX PRO 6000 Blackwell Server Edition, compute capability 12.0, VMM: yes, VRAM: 97252 MiB
version: 0 ()
built with GNU 15.2.1 for Linux x86_64

Not sure why version is broken, built from commit b8502.r5.3fc6f1aed1

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

No response

Command line

/usr/bin/llama-server --host 0.0.0.0 --port 10000 --models-dir /opt/models/ --models-max 1 --models-preset /opt/models/models.ini --jinja -to 1000 -np 8 --fit on --no-warmup -cram -1

Problem description & steps to reproduce

After a recent change, this now results in the following error upon startup:

main: failed to initialize router models: Failed to determine HF cache directory

Documentation mentioned a LLAMA_CACHE environment variable that was not relevant. Upon reviewing hf-cache.cpp, I set HF_HUB_CACHE which did make it start. I believe that this is related to user environment variables not being set since it's running as a service. If some new variable is somehow required, documentation should be updated, but I believe anyone using a service would not be using huggingface anyway and would not want this feature.

First Bad Commit

Last known working on b8495

Relevant log output

Logs

Mar 24 09:27:13 archdell systemd[1]: Started llama.cpp Server. Mar 24 09:27:13 archdell llama-server[2760178]: ggml_cuda_init: found 1 CUDA devices (Total VRAM: 97252 MiB): Mar 24 09:27:13 archdell llama-server[2760178]: Device 0: NVIDIA RTX PRO 6000 Blackwell Server Edition, compute capability 12.0, VMM: yes, VRAM: 97252 MiB Mar 24 09:27:13 archdell llama-server[2760178]: build: 0 () with GNU 15.2.1 for Linux x86_64 Mar 24 09:27:13 archdell llama-server[2760178]: system info: n_threads = 128, n_threads_batch = 128, total_threads = 256 Mar 24 09:27:13 archdell llama-server[2760178]: system_info: n_threads = 128 (n_threads_batch = 128) / 256 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 => Mar 24 09:27:13 archdell llama-server[2760178]: Running without SSL Mar 24 09:27:13 archdell llama-server[2760178]: init: api_keys: ****ikey Mar 24 09:27:13 archdell llama-server[2760178]: init: using 255 threads for HTTP server Mar 24 09:27:14 archdell llama-server[2760178]: main: failed to initialize router models: Failed to determine HF cache directory

[github-actions]

This issue might be similar or related to the following issue(s):

• Misc. bug: llama-server ignore LLAMA_CACHE env #18684: llama-server ignoring LLAMA_CACHE environment variable in router mode

This comment was auto-generated locally using llama.cpp version: 8456 (b31b30f) on NVIDIA DGX Spark

[bachp]

I encountered the same issue. I think it is caused by the change in #20775

[angt]

Hey, thanks for the issue!

I think this commit solves it: #21009 ?

[ryant00000]

Certainly makes the documentation more accurate, but I still think the server shouldn't fail to launch if there's no cache directory set. I rebuilt and confirmed setting LLAMA_CACHE does now fix the issue, but without LLAMA_CACHE or similar it still fails to launch

[angt]

I agree, how do you start it ?

$ stat /home/angt/.cache/huggingface /home/angt/.cache/llama.cpp
stat: cannot statx '/home/angt/.cache/huggingface': No such file or directory
stat: cannot statx '/home/angt/.cache/llama.cpp': No such file or directory

$ build/bin/llama-server
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 8537 (dc8d14c58) with GNU 14.2.0 for Linux x86_64
system info: n_threads = 16, n_threads_batch = 16, total_threads = 32

system_info: n_threads = 16 (n_threads_batch = 16) / 32 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | AMX_INT8 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

Running without SSL
init: using 31 threads for HTTP server
srv   load_models: Loaded 0 cached model presets
srv   load_models: Available models (0) (*: custom preset)
main: starting router server, no model will be loaded in this process
start: binding port with default address family
main: router server is listening on http://127.0.0.1:8080
main: NOTE: router mode is experimental
main:       it is not recommended to use this mode in untrusted environments

[ryant00000]

I use a systemd service to start it, which notably doesn't set environment variables that aren't explicitly set. So things like $HOME aren't set. For reference, it is below.

[Unit]
Description=llama.cpp Server
After=syslog.target network.target local-fs.target remote-fs.target nss-lookup.target

[Service]
Type=simple
EnvironmentFile=/etc/conf.d/llama.cpp
ExecStart=/usr/bin/llama-server $LLAMA_ARGS
ExecReload=/bin/kill -s HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target

from hf-cache.cpp -

            {"LLAMA_CACHE",           fs::path()},
            {"HF_HUB_CACHE",          fs::path()},
            {"HUGGINGFACE_HUB_CACHE", fs::path()},
            {"HF_HOME",               fs::path("hub")},
            {"XDG_CACHE_HOME",        fs::path("huggingface") / "hub"},
            {HOME_DIR,                fs::path(".cache") / "huggingface" / "hub"}

I suspect HOME_DIR and/or XDG_CACHE_HOME are set for you.

[angt]

Thanks 👍

[ryant00000]

Confirmed this fix worked, thanks!