Token generation broken on CUDA when offload_kqv is false

@iamlemec

Originally spotted by @iamlemec in abetlen/llama-cpp-python#1089 reproduced with llama.cpp by passing --no_kv_offload to ./main. Bug causes the model to generate repeated #'s instead of a valid completion.

@slaren

I can reproduce the bug. It seems to have been introduced in #4766 and only occurs from the second eval onwards. So ./perplexity with defaults will not detect this bug but ./perplexity -b 256 will.

cc: @slaren

Duplicate of #4983?

Yup, will be looking into this - if anyone has additional insights like which models / params work and which do not work would be helpful

@ggerganov

@ggerganov I tried with below model and params, and it ran into the generation error.

command:
            - python3
            - '-m'
            - llama_cpp.server
          args:
            - '--model'
            - openbuddy-mixtral-7bx8-v16.3-32k.Q5_K_M.gguf
            - '--chat_format'
            - openbuddy
            - '--tensor_split'
            - '0.5'
            - '0.5'
            - '--n_gpu_layers'
            - '-1'
            - '--n_threads'
            - '16'
            - '--interrupt_requests'
            - 'false'
            - '--n_ctx'
            - '32500'

Hopefully some useful info: This occurs on every type of model I've tried: llama, mistral, mixtral, even llava, regardless of quantization.

When running a modified simple program with offload_kqv = false, very short prompts work. At some point you hit a length where the logit value becomes different from the offload_kqv = true case. For any prompt length strictly higher than this cutoff the logit value is simply nan. The cutoff value varies from model to model, for llama-7b and llama-13b it's 16 and for mistral it's 4.

Looking at different values for n_gpu_layers with longer prompts and offload_kqv = false, if it's fully offloaded you get nan. Anything less than full offload you get 0.0 for the most part, unless ngl=1 when you get a non-zero but incorrect number.

This should have been fixed in #5049 (already merged), please let me know if you find any other issues.

Forgot to close this, it works, thanks for the fix!

Navigation Menu

Repository navigation

Token generation broken on CUDA when `offload_kqv` is `false` #4991

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Token generation broken on CUDA when offload_kqv is false #4991

Description

Activity

JohannesGaessler commented on Jan 17, 2024

Artefact2 commented on Jan 17, 2024

ggerganov commented on Jan 17, 2024

thiner commented on Jan 19, 2024

iamlemec commented on Jan 19, 2024

slaren commented on Jan 20, 2024

abetlen commented on Jan 25, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Token generation broken on CUDA when `offload_kqv` is `false` #4991