train-from-scratch broken when compiled with cuda

[Tostino]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

When llama.cpp is built with cuda support, train from scratch should work the same as when it is built without cuda support.

Current Behavior

I get a core dump when trying to run the test training script when compiled with cuda support. It works fine when compiled without cuda.

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

• Physical (or virtual) hardware you are using, e.g. for Linux:

$ lscpu

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  32
  On-line CPU(s) list:   0-31
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 9 7950X 16-Core Processor

• Operating System, e.g. for Linux:

$ uname -a
Linux pop-os 6.2.6-76060206-generic #202303130630~1685473338~22.04~995127e SMP PREEMPT_DYNAMIC Tue M x86_64 x86_64 x86_64 GNU/Linux

• SDK version, e.g. for Linux:

$ python3 --version
Python 3.10.6

$ make --version
GNU Make 4.3
Built for x86_64-pc-linux-gnu
$ g++ --version
g++ (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. Compile with cuda support
2. Run the example in the train-text-from-scratch example README

Failure Logs

$ ./train-text-from-scratch --vocab-model ./models/ggml-vocab.bin --ctx 64 --embd 256 --head 8 --layer 16 -checkpoint-in  chk-shakespeare-256x16.bin --checkpoint-out chk-shakespeare-256x16.bin --model-out ggml-shakespeare-256x16-f32.bin         --train-data "shakespeare.txt" -t 6 -b 16 -n 32 --seed 1 --adam-iter 16 --print-details-interval 0 --predict 16 --use-flash
main: seed: 1
llama.cpp: loading model from ./models/ggml-vocab.bin
llama_model_load_internal: format     = ggjt v1 (pre #1405)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 1 (mostly F16)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
main: tokenize training data
main: number of training tokens: 27584
print_params: n_vocab: 32000
print_params: n_ctx:   64
print_params: n_embd:  256
print_params: n_mult:  256
print_params: n_head:  8
print_params: n_ff:    768
print_params: n_layer: 16
print_params: n_rot:   32
main: number of unique tokens: 3070
ggml_init_cublas: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090
  Device 1: NVIDIA GeForce RTX 3090
main: init model
load_checkpoint: Training iterations: 0.
load_checkpoint: Training samples:    0.
load_checkpoint: Training tokens:     0.
main: opt iter 0
used_mem model+cache: 242364416 bytes
main: begin training
Segmentation fault (core dumped)

[abc-nix]

Hi, Tostino.

See if pull request #1952 helps solve the issue.

[howard0su]

verified it works with latest.