Linux (Fedora) containers uses CPU instead of GPU (Intel Arc A770 16Gb)

[Clm-1138]

Hello,

I am looking for assistance because I don't know what I am doing wrong.

I am using the following system

OS: Fedora Linux 42 (Server Edition) x86_64
Kernel: Linux 6.15.7-200.fc42.x86_64
CPU: Intel(R) Core(TM) i5-8400T (6) @ 3.30 GHz
GPU: Intel Arc A770 @ 2.40 GHz [Discrete]
Memory: 946.14 MiB / 15.56 GiB

• Intel OneAPI is installed
• drivers looks okay (I don't know how to find newer ones for Fedora 42, if any)

I am running Local-Ai through docker with the following docker file :

services:
  api:
    image: localai/localai:v3.3.0-gpu-intel-f32
    container_name: local-ai-intel-fp32
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      interval: 1m
      timeout: 20m
      retries: 5
    build:
      context: .
      dockerfile: Dockerfile
      args:
      - IMAGE_TYPE=core
      - BASE_IMAGE=ubuntu:22.04
    ports:
      - 8080:8080
    devices:
     - /dev/dri/card0
     - /dev/dri/renderD128
    env_file:
      - .env
    environment:
      - MODELS_PATH=/models
      - DEBUG=true
      - SYCL_DEVICE_FILTER=GPU
      - SYCL_BACKEND=level_zero
    volumes:
      - ./models:/models:cached
      - ./images/:/tmp/generated/images/
    command:
    - qwen3-8b
    - mistral-7b-instruct-v0.3

My GPU seems to be accessible from the container :

root@00932e8ac798:/# sycl-ls
[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Arc(TM) A770 Graphics 12.55.8 [1.6.33276+22]
[opencl:cpu][opencl:0] Intel(R) OpenCL, Intel(R) Core(TM) i5-8400T CPU @ 1.70GHz OpenCL 3.0 (Build 0) [2025.20.6.0.04_224945]
[opencl:gpu][opencl:1] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) A770 Graphics OpenCL 3.0 NEO  [25.13.33276]
[opencl:cpu][opencl:2] Intel(R) OpenCL, Intel(R) Core(TM) i5-8400T CPU @ 1.70GHz OpenCL 3.0 (Build 0) [2025.20.6.0.04_224945]

I tried the following images :

• localai/localai:v3.3.0-gpu-intel-f32
• localai/localai:master-gpu-intel-f16
• localai/localai:master-gpu-vulkan

And yet, every time, the cpu is doing all the work.
Here is something the logs keeps telling :

api-1  | 10:14AM DBG Installing backend "llama-cpp"
api-1  | 10:14AM DBG GPU vendor gpuVendor=
api-1  | 10:14AM DBG GPU vendor gpuVendor=
api-1  | 10:14AM DBG guessDefaultsFromFile: NGPULayers set NGPULayers=99999999
api-1  | 10:14AM DBG guessDefaultsFromFile: template already set name=mistral-7b-instruct-v0.3
api-1  | 10:14AM DBG guessDefaultsFromFile: NGPULayers set NGPULayers=99999999
api-1  | 10:14AM DBG guessDefaultsFromFile: template already set name=qwen3-8b
api-1  | 10:14AM DBG Registering backend name=cpu-llama-cpp runFile=/backends/cpu-llama-cpp/run.sh
api-1  | 10:14AM DBG Registering backend name=llama-cpp runFile=/backends/cpu-llama-cpp/run.sh

I really need some help here because I must be doing something wrong, but I do not know what.

[Clm-1138]

I also tried the docker run route, but without sucess :

clement@localhost:~$ sudo docker run -e DEBUG=true --privileged -ti -v $PWD/models:/models --gpus all -p 808
0:8080  -v /dev/dri:/dev/dri --rm localai/localai:master-gpu-intel-f32 qwen3-8b
Unable to find image 'localai/localai:master-gpu-intel-f32' locally
master-gpu-intel-f32: Pulling from localai/localai
Digest: sha256:91497b443fd27f0b3d07fa9ab9917b362cb44d9d6714c7faf49746add46e5666
Status: Downloaded newer image for localai/localai:master-gpu-intel-f32
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

Yet this one looks like it's telling me something I don't understand...

[emma12669]

Hello

[mudler]

Hello,

I am looking for assistance because I don't know what I am doing wrong.

I am using the following system

OS: Fedora Linux 42 (Server Edition) x86_64
Kernel: Linux 6.15.7-200.fc42.x86_64
CPU: Intel(R) Core(TM) i5-8400T (6) @ 3.30 GHz
GPU: Intel Arc A770 @ 2.40 GHz [Discrete]
Memory: 946.14 MiB / 15.56 GiB

* Intel OneAPI is installed

* drivers looks okay (I don't know how to find newer ones for Fedora 42, if any)

I am running Local-Ai through docker with the following docker file :

services:
  api:
    image: localai/localai:v3.3.0-gpu-intel-f32
    container_name: local-ai-intel-fp32
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      interval: 1m
      timeout: 20m
      retries: 5
    build:
      context: .
      dockerfile: Dockerfile
      args:
      - IMAGE_TYPE=core
      - BASE_IMAGE=ubuntu:22.04
    ports:
      - 8080:8080
    devices:
     - /dev/dri/card0
     - /dev/dri/renderD128
    env_file:
      - .env
    environment:
      - MODELS_PATH=/models
      - DEBUG=true
      - SYCL_DEVICE_FILTER=GPU
      - SYCL_BACKEND=level_zero
    volumes:
      - ./models:/models:cached
      - ./images/:/tmp/generated/images/
    command:
    - qwen3-8b
    - mistral-7b-instruct-v0.3

My GPU seems to be accessible from the container :

root@00932e8ac798:/# sycl-ls
[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Arc(TM) A770 Graphics 12.55.8 [1.6.33276+22]
[opencl:cpu][opencl:0] Intel(R) OpenCL, Intel(R) Core(TM) i5-8400T CPU @ 1.70GHz OpenCL 3.0 (Build 0) [2025.20.6.0.04_224945]
[opencl:gpu][opencl:1] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) A770 Graphics OpenCL 3.0 NEO  [25.13.33276]
[opencl:cpu][opencl:2] Intel(R) OpenCL, Intel(R) Core(TM) i5-8400T CPU @ 1.70GHz OpenCL 3.0 (Build 0) [2025.20.6.0.04_224945]

I tried the following images :

* localai/localai:v3.3.0-gpu-intel-f32

* localai/localai:master-gpu-intel-f16

* localai/localai:master-gpu-vulkan

And yet, every time, the cpu is doing all the work. Here is something the logs keeps telling :

api-1  | 10:14AM DBG Installing backend "llama-cpp"
api-1  | 10:14AM DBG GPU vendor gpuVendor=
api-1  | 10:14AM DBG GPU vendor gpuVendor=
api-1  | 10:14AM DBG guessDefaultsFromFile: NGPULayers set NGPULayers=99999999
api-1  | 10:14AM DBG guessDefaultsFromFile: template already set name=mistral-7b-instruct-v0.3
api-1  | 10:14AM DBG guessDefaultsFromFile: NGPULayers set NGPULayers=99999999
api-1  | 10:14AM DBG guessDefaultsFromFile: template already set name=qwen3-8b
api-1  | 10:14AM DBG Registering backend name=cpu-llama-cpp runFile=/backends/cpu-llama-cpp/run.sh
api-1  | 10:14AM DBG Registering backend name=llama-cpp runFile=/backends/cpu-llama-cpp/run.sh

I really need some help here because I must be doing something wrong, but I do not know what.

The docker-compose file is correct. There is currently a bug in LocalAI as such the intel gpu is not identified correctly. This is being fixed in #5945, but meanwhile, as a temporary workaround you can set the LOCALAI_FORCE_META_BACKEND_CAPABILITY variable manually to intel to force LocalAI to pull the intel backends.

This is documented in https://localai.io/features/gpu-acceleration/#automatic-backend-detection .

[Clm-1138]

Thanks a lot, I'm gonna try this out as soon as possible and will keep you posted.

[Clm-1138]

Okay, update, last time, CPU was doing the heavy lifting, this time, it offloads to the GPU... which does noting.
When using intel_gpu_top there is a very short burst of activity before nothing more happens.

GPU is working. Both Unigine Heaven and FurMark are using it properly. Performance is not stellar, but it works.
I am using pcie gen 3 and have no acess to resizeableBAR if it is of some relevance...

[kulak]

I can confirm that LocalAI uses nVidia GPU if configured correctly.

My setup:

OS: Fedora 43
GPU: Tesla P40
Container System: podman
Image: docker.io/localai/localai:latest-aio-gpu-nvidia-cuda-11

Notice 1: My old video card requires Cuda 11. While nvidia-smi reports CUDA 13 installed, CUDA-11 has actual support for the card. That part is very confusing, but it has nothing to do with LocalAI project.

Notice 2: One has to give up trying to run LocalAI on Fedora 43 directly. It is only possible for systems that statically compile CUDA support into the executable. LocalAI does not do that. It is nearly impossible to configure Fedora 43 with Cuda 11, because Cuda 11 nvcc requres older gcc, which can only be set globally. So, it is container route for Fedora 36+ users. Cuda 11 was supported on Fedora 35.