Skip to content

Latest commit

 

History

History
119 lines (72 loc) · 4.18 KB

README.md

File metadata and controls

119 lines (72 loc) · 4.18 KB

CLI Chat application

Chat application for Windows on Snapdragon® demonstrating a large language model (LLM, e.g., Llama 3.2 3B) using Genie SDK.

The app demonstrates how to use the Genie APIs from QAIRT SDK to run and accelerate LLMs using the Snapdragon® Neural Processing Unit (NPU).

Requirements

Platform

  • Snapdragon® Platform (e.g. X Elite)
  • Windows 11+

Tools and SDK

  • Visual Studio 22
    • Download any variant of Visual Studio here
    • Make sure Desktop development with C++ tools are selected during installation or installed separately later
  • QAIRT SDK: Qualcomm AI Runtime SDK (see QNN SDK for older versions)
    • Refer to Setup QAIRT SDK to install compatible QAIRT SDK for models downloaded from AI Hub.

Build App

Compile to Context Binary via AI Hub and Generate Genie Bundle

  1. Clone this repository so that you have a local copy of ChatApp.

  2. Please follow this tutorial to generate genie_bundle required by ChatApp. If you use any of the Llama 3 models, the app will work without modifications. If you use another model, you will need to update the prompt format in PromptHandler.cpp first.

  3. Copy bundle assets from step 2 to ChatApp\genie_bundle. You should see ChatApp\genie_bundle\*.bin context binary files.

Setup QAIRT SDK

Please ensure that the QAIRT (or QNN) SDK version installed on the system is the same as the one used by AI Hub for generating context binaries. You can find the AI Hub QAIRT version in the compile job page as shown in the following screenshot:

QAIRT version on AI Hub

Having different QAIRT versions could result in runtime or load-time failures.

Please follow the following steps to configure QAIRT SDKs for ChatApp:

  1. Download and install Qualcomm AI Runtime SDK (see QNN SDK for older versions)

  2. Set global environment variable QNN_SDK_ROOT to root path of QAIRT SDK e.g. C:\Qualcomm\AIStack\QAIRT\2.32.0.250228

    • Make sure you can run the following command with no error and that it prints the various libraries available with your QAIRT package:
    ls ${env:QNN_SDK_ROOT}/lib

    QNN SDK Verion check

  3. If command from step 2 succeeds, QAIRT SDK is correctly configured to work with ChatApp.

Build project in Visual Studio 22

Make sure QNN_SDK_ROOT is set globally pointing to QAIRT SDK before you build the project.

  1. Open ChatApp.sln
  2. Build project in Visual Studio

Running App

Running via Visual Studio

Click on the green play button to build and run.

Visual studio project is configured with the following command arguments:

.\ARM64\Debug\ChatApp.exe --genie-config .\\genie_bundle\\genie_config.json --base-dir .\\genie_bundle\\

Running via CLI

cd {Project directory}
.\ARM64\Debug\ChatApp.exe --genie-config .\\genie_bundle\\genie_config.json --base-dir .\\genie_bundle\\

Run --help to learn more:

.\ARM64\Debug\ChatApp.exe --help

Make sure to provide paths to local config file and models using \\ or / as a path separator and not \

Correct file path examples

1. C:\\Path\\To\\Model\\Config\\llama2_7b.json
2. C:/Path/To/Model/Config/llama2_7b.json

Incorrect file path example

1. C:\Path\To\Model\Config\llama2_7b.json

Unicode characters

To use languages that require Unicode, please follow these instructions:

Sample Output

sample_output