|
| 1 | +# QuickSatrt Guide |
| 2 | + |
| 3 | +Ver: 1.0 |
| 4 | +Last Update: 2024-Aug-21 |
| 5 | +Author: [PeterYang12](https://github.com/PeterYang12) |
| 6 | + |
| 7 | + |
| 8 | +This document is a quickstart guide for GenAIInfra deployment and test on NVIDIA GPU platform. |
| 9 | + |
| 10 | +## Prerequisite |
| 11 | + |
| 12 | +GenAIInfra uses Kubernetes as the cloud native infrastructure. Please follow the steps below to prepare the Kubernetes environment. |
| 13 | + |
| 14 | +#### Setup Kubernetes cluster |
| 15 | + |
| 16 | +Please follow [Kubernetes official setup guide](https://github.com/opea-project/GenAIInfra?tab=readme-ov-file#setup-kubernetes-cluster) to setup Kubernetes. We recommend to use Kubernetes with version >= 1.27. |
| 17 | + |
| 18 | +#### To run GenAIInfra on NVIDIA GPUs |
| 19 | + |
| 20 | +To run the workloads on NVIDIA GPUs, please follow the steps. |
| 21 | + |
| 22 | +1. Please check the [support matrix](https://docs.nvidia.com/ai-enterprise/latest/product-support-matrix/index.html) to make sure that environment meets the requirements. |
| 23 | + |
| 24 | +2. [Install the NVIDIA GPU CUDA driver and software stack](https://developer.nvidia.com/cuda-downloads). |
| 25 | + |
| 26 | +3. [Install the NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) |
| 27 | + |
| 28 | +4. [Install the NVIDIA GPU device plugin for Kubernetes](https://github.com/NVIDIA/k8s-device-plugin). |
| 29 | +5. [Install helm](https://helm.sh/docs/intro/install/) |
| 30 | + |
| 31 | +NOTE: Please make sure you configure the appropriate container runtime based on the type of container runtime you installed during Kubernetes setup. |
| 32 | + |
| 33 | +## Usages |
| 34 | + |
| 35 | +#### Use GenAI Microservices Connector (GMC) to deploy and adjust GenAIExamples on NVIDIA GPUs |
| 36 | + |
| 37 | +#### 1. Install the GMC Helm Chart |
| 38 | + |
| 39 | +**_NOTE_**: Before installingGMC, please export your own huggingface tokens, Google API KEY and Google CSE ID. If you have pre-defined directory to save the models on you cluster hosts, please also set the path. |
| 40 | + |
| 41 | +``` |
| 42 | +export YOUR_HF_TOKEN=<your hugging facetoken> |
| 43 | +export YOUR_GOOGLE_API_KEY=<your google api key> |
| 44 | +export YOUR_GOOGLE_CSE_ID=<your google cse id> |
| 45 | +export MOUNT_DIR=<your model path> |
| 46 | +``` |
| 47 | + |
| 48 | +Here also provides a simple way to install GMC using helm chart `./install-gmc.sh` |
| 49 | + |
| 50 | +> WARNING: the install-gmc.sh may fail due to OS distributions. |
| 51 | +
|
| 52 | +For more details, please refer to [GMC installation](https://github.com/opea-project/GenAIInfra/blob/main/microservices-connector/README.md) to get more details. |
| 53 | + |
| 54 | +#### 2.Use GMC to compose a ChatQnA Pipeline |
| 55 | + |
| 56 | +Please refer to [Usage guide for GMC](https://github.com/opea-project/GenAIInfra/blob/main/microservices-connector/usage_guide.md) for more details. |
| 57 | + |
| 58 | +Here provides a simple script to use GMC to compose ChatQnA pipeline. |
| 59 | + |
| 60 | +#### 3. Test ChatQnA service |
| 61 | + |
| 62 | +Please refer to [GMC ChatQnA test](https://github.com/opea-project/GenAIInfra/blob/main/microservices-connector/usage_guide.md#use-gmc-to-compose-a-chatqna-pipeline) |
| 63 | +Here provides a simple way to test the service. `./gmc-chatqna-test.sh` |
| 64 | + |
| 65 | +#### 4. Delete ChatQnA and GMC |
| 66 | + |
| 67 | +``` |
| 68 | +kubectl delete ns chatqa |
| 69 | +./delete-gmc.sh |
| 70 | +``` |
| 71 | + |
| 72 | +## FAQ and Troubleshooting |
| 73 | + |
| 74 | +The scripts are only tested on baremental **Ubuntu22.04** with **NVIDIA H100**. Please report an issue if you meet any issue. |
0 commit comments