Skip to content

Commit cad2fc3

Browse files
Add some NVIDIA platform support docs and scripts (#325)
* Add some NVIDIA platform support docs and scripts Signed-off-by: PeterYang12 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent cab7a88 commit cad2fc3

File tree

5 files changed

+148
-0
lines changed

5 files changed

+148
-0
lines changed

scripts/nvidia/README.md

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# QuickSatrt Guide
2+
3+
Ver: 1.0
4+
Last Update: 2024-Aug-21
5+
Author: [PeterYang12](https://github.com/PeterYang12)
6+
7+
8+
This document is a quickstart guide for GenAIInfra deployment and test on NVIDIA GPU platform.
9+
10+
## Prerequisite
11+
12+
GenAIInfra uses Kubernetes as the cloud native infrastructure. Please follow the steps below to prepare the Kubernetes environment.
13+
14+
#### Setup Kubernetes cluster
15+
16+
Please follow [Kubernetes official setup guide](https://github.com/opea-project/GenAIInfra?tab=readme-ov-file#setup-kubernetes-cluster) to setup Kubernetes. We recommend to use Kubernetes with version >= 1.27.
17+
18+
#### To run GenAIInfra on NVIDIA GPUs
19+
20+
To run the workloads on NVIDIA GPUs, please follow the steps.
21+
22+
1. Please check the [support matrix](https://docs.nvidia.com/ai-enterprise/latest/product-support-matrix/index.html) to make sure that environment meets the requirements.
23+
24+
2. [Install the NVIDIA GPU CUDA driver and software stack](https://developer.nvidia.com/cuda-downloads).
25+
26+
3. [Install the NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
27+
28+
4. [Install the NVIDIA GPU device plugin for Kubernetes](https://github.com/NVIDIA/k8s-device-plugin).
29+
5. [Install helm](https://helm.sh/docs/intro/install/)
30+
31+
NOTE: Please make sure you configure the appropriate container runtime based on the type of container runtime you installed during Kubernetes setup.
32+
33+
## Usages
34+
35+
#### Use GenAI Microservices Connector (GMC) to deploy and adjust GenAIExamples on NVIDIA GPUs
36+
37+
#### 1. Install the GMC Helm Chart
38+
39+
**_NOTE_**: Before installingGMC, please export your own huggingface tokens, Google API KEY and Google CSE ID. If you have pre-defined directory to save the models on you cluster hosts, please also set the path.
40+
41+
```
42+
export YOUR_HF_TOKEN=<your hugging facetoken>
43+
export YOUR_GOOGLE_API_KEY=<your google api key>
44+
export YOUR_GOOGLE_CSE_ID=<your google cse id>
45+
export MOUNT_DIR=<your model path>
46+
```
47+
48+
Here also provides a simple way to install GMC using helm chart `./install-gmc.sh`
49+
50+
> WARNING: the install-gmc.sh may fail due to OS distributions.
51+
52+
For more details, please refer to [GMC installation](https://github.com/opea-project/GenAIInfra/blob/main/microservices-connector/README.md) to get more details.
53+
54+
#### 2.Use GMC to compose a ChatQnA Pipeline
55+
56+
Please refer to [Usage guide for GMC](https://github.com/opea-project/GenAIInfra/blob/main/microservices-connector/usage_guide.md) for more details.
57+
58+
Here provides a simple script to use GMC to compose ChatQnA pipeline.
59+
60+
#### 3. Test ChatQnA service
61+
62+
Please refer to [GMC ChatQnA test](https://github.com/opea-project/GenAIInfra/blob/main/microservices-connector/usage_guide.md#use-gmc-to-compose-a-chatqna-pipeline)
63+
Here provides a simple way to test the service. `./gmc-chatqna-test.sh`
64+
65+
#### 4. Delete ChatQnA and GMC
66+
67+
```
68+
kubectl delete ns chatqa
69+
./delete-gmc.sh
70+
```
71+
72+
## FAQ and Troubleshooting
73+
74+
The scripts are only tested on baremental **Ubuntu22.04** with **NVIDIA H100**. Please report an issue if you meet any issue.

scripts/nvidia/delete-gmc.sh

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#!/usr/bin/env bash
2+
# Copyright (C) 2024 Intel Corporation
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
set -e
6+
7+
SCRIPT_DIR=$(cd $(dirname "${BASH_SOURCE[0]}") && pwd)
8+
cd $SCRIPT_DIR && cd ../../
9+
GenAIInfra_DIR=$(pwd)
10+
cd $GenAIInfra_DIR/microservices-connector
11+
12+
# kubectl delete -k config/samples/
13+
helm delete -n system gmc
14+
kubectl delete crd gmconnectors.gmc.opea.io
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/usr/bin/env bash
2+
# Copyright (C) 2024 Intel Corporation
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
set -e
6+
7+
SCRIPT_DIR=$(cd $(dirname "${BASH_SOURCE[0]}") && pwd)
8+
cd $SCRIPT_DIR && cd ../../
9+
GenAIInfra_DIR=$(pwd)
10+
cd $GenAIInfra_DIR/microservices-connector/
11+
12+
# TODO: to support more examples
13+
kubectl create ns chatqa
14+
kubectl apply -f $(pwd)/config/samples/chatQnA_nv.yaml
15+
16+
sleep 2
17+
kubectl get service -n chatqa
18+
kubectl create deployment client-test -n chatqa --image=python:3.8.13 -- sleep infinity

scripts/nvidia/gmc-chatqna-test.sh

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#!/usr/bin/env bash
2+
# Copyright (C) 2024 Intel Corporation
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
set -e
6+
7+
CLIENT_POD=$(kubectl get pod -n chatqa -l app=client-test -o jsonpath={.items..metadata.name})
8+
accessUrl=$(kubectl get gmc -n chatqa -o jsonpath="{.items[?(@.metadata.name=='chatqa')].status.accessUrl}")
9+
10+
kubectl exec "$CLIENT_POD" -n chatqa -- curl $accessUrl -X POST -d '{"text":"What is the revenue of Nike in 2023?","parameters":{"max_new_tokens":17, "do_sample": true}}' -H 'Content-Type: application/json'

scripts/nvidia/install-gmc.sh

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
#!/usr/bin/env bash
2+
# Copyright (C) 2024 Intel Corporation
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
set -e
6+
7+
SCRIPT_DIR=$(cd $(dirname "${BASH_SOURCE[0]}") && pwd)
8+
cd $SCRIPT_DIR && cd ../../
9+
GenAIInfra_DIR=$(pwd)
10+
cd $GenAIInfra_DIR/microservices-connector/helm
11+
12+
if [ -n "$YOUR_HF_TOKEN" ]; then
13+
find manifests_common/ -name '*.yaml' -type f -exec sed -i "s#insert-your-huggingface-token-here#$YOUR_HF_TOKEN#g" {} \;
14+
fi
15+
16+
if [ -n "$YOUR_GOOGLE_API_KEY" ]; then
17+
find manifests_common/ -name '*.yaml' -type f -exec sed -i "s#GOOGLE_API_KEY:.*#GOOGLE_API_KEY: "$YOUR_GOOGLE_API_KEY"#g" {} \;
18+
fi
19+
20+
if [ -n "$YOUR_GOOGLE_CSE_ID" ]; then
21+
find manifests_common/ -name '*.yaml' -type f -exec sed -i "s#GOOGLE_CSE_ID:.*#GOOGLE_CSE_ID: "$YOUR_GOOGLE_CSE_ID"#g" {} \;
22+
fi
23+
24+
25+
if [ -n "$MOUNT_DIR" ]; then
26+
find manifests_common/ -name '*.yaml' -type f -exec sed -i "s#path: /mnt/opea-models#path: $MOUNT_DIR#g" {} \;
27+
fi
28+
29+
# install GMC helm chart
30+
helm install -n system --create-namespace gmc .
31+
sleep 2
32+
kubectl get pod -n system

0 commit comments

Comments
 (0)