Skip to content

Commit 84a7e57

Browse files
add microservice for intent detection (#131)
* add microservice for intent detection Signed-off-by: Liangyx2 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update license copyright Signed-off-by: Liangyx2 <[email protected]> * add ut Signed-off-by: Liangyx2 <[email protected]> * refine Signed-off-by: Liangyx2 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update folder Signed-off-by: Liangyx2 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix test Signed-off-by: Liangyx2 <[email protected]> --------- Signed-off-by: Liangyx2 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent b541fd8 commit 84a7e57

File tree

7 files changed

+277
-0
lines changed

7 files changed

+277
-0
lines changed

comps/intent_detection/README.md

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Intent Detection Microservice by TGI
2+
3+
# 🚀1. Start Microservice with Python(Option 1)
4+
5+
## 1.1 Install Requirements
6+
7+
```bash
8+
pip install -r requirements.txt
9+
```
10+
11+
## 1.2 Start TGI Service
12+
13+
```bash
14+
export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
15+
export LANGCHAIN_TRACING_V2=true
16+
export LANGCHAIN_API_KEY=${your_langchain_api_key}
17+
export LANGCHAIN_PROJECT="opea/gen-ai-comps:llms"
18+
docker run -p 8008:80 -v ./data:/data --name tgi_service --shm-size 1g ghcr.io/huggingface/text-generation-inference:1.4 --model-id ${your_hf_llm_model}
19+
```
20+
21+
## 1.3 Verify the TGI Service
22+
23+
```bash
24+
curl http://${your_ip}:8008/generate \
25+
-X POST \
26+
-d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":17, "do_sample": true}}' \
27+
-H 'Content-Type: application/json'
28+
```
29+
30+
## 1.4 Setup Environment Variables
31+
32+
```bash
33+
export TGI_LLM_ENDPOINT="http://${your_ip}:8008"
34+
export LANGCHAIN_TRACING_V2=true
35+
export LANGCHAIN_API_KEY=${your_langchain_api_key}
36+
export LANGCHAIN_PROJECT="opea/intent"
37+
```
38+
39+
## 1.5 Start Intent Detection Microservice with Python Script
40+
41+
Start intent detection microservice with below command.
42+
43+
```bash
44+
cd /your_project_path/GenAIComps/
45+
cp comps/intent_detection/langchain/intent_detection.py .
46+
python intent_detection.py
47+
```
48+
49+
# 🚀2. Start Microservice with Docker (Option 2)
50+
51+
## 2.1 Start TGI Service
52+
53+
Please refer to 1.2.
54+
55+
## 2.2 Setup Environment Variables
56+
57+
```bash
58+
export TGI_LLM_ENDPOINT="http://${your_ip}:8008"
59+
export LANGCHAIN_TRACING_V2=true
60+
export LANGCHAIN_API_KEY=${your_langchain_api_key}
61+
export LANGCHAIN_PROJECT="opea/intent"
62+
```
63+
64+
## 2.3 Build Docker Image
65+
66+
```bash
67+
cd /your_project_path/GenAIComps
68+
docker build --no-cache -t opea/llm-tgi:latest -f comps/intent_detection/langchain/Dockerfile .
69+
```
70+
71+
## 2.4 Run Docker with CLI (Option A)
72+
73+
```bash
74+
docker run -it --name="intent-tgi-server" --net=host --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e TGI_LLM_ENDPOINT=$TGI_LLM_ENDPOINT -e HUGGINGFACEHUB_API_TOKEN=$HUGGINGFACEHUB_API_TOKEN opea/llm-tgi:latest
75+
```
76+
77+
## 2.5 Run with Docker Compose (Option B)
78+
79+
```bash
80+
cd /your_project_path/GenAIComps/comps/intent_detection/langchain
81+
export LLM_MODEL_ID=${your_hf_llm_model}
82+
export http_proxy=${your_http_proxy}
83+
export https_proxy=${your_http_proxy}
84+
export TGI_LLM_ENDPOINT="http://tgi-service:80"
85+
export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
86+
export LANGCHAIN_API_KEY=${your_langchain_api_key}
87+
docker compose -f docker_compose_intent.yaml up -d
88+
```
89+
90+
# 🚀3. Consume Microservice
91+
92+
Once intent detection microservice is started, user can use below command to invoke the microservice.
93+
94+
```bash
95+
curl http://${your_ip}:9000/v1/chat/intent\
96+
-X POST \
97+
-d '{"query":"What is Deep Learning?","max_new_tokens":10,"top_k":1,"temperature":0.001,"streaming":false}' \
98+
-H 'Content-Type: application/json'
99+
```
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
FROM langchain/langchain:latest
5+
6+
RUN apt-get update -y && apt-get install -y --no-install-recommends --fix-missing \
7+
libgl1-mesa-glx \
8+
libjemalloc-dev \
9+
vim
10+
11+
RUN useradd -m -s /bin/bash user && \
12+
mkdir -p /home/user && \
13+
chown -R user /home/user/
14+
15+
USER user
16+
17+
COPY comps /home/user/comps
18+
19+
RUN pip install --no-cache-dir --upgrade pip && \
20+
pip install --no-cache-dir -r /home/user/comps/intent_detection/langchain/requirements.txt
21+
22+
ENV PYTHONPATH=$PYTHONPATH:/home/user
23+
24+
WORKDIR /home/user/comps/intent_detection/langchain
25+
ENTRYPOINT ["python", "intent_detection.py"]
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
version: "3.8"
5+
6+
services:
7+
tgi_service:
8+
image: ghcr.io/huggingface/text-generation-inference:1.4
9+
container_name: tgi-service
10+
ports:
11+
- "8008:80"
12+
volumes:
13+
- "./data:/data"
14+
shm_size: 1g
15+
command: --model-id ${LLM_MODEL_ID}
16+
llm:
17+
image: opea/llm-tgi:latest
18+
container_name: intent-tgi-server
19+
ports:
20+
- "9000:9000"
21+
ipc: host
22+
environment:
23+
http_proxy: ${http_proxy}
24+
https_proxy: ${https_proxy}
25+
TGI_LLM_ENDPOINT: ${TGI_LLM_ENDPOINT}
26+
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
27+
LANGCHAIN_API_KEY: ${LANGCHAIN_API_KEY}
28+
restart: unless-stopped
29+
30+
networks:
31+
default:
32+
driver: bridge
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
import os
5+
6+
from langchain import LLMChain, PromptTemplate
7+
from langchain_community.llms import HuggingFaceEndpoint
8+
from langsmith import traceable
9+
10+
from comps import GeneratedDoc, LLMParamsDoc, ServiceType, opea_microservices, register_microservice
11+
from comps.intent_detection.langchain.template import IntentTemplate
12+
13+
14+
@register_microservice(
15+
name="opea_service@llm_intent",
16+
service_type=ServiceType.LLM,
17+
endpoint="/v1/chat/intent",
18+
host="0.0.0.0",
19+
port=9000,
20+
)
21+
@traceable(run_type="llm")
22+
def llm_generate(input: LLMParamsDoc):
23+
llm_endpoint = os.getenv("TGI_LLM_ENDPOINT", "http://localhost:8080")
24+
llm = HuggingFaceEndpoint(
25+
endpoint_url=llm_endpoint,
26+
max_new_tokens=input.max_new_tokens,
27+
top_k=input.top_k,
28+
top_p=input.top_p,
29+
typical_p=input.typical_p,
30+
temperature=input.temperature,
31+
repetition_penalty=input.repetition_penalty,
32+
streaming=input.streaming,
33+
timeout=600,
34+
)
35+
36+
prompt = PromptTemplate(template=IntentTemplate.generate_intent_template, input_variables=["query"])
37+
38+
llm_chain = LLMChain(prompt=prompt, llm=llm)
39+
40+
response = llm_chain.invoke(input.query)
41+
response = response["text"]
42+
print("response", response)
43+
return GeneratedDoc(text=response, prompt=input.query)
44+
45+
46+
if __name__ == "__main__":
47+
opea_microservices["opea_service@llm_intent"].start()
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
docarray[full]
2+
fastapi
3+
huggingface_hub
4+
langchain==0.1.16
5+
opentelemetry-api
6+
opentelemetry-exporter-otlp
7+
opentelemetry-sdk
8+
prometheus-fastapi-instrumentator
9+
shortuuid
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
5+
class IntentTemplate:
6+
def generate_intent_template(query):
7+
return f"""Please identify the intent of the user query. You may only respond with "chitchat" or "QA" without explanations or engaging in conversation.
8+
### User Query: {query}, ### Response: """
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
#!/bin/bash
2+
# Copyright (C) 2024 Intel Corporation
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
set -xe
6+
7+
WORKPATH=$(dirname "$PWD")
8+
ip_address=$(hostname -I | awk '{print $1}')
9+
function build_docker_images() {
10+
cd $WORKPATH
11+
docker build --no-cache -t opea/llm-tgi:latest -f comps/intent_detection/langchain/Dockerfile .
12+
}
13+
14+
function start_service() {
15+
tgi_endpoint=5004
16+
# Remember to set HF_TOKEN before invoking this test!
17+
export HUGGINGFACEHUB_API_TOKEN=${HF_TOKEN}
18+
model=Intel/neural-chat-7b-v3-3
19+
docker run -d --name="test-comps-intent-tgi-endpoint" -p $tgi_endpoint:80 -v ./data:/data --shm-size 1g ghcr.io/huggingface/text-generation-inference:1.4 --model-id $model
20+
21+
export TGI_LLM_ENDPOINT="http://${ip_address}:${tgi_endpoint}"
22+
tei_service_port=5005
23+
unset http_proxy
24+
docker run -d --name="test-comps-intent-tei-server" -p ${tei_service_port}:9000 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e TGI_LLM_ENDPOINT=$TGI_LLM_ENDPOINT -e HUGGINGFACEHUB_API_TOKEN=$HUGGINGFACEHUB_API_TOKEN opea/llm-tgi:latest
25+
sleep 3m
26+
}
27+
28+
function validate_microservice() {
29+
tei_service_port=5005
30+
http_proxy="" curl http://${ip_address}:${tei_service_port}/v1/chat/intent\
31+
-X POST \
32+
-d '{"query":"What is Deep Learning?","max_new_tokens":10,"top_k":1,"temperature":0.001,"streaming":false}' \
33+
-H 'Content-Type: application/json'
34+
docker logs test-comps-intent-tei-server
35+
docker logs test-comps-intent-tgi-endpoint
36+
}
37+
38+
function stop_docker() {
39+
cid=$(docker ps -aq --filter "name=test-comps-intent*")
40+
if [[ ! -z "$cid" ]]; then docker stop $cid && docker rm $cid && sleep 1s; fi
41+
}
42+
43+
function main() {
44+
45+
stop_docker
46+
47+
build_docker_images
48+
start_service
49+
50+
validate_microservice
51+
52+
stop_docker
53+
echo y | docker system prune
54+
55+
}
56+
57+
main

0 commit comments

Comments
 (0)