Skip to content

Commit 5fb4a38

Browse files
BaoHuilingpre-commit-ci[bot]chensuyue
authored
Add local Rerank microservice for VideoRAGQnA (#496)
* initial commit Signed-off-by: BaoHuiling <[email protected]> * save Signed-off-by: BaoHuiling <[email protected]> * add readme, test script, fix bug Signed-off-by: BaoHuiling <[email protected]> * update video URL Signed-off-by: BaoHuiling <[email protected]> * use default Signed-off-by: BaoHuiling <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update core dependency Signed-off-by: BaoHuiling <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * use p 5000 Signed-off-by: BaoHuiling <[email protected]> * use 5037 Signed-off-by: BaoHuiling <[email protected]> * update ctnr name Signed-off-by: BaoHuiling <[email protected]> * remove langsmith Signed-off-by: BaoHuiling <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add rerank algo desc in readme Signed-off-by: BaoHuiling <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: BaoHuiling <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: chen, suyue <[email protected]>
1 parent 2c2322e commit 5fb4a38

File tree

8 files changed

+307
-1
lines changed

8 files changed

+307
-1
lines changed

‎comps/__init__.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,11 @@
1212
GeneratedDoc,
1313
LLMParamsDoc,
1414
SearchedDoc,
15+
SearchedMultimodalDoc,
1516
RerankedDoc,
1617
TextDoc,
18+
ImageDoc,
19+
TextImageDoc,
1720
RAGASParams,
1821
RAGASScores,
1922
GraphDoc,

‎comps/cores/proto/docarray.py

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Copyright (C) 2024 Intel Corporation
22
# SPDX-License-Identifier: Apache-2.0
33

4-
from typing import Dict, List, Optional, Union
4+
from typing import Dict, List, Optional, Tuple, Union
55

66
import numpy as np
77
from docarray import BaseDoc, DocList
@@ -43,6 +43,14 @@ class TextImageDoc(BaseDoc):
4343
]
4444

4545

46+
class ImageDoc(BaseDoc):
47+
image_path: str
48+
49+
50+
class TextImageDoc(BaseDoc):
51+
doc: Tuple[Union[TextDoc, ImageDoc]]
52+
53+
4654
class Base64ByteStrDoc(BaseDoc):
4755
byte_str: str
4856

@@ -102,6 +110,16 @@ class Config:
102110
json_encoders = {np.ndarray: lambda x: x.tolist()}
103111

104112

113+
class SearchedMultimodalDoc(BaseDoc):
114+
retrieved_docs: List[TextImageDoc]
115+
initial_query: str
116+
top_n: int = 1
117+
metadata: Optional[List[Dict]] = None
118+
119+
class Config:
120+
json_encoders = {np.ndarray: lambda x: x.tolist()}
121+
122+
105123
class GeneratedDoc(BaseDoc):
106124
text: str
107125
prompt: str

‎comps/reranks/video-rag-qna/README.md

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Rerank Microservice
2+
3+
This is a Docker-based microservice that do result rerank for VideoRAGQnA use case. Local rerank is used rather than rerank model.
4+
5+
For the `VideoRAGQnA` usecase, during the data preparation phase, frames are extracted from videos and stored in a vector database. To identify the most relevant video, we count the occurrences of each video source among the retrieved data with rerank function `get_top_doc`. This sorts the video as a descending list of names, ranked by their degree of match with the query. Then we could send the `top_n` videos to the downstream LVM.
6+
7+
# 🚀1. Start Microservice with Docker
8+
9+
## 1.1 Build Images
10+
11+
```bash
12+
cd GenAIComps
13+
docker build --no-cache -t opea/reranking-videoragqna:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/reranks/video-rag-qna/docker/Dockerfile .
14+
```
15+
16+
## 1.2 Start Rerank Service
17+
18+
```bash
19+
docker compose -f comps/reranks/video-rag-qna/docker/docker_compose_reranking.yaml up -d
20+
# wait until ready
21+
until docker logs reranking-videoragqna-server 2>&1 | grep -q "Uvicorn running on"; do
22+
sleep 2
23+
done
24+
```
25+
26+
Available configuration by environment variable:
27+
28+
- CHUNK_DURATION: target chunk duration, should be aligned with VideoRAGQnA dataprep. Default 10s.
29+
30+
# ✅ 2. Test
31+
32+
```bash
33+
export ip_address=$(hostname -I | awk '{print $1}')
34+
curl -X 'POST' \
35+
"http://${ip_address}:8000/v1/reranking" \
36+
-H 'accept: application/json' \
37+
-H 'Content-Type: application/json' \
38+
-d '{
39+
"retrieved_docs": [{"doc": [{"text": "this is the retrieved text"}]}],
40+
"initial_query": "this is the query",
41+
"top_n": 1,
42+
"metadata": [
43+
{"other_key": "value", "video":"top_video_name", "timestamp":"20"},
44+
{"other_key": "value", "video":"second_video_name", "timestamp":"40"},
45+
{"other_key": "value", "video":"top_video_name", "timestamp":"20"}
46+
]
47+
}'
48+
```
49+
50+
The result should be:
51+
52+
```bash
53+
{"id":"random number","video_url":"http://0.0.0.0:6005/top_video_name","chunk_start":20.0,"chunk_duration":10.0,"prompt":"this is the query","max_new_tokens":512}
54+
```
55+
56+
# ♻️ 3. Clean
57+
58+
```bash
59+
# remove the container
60+
cid=$(docker ps -aq --filter "name=reranking-videoragqna-server")
61+
if [[ ! -z "$cid" ]]; then docker stop $cid && docker rm $cid && sleep 1s; fi
62+
```

‎comps/reranks/video-rag-qna/docker/Dockerfile

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
2+
# Copyright (C) 2024 Intel Corporation
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
FROM python:3.11-slim
6+
7+
ENV LANG=C.UTF-8
8+
9+
RUN useradd -m -s /bin/bash user && \
10+
mkdir -p /home/user && \
11+
chown -R user /home/user/
12+
13+
USER user
14+
15+
COPY comps /home/user/comps
16+
17+
RUN pip install --no-cache-dir --upgrade pip && \
18+
pip install --no-cache-dir -r /home/user/comps/reranks/video-rag-qna/requirements.txt
19+
20+
ENV PYTHONPATH=$PYTHONPATH:/home/user
21+
22+
WORKDIR /home/user/comps/reranks/video-rag-qna
23+
24+
ENTRYPOINT ["python", "local_reranking.py"]

‎comps/reranks/video-rag-qna/docker/docker_compose_reranking.yaml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
services:
5+
reranking:
6+
image: opea/reranking-videoragqna:latest
7+
container_name: reranking-videoragqna-server
8+
ports:
9+
- "8000:8000"
10+
ipc: host
11+
environment:
12+
no_proxy: ${no_proxy}
13+
http_proxy: ${http_proxy}
14+
https_proxy: ${https_proxy}
15+
CHUNK_DURATION: ${CHUNK_DURATION}
16+
FILE_SERVER_ENDPOINT: ${FILE_SERVER_ENDPOINT}
17+
restart: unless-stopped
18+
19+
networks:
20+
default:
21+
driver: bridge

‎comps/reranks/video-rag-qna/local_reranking.py

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
import logging
5+
import os
6+
import time
7+
8+
from comps import (
9+
LVMVideoDoc,
10+
SearchedMultimodalDoc,
11+
ServiceType,
12+
opea_microservices,
13+
register_microservice,
14+
register_statistics,
15+
statistics_dict,
16+
)
17+
18+
chunk_duration = os.getenv("CHUNK_DURATION", "10") or "10"
19+
chunk_duration = float(chunk_duration) if chunk_duration.isdigit() else 10.0
20+
21+
file_server_endpoint = os.getenv("FILE_SERVER_ENDPOINT") or "http://0.0.0.0:6005"
22+
23+
logging.basicConfig(
24+
level=logging.INFO, format="%(levelname)s: [%(asctime)s] %(message)s", datefmt="%d/%m/%Y %I:%M:%S"
25+
)
26+
27+
28+
def get_top_doc(top_n, videos) -> list:
29+
hit_score = {}
30+
if videos is None:
31+
return None
32+
for video_name in videos:
33+
try:
34+
if video_name not in hit_score.keys():
35+
hit_score[video_name] = 0
36+
hit_score[video_name] += 1
37+
except KeyError as r:
38+
logging.info(f"no video name {r}")
39+
40+
x = dict(sorted(hit_score.items(), key=lambda item: -item[1])) # sorted dict of video name and score
41+
top_n_names = list(x.keys())[:top_n]
42+
logging.info(f"top docs = {x}")
43+
logging.info(f"top n docs names = {top_n_names}")
44+
45+
return top_n_names
46+
47+
48+
def find_timestamp_from_video(metadata_list, video):
49+
return next(
50+
(metadata["timestamp"] for metadata in metadata_list if metadata["video"] == video),
51+
None,
52+
)
53+
54+
55+
@register_microservice(
56+
name="opea_service@reranking_visual_rag",
57+
service_type=ServiceType.RERANK,
58+
endpoint="/v1/reranking",
59+
host="0.0.0.0",
60+
port=8000,
61+
input_datatype=SearchedMultimodalDoc,
62+
output_datatype=LVMVideoDoc,
63+
)
64+
@register_statistics(names=["opea_service@reranking_visual_rag"])
65+
def reranking(input: SearchedMultimodalDoc) -> LVMVideoDoc:
66+
start = time.time()
67+
68+
# get top video name from metadata
69+
video_names = [meta["video"] for meta in input.metadata]
70+
top_video_names = get_top_doc(input.top_n, video_names)
71+
72+
# only use the first top video
73+
timestamp = find_timestamp_from_video(input.metadata, top_video_names[0])
74+
video_url = f"{file_server_endpoint.rstrip('/')}/{top_video_names[0]}"
75+
76+
result = LVMVideoDoc(
77+
video_url=video_url,
78+
prompt=input.initial_query,
79+
chunk_start=timestamp,
80+
chunk_duration=float(chunk_duration),
81+
max_new_tokens=512,
82+
)
83+
statistics_dict["opea_service@reranking_visual_rag"].append_latency(time.time() - start, None)
84+
85+
return result
86+
87+
88+
if __name__ == "__main__":
89+
opea_microservices["opea_service@reranking_visual_rag"].start()

‎comps/reranks/video-rag-qna/requirements.txt

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
datasets
2+
docarray
3+
fastapi
4+
opentelemetry-api
5+
opentelemetry-exporter-otlp
6+
opentelemetry-sdk
7+
Pillow
8+
prometheus-fastapi-instrumentator
9+
pydub
10+
shortuuid
11+
uvicorn

‎tests/test_reranks_video-rag-qna.sh

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
#!/bin/bash
2+
# Copyright (C) 2024 Intel Corporation
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
set -xe
6+
7+
WORKPATH=$(dirname "$PWD")
8+
ip_address=$(hostname -I | awk '{print $1}')
9+
10+
function build_docker_images() {
11+
cd $WORKPATH
12+
docker build --no-cache -t opea/reranking-videoragqna:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/reranks/video-rag-qna/docker/Dockerfile .
13+
}
14+
15+
function start_service() {
16+
docker run -d --name "test-comps-reranking-videoragqna-server" \
17+
-p 5037:8000 \
18+
--ipc=host \
19+
-e no_proxy=${no_proxy} \
20+
-e http_proxy=${http_proxy} \
21+
-e https_proxy=${https_proxy} \
22+
-e CHUNK_DURATION=${CHUNK_DURATION} \
23+
-e FILE_SERVER_ENDPOINT=${FILE_SERVER_ENDPOINT} \
24+
opea/reranking-videoragqna:latest
25+
26+
27+
until docker logs test-comps-reranking-videoragqna-server 2>&1 | grep -q "Uvicorn running on"; do
28+
sleep 2
29+
done
30+
}
31+
32+
function validate_microservice() {
33+
result=$(\
34+
http_proxy="" \
35+
curl -X 'POST' \
36+
"http://${ip_address}:5037/v1/reranking" \
37+
-H 'accept: application/json' \
38+
-H 'Content-Type: application/json' \
39+
-d '{
40+
"retrieved_docs": [
41+
{"doc": [{"text": "this is the retrieved text"}]}
42+
],
43+
"initial_query": "this is the query",
44+
"top_n": 1,
45+
"metadata": [
46+
{"other_key": "value", "video":"top_video_name", "timestamp":"20"},
47+
{"other_key": "value", "video":"second_video_name", "timestamp":"40"},
48+
{"other_key": "value", "video":"top_video_name", "timestamp":"20"}
49+
]
50+
}')
51+
if [[ $result == *"this is the query"* ]]; then
52+
echo "Result correct."
53+
else
54+
echo "Result wrong."
55+
exit 1
56+
fi
57+
}
58+
59+
function stop_docker() {
60+
cid=$(docker ps -aq --filter "name=test-comps-reranking*")
61+
if [[ ! -z "$cid" ]]; then docker stop $cid && docker rm $cid && sleep 1s; fi
62+
}
63+
64+
function main() {
65+
66+
stop_docker
67+
68+
build_docker_images
69+
start_service
70+
71+
validate_microservice
72+
73+
stop_docker
74+
echo y | docker system prune
75+
76+
}
77+
78+
main

0 commit comments

Comments
 (0)