Skip to content

Commit 2ca56f3

Browse files
tileintelpre-commit-ci[bot]
tileintel
and
pre-commit-ci[bot]
authored
Update example name into MultimodalQnA and update image names (#686)
* update image names Signed-off-by: Tiep Le <[email protected]> * update image names for dataprep service Signed-off-by: Tiep Le <[email protected]> * update image name for retriever Signed-off-by: Tiep Le <[email protected]> * update image name for lvm Signed-off-by: Tiep Le <[email protected]> * update lvm include caption in output Signed-off-by: Tiep Le <[email protected]> * update gateway name into MultimodalQnAGateway Signed-off-by: Tiep Le <[email protected]> * update test gateway for rename Signed-off-by: Tiep Le <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Tiep Le <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 2b68323 commit 2ca56f3

File tree

19 files changed

+104
-104
lines changed

19 files changed

+104
-104
lines changed

‎comps/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@
4848
FaqGenGateway,
4949
VideoQnAGateway,
5050
VisualQnAGateway,
51-
MultimodalRAGWithVideosGateway,
51+
MultimodalQnAGateway,
5252
)
5353

5454
# Telemetry

‎comps/cores/mega/constants.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ class MegaServiceEndpoint(Enum):
4343
CODE_TRANS = "/v1/codetrans"
4444
DOC_SUMMARY = "/v1/docsum"
4545
SEARCH_QNA = "/v1/searchqna"
46-
MULTIMODAL_RAG_WITH_VIDEOS = "/v1/mmragvideoqna"
46+
MULTIMODAL_QNA = "/v1/multimodalqna"
4747
TRANSLATION = "/v1/translation"
4848
RETRIEVALTOOL = "/v1/retrievaltool"
4949
FAQ_GEN = "/v1/faqgen"

‎comps/cores/mega/gateway.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -634,14 +634,14 @@ def parser_input(data, TypeClass, key):
634634
return response
635635

636636

637-
class MultimodalRAGWithVideosGateway(Gateway):
637+
class MultimodalQnAGateway(Gateway):
638638
def __init__(self, multimodal_rag_megaservice, lvm_megaservice, host="0.0.0.0", port=9999):
639639
self.lvm_megaservice = lvm_megaservice
640640
super().__init__(
641641
multimodal_rag_megaservice,
642642
host,
643643
port,
644-
str(MegaServiceEndpoint.MULTIMODAL_RAG_WITH_VIDEOS),
644+
str(MegaServiceEndpoint.MULTIMODAL_QNA),
645645
ChatCompletionRequest,
646646
ChatCompletionResponse,
647647
)
@@ -737,7 +737,7 @@ async def handle_request(self, request: Request):
737737
data = await request.json()
738738
stream_opt = bool(data.get("stream", False))
739739
if stream_opt:
740-
print("[ MultimodalRAGWithVideosGateway ] stream=True not used, this has not support streaming yet!")
740+
print("[ MultimodalQnAGateway ] stream=True not used, this has not support streaming yet!")
741741
stream_opt = False
742742
chat_request = ChatCompletionRequest.model_validate(data)
743743
# Multimodal RAG QnA With Videos has not yet accepts image as input during QnA.
@@ -803,4 +803,4 @@ async def handle_request(self, request: Request):
803803
metadata=metadata,
804804
)
805805
)
806-
return ChatCompletionResponse(model="multimodalragwithvideos", choices=choices, usage=usage)
806+
return ChatCompletionResponse(model="multimodalqna", choices=choices, usage=usage)

‎comps/dataprep/multimodal/redis/langchain/README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -83,13 +83,13 @@ export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
8383

8484
```bash
8585
cd ../../../../
86-
docker build -t opea/dataprep-redis:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/multimodal/redis/langchain/Dockerfile .
86+
docker build -t opea/dataprep-multimodal-redis:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/multimodal/redis/langchain/Dockerfile .
8787
```
8888

8989
### 2.5 Run Docker with CLI (Option A)
9090

9191
```bash
92-
docker run -d --name="dataprep-redis-server" -p 6007:6007 --runtime=runc --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e REDIS_URL=$REDIS_URL -e INDEX_NAME=$INDEX_NAME -e LVM_ENDPOINT=$LVM_ENDPOINT -e HUGGINGFACEHUB_API_TOKEN=$HUGGINGFACEHUB_API_TOKEN opea/dataprep-redis:latest
92+
docker run -d --name="dataprep-multimodal-redis" -p 6007:6007 --runtime=runc --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e REDIS_URL=$REDIS_URL -e INDEX_NAME=$INDEX_NAME -e LVM_ENDPOINT=$LVM_ENDPOINT -e HUGGINGFACEHUB_API_TOKEN=$HUGGINGFACEHUB_API_TOKEN opea/dataprep-multimodal-redis:latest
9393
```
9494

9595
### 2.6 Run with Docker Compose (Option B - deprecated, will move to genAIExample in future)
@@ -102,7 +102,7 @@ docker compose -f docker-compose-dataprep-redis.yaml up -d
102102
## 🚀3. Status Microservice
103103

104104
```bash
105-
docker container logs -f dataprep-redis-server
105+
docker container logs -f dataprep-multimodal-redis
106106
```
107107

108108
## 🚀4. Consume Microservice

‎comps/dataprep/multimodal/redis/langchain/docker-compose-dataprep-redis.yaml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,9 @@ services:
99
ports:
1010
- "6379:6379"
1111
- "8001:8001"
12-
dataprep-redis:
13-
image: opea/dataprep-redis:latest
14-
container_name: dataprep-redis-server
12+
dataprep-multimodal-redis:
13+
image: opea/dataprep-multimodal-redis:latest
14+
container_name: dataprep-multimodal-redis
1515
ports:
1616
- "6007:6007"
1717
ipc: host

‎comps/embeddings/multimodal/README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ Currently, we employ [**BridgeTower**](https://huggingface.co/BridgeTower/bridge
5252

5353
```bash
5454
cd ../../..
55-
docker build -t opea/bridgetower-embedder:latest --build-arg EMBEDDER_PORT=$EMBEDDER_PORT --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/embeddings/multimodal/bridgetower/Dockerfile.intel_hpu .
55+
docker build -t opea/embedding-multimodal-bridgetower:latest --build-arg EMBEDDER_PORT=$EMBEDDER_PORT --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/embeddings/multimodal/bridgetower/Dockerfile.intel_hpu .
5656
cd comps/embeddings/multimodal/bridgetower/
5757
docker compose -f docker_compose_bridgetower_embedding_endpoint.yaml up -d
5858
```
@@ -61,7 +61,7 @@ docker compose -f docker_compose_bridgetower_embedding_endpoint.yaml up -d
6161

6262
```bash
6363
cd ../../..
64-
docker build -t opea/bridgetower-embedder:latest --build-arg EMBEDDER_PORT=$EMBEDDER_PORT --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/embeddings/multimodal/bridgetower/Dockerfile .
64+
docker build -t opea/embedding-multimodal-bridgetower:latest --build-arg EMBEDDER_PORT=$EMBEDDER_PORT --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/embeddings/multimodal/bridgetower/Dockerfile .
6565
cd comps/embeddings/multimodal/bridgetower/
6666
docker compose -f docker_compose_bridgetower_embedding_endpoint.yaml up -d
6767
```
@@ -116,7 +116,7 @@ Currently, we employ [**BridgeTower**](https://huggingface.co/BridgeTower/bridge
116116

117117
```bash
118118
cd ../../..
119-
docker build -t opea/bridgetower-embedder:latest --build-arg EMBEDDER_PORT=$EMBEDDER_PORT --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/embeddings/multimodal/bridgetower/Dockerfile.intel_hpu .
119+
docker build -t opea/embedding-multimodal-bridgetower:latest --build-arg EMBEDDER_PORT=$EMBEDDER_PORT --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/embeddings/multimodal/bridgetower/Dockerfile.intel_hpu .
120120
cd comps/embeddings/multimodal/bridgetower/
121121
docker compose -f docker_compose_bridgetower_embedding_endpoint.yaml up -d
122122
```
@@ -125,7 +125,7 @@ docker compose -f docker_compose_bridgetower_embedding_endpoint.yaml up -d
125125

126126
```bash
127127
cd ../../..
128-
docker build -t opea/bridgetower-embedder:latest --build-arg EMBEDDER_PORT=$EMBEDDER_PORT --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/embeddings/multimodal/bridgetower/Dockerfile .
128+
docker build -t opea/embedding-multimodal-bridgetower:latest --build-arg EMBEDDER_PORT=$EMBEDDER_PORT --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/embeddings/multimodal/bridgetower/Dockerfile .
129129
cd comps/embeddings/multimodal/bridgetower/
130130
docker compose -f docker_compose_bridgetower_embedding_endpoint.yaml up -d
131131
```

‎comps/embeddings/multimodal/bridgetower/docker_compose_bridgetower_embedding_endpoint.yaml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
# SPDX-License-Identifier: Apache-2.0
33

44
services:
5-
bridgetower:
6-
image: opea/bridgetower-embedder:latest
7-
container_name: bridgetower-embedding-server
5+
embedding-multimodal-bridgetower:
6+
image: opea/embedding-multimodal-bridgetower:latest
7+
container_name: embedding-multimodal-bridgetower
88
ports:
99
- ${EMBEDDER_PORT}:${EMBEDDER_PORT}
1010
ipc: host

‎comps/embeddings/multimodal/multimodal_langchain/docker_compose_multimodal_embedding.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
# SPDX-License-Identifier: Apache-2.0
33

44
services:
5-
embedding:
5+
embedding-multimodal:
66
image: opea/embedding-multimodal:latest
7-
container_name: embedding-multimodal-server
7+
container_name: embedding-multimodal
88
ports:
99
- ${MM_EMBEDDING_PORT_MICROSERVICE}:${MM_EMBEDDING_PORT_MICROSERVICE}
1010
ipc: host

‎comps/lvms/llava/README.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -57,21 +57,21 @@ python check_lvm.py
5757

5858
```bash
5959
cd ../../../
60-
docker build -t opea/llava:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/lvms/llava/dependency/Dockerfile .
60+
docker build -t opea/lvm-llava:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/lvms/llava/dependency/Dockerfile .
6161
```
6262

6363
- Gaudi2 HPU
6464

6565
```bash
6666
cd ../../../
67-
docker build -t opea/llava:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/lvms/llava/dependency/Dockerfile.intel_hpu .
67+
docker build -t opea/lvm-llava:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/lvms/llava/dependency/Dockerfile.intel_hpu .
6868
```
6969

7070
#### 2.1.2 LVM Service Image
7171

7272
```bash
7373
cd ../../../
74-
docker build -t opea/lvm:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/lvms/llava/Dockerfile .
74+
docker build -t opea/lvm-llava-svc:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/lvms/llava/Dockerfile .
7575
```
7676

7777
### 2.2 Start LLaVA and LVM Service
@@ -81,21 +81,21 @@ docker build -t opea/lvm:latest --build-arg https_proxy=$https_proxy --build-arg
8181
- Xeon
8282

8383
```bash
84-
docker run -p 8399:8399 -e http_proxy=$http_proxy --ipc=host -e https_proxy=$https_proxy opea/llava:latest
84+
docker run -p 8399:8399 -e http_proxy=$http_proxy --ipc=host -e https_proxy=$https_proxy opea/lvm-llava:latest
8585
```
8686

8787
- Gaudi2 HPU
8888

8989
```bash
90-
docker run -p 8399:8399 --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy opea/llava:latest
90+
docker run -p 8399:8399 --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy opea/lvm-llava:latest
9191
```
9292

9393
#### 2.2.2 Start LVM service
9494

9595
```bash
9696
ip_address=$(hostname -I | awk '{print $1}')
9797

98-
docker run -p 9399:9399 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e LVM_ENDPOINT=http://$ip_address:8399 opea/lvm:latest
98+
docker run -p 9399:9399 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e LVM_ENDPOINT=http://$ip_address:8399 opea/lvm-llava-svc:latest
9999
```
100100

101101
#### 2.2.3 Test

‎comps/lvms/llava/lvm.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ async def lvm(request: Union[LVMDoc, LVMSearchedMultimodalDoc]) -> Union[TextDoc
8989
return_metadata["video_id"] = retrieved_metadata["video_id"]
9090
return_metadata["source_video"] = retrieved_metadata["source_video"]
9191
return_metadata["time_of_frame_ms"] = retrieved_metadata["time_of_frame_ms"]
92+
return_metadata["transcript_for_inference"] = retrieved_metadata["transcript_for_inference"]
9293
return MetadataTextDoc(text=result, metadata=return_metadata)
9394
else:
9495
return TextDoc(text=result)

0 commit comments

Comments
 (0)