Skip to content

Commit 191061b

Browse files
sharanshirodkar7sharanshirodkar7
authored
Prediction Guard embeddings component (#675)
* added files for PG embeddingso component Signed-off-by: sharanshirodkar7 <[email protected]> * added package Signed-off-by: sharanshirodkar7 <[email protected]> * fixed dockerfile link Signed-off-by: sharanshirodkar7 <[email protected]> * Fix pre-commit issues: end-of-file, requirements.txt, trailing whitespace, imports, and formatting Signed-off-by: sharanshirodkar7 <[email protected]> * added package Signed-off-by: sharanshirodkar7 <[email protected]> * added package Signed-off-by: sharanshirodkar7 <[email protected]> * fixed embedoc call Signed-off-by: sharanshirodkar7 <[email protected]> * file structure updated to latest Signed-off-by: sharanshirodkar7 <[email protected]> * Fix pre-commit issues: end-of-file, requirements.txt, trailing whitespace, imports, and formatting Signed-off-by: sharanshirodkar7 <[email protected]> * added package Signed-off-by: sharanshirodkar7 <[email protected]> --------- Signed-off-by: sharanshirodkar7 <[email protected]>
1 parent b4a7f26 commit 191061b

File tree

8 files changed

+218
-0
lines changed

8 files changed

+218
-0
lines changed

‎.github/workflows/docker/compose/embeddings-compose-cd.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,3 +26,7 @@ services:
2626
build:
2727
dockerfile: comps/embeddings/multimodal/multimodal_langchain/Dockerfile
2828
image: ${REGISTRY:-opea}/embedding-multimodal:${TAG:-latest}
29+
embedding-predictionguard:
30+
build:
31+
dockerfile: comps/embeddings/predictionguard/Dockerfile
32+
image: ${REGISTRY:-opea}/embedding-predictionguard:${TAG:-latest}

‎comps/embeddings/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,3 +31,7 @@ For details, please refer to this [readme](multimodal/README.md).
3131
## Embeddings Microservice with Multimodal Clip
3232

3333
For details, please refer to this [readme](multimodal_clip/README.md).
34+
35+
## Embeddings Microservice with Prediction Guard
36+
37+
For details, please refer to this [readme](predictionguard/README.md).

‎comps/embeddings/predictionguard/Dockerfile

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Copyright (C) 2024 Prediction Guard, Inc
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
FROM python:3.11-slim
5+
6+
COPY comps /home/comps
7+
8+
RUN pip install --no-cache-dir --upgrade pip && \
9+
pip install --no-cache-dir -r /home/comps/embeddings/predictionguard/requirements.txt
10+
11+
ENV PYTHONPATH=$PYTHONPATH:/home
12+
13+
WORKDIR /home/comps/embeddings/predictionguard
14+
15+
ENTRYPOINT ["python", "embedding_predictionguard.py"]
16+

‎comps/embeddings/predictionguard/README.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Embedding Generation Prediction Guard Microservice
2+
3+
[Prediction Guard](https://docs.predictionguard.com) allows you to utilize hosted open access LLMs, LVMs, and embedding functionality with seamlessly integrated safeguards. In addition to providing a scalable access to open models, Prediction Guard allows you to configure factual consistency checks, toxicity filters, PII filters, and prompt injection blocking. Join the [Prediction Guard Discord channel](https://discord.gg/TFHgnhAFKd) and request an API key to get started.
4+
5+
This embedding microservice is designed to efficiently convert text into vectorized embeddings using the [BridgeTower model](https://huggingface.co/BridgeTower/bridgetower-large-itm-mlm-itc). Thus, it is ideal for both RAG or semantic search applications.
6+
7+
**Note** - The BridgeTower model implemented in Prediction Guard can actually embed text, images, or text + images (jointly). For now this service only embeds text, but a follow on contribution will enable the multimodal functionality.
8+
9+
# 🚀 Start Microservice with Docker
10+
11+
## Setup Environment Variables
12+
13+
Setup the following environment variables first
14+
15+
```bash
16+
export PREDICTIONGUARD_API_KEY=${your_predictionguard_api_key}
17+
```
18+
19+
## Build Docker Images
20+
21+
```bash
22+
cd ../../..
23+
docker build -t opea/embedding-predictionguard:latest -f comps/embeddings/predictionguard/Dockerfile .
24+
```
25+
26+
## Start Service
27+
28+
```bash
29+
docker run -d --name="embedding-predictionguard" -p 6000:6000 -e PREDICTIONGUARD_API_KEY=$PREDICTIONGUARD_API_KEY opea/embedding-predictionguard:latest
30+
```
31+
32+
# 🚀 Consume Embeddings Service
33+
34+
```bash
35+
curl localhost:6000/v1/embeddings \
36+
-X POST \
37+
-d '{"text":"Hello, world!"}' \
38+
-H 'Content-Type: application/json'
39+
```

‎comps/embeddings/predictionguard/docker_compose_embedding.yaml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Copyright (C) 2024 Prediction Guard, Inc
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
services:
5+
embedding:
6+
image: opea/embedding-predictionguard:latest
7+
container_name: embedding-predictionguard
8+
ports:
9+
- "6000:6000"
10+
ipc: host
11+
environment:
12+
no_proxy: ${no_proxy}
13+
http_proxy: ${http_proxy}
14+
https_proxy: ${https_proxy}
15+
PG_EMBEDDING_MODEL_NAME: ${PG_EMBEDDING_MODEL_NAME}
16+
PREDICTIONGUARD_API_KEY: ${PREDICTIONGUARD_API_KEY}
17+
restart: unless-stopped
18+
19+
networks:
20+
default:
21+
driver: bridge

‎comps/embeddings/predictionguard/embedding_predictionguard.py

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Copyright (C) 2024 Prediction Guard, Inc.
2+
# SPDX-License-Identified: Apache-2.0
3+
4+
5+
import os
6+
import time
7+
8+
from predictionguard import PredictionGuard
9+
10+
from comps import (
11+
EmbedDoc,
12+
ServiceType,
13+
TextDoc,
14+
opea_microservices,
15+
register_microservice,
16+
register_statistics,
17+
statistics_dict,
18+
)
19+
20+
# Initialize Prediction Guard client
21+
client = PredictionGuard()
22+
23+
24+
@register_microservice(
25+
name="opea_service@embedding_predictionguard",
26+
service_type=ServiceType.EMBEDDING,
27+
endpoint="/v1/embeddings",
28+
host="0.0.0.0",
29+
port=6000,
30+
input_datatype=TextDoc,
31+
output_datatype=EmbedDoc,
32+
)
33+
@register_statistics(names=["opea_service@embedding_predictionguard"])
34+
def embedding(input: TextDoc) -> EmbedDoc:
35+
start = time.time()
36+
response = client.embeddings.create(model=pg_embedding_model_name, input=[{"text": input.text}])
37+
embed_vector = response["data"][0]["embedding"]
38+
embed_vector = embed_vector[:512] # Keep only the first 512 elements
39+
res = EmbedDoc(text=input.text, embedding=embed_vector)
40+
statistics_dict["opea_service@embedding_predictionguard"].append_latency(time.time() - start, None)
41+
return res
42+
43+
44+
if __name__ == "__main__":
45+
pg_embedding_model_name = os.getenv("PG_EMBEDDING_MODEL_NAME", "bridgetower-large-itm-mlm-itc")
46+
print("Prediction Guard Embedding initialized.")
47+
opea_microservices["opea_service@embedding_predictionguard"].start()

‎comps/embeddings/predictionguard/requirements.txt

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
aiohttp
2+
docarray
3+
fastapi
4+
opentelemetry-api
5+
opentelemetry-exporter-otlp
6+
opentelemetry-sdk
7+
Pillow
8+
predictionguard==2.2.1
9+
prometheus-fastapi-instrumentator
10+
PyYAML
11+
shortuuid
12+
uvicorn

‎tests/embeddings/test_embeddings_predictionguard.sh

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
#!/bin/bash
2+
# Copyright (C) 2024 Intel Corporation
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
set -x
6+
7+
WORKPATH=$(dirname "$PWD")
8+
ip_address=$(hostname -I | awk '{print $1}') # Adjust to a more reliable command
9+
if [ -z "$ip_address" ]; then
10+
ip_address="localhost" # Default to localhost if IP address is empty
11+
fi
12+
13+
function build_docker_images() {
14+
cd $WORKPATH
15+
echo $(pwd)
16+
docker build --no-cache -t opea/embedding-pg:comps -f comps/embeddings/predictionguard/Dockerfile .
17+
if [ $? -ne 0 ]; then
18+
echo "opea/embedding-pg built fail"
19+
exit 1
20+
else
21+
echo "opea/embedding-pg built successfully"
22+
fi
23+
}
24+
25+
function start_service() {
26+
tei_service_port=6000
27+
unset http_proxy
28+
docker run -d --name=test-comps-embedding-pg-server \
29+
-e http_proxy= -e https_proxy= \
30+
-e PREDICTIONGUARD_API_KEY=${PREDICTIONGUARD_API_KEY} \
31+
-p 6000:6000 --ipc=host opea/embedding-pg:comps
32+
sleep 60 # Sleep for 1 minute to allow the service to start
33+
}
34+
35+
function validate_microservice() {
36+
tei_service_port=6000
37+
result=$(http_proxy="" curl http://${ip_address}:${tei_service_port}/v1/embeddings \
38+
-X POST \
39+
-d '{"text":"What is Deep Learning?"}' \
40+
-H 'Content-Type: application/json')
41+
42+
# Check for a proper response format
43+
if [[ $result == *"embedding"* ]]; then
44+
echo "Result correct."
45+
elif [[ $result == *"error"* || $result == *"detail"* ]]; then
46+
echo "Result wrong. Error received was: $result"
47+
docker logs test-comps-embedding-pg-server
48+
exit 1
49+
else
50+
echo "Unexpected result format received was: $result"
51+
docker logs test-comps-embedding-pg-server
52+
exit 1
53+
fi
54+
}
55+
56+
function stop_docker() {
57+
cid=$(docker ps -aq --filter "name=test-comps-embedding-pg-*")
58+
if [[ ! -z "$cid" ]]; then docker stop $cid && docker rm $cid && sleep 1s; fi
59+
}
60+
61+
function main() {
62+
63+
stop_docker
64+
65+
build_docker_images
66+
start_service
67+
68+
validate_microservice
69+
70+
stop_docker
71+
echo y | docker system prune
72+
73+
}
74+
75+
main

0 commit comments

Comments
 (0)