Add redis-finance to dataprep (opea-project#1384)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* redis finance code

Signed-off-by: minmin-intel <[email protected]>

* add two new packages

Signed-off-by: minmin-intel <[email protected]>

* fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* debug ci error

Signed-off-by: minmin-intel <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix errors

Signed-off-by: minmin-intel <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add readme and rename test script to run ci on gaudi

Signed-off-by: minmin-intel <[email protected]>

---------

Signed-off-by: minmin-intel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: minmin-intel

comps/dataprep/README.md

@@ -56,3 +56,7 @@ For details, please refer to this [readme](src/README_opensearch.md)
 ## Dataprep Microservice with neo4j
 
 For details, please refer to this [readme](src/README_neo4j_llamaindex.md)
+
+## Dataprep Microservice for financial domain data
+
+For details, please refer to this [readme](src/README_finance.md)

comps/dataprep/deployment/docker_compose/compose.yaml

@@ -286,6 +286,32 @@ services:
       COLLECTION_NAME: ${INDEX_NAME}
     restart: unless-stopped
 
+  dataprep-redis-finance:
+    image: ${REGISTRY:-opea}/dataprep:${TAG:-latest}
+    container_name: dataprep-redis-server-finance
+    depends_on:
+      redis-vector-db:
+        condition: service_healthy
+      redis-kv-store:
+        condition: service_healthy
+      tei-embedding-serving:
+        condition: service_healthy
+    ports:
+      - "${DATAPREP_PORT:-11108}:5000"
+    environment:
+      no_proxy: ${no_proxy}
+      http_proxy: ${http_proxy}
+      https_proxy: ${https_proxy}
+      DATAPREP_COMPONENT_NAME: ${DATAPREP_COMPONENT_NAME}
+      REDIS_URL_VECTOR: ${REDIS_URL_VECTOR}
+      REDIS_URL_KV: ${REDIS_URL_KV}
+      TEI_EMBEDDING_ENDPOINT: ${TEI_EMBEDDING_ENDPOINT}
+      LLM_ENDPOINT: ${LLM_ENDPOINT}
+      LLM_MODEL: ${LLM_MODEL}
+      HUGGINGFACEHUB_API_TOKEN: ${HF_TOKEN}
+      HF_TOKEN: ${HF_TOKEN}
+      LOGFLAG: true
+
 networks:
   default:
     driver: bridge

comps/dataprep/src/README_finance.md

@@ -0,0 +1,101 @@
+# Dataprep microservice for financial domain data
+
+## 1. Overview
+
+We currently support ingestion of PDFs and URL links. The data should be financial domain, such as SEC filings and earnings call transcripts. If the data is not financial domain, you may encounter accuracy problems or errors.
+
+The dataprep microservice saves financial documents into two databases:
+
+1. One vector database of text chunks and tables
+2. One KV store of full-length documents
+
+Each unique company has its own index within the vector database and KV store, and metadata such as year, quarter, doc_title, doc_type and source are stored to enable metadata filtering to improve retrieval precision and recall. A company list is maintained so that when a new document comes, the document will either be mapped to an existing company or be added as a new company.
+
+An LLM is used in processing the documents, including extracting metadata and generating summaries for text chunks and tables, and deciding if the document is about an existing company in the knowledge base or not. The default LLM to be used is `meta-llama/Llama-3.3-70B-Instruct`.
+
+## 2. Deploy with docker
+
+### 2.1 Start Redis vector database and Redis KV store
+
+```bash
+docker run --name redis-db -p 6379:6379 -p 8001:8001 -d redis/redis-stack:7.2.0-v9
+docker run --name redis-kv -p 6380:6379 -p 8002:8001 -d redis/redis-stack:7.2.0-v9
+```
+
+### 2.2 Start Embedding Service
+
+First, you need to start a TEI service.
+
+```bash
+your_port=6006
+model="BAAI/bge-base-en-v1.5"
+docker run -p $your_port:80 -v ./data:/data --name tei_server -e http_proxy=$http_proxy -e https_proxy=$https_proxy --pull always ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 --model-id $model
+```
+
+Then you need to test your TEI service using the following commands:
+
+```bash
+curl localhost:$your_port/embed \
+    -X POST \
+    -d '{"inputs":"What is Deep Learning?"}' \
+    -H 'Content-Type: application/json'
+```
+
+### 2.3 Start vllm endpoint
+
+First build vllm-gaudi docker image.
+
+```bash
+cd $WORKDIR
+git clone https://github.com/HabanaAI/vllm-fork.git
+# get the latest release tag of vllm gaudi
+VLLM_VER=$(git describe --tags "$(git rev-list --tags --max-count=1)")
+echo "Check out vLLM tag ${VLLM_VER}"
+git checkout ${VLLM_VER}
+docker build --no-cache -f Dockerfile.hpu -t opea/vllm-gaudi:latest --shm-size=128g . --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy
+```
+
+Then launch vllm on Gaudi.
+
+```bash
+export vllm_port=8086
+export vllm_volume=$HF_CACHE_DIR
+export max_length=16384
+export model="meta-llama/Llama-3.3-70B-Instruct"
+export HF_TOKEN=<your-hf-token>
+docker run -d --runtime=habana --rm --name "vllm-gaudi-server" -e HABANA_VISIBLE_DEVICES=all -p $vllm_port:8000 -v $vllm_volume:/data -e HF_TOKEN=$HF_TOKEN -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN -e HF_HOME=/data -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e PT_HPU_ENABLE_LAZY_COLLECTIVES=true -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e no_proxy=$no_proxy -e VLLM_SKIP_WARMUP=true --cap-add=sys_nice --ipc=host opea/vllm-gaudi:comps --model ${model} --max-seq-len-to-capture $max_length --tensor-parallel-size 4
+```
+
+### 2.4 Build Docker Image for dataprep microservice
+
+```bash
+cd ../../ # go to GenAIComps
+docker build -t opea/dataprep:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/src/Dockerfile .
+```
+
+### 2.5 Start dataprep microservice
+
+```bash
+export ip_address=$(hostname -I | awk '{print $1}')
+export REDIS_URL_VECTOR="redis://${ip_address}:6379"
+export REDIS_URL_KV="redis://${ip_address}:6380"
+export LLM_MODEL="meta-llama/Llama-3.3-70B-Instruct"
+export LLM_ENDPOINT="http://${ip_address}:8086"
+export TEI_EMBEDDING_ENDPOINT="http://${your_ip}:6006"
+export DATAPREP_COMPONENT_NAME="OPEA_DATAPREP_REDIS_FIANANCE"
+export HUGGINGFACEHUB_API_TOKEN=<your-hf-token>
+```
+
+```bash
+docker run -d --name="dataprep-redis-server-finance" -p 6007:5000 --runtime=runc --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e REDIS_URL_VECTOR=$REDIS_URL_VECTOR -e REDIS_URL_KV=$REDIS_URL_KV -e LLM_MODEL=$LLM_MODEL -e LLM_ENDPOINT=$LLM_ENDPOINT -e TEI_EMBEDDING_ENDPOINT=$TEI_EMBEDDING_ENDPOINT -e HUGGINGFACEHUB_API_TOKEN=$HUGGINGFACEHUB_API_TOKEN -e HF_TOKEN=$HUGGINGFACEHUB_API_TOKEN -e DATAPREP_COMPONENT_NAME=$DATAPREP_COMPONENT_NAME opea/dataprep:latest
+```
+
+### 2.6 Check the status of dataprep microservice
+
+```bash
+docker container logs -f dataprep-redis-server-finance
+```
+
+## 3. Consume Microservice
+
+See example python script [here](../../../tests/dataprep/test_redis_finance.py).