[Bug] Orphaned Containers in Github Runners that Run Tests

[edlee123]

Priority

P3-Medium

OS type

Ubuntu

Hardware type

Xeon-GNR

Installation method

• Pull docker images from hub.docker.com
• Build docker images from source
• Other

Deploy method

• Docker
• Docker Compose
• Kubernetes Helm Charts
• Other

Running nodes

Single Node

What's the version?

NA

Description

Not all containers from CI tests are removed for subsequent CI tests e.g., tests/llms/test_llms_* . The impact is there might be container or port conflicts, OR some tests use containers from previous run.

I think the problem is the function stop_docker may not down all the orphan containers when using the ${service_name}:

function stop_docker() {
    cd $WORKPATH/comps/llms/deployment/docker_compose
    docker compose -f compose_text-generation.yaml down ${service_name} --remove-orphans
}

Usually it is the include containers from compose_text-generation.yaml that don't get taken down:

include:
  - ../../../third_parties/tgi/deployment/docker_compose/compose.yaml
  - ../../../third_parties/vllm/deployment/docker_compose/compose.yaml
  - ../../../third_parties/ollama/deployment/docker_compose/compose.yaml
  - ../../../third_parties/llamacpp/deployment/docker_compose/compose.yaml

I added docker ps -a to see this here example run. Please search logs 'Docker containers before stop_docker' and 'Docker containers after stop_docker' to see that containers are not taken down..

I did try leaving out the ${service_name}, below and it does help when I try locally i.e.:

docker compose -f compose_text-generation.yaml down --remove-orphans

To fix, we may need to amend the stop_docker() in the tests? Happy to discuss or clarify further.

Reproduce steps

example run (please search logs 'Docker containers before stop_docker' and 'Docker containers after stop_docker'

Can add docker ps -a before and after stop_docker.

Raw log

Attachments

No response

[xiguiw]

@edlee123

Thanks for catching this bug!
Let's find someone to fix it

[edlee123]

Hi @Brijeshthummar02, thank your help last time on the documentation. Would you be interested in this one (and if you have availability)?

[Brijeshthummar02]

@edlee123 Sure, would love to do it!

[Brijeshthummar02]

Just one more thing what is P3-Medium mean??

[edlee123]

Just one more thing what is P3-Medium mean??

It's one of the choices to rate how big an issue you think it is. P1 is biggest problem, and P3 is medium.

Please let me know if you need any help understanding the issue? TLDR: the docker compose -f compose_text-generation.yaml down ${service_name} --remove-orphans doesn't remove all the orphan containers before and after the tests run. However, I think docker compose -f compose_text-generation.yaml down --remove-orphans may work. You may have to change all the test scripts if you do it this way.

Another way I considered is docker ps -aq | xargs docker stop | xargs docker rm, but the CI runner has an injection safety check and won't allow such command.

[Brijeshthummar02]

@edlee123 Just a quick question do we need to clean not used network to after cleaning unused orphan containers

[Brijeshthummar02]

@edlee123 as you see image and containers are running fine now what should i start with, any tips to start fixing this?

[edlee123]

Hi @xiguiw , a volunteer @Brijeshthummar02 found 80+ test files that have the stop_docker function. Is it possible to remove the containers in "Clean Up Working Directory" CI step? e.g., docker ps -aq | xargs docker stop | xargs docker rm?
What are the pros and cons this way?

example run

[Brijeshthummar02]

@edlee123 , @xiguiw as best of me what i suggest we can have simply

Docker containers before stop_docker:
docker ps -a

Stopping services and removing orphaned containers...
docker compose -f compose.yaml down --remove-orphans
or in general
docker compose -f compose_text-generation.yaml down --remove-orphans || true

Removing stopped containers.......
docker container prune -f || true

Removing unused networks..
docker network prune -f || true

Docker containers after stop_docker:
docker ps -a

so maybe we can have something like this by changing all the stop_docker() function with above. Maybe you can have a better way to fix please mention this is initially my suggestions.