You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature] Integration test case with Data Science Pipeline, CodeFlare and KubeRay #425
Ray driver launches two Ray actors, deployed to a pod each. The first actor runs TfidfVectorizer, followed by Kmeans clustering and evaluation. The second actor runs HashingVectorizer, followed by Kmeans clustering and evaluation.
Ray driver collects evaluation results from the two actors. Then it reports the summaries.
Ray cluster is stopped and shutdown.
Pipeline run is completed.
Expected test assets:
DSP pipeline yaml to deploy and kick off test runs.
Test image with Ray and document clustering code.
CodeFlare image to deploy the test image.
Preconfigured credentials and configmaps in the test environment.
Activity
yuanchi2807 commentedon Dec 11, 2023
Cross posting from opendatahub-io/data-science-pipelines#179
A prototype following the above solution design can be found at this link.
https://github.com/yuanchi2807/dsp_codeflare_int_testing
Ray application image can be pulled from quay.io/yuanchichang_ibm/integration_testing/dsp_codeflare_int_testing:0.1
The pipeline definition yet_another_ray_integration_test.py is modified from https://github.com/diegolovison/ods-ci/blob/ray_integration/ods_ci/tests/Resources/Files/pipeline-samples/ray_integration.py to point to the custom image and invokes docker_clustering_driver.py through Ray jobs API.
Please feel free to comment.
anishasthana commentedon Dec 13, 2023
fyi @sutaakar
sutaakar commentedon Dec 13, 2023
On the first look it looks fine to me. I will try to run it this week.
Waiting for feedback from Diego, as he has more experience with Pipelines.
yuanchi2807 commentedon Dec 13, 2023
My prototype is to test the water and can be enhanced to lengthen the pipeline.