Skip to content

Commit 83d733e

Browse files
sutaakarsutaakar
committed
Introduce ADR for Project CodeFlare test strategy
Signed-off-by: Karel Suta <[email protected]>
1 parent 9105afa commit 83d733e

File tree

1 file changed

+266
-0
lines changed

1 file changed

+266
-0
lines changed

‎PCF-ADR-0004-codeflare-test-strategy.md

+266
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,266 @@
1+
# Define testing strategy for Project CodeFlare
2+
3+
| | |
4+
| -------------- | ------------------------ |
5+
| Date | 06/14/2023 |
6+
| Scope | |
7+
| Status | Proposed |
8+
| Authors | [Karel Suta](@sutaakar), [Antonin Stefanutti](@astefanutti) |
9+
| Supersedes | N/A |
10+
| Superseded by: | N/A |
11+
| Issues | |
12+
| Other docs: | [PCF-ADR-0003](https://github.com/project-codeflare/adr/blob/main/PCF-ADR-0003-codeflare-release-process.md) |
13+
14+
## What
15+
16+
This ADR introduces an overview of testing strategy and approach for Project CodeFlare.
17+
18+
## Why
19+
20+
Various components of Project CodeFlare currently handle their testing in individual and uncoordinated way. Additionally, there are no tests verifying interactions and compatibility between different components.
21+
This ADR aims to document a unified approach for testing of various components, extending the testing section of [Release process ADR](https://github.com/project-codeflare/adr/blob/main/PCF-ADR-0003-codeflare-release-process.md#testing).
22+
23+
## Goals
24+
25+
* Establish a common test approach for CodeFlare components
26+
* Define a testing scope and priorities
27+
* Analyze and decide test automation environment and tools
28+
29+
## Non-Goals
30+
31+
* Cover certification testing
32+
* Cover non functional testing (performance, security....)
33+
34+
## How
35+
36+
37+
### Scope
38+
39+
High priority components:
40+
* CodeFlare operator
41+
* KubeRay
42+
* MCAD
43+
* ODH Distributed Workloads Component
44+
45+
Lower priority components
46+
* CodeFlare SDK
47+
* MCAD dashboard
48+
* InstaScale
49+
50+
51+
### Test Approach
52+
53+
#### Test levels and requirements:
54+
55+
PR automated check:
56+
* Test execution speed is critical to provide fast feedback (less than 30 minutes)
57+
* Focusing on functional testing, avoid long running tests with lower value
58+
* Test composition (first execute unit tests, when unit tests pass then run e2e tests)
59+
* Use parallelization where possible to speed up execution
60+
61+
Nightly build check (to be implemented once nightly builds are available):
62+
* Relaxed test execution speed requirements (i.e. 2 hours)
63+
* Running complete test suite
64+
* Test also integration with ODH Distributed Workloads Component
65+
66+
Release build check:
67+
* Test speed is important to be able to provide a build fast (less than 1 hour):
68+
* Focusing on functional testing, avoid long running tests with lower value (tested by nightly checks)
69+
70+
#### Testing levels and Type of testing:
71+
72+
| Test type/development cycle | Feature developer | PR automated checks | PR review by reviewer | Nightly automated checks | Release automated checks |
73+
| ---------------------------- | ----------------- | -------------------- | --------------------- | ------------------------- | ------------------------ |
74+
| Manual testing | Yes | No | Yes | No | No |
75+
| Unit testing | Yes | Yes | No | Yes | Yes |
76+
| End to end testing | Yes | Yes | No | Yes | Yes |
77+
| Integration testing | Optional | Yes | No | Yes | Yes |
78+
| Upgrade testing | Optional | Yes | No | Yes | Yes |
79+
80+
Glossary:
81+
82+
Manual testing - Manual verification of the functionality.
83+
84+
Unit testing - Unit tests covering the base functionality.
85+
86+
End to end testing - Testing including all CodeFlare components, running in a cloud.
87+
88+
Integration testing - Testing of interactions between ODH components and CodeFlare components in a cloud.
89+
90+
Upgrade testing - Testing of component upgrades (deploy new operator version) in a cloud.
91+
92+
### Test Environment
93+
94+
Test environment depends on resource requirements of executed tests. For unit tests we will leverage GitHub actions runner. For end to end/integration/upgrade testing we will currently use KinD cluster running on GitHub actions runner. In case the resources provided by GitHub actions runner are not sufficient we can consider alternatives like [testing farm](https://docs.testing-farm.io/general/0.1/index.html).
95+
96+
For OpenShift testing we consider leveraging [OpenShift-CI](https://github.com/openshift/release/tree/master/ci-operator).
97+
98+
### Defect Management
99+
100+
#### Bug logging
101+
Bugs found by testing are reported as GitHub issues in respective repository. Every bug description has to contain steps to reproduce.
102+
103+
#### Bug fixing
104+
Bugs are going to be fixed in a similar approach as new features - planned as part of sprint planning, fixed through PRs. If possible the fix contains test coverage, preventing the issue from happening again.
105+
106+
107+
### Test Suite
108+
109+
#### E2E Test Cases
110+
111+
##### Setup
112+
113+
The following steps must be executed once, with a cluster admin role, as prerequisites to running the e2e test cases:
114+
1. Provision a test Kubernetes cluster (resp. an OpenShift cluster)
115+
2. Build the CodeFlare operator container image:
116+
* Clone the CodeFlare operator source code repository
117+
* Checkout the branch to be tested, e.g., the PR feature branch, i.e. `git clone --branch mytag0.1 --depth 1 https://example.com/my/repo.git`
118+
* Build the CodeFlare operator container image at HEAD revision
119+
* Push the CodeFlare operator container image into a test container image registry (resp. the OpenShift cluster internal image registry)
120+
3. Install the CodeFlare stack components into the test cluster:
121+
* Configure the CodeFlare operator deployment with Kustomize, to use the previously built container image
122+
* Create the codeflare-system Namespace (resp. Project)
123+
* Deploy the CodeFlare operator using Kustomize
124+
Note: the installation using OLM is covered as part of the installation / upgrade test cases
125+
* Deploy the KubeRay operator
126+
* Create a default MCAD resource
127+
* Grant the MCAD controller ServiceAccount edit permission on RayCluster resources
128+
* Create a default InstaScale resource
129+
Wait until all the components are ready
130+
The e2e test cases can now be executed, with a **standard user role**, ideally in parallel, to speed the execution time up and shorten the feedback loop as much as possible.
131+
> **Note**
132+
> The test cases should define lean / minimal compute resources, relative to the provisioned cluster, so parallel execution / throughput of the batch jobs submitted via the MCAD scheduling queue is maximized.
133+
134+
##### Submit a Sample PyTorch Job in a managed Ray Cluster
135+
136+
###### Description
137+
138+
Submit a test PyTorch batch job, to a Ray cluster managed by MCAD, in a user tenant, and assert successful completion of the job. Shutdown the Ray cluster, and assert successful freeing of resources.
139+
140+
###### Scenario
141+
142+
1. Create a test Namespace (resp. Project)
143+
2. Create a test AppWrapper resource with the following specifications: https://github.com/project-codeflare/multi-cluster-app-dispatcher/blob/14d569bec1cd016dd41352e3c026f461d851f480/doc/usage/examples/kuberay/config/aw-raycluster.yaml
144+
3. Wait until the Ray cluster is ready
145+
4. Submit the test batch job to the Ray cluster
146+
5. Wait until the job has completed
147+
6. Assert the job status is successful
148+
7. Delete the test AppWrapper resource
149+
8. Assert all the resources have been successfully freed
150+
151+
##### Submit a Sample PyTorch Job directly via MCAD
152+
153+
Lower priority.
154+
155+
###### Description
156+
157+
Submit a test PyTorch batch job to the MCAD scheduler, in a user tenant, and assert successful completion of the job. Assert successful execution of the job, and clean-up of resources upon the batch job completion.
158+
159+
###### Scenario
160+
161+
1. Create a test Namespace (resp. Project)
162+
2. Create an AppWrapper that allocates resources for running the sample PyTorch script as a batch Job
163+
3. Wait until the test job has completed
164+
4. Assert the job status is successful
165+
5. Delete the test AppWrapper resource
166+
6. Assert all the resources have been successfully freed
167+
168+
##### Run a Sample PyTorch Job in a Ray Cluster using the CodeFlare SDK
169+
170+
###### Description
171+
172+
In an isolated, vanilla Python environment, connected to the test Kubernetes cluster, and impersonated as a standard user, i.e., granted the `edit` role in the testing Namespace (resp. Project), submit a sample PyTorch job in a Ray cluster using the CodeFlare SDK.
173+
Assert successful execution of the job, and clean-up of resources upon the job completion.
174+
175+
###### Scenario
176+
177+
1. Install the CodeFlare SDK, e.g., with `pip install codeflare_sdk==<sdk_version>`
178+
2. Create a test Namespace (resp. Project)
179+
3. Create a Ray cluster using the SDK Cluster API in this namespace
180+
4. Wait until the Ray Cluster is ready
181+
5. Create a job using the SDK Job API, by providing the sample job pip requirements and Python script
182+
6. Submit the job into the previously created Ray cluster
183+
7. Wait until test job has completed
184+
8. Assert the job status is successful
185+
9. Retrieve the job logs using the SDK Job API
186+
10. Tear down the ray cluster using the SDK Cluster API
187+
11. Assert all the resources have been successfully freed
188+
189+
#### OLM Installation / Upgrade Test Cases
190+
191+
##### Setup
192+
193+
The following steps must be executed once, with a **cluster admin role**, as prerequisites to running the OLM installation / upgrade test cases:
194+
195+
1. Provision a test Kubernetes cluster (resp. an OpenShift cluster)
196+
2. For Kubernetes cluster only: install OLM using [installation instructions](https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/install/install.md)
197+
3. Build the CodeFlare operator OLM bundle:
198+
* Clone the CodeFlare operator source code repository
199+
* Checkout the branch to be tested, e.g., the PR feature branch, i.e. `git clone --branch mytag0.1 --depth 1 https://example.com/my/repo.git`
200+
* Build the CodeFlare operator container image at HEAD revision
201+
* Push the CodeFlare operator container image into a test container image registry (resp. the OpenShift cluster internal image registry)
202+
* Build the OLM bundle image
203+
* Push the OLM bundle image into a test container image registry (resp. the OpenShift cluster internal image registry)
204+
4. Build the OLM index image:
205+
* Build the OLM index/catalog image with OPM from the latest Operator Hub community catalog (resp. OpenShift catalog)
206+
207+
> **Note**
208+
> the above should be done according to https://github.com/operator-framework/operator-sdk/issues/5832 when available.
209+
210+
> **Note**
211+
> the channel must be adapted according to https://github.com/project-codeflare/codeflare-operator/issues/126.
212+
213+
* Push the OLM index/catalog image that’s been built at the previous step to the test container image registry (resp. The OpenShift cluster internal container image registry)
214+
215+
216+
The test cases can now be executed, with a **cluster admin role**, in sequence, as the upgrade test case depends on the installation one.
217+
218+
##### Installation of the CodeFlare Operator with OLM
219+
220+
###### Description
221+
222+
Install the latest released version of the CodeFlare operator using OLM, and assert successful deployment of the operator. Run a smoke test, to make sure the current version of the operator is working as expected.
223+
224+
###### Scenario
225+
226+
1. Create a codeflare-system test Namespace (resp. Project)
227+
2. Create a CatalogSource resource pointing to the latest release index image
228+
3. Create a Subscription resource, that points to the CatalogSource resource created at the previous step
229+
4. Run a smoke test, e.g. by running one of the e2e test cases, to make sure the operator current version is working correctly.
230+
231+
##### Upgrade of the CodeFlare Operator installed via OLM
232+
233+
###### Description
234+
235+
Add a newer version of the CodeFlare operator bundle, replacing the latest released version from the channel subscribed to during the Installation of the CodeFlare Operator with OLM test case. Assert successful upgrade of the operator to the newer version, and run a smoke test, to make sure that newer version is working as expected.
236+
237+
###### Scenario
238+
239+
> **Note**
240+
> Note: Make sure to execute the Installation of the CodeFlare Operator with OLM test case first.
241+
242+
1. In the same codeflare-system Namespace (resp. Project), update the existing test-catalog CatalogSource resource, by pointing to the newly built OLM index image
243+
2. Assert a new ClusterServiceVersion resource has been created for the newer operator version, and eventually reaches the CSVPhaseSucceeded phase
244+
3. Run a smoke test, e.g., by running one of the e2e test cases, to make sure the operator current version is working correctly.
245+
246+
## Open Questions
247+
248+
1. [Add observability test cases](https://github.com/project-codeflare/codeflare-operator/issues/158)
249+
2. Add integration test cases
250+
251+
## Alternatives
252+
253+
We didn't consider any other alternatives
254+
255+
## Stakeholder Impacts
256+
257+
| Group | Key Contacts | Date | Impacted? |
258+
| ----------------------------- | ------------------ | ---------- | --------- |
259+
| CodeFlare SDK | Mustafa Eyceoz | | yes |
260+
| MCAD | Abhishek Malvankar | | yes |
261+
| InstaScale | Abhishek Malvankar | | yes |
262+
| CodeFlare Operator | Anish Asthana | | yes |
263+
264+
## Reviews
265+
266+
Reviews on the pull request will suffice for the approval process. At least 2 approvals are required prior to this ADR being merged. The ADR must also remain open for at least one week.

0 commit comments

Comments
 (0)