Skip to content

Commit 0379f65

Browse files
sutaakarsutaakar
committed
Introduce ADR for Project CodeFlare test strategy
Signed-off-by: Karel Suta <[email protected]>
1 parent 9105afa commit 0379f65

File tree

1 file changed

+376
-0
lines changed

1 file changed

+376
-0
lines changed

‎PCF-ADR-0004-codeflare-test-strategy.md

Lines changed: 376 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,376 @@
1+
# Define testing strategy for Project CodeFlare
2+
3+
| | |
4+
| -------------- | ------------------------ |
5+
| Date | 06/14/2023 |
6+
| Scope | |
7+
| Status | Proposed |
8+
| Authors | [Karel Suta](@sutaakar), [Antonin Stefanutti](@astefanutti) |
9+
| Supersedes | N/A |
10+
| Superseded by: | N/A |
11+
| Issues | |
12+
| Other docs: | [PCF-ADR-0003](https://github.com/project-codeflare/adr/blob/main/PCF-ADR-0003-codeflare-release-process.md) |
13+
14+
## What
15+
16+
This ADR introduces an overview of testing strategy and approach for Project CodeFlare.
17+
18+
## Why
19+
20+
Various components of Project CodeFlare currently handle their testing in individual and uncoordinated way. Additionally, there are no tests verifying interactions and compatibility between different components.
21+
This ADR aims to document a unified approach for testing of various components, extending the testing section of [Release process ADR](https://github.com/project-codeflare/adr/blob/main/PCF-ADR-0003-codeflare-release-process.md#testing).
22+
23+
## Goals
24+
25+
* Establish a common test approach for CodeFlare components
26+
* Define a testing scope and priorities
27+
* Analyze and decide test automation environment and tools
28+
29+
## Non-Goals
30+
31+
* Cover certification testing
32+
* Cover non functional testing (performance, security....)
33+
34+
## How
35+
36+
37+
### Scope
38+
39+
High priority components:
40+
* CodeFlare operator
41+
* KubeRay
42+
* MCAD
43+
* ODH Distributed Workloads Component
44+
45+
Lower priority components
46+
* CodeFlare SDK
47+
* MCAD dashboard
48+
* InstaScale
49+
50+
51+
### Test Approach
52+
53+
#### Test levels and requirements:
54+
55+
PR automated check:
56+
* Test execution speed is critical to provide fast feedback (less than 30 minutes)
57+
* Focusing on functional testing, avoid long running tests with lower value
58+
* Test composition (first execute unit tests, when unit tests pass then run e2e tests)
59+
* Use parallelization where possible to speed up execution
60+
61+
Nightly build check (to be implemented once nightly builds are available):
62+
* Relaxed test execution speed requirements (i.e. 2 hours)
63+
* Running complete test suite
64+
* Test also integration with ODH Distributed Workloads Component
65+
66+
Release build check:
67+
* Test speed is important to be able to provide a build fast (less than 1 hour):
68+
* Focusing on functional testing, avoid long running tests with lower value (tested by nightly checks)
69+
70+
#### Testing levels and Type of testing:
71+
72+
| Test type/development cycle | Feature developer | PR automated checks | PR review by reviewer | Nightly automated checks | Release automated checks |
73+
| ---------------------------- | ----------------- | -------------------- | --------------------- | ------------------------- | ------------------------ |
74+
| Manual testing | Yes | No | Yes | No | No |
75+
| Unit testing | Yes | Yes | No | Yes | Yes |
76+
| End to end testing | Yes | Yes | No | Yes | Yes |
77+
| Integration testing | Optional | Yes | No | Yes | Yes |
78+
| Upgrade testing | Optional | Yes | No | Yes | Yes |
79+
80+
Glossary:
81+
82+
Manual testing - Manual verification of the functionality.
83+
84+
Unit testing - Unit tests covering the base functionality.
85+
86+
End to end testing - Testing including all CodeFlare components, running in a cloud.
87+
88+
Integration testing - Testing of interactions between ODH components and CodeFlare components in a cloud.
89+
90+
Upgrade testing - Testing of component upgrades (deploy new operator version) in a cloud.
91+
92+
### Test Environment
93+
94+
Test environment depends on resource requirements of executed tests. For unit tests we will leverage GitHub actions runner. For end to end/integration/upgrade testing we will currently use KinD cluster running on GitHub actions runner. In case the resources provided by GitHub actions runner are not sufficient we can consider alternatives like [testing farm](https://docs.testing-farm.io/general/0.1/index.html).
95+
96+
For OpenShift testing we consider leveraging [OpenShift-CI](https://github.com/openshift/release/tree/master/ci-operator).
97+
98+
### Defect Management
99+
100+
#### Bug logging
101+
Bugs found by testing are reported as GitHub issues in respective repository. Every bug description has to contain steps to reproduce.
102+
103+
#### Bug fixing
104+
Bugs are going to be fixed in a similar approach as new features - planned as part of sprint planning, fixed through PRs. If possible the fix contains test coverage, preventing the issue from happening again.
105+
106+
107+
### Test Suite
108+
109+
#### E2E Test Cases
110+
111+
##### Setup
112+
113+
The following steps must be executed once, with a cluster admin role, as prerequisites to running the e2e test cases:
114+
1. Provision a test Kubernetes cluster (resp. an OpenShift cluster)
115+
2. Build the CodeFlare operator container image:
116+
* Clone the CodeFlare operator source code repository
117+
* Checkout the branch to be tested, e.g., the PR feature branch, i.e. `git clone --branch mytag0.1 --depth 1 https://example.com/my/repo.git`
118+
* Build the CodeFlare operator container image at HEAD revision
119+
* Push the CodeFlare operator container image into a test container image registry (resp. the OpenShift cluster internal image registry)
120+
3. Install the CodeFlare stack components into the test cluster:
121+
* Configure the CodeFlare operator deployment with Kustomize, to use the previously built container image
122+
* Create the codeflare-system Namespace (resp. Project)
123+
* Deploy the CodeFlare operator using Kustomize
124+
Note: the installation using OLM is covered as part of the installation / upgrade test cases
125+
* Deploy the KubeRay operator
126+
* Create a default MCAD resource
127+
* Grant the MCAD controller ServiceAccount edit permission on RayCluster resources
128+
* Create a default InstaScale resource
129+
Wait until all the components are ready
130+
The e2e test cases can now be executed, with a **standard user role**, ideally in parallel, to speed the execution time up and shorten the feedback loop as much as possible.
131+
> **Note**
132+
> The test cases should define lean / minimal compute resources, relative to the provisioned cluster, so parallel execution / throughput of the batch jobs submitted via the MCAD scheduling queue is maximized.
133+
134+
##### Submit a Sample PyTorch Job in a managed Ray Cluster
135+
136+
###### Description
137+
138+
Submit a test PyTorch batch job, to a Ray cluster managed by MCAD, in a user tenant, and assert successful completion of the job. Shutdown the Ray cluster, and assert successful freeing of resources.
139+
140+
###### Scenario
141+
142+
1. Create a test Namespace (resp. Project)
143+
2. Create a test AppWrapper resource with the following specifications: https://github.com/project-codeflare/multi-cluster-app-dispatcher/blob/14d569bec1cd016dd41352e3c026f461d851f480/doc/usage/examples/kuberay/config/aw-raycluster.yaml
144+
3. Wait until the Ray cluster is ready
145+
4. Submit the test batch job to the Ray cluster
146+
5. Wait until the job has completed
147+
6. Assert the job status is successful
148+
7. Delete the test AppWrapper resource
149+
8. Assert all the resources have been successfully freed
150+
151+
##### Submit a Sample PyTorch Job directly via MCAD
152+
153+
Lower priority.
154+
155+
###### Description
156+
157+
Submit a test PyTorch batch job to the MCAD scheduler, in a user tenant, and assert successful completion of the job. Assert successful execution of the job, and clean-up of resources upon the batch job completion.
158+
159+
###### Scenario
160+
161+
1. Create a test Namespace (resp. Project)
162+
2. Create a test AppWrapper resource with the following specifications:
163+
```
164+
apiVersion: mcad.ibm.com/v1beta1
165+
kind: AppWrapper
166+
spec:
167+
resources:
168+
GenericItems:
169+
- allocated: 0
170+
generictemplate:
171+
apiVersion: v1
172+
kind: Pod
173+
metadata:
174+
spec:
175+
containers:
176+
- command:
177+
- bash
178+
- '-c'
179+
- >-
180+
torchrun --rdzv_backend c10d --rdzv_endpoint
181+
$TORCHX_RANK0_HOST:49782 --rdzv_id 'test-job'
182+
--nnodes 1 --nproc_per_node 1 --node_rank '0' --tee 3 --role
183+
'' mnist.py
184+
env:
185+
- name: TORCHX_TRACKING_EXPERIMENT_NAME
186+
value: default-experiment
187+
- name: LOGLEVEL
188+
value: WARNING
189+
- name: TORCHX_JOB_ID
190+
value: 'kubernetes_mcad://torchx/test-job'
191+
- name: TORCHX_RANK0_HOST
192+
value: localhost
193+
- name: TORCHX_MCAD_MNIST_0_HOSTS
194+
value: test-job
195+
image: 'quay.io/michaelclifford/mnist-test:latest'
196+
name: test-job
197+
ports:
198+
- containerPort: 29500
199+
name: c10d
200+
resources:
201+
limits:
202+
cpu: 1000m
203+
memory: 4000M
204+
requests:
205+
cpu: 900m
206+
memory: 2976M
207+
volumeMounts:
208+
- mountPath: /dev/shm
209+
name: dshm
210+
hostname: test-job
211+
restartPolicy: Never
212+
subdomain: test-job
213+
volumes:
214+
- emptyDir:
215+
medium: Memory
216+
name: dshm
217+
priority: 0
218+
priorityslope: 0
219+
replicas: 1
220+
- allocated: 0
221+
generictemplate:
222+
apiVersion: v1
223+
kind: Service
224+
metadata:
225+
name: test-job
226+
spec:
227+
clusterIP: None
228+
ports:
229+
- port: 29500
230+
protocol: TCP
231+
targetPort: 29500
232+
publishNotReadyAddresses: true
233+
selector:
234+
appwrapper.mcad.ibm.com: test-job
235+
sessionAffinity: None
236+
type: ClusterIP
237+
schedulingSpec:
238+
requeuing:
239+
growthType: exponential
240+
maxNumRequeuings: 0
241+
maxTimeInSeconds: 0
242+
numRequeuings: 0
243+
timeInSeconds: 300
244+
```
245+
246+
3. Wait until the test job has completed
247+
4. Assert the job status is successful
248+
5. Delete the test AppWrapper resource
249+
6. Assert all the resources have been successfully freed
250+
251+
#### OLM Installation / Upgrade Test Cases
252+
253+
##### Setup
254+
255+
The following steps must be executed once, with a **cluster admin role**, as prerequisites to running the OLM installation / upgrade test cases:
256+
257+
1. Provision a test Kubernetes cluster (resp. an OpenShift cluster)
258+
2. For Kubernetes cluster only: install OLM by executing the following commands:
259+
```
260+
$ kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.24.0/crds.yaml
261+
$ kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.24.0/olm.yaml
262+
```
263+
3. Build the CodeFlare operator OLM bundle:
264+
* Clone the CodeFlare operator source code repository
265+
* Checkout the branch to be tested, e.g., the PR feature branch, i.e. `git clone --branch mytag0.1 --depth 1 https://example.com/my/repo.git`
266+
* Build the CodeFlare operator container image at HEAD revision
267+
* Push the CodeFlare operator container image into a test container image registry (resp. the OpenShift cluster internal image registry)
268+
* Build the OLM bundle image
269+
* Push the OLM bundle image into a test container image registry (resp. the OpenShift cluster internal image registry)
270+
271+
4. Build the OLM index image:
272+
* Build the OLM index/catalog image with OPM from the latest Operator Hub community catalog (resp. OpenShift catalog), e.g.:
273+
```
274+
$ mkdir catalog
275+
$ opm render registry.redhat.io/redhat/redhat-operator-index:<OCP_VERSION> -o yaml > catalog/bundles.yaml
276+
$ opm render $BUNDLE_IMAGE > catalog/codeflare-operator-bundle.yaml
277+
$ sed -i -E "s/(.*)(- name: codeflare-operator.$(PREVIOUS_VERSION).*)/\1- name: codeflare-operator.$(VERSION)\n replaces: codeflare-operator.$(PREVIOUS_VERSION)\n\2/" catalog/bundles.yaml
278+
$ opm validate catalog
279+
$ opm generate dockerfile catalog
280+
$ podman build . -f catalog.Dockerfile -t <CATALOG_IMG>
281+
```
282+
283+
> **Note**
284+
> the above should be done according to https://github.com/operator-framework/operator-sdk/issues/5832 when available.
285+
286+
> **Note**
287+
> the channel must be adapted according to https://github.com/project-codeflare/codeflare-operator/issues/126.
288+
289+
* Push the OLM index/catalog image that’s been built at the previous step to the test container image registry (resp. The OpenShift cluster internal container image registry)
290+
291+
292+
The test cases can now be executed, with a **cluster admin role**, in sequence, as the upgrade test case depends on the installation one.
293+
294+
##### Installation of the CodeFlare Operator with OLM
295+
296+
###### Description
297+
298+
Install the latest released version of the CodeFlare operator using OLM, and assert successful deployment of the operator. Run a smoke test, to make sure the current version of the operator is working as expected.
299+
300+
###### Scenario
301+
302+
1. Create a codeflare-system test Namespace (resp. Project)
303+
2. Create a CatalogSource resource with the following specifications:
304+
```
305+
apiVersion: operators.coreos.com/v1alpha1
306+
kind: CatalogSource
307+
metadata:
308+
name: test-catalog
309+
spec:
310+
displayName: CodeFlare OLM Upgrade Tests
311+
image: registry.redhat.io/redhat/redhat-operator-index
312+
publisher: CodeFlare Team
313+
sourceType: grpc
314+
```
315+
3. Create a Subscription resource, that points to the CatalogSource resource created at the previous step, e.g.:
316+
```
317+
apiVersion: operators.coreos.com/v1alpha1
318+
kind: Subscription
319+
metadata:
320+
name: codeflare-operator
321+
spec:
322+
channel: alpha
323+
installPlanApproval: Automatic
324+
name: codeflare-operator
325+
source: test-catalog
326+
startingCSV: codeflare-operator.<LATEST_VERSION>
327+
```
328+
4. Run a smoke test, e.g. by running one of the e2e test cases, to make sure the operator current version is working correctly.
329+
330+
##### Upgrade of the CodeFlare Operator installed via OLM
331+
332+
###### Description
333+
334+
Add a newer version of the CodeFlare operator bundle, replacing the latest released version from the channel subscribed to during the Installation of the CodeFlare Operator with OLM test case. Assert successful upgrade of the operator to the newer version, and run a smoke test, to make sure that newer version is working as expected.
335+
336+
###### Scenario
337+
338+
> **Note**
339+
> Note: Make sure to execute the Installation of the CodeFlare Operator with OLM test case first.
340+
341+
1. In the same codeflare-system Namespace (resp. Project), update the existing test-catalog CatalogSource resource, by pointing to the new OLM index image, e.g.:
342+
```
343+
apiVersion: operators.coreos.com/v1alpha1
344+
kind: CatalogSource
345+
metadata:
346+
name: test-catalog
347+
spec:
348+
displayName: CodeFlare OLM Upgrade Tests
349+
image: <CATALOG_IMG>
350+
publisher: CodeFlare Team
351+
sourceType: grpc
352+
```
353+
2. Assert a new ClusterServiceVersion resource has been created for the newer operator version, and eventually reaches the CSVPhaseSucceeded phase
354+
3. Run a smoke test, e.g., by running one of the e2e test cases, to make sure the operator current version is working correctly.
355+
356+
## Open Questions
357+
358+
1. Add observability test cases
359+
2. Add integration test cases
360+
361+
## Alternatives
362+
363+
We didn't consider any other alternatives
364+
365+
## Stakeholder Impacts
366+
367+
| Group | Key Contacts | Date | Impacted? |
368+
| ----------------------------- | ------------------ | ---------- | --------- |
369+
| CodeFlare SDK | Mustafa Eyceoz | | yes |
370+
| MCAD | Abhishek Malvankar | | yes |
371+
| InstaScale | Abhishek Malvankar | | yes |
372+
| CodeFlare Operator | Anish Asthana | | yes |
373+
374+
## Reviews
375+
376+
Reviews on the pull request will suffice for the approval process. At least 2 approvals are required prior to this ADR being merged. The ADR must also remain open for at least one week.

0 commit comments

Comments
 (0)