NC | Lifecycle | Adjust expire/noncurrent state properties to GPFS flow + fix of 2 small bugs

[romayalon]

Describe the Problem

GPFS flow still uses old inline state usage for maintaining expiration state.

Explain the Changes

1. parse_candidates_from_gpfs_ilm_policy() - currently GPFS flow supports only expiration rule, but for modularity purposes checked the rule type and used it for setting/getting the rule state based on the type - expire/noncurrent.

2. _set_rule_state() - updated the whole state while we are calling it for a specific action, therefore we should update only the relevant part and keep the existing state that is irrelevant to the action (expire/noncurrent).

3. create_gpfs_candidates_files () - we were not skipping correctly buckets that don't have any bucket lifecycle configuration - added a check + continue for that.

4. Refactoring - replaced this.lifecycle_run_status.buckets_statuses[bucket_json.name].rules_statuses[lifecycle_rule.id].state with a call to _get_rule_state() function where needed.

Issues: Fixed #xxx / Gap #xxx

1. Unlreated Gap - during those tests I saw that sometimes mmapplypolicy doesn't return all 1500 objects that should be deleted. need further debugging.
2. Unrelated Gap - if delete_candidates list contains more than 1000 objects (by having multiple actions) we need to check if it deletes them all.

Testing Instructions:

Manual tests on a GPFS machine -

1. create account
2. create 2 buckets that exports the paths -/mnt/gpfs0/romy/bucket1_storage/ and /mnt/gpfs0/romy/bucket2_storage/
3. set bucket lifecycle rule on each bucket - expiry set to the past + prefix b
4. create 1500 objects that should be deleted per expire rule -
   for i in $(seq 1 1500); do echo "blalalalla" > /mnt/gpfs0/romy/bucket1_storage/bav$i.txt ; done
5. run the lifecycle process manually - noobaa-cli lifecycle --disable_service_validation --disable_runtime_validation --debug 5 &> lifecycle_logs1.logs
6. check 3000 objects were deleted and check the logs for 2 delete cycles per bucket.

cat lifecycle_logs.logs | grep "calling delete"
Apr-30 8:36:17.147 [noobaa-cli/3735888]    [L0] core.manage_nsfs.nc_lifecycle:: process_rule: calling delete_multiple_objects, num of objects to be deleted 1000
Apr-30 8:36:17.148 [noobaa-cli/3735888]    [L0] core.manage_nsfs.nc_lifecycle:: process_rule: calling delete_multiple_objects, num of objects to be deleted 1000
Apr-30 8:36:21.343 [noobaa-cli/3735888]    [L0] core.manage_nsfs.nc_lifecycle:: process_rule: calling delete_multiple_objects, num of objects to be deleted 500
Apr-30 8:36:21.493 [noobaa-cli/3735888]    [L0] core.manage_nsfs.nc_lifecycle:: process_rule: calling delete_multiple_objects, num of objects to be deleted 500

• Doc added/updated
• Tests added

[shirady]

LGTM
I raised a couple of questions and comments