NC | Online Upgrade | Fail config directory upgrade if upgrade is already in progress

[romayalon]

Explain the changes

1. Add a condition in the verification step of the config dir upgrade that checks if there is already in progress upgrade based on system.json.
2. Added a unit test.

Issues: Fixed #xxx / Gap #xxx

1. Fixed NC | Online upgrade improvements #8586 partially
2. Gap - this check still can have a race condition while running config dir upgrade in parallel, this can be improved by using lock - lockfd.fcntllock().

Testing Instructions:

Automatic tests - sudo jest --testRegex=jest_tests/test_nc_upgrade_manager.test.js -t "fail on already in progress upgrade"

Manual test artificially -

1. Checkout to old branch like 5.16/5.17
2. run noobaa - sudo node src/cmd/nsfs.js --debug 5, stop noobaa Ctrl+C
3. Checkout to 5.18 branch
4. run noobaa - sudo node src/cmd/nsfs.js --debug 5, stop noobaa Ctrl+C
5. add a delay in NCUpgradeManager for making the upgrade take 2 minutes -

+ const P = require('../util/promise');
....
+ await P.delay(120 * 1000);
   try {
      await this._run_nc_upgrade_scripts(this_upgrade);

7. run upgrade config dir on tab 1 - noobaa-cli upgrade start --expected_version 5.18.0 --expected_hosts <hostname>
8. run immediately upgrade config dir on tab 2 - noobaa-cli upgrade start --expected_version 5.18.0 --expected_hosts <hostname> and expect a failure with error message -

 "error": {
    "code": "UpgradeFailed",
    "message": "Upgrade request failed",
    "cause": "Error: config dir upgrade can not be started - there is already an ongoing upgrade system_data.config_directory.in_progess_upgrade .....

• Doc added/updated
• Tests added

[shirady]

LGTM

[shirady]

@romayalon in the testing instructions (inside the PR description):
Do you open four tabs? (two for the endpoints and two according to steps 6+7), if so could you mention it explicitly in the instructions?

[romayalon]

@romayalon in the testing instructions (inside the PR description): Do you open four tabs? (two for the endpoints and two according to steps 6+7), if so could you mention it explicitly in the instructions?

no, I use a tab1 for the upgrade and a second tab for running another config dir upgrade start that I expect to fail.