Skip to content

feat(cmapi): MCOL-6006: Disable failover when shared storage not detected #3661

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 15 commits into
base: stable-23.10
Choose a base branch
from
Open

feat(cmapi): MCOL-6006: Disable failover when shared storage not detected #3661

wants to merge 15 commits into from

Conversation

mariadb-AlanMologorsky
Copy link
Contributor

  • add Base api client
  • add NodeController api client
  • fix ClusterController Api client
  • add NodeController in endpoints.py
  • add 2 new endpoints to dispatcher: cluster/check-shared-storage and node/check-shared-file
  • add check_shared_storage method into ClusterHandler class
  • add set_shared_storage to node_manipulation t
  • add reading shared_storage in failover/config.py
  • fix is_shared_storage method in failover/config.py
  • add check_shared_storage in NodeMonitor class
  • fix some minor styling

@mariadb-LeonidFedorov mariadb-LeonidFedorov changed the title feat(cmapi): MCOL-6006 Disable failover when shared storage not detected feat(cmapi): MCOL-6006: Disable failover when shared storage not detected Jul 29, 2025
@mariadb-AlanMologorsky mariadb-AlanMologorsky force-pushed the feature/MCOL-6006-failover-detect-shared-storage branch 2 times, most recently from ff027b5 to 92275c8 Compare August 8, 2025 09:05
@mariadb-AlanMologorsky mariadb-AlanMologorsky force-pushed the feature/MCOL-6006-failover-detect-shared-storage branch 3 times, most recently from d70dff2 to 73a5abd Compare August 15, 2025 12:50
mariadb-AlexanderPresniakov
mariadb-AlexanderPresniakov requested changes Aug 19, 2025
View reviewed changes
@mariadb-AlanMologorsky mariadb-AlanMologorsky force-pushed the feature/MCOL-6006-failover-detect-shared-storage branch 2 times, most recently from 92ab8a2 to 5f24507 Compare August 24, 2025 13:02
@mariadb-AlanMologorsky mariadb-AlanMologorsky force-pushed the feature/MCOL-6006-failover-detect-shared-storage branch from 5f24507 to 440173e Compare August 24, 2025 20:18
@mariadb-AlanMologorsky mariadb-AlanMologorsky force-pushed the feature/MCOL-6006-failover-detect-shared-storage branch from 440173e to a94684c Compare September 2, 2025 13:57
@mariadb-AlanMologorsky mariadb-AlanMologorsky force-pushed the feature/MCOL-6006-failover-detect-shared-storage branch 2 times, most recently from 4ace6f1 to 1a1e03d Compare September 4, 2025 18:57
mariadb-AlexanderPresniakov
mariadb-AlexanderPresniakov approved these changes Sep 8, 2025
View reviewed changes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
(but the test needs fixing)

@mariadb-AlanMologorsky mariadb-AlanMologorsky force-pushed the feature/MCOL-6006-failover-detect-shared-storage branch from 4bd79c0 to 9075d7f Compare September 9, 2025 12:39
mariadb-LeonidFedorov
mariadb-LeonidFedorov approved these changes Sep 9, 2025
View reviewed changes
Copy link
Collaborator

@mariadb-LeonidFedorov mariadb-LeonidFedorov left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current state of code looks good to me, the approval doesn't consider merge without fixes of tests, but the approach and the implementation look neat

@mariadb-AlanMologorsky mariadb-AlanMologorsky force-pushed the feature/MCOL-6006-failover-detect-shared-storage branch 2 times, most recently from eb0b2cf to 2a0675e Compare September 9, 2025 20:37
@mariadb-AlanMologorsky mariadb-AlanMologorsky force-pushed the feature/MCOL-6006-failover-detect-shared-storage branch 5 times, most recently from b061400 to 8bfb2ce Compare September 12, 2025 12:45
@mariadb-AlanMologorsky
feat(cmapi): MCOL-6006 Disable failover when shared storage not detected
d0d6075 
add Base api client
add NodeController api client
fix ClusterController Api client
add NodeController in endpoints.py
add 2 new endpoints to dispatcher: cluster/check-shared-storage and node/check-shared-file
add check_shared_storage method into ClusterHandler class
add set_shared_storage to node_manipulation t
add reading shared_storage in failover/config.py
fix is_shared_storage method in failover/config.py
add check_shared_storage in NodeMonitor class
fix some minor styling
@mariadb-AlanMologorsky
addition
b403406
@mariadb-AlanMologorsky
- add AppStatefulConfig, StateConfigVersion classes to implement CMAP…
d722499 
…I in memory stateful configuration

- add stateful config distribution logic into endpoints.ConfigController (node endpoint) with fast "only_stateful_config" update logic
- add stateful config logic into helpers.broadcast_new_config
- add new function broadcast_stateful_config into helpers.py
- port Alexander Presnyakov get_existing_db_roots function into node_manipulation.py from other branch
- add new SharedStorageMonitor threaded compomemt for Failover it starts working with NodeMonitor component as a separate thread that checks shared storage state/health each 5 mins.
@mariadb-AlanMologorsky
Review fixes.
767b08d 
- Implemented Pydantic models and validation.
- Added separated endpoint for broadcasting stateful config (next releases)
- Fixed some not threadsafe AppStatefulConfig methods.
@mariadb-AlanMologorsky
Fix broken after rebase requirements.txt.
2f1b1d0
@mariadb-AlanMologorsky
Fix another broken rebase + more logging.
038adc8
@mariadb-AlanMologorsky
Added Pydantic Models for Config put endpoint.
59369b3 
Changed config put request body validation to use Pydantic models.
@mariadb-AlanMologorsky
Fixes and improvements after tests.
9715c48 
- fixed enpoints.py put_config
- improved request_models to support set_cluster_mode command
- improved logging for shared file check
- fix race condition while writing and reading shared temp file
- changed delay between checking shared storage to 10 sec
- fix tests
@mariadb-AlanMologorsky
Another fixes for tests.
765977e
@mariadb-AlanMologorsky
temp fix deactivate_only.
71d57f4
@mariadb-AlanMologorsky
Fix failover monitpring thread.
993793a 
Was rebase error.
Missed early stop the loop.
@mariadb-AlanMologorsky
Fix _rebalance_dbroots function formatting
4ef9655 
Add test_rebalance_noop_when_shared_storage_off
@mariadb-AlanMologorsky
Add state field for each node in mcs cluster command response.
56e245d 
Now if failover turned off and one of the nodes failed we can see it in the mcs status respose as a offline value of the node state field.
Optionally there could be error field with information.
@mariadb-AlanMologorsky mariadb-AlanMologorsky force-pushed the feature/MCOL-6006-failover-detect-shared-storage branch from ca2a038 to 56e245d Compare September 18, 2025 06:27
@mariadb-AlanMologorsky mariadb-AlanMologorsky force-pushed the feature/MCOL-6006-failover-detect-shared-storage branch from 6b7c324 to d644d80 Compare September 18, 2025 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mariadb-LeonidFedorov mariadb-LeonidFedorov mariadb-LeonidFedorov approved these changes

@mariadb-AlexanderPresniakov mariadb-AlexanderPresniakov mariadb-AlexanderPresniakov approved these changes

@drrtuy drrtuy Awaiting requested review from drrtuy

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mariadb-AlanMologorsky @mariadb-LeonidFedorov @mariadb-AlexanderPresniakov