util: enhance tracevol.py script to work with volumesnapshots #5049

black-dragon74 · 2025-01-02T11:50:02Z

Describe what this PR does

This patch adds the functionality to map the k8s volumesnapshots to the cephfs/rbd snapshots.

This patch also adds a wrapper around oc/kubectl called kube_client which will help get rid of code duplication.

Closes: #3344

Future concerns

I will follow up the refactors around kube_client in a separate PR

black-dragon74 · 2025-01-02T11:52:12Z

Testing

Script output

❯ python3 troubleshooting/tools/tracevol.py -k ~/.kube/config -c kubectl -rn rook-ceph -n rook-ceph -cm rook-ceph-csi-config -cmn rook-ceph -d True
+------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                            RBD                                                                             |
+----------+------------------------------------------+----------------------------------------------+-----------------+------------------+------------------+
| PVC Name |                 PV Name                  |                  Image Name                  | PV name in omap | Image ID in omap | Image in cluster |
+----------+------------------------------------------+----------------------------------------------+-----------------+------------------+------------------+
| rbd-pvc  | pvc-234b935e-b3e8-4aeb-8369-f14d8f7ef87e | csi-vol-8cf8a069-6c2f-4c5a-b4db-2c4f54457009 |       True      |       True       |       True       |
+----------+------------------------------------------+----------------------------------------------+-----------------+------------------+------------------+
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                    CephFS                                                                                    |
+--------------------+------------------------------------------+----------------------------------------------+-----------------+----------------------+----------------------+
|      PVC Name      |                 PV Name                  |                Subvolume Name                | PV name in omap | Subvolume ID in omap | Subvolume in cluster |
+--------------------+------------------------------------------+----------------------------------------------+-----------------+----------------------+----------------------+
|     cephfs-pvc     | pvc-d3f5902e-c8aa-4651-96c7-a6b8d00af97a | csi-vol-238df571-8de6-4a71-8795-3e2b9cab3b1a |       True      |         True         |         True         |
|  cephfs-pvc-clone  | pvc-28a47204-b880-43ef-aa1c-574a8ccd5cd5 | csi-vol-8325f40c-91a3-4a44-a3cd-84f0b1b8c664 |       True      |         True         |         True         |
| cephfs-pvc-restore | pvc-9f8acd0a-322c-4188-b02a-a37e26db0db8 | csi-vol-083f7338-00e7-4d1b-ac50-b01222130659 |       True      |         True         |         True         |
+--------------------+------------------------------------------+----------------------------------------------+-----------------+----------------------+----------------------+
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                           RBD Volume Snapshots                                                                          |
+--------------------+---------+----------------------------------------------+-----------------------------------------------+---------------------+---------------------+
|        Name        |   PVC   |                  Image Name                  |                  Snapshot ID                  | Snapshot ID in omap | Snapcontent in omap |
+--------------------+---------+----------------------------------------------+-----------------------------------------------+---------------------+---------------------+
|  rbd-pvc-snapshot  | rbd-pvc | csi-vol-8cf8a069-6c2f-4c5a-b4db-2c4f54457009 | csi-snap-657a613f-66c4-4c3c-8de4-28cf892c6aff |         True        |         True        |
| rbd-pvc-snapshot-1 | rbd-pvc | csi-vol-8cf8a069-6c2f-4c5a-b4db-2c4f54457009 | csi-snap-4114df25-3120-4544-8a55-a7fde19f67c9 |         True        |         True        |
+--------------------+---------+----------------------------------------------+-----------------------------------------------+---------------------+---------------------+
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                            CephFS Volume Snapshots                                                                            |
+-----------------------+------------+----------------------------------------------+-----------------------------------------------+---------------------+---------------------+
|          Name         |    PVC     |                Subvolume Name                |                  Snapshot ID                  | Snapshot ID in omap | Snapcontent in omap |
+-----------------------+------------+----------------------------------------------+-----------------------------------------------+---------------------+---------------------+
|  cephfs-pvc-snapshot  | cephfs-pvc | csi-vol-238df571-8de6-4a71-8795-3e2b9cab3b1a | csi-snap-9cbfd4c7-d448-4ea2-b2bf-087c7b3cb418 |         True        |         True        |
| cephfs-pvc-snapshot-1 | cephfs-pvc | csi-vol-238df571-8de6-4a71-8795-3e2b9cab3b1a | csi-snap-2470db35-8f8a-4100-826d-9d9ce46b1bf3 |         True        |         True        |
+-----------------------+------------+----------------------------------------------+-----------------------------------------------+---------------------+---------------------+

RBD snapshot inside ceph

❯ oc exec -it rook-ceph-tools-68bf47bc65-486jp -- rbd children replicapool/csi-vol-ec58aaf0-2f28-45fc-95f8-67775e0be615
replicapool/csi-snap-dab553a8-9184-45ab-ba43-dde720fa45ea

CephFS snapshot inside ceph

❯ oc exec -it rook-ceph-tools-68bf47bc65-486jp -- ceph fs subvolume snapshot ls myfs csi-vol-595c630d-6e17-4c00-a66e-91785fb01c6d csi
[
    {
        "name": "csi-snap-7a9fa01e-8577-4829-bb30-73b0ac94b534"
    }
]

Regards

Madhu-1 · 2025-01-03T08:22:26Z

+---------------------------------------------------------------------------+------------+----------+----------------------------------------------+-----------------------------------------------+
| Name | PVC | PVC Type | Subvolume Name | Snapshot ID |
+---------------------------------------------------------------------------+------------+----------+----------------------------------------------+-----------------------------------------------+
| rbd-pvc-snapshot | rbd-pvc | RBD | csi-vol-ec58aaf0-2f28-45fc-95f8-67775e0be615 | csi-snap-dab553a8-9184-45ab-ba43-dde720fa45ea |
| snapshot-f64ca5dd7f01b816f689a0c44423338452e82ae57498cfd449949c1dc399fb68 | cephfs-pvc | CephFS | csi-vol-595c630d-6e17-4c00-a66e-91785fb01c6d | csi-snap-7a9fa01e-8577-4829-bb30-73b0ac94b534 |
+---------------------------------------------------------------------------+------------+----------+----------------------------------------------+-----------------------------------------------+

can we have different table for cephfs and rbd just like we have for PVC? Subvolume Name is cephfs specific, if the type is RBD it doesn't make sense to print rbd details under subvolume name

black-dragon74 · 2025-01-06T10:22:43Z

can we have different table for cephfs and rbd just like we have for PVC? Subvolume Name is cephfs specific, if the type is RBD it doesn't make sense to print rbd details under subvolume name

Done. I have updated the test results with the most recent output :)

Madhu-1 · 2025-01-06T10:40:11Z

troubleshooting/tools/tracevol.py

+        pv_data = kube_client("get", "pv", pvc['spec']['volumeName'])
+
+        # Find the type of the PVC (RBD or CephFS)
+        fs_type = "CephFS" if "fsName" in pv_data['spec']['csi']['volumeAttributes'] else "RBD"


what happens if there are non-csi pvc or pvc's belongs to other storage provider?

Updated it like: If the encountered PVC is not a CSI volume, we skip it.

belongs to other storage provider?

The existing checks in the script only contain cases for either CephFS or RBD. I can refactor the detection logic to be more precise in the upcoming PR?

Madhu-1 · 2025-01-06T10:41:35Z

troubleshooting/tools/tracevol.py

+        # Alternatively, we can use the toolbox pod to fetch these values from ceph directly
+        snap_content_name = snapshot['status']['boundVolumeSnapshotContentName']
+        snap_content = kube_client("get", "volumesnapshotcontent", snap_content_name)
+        snap_id = "csi-snap-" + get_image_uuid(snap_content['status']['snapshotHandle'])


csi-snap- is default but you can change the snap prefix in the volumesnapshotclass

Madhu-1 · 2025-01-06T10:42:46Z

what about the column for omap and if the snapshots is present on the cluster or not?

black-dragon74 · 2025-01-07T12:06:35Z

what about the column for omap and if the snapshots is present on the cluster or not?

Since the backing subvolume/image inside ceph is same to that of the PVC, is it necessary to duplicate that info?

Madhu-1 · 2025-01-07T12:26:54Z

troubleshooting/tools/tracevol.py

+        # The snapshot prefix can be optionally defined in the volumesnapshotclass paramaters
+        snap_handle = snap_content['status']['snapshotHandle']
+        snap_uuid = get_image_uuid(snap_handle)
+        snap_prefix = snap_class['parameters'].get('snapshotNamePrefix') or 'csi-snap-'


we need to make it more flexible to fetch it from omap rather from the snap class as snap class can be deleted post creation and recreated post snapshot creation with different prefix.

Ack. On it.

iPraveenParihar · 2025-01-09T09:09:17Z

troubleshooting/tools/tracevol.py

+    Retrieve and parse the list of volume snapshots from the cluster.
+    """
+    if ARGS.namespace:
+        volumesnapshots = kube_client("get", "volumesnapshots", "-n", ARGS.namespace)


why is this required? why not directly use ARGS.namespace in kube_client method?

kube_client is supposed to be a simple wrapper around kubectl or oc. Doing it like this allows us to fetch both namespaced and clusterscoped resources and also keep the syntax similar to kc/oc.

Then why is this code in kube_client? mistake?

cmd += ["--namespace", arg.namespace]t cmd += list(commands)

Whoops, missed it. Thank you!

if ARGS.namespace: volumesnapshots = kube_client("get", "volumesnapshots", "-n", ARGS.namespace)

🤔 But still this conditioning will not work as per your requirement.
if ARGS.namespace is not provided, default value will be default namespace and if condition will always be true?

Madhu-1 · 2025-01-27T09:14:01Z

@Mergifyio queue

mergify · 2025-01-27T09:14:13Z

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at ec5fefc

This patch adds the functionality to map the k8s volumesnapshots to the cephfs/rbd snapshots. This patch also adds a wrapper around oc/kubectl called `kube_client` which will help get rid of code duplication. Closes: ceph#3344 Signed-off-by: Niraj Yadav <[email protected]>

ceph-csi-bot · 2025-01-27T09:14:50Z

/test ci/centos/k8s-e2e-external-storage/1.30

ceph-csi-bot · 2025-01-27T09:14:50Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2025-01-27T09:14:50Z

/test ci/centos/mini-e2e-helm/k8s-1.30

ceph-csi-bot · 2025-01-27T09:14:51Z

/test ci/centos/upgrade-tests-rbd

ceph-csi-bot · 2025-01-27T09:14:51Z

/test ci/centos/mini-e2e/k8s-1.30

ceph-csi-bot · 2025-01-27T09:14:51Z

/test ci/centos/k8s-e2e-external-storage/1.32

ceph-csi-bot · 2025-01-27T09:14:52Z

/test ci/centos/k8s-e2e-external-storage/1.31

ceph-csi-bot · 2025-01-27T09:14:52Z

/test ci/centos/mini-e2e-helm/k8s-1.32

ceph-csi-bot · 2025-01-27T09:14:53Z

/test ci/centos/mini-e2e-helm/k8s-1.31

ceph-csi-bot · 2025-01-27T09:14:53Z

/test ci/centos/mini-e2e/k8s-1.32

ceph-csi-bot · 2025-01-27T09:14:54Z

/test ci/centos/mini-e2e/k8s-1.31

black-dragon74 requested a review from a team January 2, 2025 11:52

black-dragon74 force-pushed the trace-add-vs branch from 5ff5ab6 to a85bfca Compare January 2, 2025 12:05

nixpanic added ci/skip/e2e skip running e2e CI jobs ci/skip/multi-arch-build skip building on multiple architectures labels Jan 6, 2025

black-dragon74 force-pushed the trace-add-vs branch from a85bfca to a82f7ab Compare January 6, 2025 10:20

black-dragon74 requested a review from Madhu-1 January 6, 2025 10:29

Madhu-1 requested changes Jan 6, 2025

View reviewed changes

black-dragon74 force-pushed the trace-add-vs branch from a82f7ab to f7c24f7 Compare January 7, 2025 12:24

Madhu-1 reviewed Jan 7, 2025

View reviewed changes

black-dragon74 force-pushed the trace-add-vs branch 2 times, most recently from e6bb877 to c2adb42 Compare January 7, 2025 15:28

black-dragon74 requested a review from Madhu-1 January 7, 2025 15:30

iPraveenParihar reviewed Jan 9, 2025

View reviewed changes

black-dragon74 force-pushed the trace-add-vs branch 2 times, most recently from 295bfdb to b9f9e8b Compare January 9, 2025 10:06

black-dragon74 requested a review from iPraveenParihar January 9, 2025 10:08

Madhu-1 approved these changes Jan 24, 2025

View reviewed changes

iPraveenParihar approved these changes Jan 27, 2025

View reviewed changes

black-dragon74 force-pushed the trace-add-vs branch from b9f9e8b to dcd7adf Compare January 27, 2025 09:14

mergify bot added the ok-to-test Label to trigger E2E tests label Jan 27, 2025

ceph-csi-bot removed the ok-to-test Label to trigger E2E tests label Jan 27, 2025

mergify bot merged commit ec5fefc into ceph:devel Jan 27, 2025
37 checks passed

Uh oh!

util: enhance tracevol.py script to work with volumesnapshots #5049

util: enhance tracevol.py script to work with volumesnapshots #5049

Uh oh!

Conversation

black-dragon74 commented Jan 2, 2025

Describe what this PR does

Future concerns

Uh oh!

black-dragon74 commented Jan 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Script output

RBD snapshot inside ceph

CephFS snapshot inside ceph

Uh oh!

Madhu-1 commented Jan 3, 2025

Uh oh!

black-dragon74 commented Jan 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Madhu-1 commented Jan 6, 2025

Uh oh!

black-dragon74 commented Jan 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Madhu-1 commented Jan 27, 2025

Uh oh!

mergify bot commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ The pull request has been merged automatically

Uh oh!

ceph-csi-bot commented Jan 27, 2025

Uh oh!

ceph-csi-bot commented Jan 27, 2025

Uh oh!

ceph-csi-bot commented Jan 27, 2025

Uh oh!

ceph-csi-bot commented Jan 27, 2025

Uh oh!

ceph-csi-bot commented Jan 27, 2025

Uh oh!

ceph-csi-bot commented Jan 27, 2025

Uh oh!

ceph-csi-bot commented Jan 27, 2025

Uh oh!

ceph-csi-bot commented Jan 27, 2025

Uh oh!

ceph-csi-bot commented Jan 27, 2025

Uh oh!

ceph-csi-bot commented Jan 27, 2025

Uh oh!

ceph-csi-bot commented Jan 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

black-dragon74 commented Jan 2, 2025 •

edited

Loading

mergify bot commented Jan 27, 2025 •

edited

Loading