Feature: Check if disk rule cleanup can be done in Rucio after tape rule success

[haozturk]

Feature Description

When WM produce datasets, they create a disk rule as well as a tape rule. Disk rule is deleted by a WM service when the tape rule is OK. The question is whether this can be done in rucio such that we don't have to maintain an extra service/functionality in WM

Use Case

Moving DM functionality from WM to Rucio

Possible Solution

According to a mattermost chat with Martin [1], it should be possible by linking the disk rule and the tape rule in such a way that the parent rule (disk) is deleted by rucio when the child rule (tape) succeeds. We need to check how this will work and guarantee that it's reliable.

[1] https://mattermost.web.cern.ch/rucio/pl/ap9i3ooscbrr7bwke87y9cz8qo

Related Issues

No response

[amaltaro]

@haozturk thank you for creating this issue.

I just wanted to add that the list of children rules might be very large, as we create one wma_prod rule for each block that gets produced in the agent. So, in practice, it could be that a Tape rule would have a very large list of wma_prod rules, containing a rule for every single block plus a potential rule for the whole container.
Hopefully we don't have to iterate over the rules for the container and every single block in the container, otherwise it can be a fairly inefficient process.

[haozturk]

Sorry for very late reply on this Alan. I started looking into this again. Unfortunately, rucio doesn't support multiple child rules. A parent rule can only have one child rule and both rules should be on the same dataset. With this constraint in mind, how about the following idea for the next-gen WM?

WM creates a single rule wma_prod rule to lock the disk copy. It will be grouping=DATASET. This rule will be created when the first block is created. When the new blocks are created, their replicas will be injected into Rucio, they will be attached to the dataset and their locks will be automatically created by the rule. It will not trigger a new replica as long as the RSE expression of the disk rule contains all the production disk RSEs. This way, we will have a single disk rule and a single tape rule. They can be linked to each other as we discussed and rucio would handle the cleanup of the disk rule once the tape rule succeeds. Please let me know what you think.

[haozturk]

Second thought: this contradicts with our desire of managing outputs in block level, e.g. if a block is on tape, unlock it from disk. So, it's not an ideal solution.