Prevent users from setting the same name for final and staging dataset

[sh-rp]

TLDR
We use staging datasets for transactional safety on merge write_dispositions as well as some variants of the replace write_disposition. The default behavior is to have a second dataset called "<dataset_name>_staging". Users can change this name which can lead to a setup where final and staging datasets have the same name. We should prevent this or at least print a big fat warning if users try to do this, as data in the final dataset will be truncated by the setup commands that should only truncate the staging dataset.

ToDo

• Learn about the staging dataset: https://dlthub.com/docs/dlt-ecosystem/staging#staging-dataset
• Add a new method to the WithStagingDataset class: def create_dataset_names(self, schema: Schema, config: DestinationClientDwhConfiguration) -> Tuple[str, str]:, which creates the regular and the staging dataset names for a given schema and config, this method should also raise an Exception if both are the same. See the point below to find the places where these normalized names are created.
• Use this new method to create the normalized regular and staging dataset names in for all destinations (including the filesystem destination). You can find all destination implementations under dlt/destinations/impl, or just search for all the places where normalize_staging_dataset_name() is used.
• Write tests that demonstrate that this exception is raised if both datasets end up having the same name after normalization.

[rakesh-tmdc]

I'd like to get your feedback on my approach -

Instead of the originally suggested method (adding create_dataset_names() to WithStagingDataset + updating all destinations), I implemented centralized validation in the existing normalize_staging_dataset_name() method:

def normalize_staging_dataset_name(self, schema: Schema) -> str:
    # ... existing logic ...
    
    staging_dataset_name = self._normalize_identifier(dataset_name, schema.naming)
    regular_dataset_name = self.normalize_dataset_name(schema)
    
    # Validate names don't conflict
    if regular_dataset_name == staging_dataset_name:
        raise ValueError(f"Dataset names conflict: '{regular_dataset_name}' ...")
    
    return staging_dataset_name

Does this centralized approach fit well with dlt's design patterns or Would you prefer the original approach of updating all destinations individually, or is centralized validation better?

[alkaline-0]

Hello @rakesh-tmdc, thanks for your suggestion! your idea does provide a quick fix, however, it brings potential side effects. By implementing your solution I'd be delegating more than one task to normalize_staging_dataset_name which would make it more complicated to trace/test in the future aka breaking separation of concerns (since you asked about design patterns)

I'd go with the first suggestion because it's cleaner and ensures that the existing logic remains unchanged as well since it's not broken :D

Have a good day!