[BUG]: Unable to install dqx as a tool because of AnomalyWorkflow

[cornzyblack]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I am encountering an issue when trying to install dqx as a tool in a workspace.

When I run databricks labs install dqx@v0.14.0. I believe it has something to do with the ANOMALY_AVAILABLE flag not being checked before returning in the class method all

# Try to import anomaly workflow (requires anomaly extras)
try:
    from databricks.labs.dqx.anomaly.anomaly_workflow import AnomalyTrainerWorkflow

    ANOMALY_AVAILABLE = True
except ImportError:
    ANOMALY_AVAILABLE = False

logger = logging.getLogger(__name__)


class WorkflowsRunner:
    def __init__(self, workflows: list[Workflow]):
        self._tasks: list[Task] = []
        self._workflows: dict[str, Workflow] = {}
        for workflow in workflows:
            self._workflows[workflow.name] = workflow
            for task_definition in workflow.tasks():
                # Add the workflow params to the task definition, because we cannot access
                # them from the workflow_task decorator
                with_workflow = dataclasses.replace(
                    task_definition,
                    workflow=workflow.name,
                    spark_conf=workflow.spark_conf,
                    override_clusters=workflow.override_clusters,
                )
                self._tasks.append(with_workflow)

    @classmethod
    def all(cls, config: WorkspaceConfig) -> "WorkflowsRunner":
        """Return all workflows."""
        profiler = ProfilerWorkflow(
            spark_conf=config.profiler_spark_conf,
            override_clusters=config.profiler_override_clusters,
        )
        quality_checker = DataQualityWorkflow(
            spark_conf=config.quality_checker_spark_conf,
            override_clusters=config.quality_checker_override_clusters,
        )
        e2e = EndToEndWorkflow(
            profiler,
            quality_checker,
            spark_conf=config.e2e_spark_conf,
            override_clusters=config.e2e_override_clusters,
        )
        anomaly_trainer = AnomalyTrainerWorkflow(
            spark_conf=config.anomaly_spark_conf,
            override_clusters=config.anomaly_override_clusters,
        )
        workflows: list[Workflow] = [profiler, quality_checker, e2e, anomaly_trainer]

        return cls(workflows)

Proposed fix:

    @classmethod
    def all(cls, config: WorkspaceConfig) -> "WorkflowsRunner":
        """Return all workflows."""
        profiler = ProfilerWorkflow(
            spark_conf=config.profiler_spark_conf,
            override_clusters=config.profiler_override_clusters,
        )
        quality_checker = DataQualityWorkflow(
            spark_conf=config.quality_checker_spark_conf,
            override_clusters=config.quality_checker_override_clusters,
        )
        e2e = EndToEndWorkflow(
            profiler,
            quality_checker,
            spark_conf=config.e2e_spark_conf,
            override_clusters=config.e2e_override_clusters,
        )
        workflows: list[Workflow] = [profiler, quality_checker, e2e]

        # Check if Anomaly is available
        if ANOMALY_AVAILABLE:
            anomaly_trainer = AnomalyTrainerWorkflow(
                spark_conf=config.anomaly_spark_conf,
                override_clusters=config.anomaly_override_clusters,
            )
            # Append the trainer workflow
            workflows.append(anomaly_trainer)

        return cls(workflows)

I manually modified ~/.databricks/labs/dqx/lib/src/databricks/labs/dqx/workflows_runner.py and then ran

databricks labs install dqx@v0.14.0 --offline so it doesn't redownload the tar.gz file and overwrite my changes, and it worked.

Expected Behavior

To successfully install DQX as a tool in the Databricks workspace

Steps To Reproduce

databricks labs install dqx@v0.14.0
Enter a workspace path for DQX installation (leave empty to install in user's home or global directory) (default: empty): /Shared/dqx-projects
21:29:05     INFO [databricks.sdk] Using Databricks CLI authentication
21:29:05     INFO [d.l.d.installer.install] Installing DQX v0.14.0
21:29:05     INFO [d.l.d.installer.install] Please answer a couple of questions to provide default DQX run configuration. The configuration can also be updated manually after the installation.
21:29:05     INFO [d.l.d.installer.install] DQX will be installed in folder '/Shared/dqx-projects'
Log level (default: INFO):
Should the input data be read using streaming? (default: no):
Provide location for the input data as a path or table in the fully qualified format `catalog.schema.table` or `schema.table` (default: skipped):
Provide output table in the fully qualified format `catalog.schema.table` or `schema.table`: dqx.default.output
Provide write mode for output table (e.g. 'append' or 'overwrite') (default: append):
Provide format for the output data (e.g. delta, parquet) (default: delta):
Provide additional options for writing the output data (e.g. {"mergeSchema": "true"}) (default: {}):
Provide quarantined table in the fully qualified format `catalog.schema.table` or `schema.table` (use output table if skipped) (default: skipped):
Do you want to store summary metrics from data quality checking in a table? (default: no):
Provide location of the quality checks definitions, either:
- a filename for storing data quality rules (e.g. checks.yml),
- or a table for storing checks in the format `catalog.schema.table` or `schema.table`,
- or a full volume path in the format /Volumes/catalog/schema/volume/<folder_path>/<file_name_with_extension>,
 (default: checks.yml):
Provide filename for storing profile summary statistics (default: profile_summary_stats.yml):
Do you want to use standard job clusters for the workflows execution (not Serverless)? (default: no):
Provide reference tables to use for checks as a dictionary that maps reference table name to reference data location. The specification can contain fields from InputConfig such as: location, format, schema, options and is_streaming fields (e.g. {"reference_vendor":{"location": "catalog.schema.table", "format": "delta"}}) (default: {}):
Provide custom check functions as a dictionary that maps function name to a python module located in the workspace file (relative or absolute workspace path) or volume (e.g. {"my_func": "/Workspace/Shared/my_module.py"}),  (default: {}):
Select PRO or SERVERLESS SQL warehouse to run data quality dashboards on
[0]  [Create new PRO or SERVERLESS SQL warehouse ]
[1] DQX Dashboard xxxx (xxxx, PRO, STOPPED)
[2] Serverless Starter Warehouse (xxxx, SERVERLESS, RUNNING)
Enter a number between 0 and 2: 2
Does the given workspace block Internet access? (default: no):
Open config file in the browser and continue installing? https://dbc-xxxx.cloud.databricks.com/#workspace/Shared/dqx-projects/config.yml (default: no):
Traceback (most recent call last):
  File "/Users/xxxx/.databricks/labs/dqx/lib/src/databricks/labs/dqx/installer/install.py", line 385, in <module>
    workspace_installer.run()
  File "/Users/xxxx/.databricks/labs/dqx/lib/src/databricks/labs/dqx/installer/install.py", line 116, in run
    tasks = WorkflowsRunner.all(config).tasks()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxxx/.databricks/labs/dqx/lib/src/databricks/labs/dqx/workflows_runner.py", line 62, in all
    anomaly_trainer = AnomalyTrainerWorkflow(
                      ^^^^^^^^^^^^^^^^^^^^^^
NameError: name 'AnomalyTrainerWorkflow' is not defined
Error: installer: exit status 1

Cloud

AWS

Operating System

macOS

Relevant log output, error message and full stacktrace

Traceback (most recent call last):
  File "/Users/xxxx/.databricks/labs/dqx/lib/src/databricks/labs/dqx/installer/install.py", line 385, in <module>
    workspace_installer.run()
  File "/Users/xxxx/.databricks/labs/dqx/lib/src/databricks/labs/dqx/installer/install.py", line 116, in run
    tasks = WorkflowsRunner.all(config).tasks()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxxx/.databricks/labs/dqx/lib/src/databricks/labs/dqx/workflows_runner.py", line 62, in all
    anomaly_trainer = AnomalyTrainerWorkflow(
                      ^^^^^^^^^^^^^^^^^^^^^^
NameError: name 'AnomalyTrainerWorkflow' is not defined
Error: installer: exit status 1

[ghanse]

@cornzyblack thanks for raising, would you like to work on this?

[cornzyblack]

yeah sure!