Unity Catalog and Labs CLI Support by ravi-databricks · Pull Request #28 · databrickslabs/dlt-meta

ravi-databricks · 2023-12-15T23:41:33Z

Unity Catalog Support

Added uc_enabled flag so that existing customers wont face any issues
Based on uc_enabled flag Onboarding and DataflowPipeline will use paths or uc namespaces
Created integration tests framework which can be used to launch demos
Added integration tests for uc
Modified existing demos to incorporate unity catalog and integration test framework
Added unit test coverage for Dataflowpipeline
Modified documentation

Databricks Labs CLI Support

Added two commands for DLT-META

onboard: Captures all onboarding details from command line and launch onboarding job to your databricks workspace
deploy: Captures all DLT pipeline details from command line and launch DLT pipeline to your databricks workspace
Customers can use onboard and deploy as API if dlt-meta is installed using pypi

1.uc table scheme for dlt reader writer 2.onboarding modification to use table instead of paths

2. Fixed unit tests to include table instread of paths

2. Added databricks SDK in integration tests 3. Added docs for running integration tests with profiles

1. uc_enabled flag for supporting UC + nonUC dlt 2. used isinstance instead of types

1.Dataflowpipeline silver schema read for uc_enabled 2.Fixed integration tests with databricks sdk for cloudFiles 3.Fixed demos for dais demo for uc_enabled=true and tech_summit demo for uc_enabled=False

2.Added readme for running demo 3.Added docs to code

2.integration tests restructring as per cli changes

1.Labs CLI: onboard command with uc 2.Labs CLI: deploy command with uc

1.CLI: onboard for nonuc 2.CLI: deploy for nonuc

… uploader 2.correcty labs.yaml typo to labs.yml 3.correced logger to databricks.labs.dltmeta

Added - dtlmeta labs cli feature - unit test coverage for uc_enabled feature in onboarding and dataflowpipeline

- Added readme instructions for using cli with example

- Added readme instructions for using cli with example - Added cli to use existing demo folder as default

-Fixed cli for non uc flows

-Fixed integration tests for non-uc flows

codecov · 2023-12-15T23:47:38Z

Codecov Report

Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (f5f6c34) 85.00% compared to head (b0e2e31) 90.28%.

Files	Patch %	Lines
src/onboard_dataflowspec.py	94.28%	4 Missing ⚠️
src/dataflow_pipeline.py	93.33%	1 Missing and 2 partials ⚠️
src/__main__.py	83.33%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #28      +/-   ##
==========================================
+ Coverage   85.00%   90.28%   +5.28%     
==========================================
  Files           7        8       +1     
  Lines         740      803      +63     
  Branches      128      149      +21     
==========================================
+ Hits          629      725      +96     
+ Misses         61       31      -30     
+ Partials       50       47       -3

Flag	Coverage Δ
unittests	`90.28% <93.23%> (+5.28%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

1.Readme doc for databricks labs cli 2.Unit tests

Added unit tests and readme

howardwu-db

Looks good

- Added change log - added version 0.0.5

- Added hugo docs for databricks labs cli option

neil90-db

Tested UC and CLI Supported, works great! LGTM!

ganeshchand · 2024-01-05T04:31:35Z

README.md

+### Databricks Labs DLT-META CLI lets you run onboard and deploy in interactive python terminal
+#### pre-requisites:
+- [Databricks CLI](https://docs.databricks.com/en/dev-tools/cli/tutorial.html)
+- Python 3.8.0 +


Is this the exaction version required or the minimum version required? It would be good to clarify this.

its minimum thats why python3.8.0 +
I saw this as notation in other projects

ganeshchand · 2024-01-05T04:36:04Z

README.md

 Refer to the [Getting Started](https://databrickslabs.github.io/dlt-meta/getting_started)
+### Databricks Labs DLT-META CLI lets you run onboard and deploy in interactive python terminal
+#### pre-requisites:
+- [Databricks CLI](https://docs.databricks.com/en/dev-tools/cli/tutorial.html)


Databricks CLI is still in public preview and releases are happening every month (sometime multiple times within a month). Unless we provide a guarantee that the CLI integration with dlt-meta is forward compatible with all future databricks-cli releases, we should add a specific version(s) we have tested the integration with.

its forward looking since we have used basic databricks sdk apis

ganeshchand · 2024-01-05T04:38:49Z

README.md

+- ``` cd dlt-meta ```
+- ``` python -m venv .venv ```
+- ```source .venv/bin/activate ```
+- ``` pip install databricks-sdk ```


same comment as above with regards to the versions.

ganeshchand · 2024-01-05T04:43:22Z

README.md


+```     Provide onboarding file path (default: demo/conf/onboarding.template): 
+        Provide onboarding files local directory (default: demo/): 
+        Provide dbfs path (default: dbfs:/dlt-meta_cli_demo): 


For HMS, this default makes sense. However, for UC, we should default to some default volume?

not using volumes for uc. If you look code its default workspace location.

ganeshchand · 2024-01-05T04:50:06Z

README.md


+```     Provide onboarding file path (default: demo/conf/onboarding.template): 
+        Provide onboarding files local directory (default: demo/): 
+        Provide dbfs path (default: dbfs:/dlt-meta_cli_demo): 


This command fails and the program exits if there is any databricks cli authentication issue. In my case, the default profile was being used which had expired token. I had to explicitly provide --profile <profile_name and it got past this command

yes. either provide --profile or it will use default

ganeshchand · 2024-01-05T05:00:19Z

README.md

+        Provide dataflow spec version (default: v1): 
+        Provide environment name (default: prod): prod
+        Provide import author name (default: ravi.gawai): 
+        Provide cloud provider name


We should be able to infer this using Databricks SDK. By this time, we should have information about the target environment available to us.

Did not see any API which returns cloud provider using cli or apis

ganeshchand · 2024-01-05T05:02:09Z

README.md

+        [1] azure
+        [2] gcp
+        Enter a number between 0 and 2: 0
+        Do you want to update ws paths, catalog, schema details to your onboarding file?


It looks like the actual text is "Update workspace/dbfs paths, unity catalog name, bronze/silver schema names in onboarding file?"

Also, do we need to pompt this option? In what situation would user not want to update the onboarding file with path, UC catalog name, etc.

Some customers generates onboarding files and its par of repo in that case they do not need to overwrite paths

ganeshchand · 2024-01-05T05:09:31Z

README.md

+        [0] False
+        [1] True
+```
+- Goto your databricks workspace and located onboarding job under: Workflow->Jobs runs


The databricks labs dlt-meta onboard command seems to exit without any confirmation as shown below:

Update workspace/dbfs paths, unity catalog name, bronze/silver schema names in onboarding file? [0] False [1] True Enter a number between 0 and 1: 1 (.venv) ➜ dlt-meta git:(feature/dlt-meta-uc)

I didn't know whether the command completed successfully or not. I had to read this README to figure out I have to go check the workflow UI to find the onboarding job.
We should at least try to print a successful message with the expected job name. It would be great if we can also print the job URL.

I had prints but @nfx suggested to remove since labs cli will print it as part of logs.

ganeshchand · 2024-01-05T05:19:25Z

README.md

+```
+- Goto your databricks workspace and located onboarding job under: Workflow->Jobs runs
+- Once onboarding jobs is finished deploy `bronze` and `silver` DLT using below command
+- ```databricks labs dlt-meta deploy```


This command also silently exits. Would be great if we can print some confirmation message or even better print the pipeline URL.

ganeshchand · 2024-01-05T05:19:41Z

README.md

+```
+
+- Silver DLT
+- - ```databricks labs dlt-meta deploy```


Same as above. Would be great if we can print some confirmation message or even better print the pipeline URL.

nfx

lgtm for labs.yml

ganeshchand

Ran the onboard and deploy workflow. LGTM.

ravi-db and others added 30 commits July 20, 2023 15:08

Adding :

cb24c80

1.uc table scheme for dlt reader writer 2.onboarding modification to use table instead of paths

1. Added onboarding to support UC table schemes

a70074f

2. Fixed unit tests to include table instread of paths

Corrected Validate notebook for all int test dbc files

89441b7

Added unity catalog support for integration tests

0105d1c

1. Removed paths from dataflowspecs

db19906

2. Added databricks SDK in integration tests 3. Added docs for running integration tests with profiles

Merge branch 'main' into feature/dlt-meta-uc

7833a38

Added:

f3f3487

1. uc_enabled flag for supporting UC + nonUC dlt 2. used isinstance instead of types

Modified dlt view method signiture

33192c2

Correct QurantineTable path for non-uc

fa49fe4

Fixed:

57e07b4

1.Dataflowpipeline silver schema read for uc_enabled 2.Fixed integration tests with databricks sdk for cloudFiles 3.Fixed demos for dais demo for uc_enabled=true and tech_summit demo for uc_enabled=False

Fixed unit tests and cleaned up integration tests + demo

fc00157

1.Renamed Runners and confs

ff03b9c

2.Added readme for running demo 3.Added docs to code

1.adding databricks labs cli changes

052b197

2.integration tests restructring as per cli changes

added logic to map to commands

c8f5f8e

Added validation for cli commands

ac50f77

Fixed:

78cbbfb

1.Labs CLI: onboard command with uc 2.Labs CLI: deploy command with uc

Added:

5906961

1.CLI: onboard for nonuc 2.CLI: deploy for nonuc

1.Modified dltmeta cli wrapping in class and using installer whl file…

075a280

… uploader 2.correcty labs.yaml typo to labs.yml 3.correced logger to databricks.labs.dltmeta

Changed whl version to include 0.42.0

bfd0131

Removed linting

b8951e1

Added unit tests

1aaac7c

Removed linting

a2adbd3

Fixed unit test

81246be

Added more unittests for dataflow pipeline type2 merge

0f32031

Added unit tests for bronze merge

555d466

Removed linting error

516607b

Removed linting

a244ee4

Merge pull request #25 from databrickslabs/feature/dlt-meta-uc-cli

5e0f031

Added - dtlmeta labs cli feature - unit test coverage for uc_enabled feature in onboarding and dataflowpipeline

- Added updating paths to onboarding template under cli

9073224

- Added readme instructions for using cli with example

Merge pull request #26 from databrickslabs/feature/dlt-meta-uc-cli

33a8e16

- Added readme instructions for using cli with example - Added cli to use existing demo folder as default

ravi-databricks added 4 commits December 14, 2023 22:08

-Fixed integration tests for non-uc flows

3a180ee

-Fixed cli for non uc flows

Added docs and readme

6816265

Adding unit tests for cli.py

31bc10b

Merge pull request #27 from databrickslabs/feature/dlt-meta-uc-cli

9f1521c

-Fixed integration tests for non-uc flows

ravi-databricks requested review from alexott, ganeshchand, neil90-db and nfx December 15, 2023 23:41

ravi-databricks self-assigned this Dec 15, 2023

ravi-databricks requested a review from howardwu-db December 18, 2023 18:40

ravi-databricks added 2 commits December 20, 2023 10:57

Added:

e0932fe

1.Readme doc for databricks labs cli 2.Unit tests

Merge pull request #30 from databrickslabs/feature/dlt-meta-uc-cli

2214076

Added unit tests and readme

howardwu-db approved these changes Dec 20, 2023

View reviewed changes

ravi-databricks added 2 commits December 20, 2023 21:19

- Added hugo docs for databricks labs cli option

0ad5d67

- Added change log - added version 0.0.5

Merge pull request #31 from databrickslabs/feature/dlt-meta-uc-cli

b0e2e31

- Added hugo docs for databricks labs cli option

ravi-databricks mentioned this pull request Dec 21, 2023

Unity dataflowspec tables #3

Closed

neil90-db approved these changes Jan 5, 2024

View reviewed changes

ganeshchand reviewed Jan 5, 2024

View reviewed changes

nfx approved these changes Jan 5, 2024

View reviewed changes

ganeshchand approved these changes Jan 5, 2024

View reviewed changes

ravi-databricks merged commit 2a93dd9 into main Jan 5, 2024

ravi-databricks deleted the feature/dlt-meta-uc branch April 30, 2024 21:20

Conversation

ravi-databricks commented Dec 15, 2023

Unity Catalog Support

Databricks Labs CLI Support

Added two commands for DLT-META

Uh oh!

codecov bot commented Dec 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

howardwu-db left a comment

Choose a reason for hiding this comment

Uh oh!

neil90-db left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nfx left a comment

Choose a reason for hiding this comment

Uh oh!

ganeshchand left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov bot commented Dec 15, 2023 •

edited

Loading