Fix databricks pandas error #1443
Conversation
✅ Deploy Preview for dlt-hub-docs canceled.
|
steinitzu
force-pushed
the
sthor/fix-databricks-pandas-error
branch
from
e3c5d68 to
b22f4aa
Compare
steinitzu
force-pushed
the
sthor/fix-databricks-pandas-error
branch
4 times, most recently
from
4cfbb87 to
06de0b3
Compare
steinitzu
force-pushed
the
sthor/fix-databricks-pandas-error
branch
from
06de0b3 to
41a8a6f
Compare
| super().__init__(schema, config, sql_client) | ||
| self.config: DatabricksClientConfiguration = config | ||
| self.sql_client: DatabricksSqlClient = sql_client | ||
| self.sql_client: DatabricksSqlClient = sql_client # type: ignore[assignment] |
There was a problem hiding this comment.
I have no idea tbh. Maybe something to do with 2.9 driver not having type hints? 3.x does.
| def open_connection(self) -> DatabricksSqlConnection: | ||
| conn_params = self.credentials.to_connector_params() | ||
| self._conn = databricks_lib.connect(**conn_params, schema=self.dataset_name) | ||
| self._conn = databricks_lib.connect( |
There was a problem hiding this comment.
this is my only issue here: could we support 2.9 driver? the 3.x has
paramstyle = "named"
and 2.9 pyformat so it is trivial to distinguish them
why: I bet there are people with constrained environments (ie Airflow) where upgrading dependencies is hard. and 3.0 was released only in march
There was a problem hiding this comment.
2.9 still works if you have it installed. There's no change in our code except this argument that's ignored in 2.9.
paramstyle doesn't matter with use_inline_params, this makes 3.x work like 2.x which does not use parametrized queries at all, it's some custom escaping and %s string formatting under the hood. 🤷♂️
There was a problem hiding this comment.
If we want to switch to named paramstyle we can detect driver version and cluster version. But I don't know enough about it to tell if this is reliable in all environments: https://docs.databricks.com/en/sql/language-manual/functions/current_version.html
pyproject.toml
Outdated
| qdrant-client = {version = "^1.6.4", optional = true, extras = ["fastembed"]} | ||
| databricks-sql-connector = {version = ">=2.9.3,<3.0.0", optional = true} | ||
| dbt-databricks = {version = ">=1.7.3", optional = true} | ||
| databricks-sql-connector = {version = "^3", optional = true} |
rudolfix
left a comment
rudolfix
left a comment
There was a problem hiding this comment.
LGTM!
I cleaned up dbt stuff (moved from extras to dependency group), allowed databricks > 2.9.3 bumped dependency to >3.
all seems to work together
Description
Continued from #1436 without fork
Related Issues
Additional Context