Do not suppress logging information from Python code

[jdavidheiser]

Describe the issue

Using the experimental Python interface, I would like to be able to show logs for my Python code, without showing logs for all of the internals of the CLI code. Would it be possible to stop suppressing stderr / stdout from the Python code when running databricks bundle commands in the CLI?

Steps to reproduce the behavior

Add a print or logger.log call to Python. It will only be shown when running the Databricks CLI with --debug, which is very noisy because it shows all the CLI log info too, and there's no way to set log level like you can in Python.

Expected Behavior

I would like to be able to see Python logs.

Actual Behavior

Some flag to enable Python log output.

OS and CLI version

Any

Is this a regression?

No

Debug Logs

NA

[kanterov]

@jdavidheiser We don't configure logging inside the Python process or dictate how it should be configured. If we start showing output, we have to print all of the Python process's stderr/stdout, which can be noisy and a bad default. With that, we are left conditionally including output with --debug together with less relevant logs for you.

You can make the output less noisy by enabling JSON format and transforming structured logs, for example:

databricks bundle validate --log-format json --debug 2> >(jq 'select(.mutator == "PythonMutator(load_resources)")')

For convenience, I suggest adding a command alias for that, e.g., bundle-debug. You can add such an alias as part of projects.scripts in pyproject.toml that will make it available if the virtual environment is activated or commands like uv run are used.

[jdavidheiser]

I think my point is, if you don't suppress the python output, it allows the author of the python code to configure the logging how they see fit. By suppressing all output from the python process unless debug mode is enabled, I lose that control.

[kanterov]

@jdavidheiser It's understandable, but we also don't want to include output by default. We are considering adding a verbose mode that will be a middle ground between --debug and default logging, and will apply to all CLI commands.