[Bug] Environment variable NO_PROXY is not honored when using thin client

[patped]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The https proxy specified in the environment variable HTTPS_PROXY is used regardless of DNS specified in the environment variable NO_PROXY when using thin client. Thick (CX) client is honoring the NO_PROXY variable.

This behavior results in not being able to connect to databases on our local network.

Expected Behavior

When running thin client, the https proxy should not be used if the dns is listed in the environment variable NO_PROXY.

Steps To Reproduce

1. Set environment variables in your shell
   Example for posix-shell

   HTTPS_PROXY="my.proxy"
   NO_PROXY="database.local"

2. Have a oracle-database running on your local network with the DNS you specify in NO_PROXY from step 1.
3. Configure dbt to target the oracle-database you configured in step 2.
4. Run dbt

Relevant log output using --debug flag enabled

10:32:10  dbt was unable to connect to the specified database.
The database returned the following error:

  >Database Error
  DPY-6005: cannot connect to database (CONNECTION_ID=dbt-oracle-1.9.2-caot9iNuzx7+XxJViaAdSA==).
  DPY-2029: https_proxy requires use of the tcps protocol

Environment

- OS: Not relevant
- Python: Not relevant
- dbt: Not relevant

What Oracle database version are you using dbt with?

Not relevant

Additional Context

A workaround is to unset the HTTPS_PROXY variable when we are connecting to a local database or use thick client.

This issue have been discussed before on slack with a proposed solution, ref: https://getdbt.slack.com/archives/C01PWH4TXLY/p1683206320243519

[aosingh]

@patped

The fix is for users to provide http proxy details in dbt profiles.yml file.

HTTP PROXY environment variables are ignored. Below is an example how you can pass the http proxy details

   outputs:
      dev:
         type: oracle
         user: "{{ env_var('DBT_ORACLE_USER') }}"
         pass: "{{ env_var('DBT_ORACLE_PASSWORD') }}"
         ..
         https_proxy: "www-proxy.com"
         https_proxy_port: 80

The main reason to do this is because, I don't see no_proxy parameter in python-oracledb's connect() method.

[patped]

Thanks for looking into this! This proposed breaking change is fine for me and my company's use case, but maybe other people are using the variables to connect to the database?

The main reason to do this is because, I don't see no_proxy parameter in python-oracledb's connect() method.

Why the connect() doesn't take no_proxy as a parameter kinda makes sense from an api perspective. "If you are not gonna use a proxy to connect to the database, why would you provide it?"

I understand you'r decision to not make some extra logic as proposed on slack. But feel free to reach out if you want me to make a draft of a working proposal based on the slack discussion. I would be happy to work in it.

Either way, we are happy as long as it's fixed. :)

[aosingh]

@patped

Thank you, I remember our discussion on Slack. For now, we decided for a simpler change to fix this.

If someone complains about a breaking change we can release a patch with your proposed solution of parse_connect_string() and determining if we need to bypass the proxy. I like the use of parse_connect_string() which covers all cases.

We just need test all cases for should_bypass_proxy(uri)e.g. when NO_PROXY has wildcard, trailing dots, multiple hosts, I.P address, addresses in a subnet, etc.

no_proxy=example.com
no_proxy=anotherdomain,com,localhost,example.com,*.example.com
no_proxy=*
no_proxy=192.168.1.1,10.0.0.0/8

Weighing in all these factors, for now, we went ahead with a simpler solution but let's see :)

[patped]

Yeah, *_PROXY variables are messy. They don't have a clear spec so they are implemented slightly different all over the place. For future reference the stdlib and requests have function for checking this if you don't want to make your own or need inspiration. See urllib.request.proxy_bypass(host) or requests.utils.should_bypass_proxies(url, no_proxy).