-
Notifications
You must be signed in to change notification settings - Fork 333
Abstract common OAuth handling and add OAuth support to Athena DataLake Gen2 Connector #2932
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
⏳ I'm reviewing this pull request for security vulnerabilities and code quality issues. I'll provide an update when I'm done |
1 similar comment
|
⏳ I'm reviewing this pull request for security vulnerabilities and code quality issues. I'll provide an update when I'm done |
|
✅ I finished the code review, and didn't find any security or code quality issues. |
1 similar comment
|
✅ I finished the code review, and didn't find any security or code quality issues. |
| { | ||
| this(databaseConnectionConfig, S3Client.create(), SecretsManagerClient.create(), AthenaClient.create(), | ||
| new GenericJdbcConnectionFactory(databaseConnectionConfig, DataLakeGen2MetadataHandler.JDBC_PROPERTIES, | ||
| new DataLakeGen2JdbcConnectionFactory(databaseConnectionConfig, DataLakeGen2MetadataHandler.JDBC_PROPERTIES, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How come DataLakeGen2RecordHandler was using GenericJdbcConnectionFactory when there is a DataLakeGen2JdbcConnectionFactory class being used by MetadataHandler before?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| public static final String QUOTE_CHARACTER = "\""; | ||
|
|
||
| //Property name used to set OAuth access token in JDBC connection properties | ||
| public static final String ACCESS_TOKEN_PROPERTY = "accessToken"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will this property be used by other connectors that can use OAuth? Can we move this constant variable somewhere else so connectors like Synapse can use this (if the names are the same)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, lets think about moving this to the federation SDK since we're already extracting common OAuth code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| protected OAuthCredentialsProvider(String secretName, SecretsManagerClient secretsClient, HttpClient httpClient) | ||
| { | ||
| this.secretName = secretName; | ||
| this.secretsManager = new CachableSecretsManager(secretsClient); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we re-use the secretsManager object from MetadataHandler and RecordHandler?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, added getCachableSecretsManager() method to return same secretManger instance of MetadataHandler and RecordHandler.
| /** | ||
| * Checks if OAuth is configured by verifying required fields exist. | ||
| */ | ||
| protected abstract boolean isOAuthConfigured(Map<String, String> secretMap); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does this need to be abstract? Do two connectors check if oauth is configured in different ways?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this has to be abstract. For example SAP HANA uses client_id, client_secret and token_url.
| public static final String QUOTE_CHARACTER = "\""; | ||
|
|
||
| //Property name used to set OAuth access token in JDBC connection properties | ||
| public static final String ACCESS_TOKEN_PROPERTY = "accessToken"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, lets think about moving this to the federation SDK since we're already extracting common OAuth code.
| public static final String CLIENT_SECRET = "client_secret"; | ||
|
|
||
| // Constants for basic authentication fields | ||
| public static final String USERNAME = "username"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CredentialsProvider already exposes username, password
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated code. Thank you!
| } | ||
|
|
||
| @Override | ||
| public DefaultCredentials getCredential() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should improve the abstraction here. I suggest the following:
- Credential interface -> UserNamePasswordCredential and OAuthAccessTokenCredential
- OAuthCredentialsProvider returns a Credential object, and subclasses for OAuthCredentialsProvider will return the actual shape of the credential. A separate UserNamePasswordCredentialProvider should handle username/password separately. CredentialProvider can then be simplified
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@burhan94 ,
We have created a Credentials interface, which is implemented by both OAuthAccessTokenCredentials and the existing DefaultCredentials (for username/password).
We also created an OAuthCredentialsProvider that returns an OAuthAccessTokenCredentials object.
Instead of creating a new class for UserNamePasswordCredentialProvider, we have used the existing DefaultCredentialsProvider. Thank you!
3286ffa to
acc3ba4
Compare
|
Can you add some more tests? looks like code coverage check is failing |
acc3ba4 to
c10151e
Compare
Hi @burhan94, thank you for reviewing the PR. Initially, we added unit tests to cover the major OAuth code. However, as the design was further updated, it resulted in multiple changes across classes. Unit Tests have been added to improve the coverage of these changed classes, including the scenarios for error handling. |
06c1dab to
50fdc89
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #2932 +/- ##
============================================
+ Coverage 63.67% 64.35% +0.68%
- Complexity 4344 4550 +206
============================================
Files 621 632 +11
Lines 23286 23972 +686
Branches 2859 2954 +95
============================================
+ Hits 14827 15428 +601
- Misses 7070 7124 +54
- Partials 1389 1420 +31 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
50fdc89 to
1a66707
Compare
| /** | ||
| * Utility class for handling credential provider functionality. | ||
| */ | ||
| public final class DataLakeGen2CredentialProviderUtils |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class seems pretty generic. Seems like it returns oauthcredentialsprovider or returns defaultcredentialprovider. Any way to make this re-usable by synapse and other connectors who support oauth?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm thinking this part can be abstracted out, and all connectors can eventually call this utils class to get default/oauth credentials
if (DataLakeGen2OAuthCredentialsProvider.isOAuthConfigured(secretMap)) {
return new DataLakeGen2OAuthCredentialsProvider(secretName, secretMap, secretsManager);
}
```
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @ejeffrli , I've now moved this logic into the CredentialsProviderFactory class in the federation SDK module, making it reusable across all connectors that need OAuth support. Thank you!
|
Can you also provide an up-to-date testing document of |
Hi @ejeffrli , attached test documents to the PR. Thank you! |
fb7c2f6 to
a0db54b
Compare
| public static <T extends OAuthCredentialsProvider> CredentialsProvider createCredentialProvider( | ||
| String secretName, | ||
| CachableSecretsManager secretsManager, | ||
| Class<T> oAuthProviderClass) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really think we shouldn't use reflection when we have control over the lifecycle of these objects. here are my thoughts:
-
Current Issue:
- The code unnecessarily uses reflection when we have full control over all intended classes
- Passing class objects adds complexity without benefits in this controlled environment
-
Proposed Solution:
- Replace reflection with an interface-based approach
- Create an interface (e.g.,
AthenaFederationCredentialProvider) that all credential providers implement (if you find a better name go for it) - The interface should define the setup/initialization behavior
-
Implementation Suggestion:
- Instead of:
Class<T> oAuthProviderClass // as parameter
- Use:
AthenaFederationCredentialProvider provider // as parameter
- Each provider (like DataLakeGen2CredentialsProvider) would implement this interface
- The CredentialProviderFactory would work with the interface directly, eliminating the need for reflection
- Instead of:
-
Benefits:
- More straightforward and type-safe implementation
- Better compile-time checking
- Easier to maintain and extend
- More explicit contract between components
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @AbdulR3hman,Thank you for your suggestion. we’ve removed reflection and refactored to an interface-based approach (InitializableCredentialsProvider). All advanced providers (e.g., OAuthCredentialsProvider) will implement this interface, leveraging properties from Secrets Manager for initialization. Thank you!
d7475d0 to
f538ea6
Compare
| { | ||
| private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); | ||
|
|
||
| private CredentialsProviderFactory() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
more of a dependency injection approach than a static factory function.
Hi @ejeffrli, it seems you have deleted the two review comments above which are related to this comment. We just want to confirm is this still valid? |
|
Met with Abdul and Jeff, we are good to approve. |
- Updating Zookeeper to latest version 3.9.4 (#3005) - build(deps): bump com.google.protobuf:protobuf-bom from 4.29.3 to 4.32.0 (#2991) - Add serverless datalakegen2 support (#2973) - Abstract common OAuth handling and add OAuth support to Athena DataLake Gen2 Connector (#2932) - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.32.29 to 2.33.4 (#2992) - build(deps): bump org.elasticsearch.client:elasticsearch-rest-client from 9.1.2 to 9.1.3 (#2993) - build(deps): bump io.lettuce:lettuce-core from 6.8.0.RELEASE to 6.8.1.RELEASE (#2994) - Hbase namespace issue (#2996) - build(deps): bump actions/setup-java from 4 to 5 (#2960) - Adding support for Substrait query plan and update DDb connector implementation to use the plan (#2966) - update cloudera driver to address zookeeper CVE (#2948)
…ke Gen2 Connector (awslabs#2932) Co-authored-by: burhan94 <[email protected]>
- Update runner and slug - Update GitHub Actions workflows to build with Java 11 and 17 - Remove hard-coded Glue list-jobs --max-results 100 to find all Glue jobs (awslabs#3127) - Added unit tests in athena-synapse Connector (awslabs#2963) - Updating EncryptionKeyFactory to add overirde AWS request configuration in KMS calls made generate encryption key (awslabs#3103) - Fix epoch date conversion correctness issue when machine time zone is not in UTC (awslabs#3108) - Fix CVE-2025-48924: Upgrade Apache Commons Lang3 to 3.19.0 (awslabs#3100) - Revise Athena Federated Queries instructions in README (awslabs#3069) - fix snowflake QPT return empty result (awslabs#3106) - add view into oracle paginated query (awslabs#3107) - build(deps): bump com.google.protobuf:protobuf-bom from 4.32.1 to 4.33.0 (awslabs#3071) - build(deps): bump com.github.spotbugs:spotbugs-annotations from 4.9.7 to 4.9.8 (awslabs#3076) - build(deps): bump org.apache.avro:avro from 1.12.0 to 1.12.1 (awslabs#3075) - Handle ResourceNotFoundException from Dynamodb as AthenaConnectorExce… (awslabs#3098) - Added unit tests for bigquery (awslabs#2950) - Update PostgreSQL engine version to 15.10 (awslabs#3099) - Add unit tests for athena-oracle. (awslabs#2836) - Adding support to use custom SecretManagerClient for Google Big Query (awslabs#2846) - Added unit tests for athena-cloudera-impala Connector (awslabs#2880) - added unit tests for athena-vertica. (awslabs#2783) - added unit tests for athena-redshift. (awslabs#2733) - Always include partition column when get-table-layout (awslabs#3045) - fix cloudwatch glue connection cfn template (awslabs#3013) - Added unit tests for JDBC module (awslabs#2732) - Added pagination for Db2 connector (awslabs#2772) - [Fix] Include default truststore path when passing JAVA_TOOL_OPTIONS for Java 17 image (awslabs#3007) - build(deps): bump com.google.guava:guava from 33.4.0-jre to 33.4.8-jre (awslabs#2728) - Handle KMS and DDB NotFoundExceptions by throwing AthenaConnectorException (awslabs#3064) - build(deps): bump aws-sdk-v2.version from 2.35.1 to 2.35.5 (awslabs#3047) - build(deps): bump io.confluent:kafka-avro-serializer from 8.0.0 to 8.0.2 (awslabs#3055) - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.35.0 to 2.35.5 (awslabs#3057) - build(deps): bump org.apache.maven.plugins:maven-dependency-plugin from 3.8.1 to 3.9.0 (awslabs#3061) - Remove timestamp case from SnowflakeQueryStringBuilder (awslabs#2997) - build(deps): bump io.substrait.version from 0.65.0 to 0.66.0 (awslabs#3051) - build(deps): bump com.google.cloud:google-cloud-storage from 2.58.0 to 2.58.1 (awslabs#3059) - build(deps): bump org.postgresql:postgresql from 42.7.7 to 42.7.8 (awslabs#3063) - build(deps): bump org.apache.maven.plugins:maven-javadoc-plugin from 3.11.3 to 3.12.0 (awslabs#3062) - build(deps): bump com.github.spotbugs:spotbugs-annotations from 4.9.4 to 4.9.6 (awslabs#3058) - build(deps): bump org.jacoco:jacoco-maven-plugin from 0.8.13 to 0.8.14 (awslabs#3060) - build(deps): bump net.sf.jt400:jt400 from 21.0.5 to 21.0.6 (awslabs#3053) - build(deps): bump com.teradata.jdbc:terajdbc from 20.00.00.50 to 20.00.00.51 (awslabs#3054) - build(deps): bump org.elasticsearch.client:elasticsearch-rest-client from 9.1.3 to 9.1.5 (awslabs#3056) - build(deps): bump org.bouncycastle:bcpkix-jdk18on from 1.81 to 1.82 (awslabs#3050) - build(deps): bump org.eclipse.rdf4j:rdf4j-repository-sparql from 5.1.6 to 5.2.0 (awslabs#3052) - build(deps): bump software.amazon.jsii:jsii-runtime from 1.114.1 to 1.115.0 (awslabs#3048) - build(deps): bump aws-sdk-v2.version from 2.34.5 to 2.35.0 (awslabs#3039) - build(deps-dev): bump nl.jqno.equalsverifier:equalsverifier from 4.1.1 to 4.2 (awslabs#3037) - build(deps): bump aws-sdk.version from 1.12.791 to 1.12.792 (awslabs#3035) - build(deps): bump net.java.dev.jna:jna-platform from 5.17.0 to 5.18.1 (awslabs#3038) - build(deps-dev): bump log4j2Version from 2.25.1 to 2.25.2 (awslabs#3029) - build(deps): bump io.substrait.version from 0.52.0 to 0.65.0 (awslabs#3021) - build(deps): bump org.assertj:assertj-core from 3.27.4 to 3.27.6 (awslabs#3019) - build(deps): bump com.amazonaws:aws-lambda-java-core from 1.3.0 to 1.4.0 (awslabs#3020) - build(deps): bump net.java.dev.jna:jna from 5.17.0 to 5.18.1 (awslabs#3034) - build(deps): bump org.jetbrains.kotlin:kotlin-stdlib from 2.2.10 to 2.2.20 (awslabs#3027) - build(deps): bump org.apache.commons:commons-lang3 from 3.18.0 to 3.19.0 (awslabs#3033) - build(deps): bump org.bouncycastle:bcprov-jdk18on from 1.81 to 1.82 (awslabs#3022) - build(deps): bump org.bouncycastle:bcutil-jdk18on from 1.81 to 1.82 (awslabs#3024) - build(deps): bump org.sonatype.central:central-publishing-maven-plugin from 0.8.0 to 0.9.0 (awslabs#3026) - build(deps): bump org.codehaus.mojo:license-maven-plugin from 2.6.0 to 2.7.0 (awslabs#3023) - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.33.8 to 2.35.0 (awslabs#3040) - Add Support for OAuth in athena-saphana Connector (awslabs#2894) - build(deps): bump aws-actions/configure-aws-credentials from 4 to 5 (awslabs#2975) - build(deps): bump actions/setup-node from 4 to 5 (awslabs#2976) - [Neptune] Add doc details on how multi-valued properties are handled. (awslabs#2995) - build(deps): bump org.jetbrains.kotlin:kotlin-reflect from 2.2.10 to 2.2.20 (awslabs#3028) - build(deps): bump software.amazon.glue:schema-registry-serde from 1.1.24 to 1.1.25 (awslabs#3030) - build(deps): bump org.jetbrains.kotlin:kotlin-stdlib-jdk8 from 2.2.10 to 2.2.20 (awslabs#3031) - build(deps): bump com.teradata.jdbc:terajdbc from 20.00.00.49 to 20.00.00.50 (awslabs#3032) - build(deps): bump aws-sdk-v2.version from 2.34.3 to 2.34.5 (awslabs#3017) - Add Support for OAuth in athena-sqlserver Connector (awslabs#3006) - Add Support for OAuth in athena-synapse Connector (awslabs#2904) - build(deps): bump software.amazon.jsii:jsii-runtime from 1.113.0 to 1.114.1 (awslabs#2979) - build(deps): bump org.apache.calcite.version from 1.39.0 to 1.40.0 (awslabs#2980) - build(deps): bump aws-sdk.version from 1.12.788 to 1.12.791 (awslabs#2981) - build(deps): bump org.eclipse.rdf4j:rdf4j-repository-sparql from 5.1.4 to 5.1.5 (awslabs#2983) - build(deps): bump com.clickhouse:clickhouse-jdbc from 0.9.1 to 0.9.2 (awslabs#2985) - build(deps): bump net.snowflake:snowflake-jdbc from 3.26.0 to 3.26.1 (awslabs#2987) - build(deps): bump org.yaml:snakeyaml from 2.4 to 2.5 (awslabs#2988) - build(deps-dev): bump nl.jqno.equalsverifier:equalsverifier from 4.0.9 to 4.1 (awslabs#2998) - build(deps): bump aws-sdk-v2.version from 2.32.29 to 2.33.9 (awslabs#2999) - build(deps): bump org.apache.kafka:kafka-clients from 4.0.0 to 4.1.0 (awslabs#3000) - build(deps): bump surefire.failsafe.version from 3.5.3 to 3.5.4 (awslabs#3001) - build(deps): bump com.microsoft.azure:msal4j from 1.22.0 to 1.23.1 (awslabs#3002) - build(deps): bump org.apache.maven.plugins:maven-shade-plugin from 3.6.0 to 3.6.1 (awslabs#3003) - build(deps): bump com.google.cloud:google-cloud-storage from 2.55.0 to 2.57.0 (awslabs#3004) - build(deps): bump com.sap.cloud.db.jdbc:ngdbc from 2.25.12 to 2.26.6 (awslabs#3012) - build(deps): bump org.apache.maven.plugins:maven-compiler-plugin from 3.14.0 to 3.14.1 (awslabs#3011) - Fixing error messages to not leak sensitive info (awslabs#3008) - Include linked accounts option when querying metric_samples table. (awslabs#2922) - Updating Zookeeper to latest version 3.9.4 (awslabs#3005) - build(deps): bump com.google.protobuf:protobuf-bom from 4.29.3 to 4.32.0 (awslabs#2991) - Add serverless datalakegen2 support (awslabs#2973) - Abstract common OAuth handling and add OAuth support to Athena DataLake Gen2 Connector (awslabs#2932) - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.32.29 to 2.33.4 (awslabs#2992) - build(deps): bump org.elasticsearch.client:elasticsearch-rest-client from 9.1.2 to 9.1.3 (awslabs#2993) - build(deps): bump io.lettuce:lettuce-core from 6.8.0.RELEASE to 6.8.1.RELEASE (awslabs#2994) - Hbase namespace issue (awslabs#2996)
- wait for release branch and checkout - wait for release branch and checkout - Update runner and slug - Update GitHub Actions workflows to build with Java 11 and 17 - Remove hard-coded Glue list-jobs --max-results 100 to find all Glue jobs (awslabs#3127) - Added unit tests in athena-synapse Connector (awslabs#2963) - Updating EncryptionKeyFactory to add overirde AWS request configuration in KMS calls made generate encryption key (awslabs#3103) - Fix epoch date conversion correctness issue when machine time zone is not in UTC (awslabs#3108) - Fix CVE-2025-48924: Upgrade Apache Commons Lang3 to 3.19.0 (awslabs#3100) - Revise Athena Federated Queries instructions in README (awslabs#3069) - fix snowflake QPT return empty result (awslabs#3106) - add view into oracle paginated query (awslabs#3107) - build(deps): bump com.google.protobuf:protobuf-bom from 4.32.1 to 4.33.0 (awslabs#3071) - build(deps): bump com.github.spotbugs:spotbugs-annotations from 4.9.7 to 4.9.8 (awslabs#3076) - build(deps): bump org.apache.avro:avro from 1.12.0 to 1.12.1 (awslabs#3075) - Handle ResourceNotFoundException from Dynamodb as AthenaConnectorExce… (awslabs#3098) - Added unit tests for bigquery (awslabs#2950) - Update PostgreSQL engine version to 15.10 (awslabs#3099) - Add unit tests for athena-oracle. (awslabs#2836) - Adding support to use custom SecretManagerClient for Google Big Query (awslabs#2846) - Added unit tests for athena-cloudera-impala Connector (awslabs#2880) - added unit tests for athena-vertica. (awslabs#2783) - added unit tests for athena-redshift. (awslabs#2733) - Always include partition column when get-table-layout (awslabs#3045) - fix cloudwatch glue connection cfn template (awslabs#3013) - Added unit tests for JDBC module (awslabs#2732) - Added pagination for Db2 connector (awslabs#2772) - [Fix] Include default truststore path when passing JAVA_TOOL_OPTIONS for Java 17 image (awslabs#3007) - build(deps): bump com.google.guava:guava from 33.4.0-jre to 33.4.8-jre (awslabs#2728) - Handle KMS and DDB NotFoundExceptions by throwing AthenaConnectorException (awslabs#3064) - build(deps): bump aws-sdk-v2.version from 2.35.1 to 2.35.5 (awslabs#3047) - build(deps): bump io.confluent:kafka-avro-serializer from 8.0.0 to 8.0.2 (awslabs#3055) - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.35.0 to 2.35.5 (awslabs#3057) - build(deps): bump org.apache.maven.plugins:maven-dependency-plugin from 3.8.1 to 3.9.0 (awslabs#3061) - Remove timestamp case from SnowflakeQueryStringBuilder (awslabs#2997) - build(deps): bump io.substrait.version from 0.65.0 to 0.66.0 (awslabs#3051) - build(deps): bump com.google.cloud:google-cloud-storage from 2.58.0 to 2.58.1 (awslabs#3059) - build(deps): bump org.postgresql:postgresql from 42.7.7 to 42.7.8 (awslabs#3063) - build(deps): bump org.apache.maven.plugins:maven-javadoc-plugin from 3.11.3 to 3.12.0 (awslabs#3062) - build(deps): bump com.github.spotbugs:spotbugs-annotations from 4.9.4 to 4.9.6 (awslabs#3058) - build(deps): bump org.jacoco:jacoco-maven-plugin from 0.8.13 to 0.8.14 (awslabs#3060) - build(deps): bump net.sf.jt400:jt400 from 21.0.5 to 21.0.6 (awslabs#3053) - build(deps): bump com.teradata.jdbc:terajdbc from 20.00.00.50 to 20.00.00.51 (awslabs#3054) - build(deps): bump org.elasticsearch.client:elasticsearch-rest-client from 9.1.3 to 9.1.5 (awslabs#3056) - build(deps): bump org.bouncycastle:bcpkix-jdk18on from 1.81 to 1.82 (awslabs#3050) - build(deps): bump org.eclipse.rdf4j:rdf4j-repository-sparql from 5.1.6 to 5.2.0 (awslabs#3052) - build(deps): bump software.amazon.jsii:jsii-runtime from 1.114.1 to 1.115.0 (awslabs#3048) - build(deps): bump aws-sdk-v2.version from 2.34.5 to 2.35.0 (awslabs#3039) - build(deps-dev): bump nl.jqno.equalsverifier:equalsverifier from 4.1.1 to 4.2 (awslabs#3037) - build(deps): bump aws-sdk.version from 1.12.791 to 1.12.792 (awslabs#3035) - build(deps): bump net.java.dev.jna:jna-platform from 5.17.0 to 5.18.1 (awslabs#3038) - build(deps-dev): bump log4j2Version from 2.25.1 to 2.25.2 (awslabs#3029) - build(deps): bump io.substrait.version from 0.52.0 to 0.65.0 (awslabs#3021) - build(deps): bump org.assertj:assertj-core from 3.27.4 to 3.27.6 (awslabs#3019) - build(deps): bump com.amazonaws:aws-lambda-java-core from 1.3.0 to 1.4.0 (awslabs#3020) - build(deps): bump net.java.dev.jna:jna from 5.17.0 to 5.18.1 (awslabs#3034) - build(deps): bump org.jetbrains.kotlin:kotlin-stdlib from 2.2.10 to 2.2.20 (awslabs#3027) - build(deps): bump org.apache.commons:commons-lang3 from 3.18.0 to 3.19.0 (awslabs#3033) - build(deps): bump org.bouncycastle:bcprov-jdk18on from 1.81 to 1.82 (awslabs#3022) - build(deps): bump org.bouncycastle:bcutil-jdk18on from 1.81 to 1.82 (awslabs#3024) - build(deps): bump org.sonatype.central:central-publishing-maven-plugin from 0.8.0 to 0.9.0 (awslabs#3026) - build(deps): bump org.codehaus.mojo:license-maven-plugin from 2.6.0 to 2.7.0 (awslabs#3023) - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.33.8 to 2.35.0 (awslabs#3040) - Add Support for OAuth in athena-saphana Connector (awslabs#2894) - build(deps): bump aws-actions/configure-aws-credentials from 4 to 5 (awslabs#2975) - build(deps): bump actions/setup-node from 4 to 5 (awslabs#2976) - [Neptune] Add doc details on how multi-valued properties are handled. (awslabs#2995) - build(deps): bump org.jetbrains.kotlin:kotlin-reflect from 2.2.10 to 2.2.20 (awslabs#3028) - build(deps): bump software.amazon.glue:schema-registry-serde from 1.1.24 to 1.1.25 (awslabs#3030) - build(deps): bump org.jetbrains.kotlin:kotlin-stdlib-jdk8 from 2.2.10 to 2.2.20 (awslabs#3031) - build(deps): bump com.teradata.jdbc:terajdbc from 20.00.00.49 to 20.00.00.50 (awslabs#3032) - build(deps): bump aws-sdk-v2.version from 2.34.3 to 2.34.5 (awslabs#3017) - Add Support for OAuth in athena-sqlserver Connector (awslabs#3006) - Add Support for OAuth in athena-synapse Connector (awslabs#2904) - build(deps): bump software.amazon.jsii:jsii-runtime from 1.113.0 to 1.114.1 (awslabs#2979) - build(deps): bump org.apache.calcite.version from 1.39.0 to 1.40.0 (awslabs#2980) - build(deps): bump aws-sdk.version from 1.12.788 to 1.12.791 (awslabs#2981) - build(deps): bump org.eclipse.rdf4j:rdf4j-repository-sparql from 5.1.4 to 5.1.5 (awslabs#2983) - build(deps): bump com.clickhouse:clickhouse-jdbc from 0.9.1 to 0.9.2 (awslabs#2985) - build(deps): bump net.snowflake:snowflake-jdbc from 3.26.0 to 3.26.1 (awslabs#2987) - build(deps): bump org.yaml:snakeyaml from 2.4 to 2.5 (awslabs#2988) - build(deps-dev): bump nl.jqno.equalsverifier:equalsverifier from 4.0.9 to 4.1 (awslabs#2998) - build(deps): bump aws-sdk-v2.version from 2.32.29 to 2.33.9 (awslabs#2999) - build(deps): bump org.apache.kafka:kafka-clients from 4.0.0 to 4.1.0 (awslabs#3000) - build(deps): bump surefire.failsafe.version from 3.5.3 to 3.5.4 (awslabs#3001) - build(deps): bump com.microsoft.azure:msal4j from 1.22.0 to 1.23.1 (awslabs#3002) - build(deps): bump org.apache.maven.plugins:maven-shade-plugin from 3.6.0 to 3.6.1 (awslabs#3003) - build(deps): bump com.google.cloud:google-cloud-storage from 2.55.0 to 2.57.0 (awslabs#3004) - build(deps): bump com.sap.cloud.db.jdbc:ngdbc from 2.25.12 to 2.26.6 (awslabs#3012) - build(deps): bump org.apache.maven.plugins:maven-compiler-plugin from 3.14.0 to 3.14.1 (awslabs#3011) - Fixing error messages to not leak sensitive info (awslabs#3008) - Include linked accounts option when querying metric_samples table. (awslabs#2922) - Updating Zookeeper to latest version 3.9.4 (awslabs#3005) - build(deps): bump com.google.protobuf:protobuf-bom from 4.29.3 to 4.32.0 (awslabs#2991) - Add serverless datalakegen2 support (awslabs#2973) - Abstract common OAuth handling and add OAuth support to Athena DataLake Gen2 Connector (awslabs#2932) - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.32.29 to 2.33.4 (awslabs#2992) - build(deps): bump org.elasticsearch.client:elasticsearch-rest-client from 9.1.2 to 9.1.3 (awslabs#2993) - build(deps): bump io.lettuce:lettuce-core from 6.8.0.RELEASE to 6.8.1.RELEASE (awslabs#2994) - Hbase namespace issue (awslabs#2996)
- Update release workflow for Java version builds - Enhance cut_release workflow for Java 11 and 17 - wait for release branch and checkout - wait for release branch and checkout - Update runner and slug - Update GitHub Actions workflows to build with Java 11 and 17 - Remove hard-coded Glue list-jobs --max-results 100 to find all Glue jobs (awslabs#3127) - Added unit tests in athena-synapse Connector (awslabs#2963) - Updating EncryptionKeyFactory to add overirde AWS request configuration in KMS calls made generate encryption key (awslabs#3103) - Fix epoch date conversion correctness issue when machine time zone is not in UTC (awslabs#3108) - Fix CVE-2025-48924: Upgrade Apache Commons Lang3 to 3.19.0 (awslabs#3100) - Revise Athena Federated Queries instructions in README (awslabs#3069) - fix snowflake QPT return empty result (awslabs#3106) - add view into oracle paginated query (awslabs#3107) - build(deps): bump com.google.protobuf:protobuf-bom from 4.32.1 to 4.33.0 (awslabs#3071) - build(deps): bump com.github.spotbugs:spotbugs-annotations from 4.9.7 to 4.9.8 (awslabs#3076) - build(deps): bump org.apache.avro:avro from 1.12.0 to 1.12.1 (awslabs#3075) - Handle ResourceNotFoundException from Dynamodb as AthenaConnectorExce… (awslabs#3098) - Added unit tests for bigquery (awslabs#2950) - Update PostgreSQL engine version to 15.10 (awslabs#3099) - Add unit tests for athena-oracle. (awslabs#2836) - Adding support to use custom SecretManagerClient for Google Big Query (awslabs#2846) - Added unit tests for athena-cloudera-impala Connector (awslabs#2880) - added unit tests for athena-vertica. (awslabs#2783) - added unit tests for athena-redshift. (awslabs#2733) - Always include partition column when get-table-layout (awslabs#3045) - fix cloudwatch glue connection cfn template (awslabs#3013) - Added unit tests for JDBC module (awslabs#2732) - Added pagination for Db2 connector (awslabs#2772) - [Fix] Include default truststore path when passing JAVA_TOOL_OPTIONS for Java 17 image (awslabs#3007) - build(deps): bump com.google.guava:guava from 33.4.0-jre to 33.4.8-jre (awslabs#2728) - Handle KMS and DDB NotFoundExceptions by throwing AthenaConnectorException (awslabs#3064) - build(deps): bump aws-sdk-v2.version from 2.35.1 to 2.35.5 (awslabs#3047) - build(deps): bump io.confluent:kafka-avro-serializer from 8.0.0 to 8.0.2 (awslabs#3055) - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.35.0 to 2.35.5 (awslabs#3057) - build(deps): bump org.apache.maven.plugins:maven-dependency-plugin from 3.8.1 to 3.9.0 (awslabs#3061) - Remove timestamp case from SnowflakeQueryStringBuilder (awslabs#2997) - build(deps): bump io.substrait.version from 0.65.0 to 0.66.0 (awslabs#3051) - build(deps): bump com.google.cloud:google-cloud-storage from 2.58.0 to 2.58.1 (awslabs#3059) - build(deps): bump org.postgresql:postgresql from 42.7.7 to 42.7.8 (awslabs#3063) - build(deps): bump org.apache.maven.plugins:maven-javadoc-plugin from 3.11.3 to 3.12.0 (awslabs#3062) - build(deps): bump com.github.spotbugs:spotbugs-annotations from 4.9.4 to 4.9.6 (awslabs#3058) - build(deps): bump org.jacoco:jacoco-maven-plugin from 0.8.13 to 0.8.14 (awslabs#3060) - build(deps): bump net.sf.jt400:jt400 from 21.0.5 to 21.0.6 (awslabs#3053) - build(deps): bump com.teradata.jdbc:terajdbc from 20.00.00.50 to 20.00.00.51 (awslabs#3054) - build(deps): bump org.elasticsearch.client:elasticsearch-rest-client from 9.1.3 to 9.1.5 (awslabs#3056) - build(deps): bump org.bouncycastle:bcpkix-jdk18on from 1.81 to 1.82 (awslabs#3050) - build(deps): bump org.eclipse.rdf4j:rdf4j-repository-sparql from 5.1.6 to 5.2.0 (awslabs#3052) - build(deps): bump software.amazon.jsii:jsii-runtime from 1.114.1 to 1.115.0 (awslabs#3048) - build(deps): bump aws-sdk-v2.version from 2.34.5 to 2.35.0 (awslabs#3039) - build(deps-dev): bump nl.jqno.equalsverifier:equalsverifier from 4.1.1 to 4.2 (awslabs#3037) - build(deps): bump aws-sdk.version from 1.12.791 to 1.12.792 (awslabs#3035) - build(deps): bump net.java.dev.jna:jna-platform from 5.17.0 to 5.18.1 (awslabs#3038) - build(deps-dev): bump log4j2Version from 2.25.1 to 2.25.2 (awslabs#3029) - build(deps): bump io.substrait.version from 0.52.0 to 0.65.0 (awslabs#3021) - build(deps): bump org.assertj:assertj-core from 3.27.4 to 3.27.6 (awslabs#3019) - build(deps): bump com.amazonaws:aws-lambda-java-core from 1.3.0 to 1.4.0 (awslabs#3020) - build(deps): bump net.java.dev.jna:jna from 5.17.0 to 5.18.1 (awslabs#3034) - build(deps): bump org.jetbrains.kotlin:kotlin-stdlib from 2.2.10 to 2.2.20 (awslabs#3027) - build(deps): bump org.apache.commons:commons-lang3 from 3.18.0 to 3.19.0 (awslabs#3033) - build(deps): bump org.bouncycastle:bcprov-jdk18on from 1.81 to 1.82 (awslabs#3022) - build(deps): bump org.bouncycastle:bcutil-jdk18on from 1.81 to 1.82 (awslabs#3024) - build(deps): bump org.sonatype.central:central-publishing-maven-plugin from 0.8.0 to 0.9.0 (awslabs#3026) - build(deps): bump org.codehaus.mojo:license-maven-plugin from 2.6.0 to 2.7.0 (awslabs#3023) - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.33.8 to 2.35.0 (awslabs#3040) - Add Support for OAuth in athena-saphana Connector (awslabs#2894) - build(deps): bump aws-actions/configure-aws-credentials from 4 to 5 (awslabs#2975) - build(deps): bump actions/setup-node from 4 to 5 (awslabs#2976) - [Neptune] Add doc details on how multi-valued properties are handled. (awslabs#2995) - build(deps): bump org.jetbrains.kotlin:kotlin-reflect from 2.2.10 to 2.2.20 (awslabs#3028) - build(deps): bump software.amazon.glue:schema-registry-serde from 1.1.24 to 1.1.25 (awslabs#3030) - build(deps): bump org.jetbrains.kotlin:kotlin-stdlib-jdk8 from 2.2.10 to 2.2.20 (awslabs#3031) - build(deps): bump com.teradata.jdbc:terajdbc from 20.00.00.49 to 20.00.00.50 (awslabs#3032) - build(deps): bump aws-sdk-v2.version from 2.34.3 to 2.34.5 (awslabs#3017) - Add Support for OAuth in athena-sqlserver Connector (awslabs#3006) - Add Support for OAuth in athena-synapse Connector (awslabs#2904) - build(deps): bump software.amazon.jsii:jsii-runtime from 1.113.0 to 1.114.1 (awslabs#2979) - build(deps): bump org.apache.calcite.version from 1.39.0 to 1.40.0 (awslabs#2980) - build(deps): bump aws-sdk.version from 1.12.788 to 1.12.791 (awslabs#2981) - build(deps): bump org.eclipse.rdf4j:rdf4j-repository-sparql from 5.1.4 to 5.1.5 (awslabs#2983) - build(deps): bump com.clickhouse:clickhouse-jdbc from 0.9.1 to 0.9.2 (awslabs#2985) - build(deps): bump net.snowflake:snowflake-jdbc from 3.26.0 to 3.26.1 (awslabs#2987) - build(deps): bump org.yaml:snakeyaml from 2.4 to 2.5 (awslabs#2988) - build(deps-dev): bump nl.jqno.equalsverifier:equalsverifier from 4.0.9 to 4.1 (awslabs#2998) - build(deps): bump aws-sdk-v2.version from 2.32.29 to 2.33.9 (awslabs#2999) - build(deps): bump org.apache.kafka:kafka-clients from 4.0.0 to 4.1.0 (awslabs#3000) - build(deps): bump surefire.failsafe.version from 3.5.3 to 3.5.4 (awslabs#3001) - build(deps): bump com.microsoft.azure:msal4j from 1.22.0 to 1.23.1 (awslabs#3002) - build(deps): bump org.apache.maven.plugins:maven-shade-plugin from 3.6.0 to 3.6.1 (awslabs#3003) - build(deps): bump com.google.cloud:google-cloud-storage from 2.55.0 to 2.57.0 (awslabs#3004) - build(deps): bump com.sap.cloud.db.jdbc:ngdbc from 2.25.12 to 2.26.6 (awslabs#3012) - build(deps): bump org.apache.maven.plugins:maven-compiler-plugin from 3.14.0 to 3.14.1 (awslabs#3011) - Fixing error messages to not leak sensitive info (awslabs#3008) - Include linked accounts option when querying metric_samples table. (awslabs#2922) - Updating Zookeeper to latest version 3.9.4 (awslabs#3005) - build(deps): bump com.google.protobuf:protobuf-bom from 4.29.3 to 4.32.0 (awslabs#2991) - Add serverless datalakegen2 support (awslabs#2973) - Abstract common OAuth handling and add OAuth support to Athena DataLake Gen2 Connector (awslabs#2932) - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.32.29 to 2.33.4 (awslabs#2992) - build(deps): bump org.elasticsearch.client:elasticsearch-rest-client from 9.1.2 to 9.1.3 (awslabs#2993) - build(deps): bump io.lettuce:lettuce-core from 6.8.0.RELEASE to 6.8.1.RELEASE (awslabs#2994) - Hbase namespace issue (awslabs#2996)
Issue #, if available:
Description of changes:
Abstracted common OAuth handling into a new OAuthCredentialsProvider base class within the SDK module to enable reuse across multiple connectors.
Added support for OAuth-based authentication in the DataLake Gen2 connector using Microsoft Entra ID with the client_credentials grant type. The connector now obtains Bearer tokens directly from Entra ID and securely manages them via AWS Secrets Manager.
Users must store the following credentials in AWS Secrets Manager:
{
"client_id": "<your-entra-id-app-client-id>",
"client_secret": "<your-entra-id-app-client-secret>",
"tenant_id": "<your-entra-id-app-tenant-id>"
}
Please find attached reference and test documents.
gen2-oauth-testing.odt
OAuth Integration in datalakegen2.docx
SYNAPSE_FUNCTIONAL_TEST_2025-08-29.xlsx
SQLSERVER_FUNCTIONAL_TEST_2025-08-29.xlsx
DATALAKEGEN2_FUNCTIONAL_TEST_2025-08-29.xlsx
SNOWFLAKE_FUNCTIONAL_TEST_2025-08-29.xlsx
ORACLE_FUNCTIONAL_TEST_2025-08-29.xlsx
CLOUDERAHIVE_FUNCTIONAL_TEST_2025-08-29.xlsx
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.