Asynchronous DynamoDB Client V2 is slower than V1

[pin2t]

Describe the bug

I tried the DynamoDB asynchronous client v. 2.17.271 and noticed than requests runs slower than on version 1.11.698 (synchronous one)

I also wrote a simple tests for both client versions which calls GetItem request and Scan request in a loop and measure the time.
I used the same HTTP settings, tried on both DynamoDBLocal emulator and actual cloud DynamoDB.
The test for async v2 client runs slower than the test for sync v1 both on DynamoDBLocal and cloud one.

The bigger request (Scan with limit 100) time difference is higher than the smalle one (GetItem). But they both runs slower.

I tried different HTTP netty client configuration (OpenSSL for SSL, completion on thread pool and on event loop thread). Results are the same
What else should I look for (maybe turn on some metrics)? to find where the bottleneck is

Expected Behavior

At least the same performance. And for multiple concurrent requests should be better due to nature of asynchronous requests.

Current Behavior

Current measurements (on DynamoDBLocal), smaller number is better

`
version 2.17.271

scan sequential 200 total: 933ms consumed 0 rcu/0 wcu
scan sequential percentiles: 90th 7ms, 50th 5ms
scan parallel 200 total: 226ms consumed 0 rcu/0 wcu
scan parallel percentiles: 90th 61ms, 50th 29ms
getitem sequential 200 total: 357ms consumed 0 rcu/0 wcu
getitem sequential percentiles: 90th 3ms, 50th 3ms
getitem parallel 200 total: 103ms consumed 0 rcu/0 wcu
getitem parallel percentiles: 90th 9ms, 50th 4ms

version 1.11.698
scan sequential 200 total: 540ms consumed 0.0 rcu/0.0 wcu
scan sequential percentiles: 90th 4ms, 50th 4ms
scan parallel 200 total: 215ms consumed 0.0 rcu/0.0 wcu
scan parallel percentiles: 90th 45ms, 50th 38ms
getitem sequential 200 total: 225ms consumed 0.0 rcu/0.0 wcu
getitem sequential percentiles: 90th 6ms, 50th 2ms
getitem parallel 200 total: 51ms consumed 0.0 rcu/0.0 wcu
getitem parallel percentiles: 90th 20ms, 50th 6ms

`

Reproduction Steps

The code is in v1 and v2

Basically it measures the time for running a loop

for (int i = 0; i < nrequests; i++) {
    stat.measure(() -> {
        ScanResult rs = test.client.scan(new ScanRequest(TABLE)
            .withConsistentRead(FALSE)
            .withLimit(limit)
            .withFilterExpression("#nid > :n")
            .withExpressionAttributeNames(Collections.singletonMap("#nid", "nid"))
            .withExpressionAttributeValues(Collections.singletonMap(":n", new AttributeValue().withN(Long.toString(test.rnd.nextInt(8000)))))
        );
        if (rs.getCount() == 0 || rs.getCount() > limit) throw new RuntimeException("invalid number of rows returned " + rs.getCount());
    });
}

or

for (int i = 0; i < nrequests; i++) {
    ScanResponse rs = stat.measure(
        () -> test.client.scan(ScanRequest.builder()
                .tableName(TABLE)
                .limit(limit)
                .filterExpression("#nid > :n")
                .expressionAttributeNames(Collections.singletonMap("#nid", "nid"))
                .expressionAttributeValues(Collections.singletonMap(":n", AttributeValue.fromN(Long.toString(test.rnd.nextInt(8000)))))
                .consistentRead(FALSE)
                .returnConsumedCapacity(ReturnConsumedCapacity.TOTAL)
                .build())
        .join()
    );
    stat.addConsumed(rs.consumedCapacity());
    if (rs.count() == 0 || rs.count() > limit) throw new RuntimeException("wrong number of rows returned " + rs.count());
}

Client configuration besides authentication

builder
            .withClientConfiguration(new ClientConfiguration()
                .withMaxConnections(200)
                .withConnectionMaxIdleMillis(60_000)
                .withConnectionTimeout(30_000)
                .withClientExecutionTimeout(30_000)
                .withConnectionTTL(60_000)
                .withSocketTimeout(30_000)
                .withTcpKeepAlive(true)
                .withThrottledRetries(true)
            )
            .build()

builder
            .httpClientBuilder(NettyNioAsyncHttpClient.builder()
                .maxConcurrency(200)
                .maxPendingConnectionAcquires(10_000)
                .connectionMaxIdleTime(Duration.ofSeconds(60))
                .connectionTimeout(Duration.ofSeconds(30))
                .connectionAcquisitionTimeout(Duration.ofSeconds(30))
                .readTimeout(Duration.ofSeconds(30)))
            .asyncConfiguration(b -> b.advancedOption(SdkAdvancedAsyncClientOption.FUTURE_COMPLETION_EXECUTOR, Runnable::run))
            .build()

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.17.271

JDK version used

openjdk version "1.8.0_342", openjdk version "11.0.16" 2022-07-19

Operating System and version

Ubuntu Linux 22.04 x86_64

[debora-ito]

Quick (and overdue) update: we are looking into this and other reports of performance issues in v2.

For anyone that is impacted by this, please add a 👍 to the top comment, and share your use case here if you'd like.

[batrived]

Production profiling data: AwsExecutionContextBuilder overhead on DynamoDB-heavy workload

We've been profiling a JanusGraph server that uses the DynamoDB storage backend heavily (Query-dominated workload, ~thousands of DynamoDB calls/sec) on SDK version 2.42.28. Sharing concrete flame graph numbers in case they're useful for this investigation.

What we measured

Using async-profiler on a multi-core JVM (total samples = 7185s wall-clock CPU across all threads over a 43-minute window):

Stack / Method	Cumulative CPU
AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext	6.44%
DynamoDbResolveEndpointInterceptor.modifyRequest (within above)	1.97%
DynamoDbAuthSchemeInterceptor (within above)	~1.0%
ExecutionInterceptorChain$$Lambda_.accept	1.75%
ExecutionAttributes.getAttribute	1.62%

So ~6-7% of all JVM CPU is spent constructing per-request execution context — for a workload where DynamoDB calls are the dominant operation.

Root cause analysis

1. DynamoDbResolveEndpointInterceptor calls DefaultDynamoDbEndpointProvider.resolveEndpoint() on every single request, which re-evaluates the endpoint rules engine and re-parses the URI, even when endpointOverride is statically configured at client construction time.

2. DynamoDbAuthSchemeInterceptor re-resolves auth scheme per request.

3. ExecutionAttributes is constructed as new HashMap<>() (default capacity 16) and then has ~30–40 attributes added during the pipeline, causing multiple resize/rehash cycles per request.

Workaround we found for #1

Bypassing endpointOverride in favor of a custom endpointProvider eliminates the per-request endpoint resolution cost:

URI resolvedEndpoint = URI.create("https://dynamodb.us-east-1.amazonaws.com");
Endpoint staticEndpoint = Endpoint.builder().url(resolvedEndpoint).build();

DynamoDbClient.builder()
    .endpointProvider(params -> CompletableFuture.completedFuture(staticEndpoint))
    // ... rest of config
    .build();

This saves ~0.5% CPU on our workload by short-circuiting DynamoDbResolveEndpointInterceptor entirely.

Note on SDK version

All the improvements from PRs #5643 (unmarshalling), #6383 (SigV4 25% speedup), #4867 are already included in 2.42.28 — the numbers above reflect the current state after those fixes.

Hope this data point helps the investigation. Happy to share more profiling details if useful.