Pure python request pipeline #429

JordonPhillips · 2025-03-12T12:58:33Z

This replaces the generated request pipeline with one that is hand-written. Look at all those removed lines!

Note that CI won't pass because we need quite a few fixes. I'm splitting those off so that this isn't too big, and I've changed the target branch to a feature branch.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

nateprewitt

I think this looks fine conceptually.

packages/smithy-core/src/smithy_core/aio/client.py

JordonPhillips · 2025-05-27T09:52:05Z

This is good to look at now. I'm working on the codegen changeover, which will bloat the pr size since so much is going to be deleted. I would prefer that to be a separate PR though.

nateprewitt

Minor comment but otherwise I think looks alright.

packages/smithy-core/src/smithy_core/aio/client.py

This replaces the code generated request pipeline with the hand-written one.

nateprewitt

I still need to pull this down and generate a client to see what the output delta is. My only main concern/question right now is how we're tracking behavioral/API changes for the larger refactor.

codegen/core/src/main/java/software/amazon/smithy/python/codegen/HttpProtocolTestGenerator.java

nateprewitt · 2025-07-09T19:53:46Z

.../main/java/software/amazon/smithy/python/codegen/generators/MemberDeserializerGenerator.java

+        if (shape.hasTrait(StreamingTrait.class)) {
+            writeDeserializer("data_stream");
+        } else {
+            writeDeserializer(shape);
+        }


What's unique with the data_stream here that shape isn't accounting for? Just that it's an unknown length/shape?

read_blob expects bytes, read_data_stream expects a stream. As for writeDeserializer, when given a shape it uses the name of the shape type as the read method to dispatch to. When given a string, it just does read_{name}. We wouldn't want it to check streaming automatically since it should only apply to blobs.

packages/smithy-core/src/smithy_core/aio/client.py

This adds three top-level config options when sigv4 auth is enabled: * `aws_access_key_id` * `aws_secret_access_key` * `aws_session_token` These are used for static credentials. Names are identical to botocore.

This updates several dataclasses to avoid including schemas in their default `repr` so that they don't get logged without deliberate intent. Logging schemas can be very messy as they are huge.

This updates AsyncBytesReader to expose the underlying seek method of the data source, if it exists.

This updates the error deserializer to check that the content type of a response matches the protocol's content type before it tries searching the payload for a shape id.

This updates query serialization to no longer add an extraneous `&` as well as ensures that the path starts with `/`.

nateprewitt · 2025-08-11T18:37:01Z

packages/smithy-core/src/smithy_core/aio/client.py

+            self._await_output(self._execute_request(call, request_future))
+        )
+        request_context = await request_future
+        input_stream = self.protocol.create_event_publisher(


It doesn't look like we have a concrete implementation for either the publisher or receiver. The old workflow looks like it was calling execute on the operation directly from the _duplex_stream method. Is the intent we need to implement that for every protocol going forward?

# Conflicts: # packages/smithy-http/src/smithy_http/serializers.py

This updates the async signer to use the special payload hash for event stream operations.

JordonPhillips force-pushed the pure-python-request-pipeline branch from bb29685 to 3a6df83 Compare March 12, 2025 17:16

JordonPhillips force-pushed the pure-python-request-pipeline branch 2 times, most recently from ec58e87 to 1890159 Compare April 11, 2025 13:39

JordonPhillips changed the base branch from develop to no-http-auth April 11, 2025 13:40

JordonPhillips force-pushed the pure-python-request-pipeline branch from 1890159 to 0e83eeb Compare April 11, 2025 13:43

nateprewitt reviewed Apr 11, 2025

View reviewed changes

packages/smithy-core/src/smithy_core/aio/client.py Show resolved Hide resolved

JordonPhillips force-pushed the no-http-auth branch from 7ad2adf to 6f474c6 Compare April 16, 2025 13:31

JordonPhillips force-pushed the no-http-auth branch 2 times, most recently from dbae11e to 6dbca91 Compare May 2, 2025 15:06

JordonPhillips force-pushed the pure-python-request-pipeline branch from 0e83eeb to 5d189d2 Compare May 2, 2025 15:10

JordonPhillips force-pushed the no-http-auth branch from 75e5630 to f00644a Compare May 9, 2025 14:27

JordonPhillips force-pushed the pure-python-request-pipeline branch 2 times, most recently from aa96919 to 31a635c Compare May 9, 2025 14:35

Base automatically changed from no-http-auth to develop May 26, 2025 16:02

JordonPhillips force-pushed the pure-python-request-pipeline branch from 31a635c to 6aa97b3 Compare May 26, 2025 16:36

JordonPhillips marked this pull request as ready for review May 27, 2025 09:50

JordonPhillips requested a review from a team as a code owner May 27, 2025 09:50

JordonPhillips requested a review from nateprewitt May 27, 2025 09:50

JordonPhillips changed the title ~~[DRAFT] Pure python request pipeline~~ Pure python request pipeline May 28, 2025

nateprewitt previously approved these changes Jun 2, 2025

View reviewed changes

packages/smithy-core/src/smithy_core/aio/client.py Outdated Show resolved Hide resolved

JordonPhillips dismissed nateprewitt’s stale review via 588c3f2 June 3, 2025 09:15

JordonPhillips force-pushed the pure-python-request-pipeline branch from 6aa97b3 to 588c3f2 Compare June 3, 2025 09:15

JordonPhillips changed the base branch from develop to replace-generated-request-pipeline June 3, 2025 09:16

SamRemis reviewed Jun 3, 2025

View reviewed changes

packages/smithy-core/src/smithy_core/aio/client.py Outdated Show resolved Hide resolved

JordonPhillips changed the base branch from replace-generated-request-pipeline to develop July 1, 2025 17:14

Implement the request pipeline in pure python

2cacaf3

JordonPhillips force-pushed the pure-python-request-pipeline branch from 588c3f2 to 59307ce Compare July 9, 2025 08:49

Replace code generated request pipeline

a07082b

This replaces the code generated request pipeline with the hand-written one.

JordonPhillips force-pushed the pure-python-request-pipeline branch from 59307ce to a07082b Compare July 9, 2025 08:52

JordonPhillips force-pushed the pure-python-request-pipeline branch from 9a777e0 to 7e34bde Compare July 9, 2025 11:49

JordonPhillips added 2 commits July 9, 2025 14:03

Merge signer/identity properties

fdf7ee2

Log endpoint params

2dce401

JordonPhillips requested review from SamRemis and nateprewitt July 9, 2025 12:04

nateprewitt reviewed Jul 10, 2025

View reviewed changes

Update changelogs with unreleased changes

038ae87

JordonPhillips requested a review from nateprewitt July 16, 2025 15:09

nateprewitt and others added 12 commits July 25, 2025 18:20

Fix AuthOptions generator

ac9c6dd

Don't use result of update

63b71a7

Add AWS static config

0a24ac4

This adds three top-level config options when sigv4 auth is enabled: * `aws_access_key_id` * `aws_secret_access_key` * `aws_session_token` These are used for static credentials. Names are identical to botocore.

Avoid logging schemas

924e169

This updates several dataclasses to avoid including schemas in their default `repr` so that they don't get logged without deliberate intent. Logging schemas can be very messy as they are huge.

Wrap HTTP payloads in async readables

b011eb8

Expose underlying seek on AsyncBytesReader

e409916

This updates AsyncBytesReader to expose the underlying seek method of the data source, if it exists.

Check seekable in aws signers

423dc9e

Make ConditionallySeekable a Protocol

d375432

Match test cases to new wrapping behavior in smithy-http

e831bdf

Check unknown error content type before read

c14a0dd

This updates the error deserializer to check that the content type of a response matches the protocol's content type before it tries searching the payload for a shape id.

Don't pass on None payloads

16f4b32

Fix path and query serialization

3a2dff9

This updates query serialization to no longer add an extraneous `&` as well as ensures that the path starts with `/`.

nateprewitt reviewed Aug 11, 2025

View reviewed changes

JordonPhillips added 5 commits August 12, 2025 11:58

Implement event stream methods fro restJson1

f5b9557

Use AsyncBytesProvider for event streams

8561a1c

# Conflicts: # packages/smithy-http/src/smithy_http/serializers.py

Use special payload hash for event streams

39b018f

This updates the async signer to use the special payload hash for event stream operations.

Pass encoded event to signer

48efefc

Don't try to deserialize an event stream as a normal payload

cff549c

nateprewitt approved these changes Aug 12, 2025

View reviewed changes

nateprewitt merged commit f578cb1 into develop Aug 12, 2025
4 checks passed

nateprewitt deleted the pure-python-request-pipeline branch August 12, 2025 19:02

jonathan343 mentioned this pull request Oct 15, 2025

Remove unused serialize.py and deserialize.py modules awslabs/aws-sdk-python#22

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Pure python request pipeline #429

Pure python request pipeline #429

Uh oh!

JordonPhillips commented Mar 12, 2025 •

edited

Loading

Uh oh!

nateprewitt left a comment

Uh oh!

Uh oh!

JordonPhillips commented May 27, 2025

Uh oh!

nateprewitt left a comment

Uh oh!

Uh oh!

Uh oh!

nateprewitt left a comment

Uh oh!

Uh oh!

nateprewitt Jul 9, 2025

Uh oh!

JordonPhillips Jul 11, 2025

Uh oh!

Uh oh!

nateprewitt Aug 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Pure python request pipeline #429

Pure python request pipeline #429

Uh oh!

Conversation

JordonPhillips commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nateprewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JordonPhillips commented May 27, 2025

Uh oh!

nateprewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nateprewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nateprewitt Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

JordonPhillips Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nateprewitt Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JordonPhillips commented Mar 12, 2025 •

edited

Loading