add only cross attention to simple attention blocks #3011
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
williamberman
force-pushed
the
only_cross_attention_simple_attention_blocks
branch
from
c7d7ad9 to
9a56e8f
Compare
williamberman
commented
Apr 7, 2023
| @@ -321,6 +322,7 @@ def __init__( | |||
| attn_num_head_channels=attention_head_dim[-1], | |||
| resnet_groups=norm_num_groups, | |||
| resnet_time_scale_shift=resnet_time_scale_shift, | |||
| only_cross_attention=mid_block_only_cross_attention, | |||
There was a problem hiding this comment.
@patrickvonplaten you had used only_cross_attention[-1] for this argument but I think it makes sense to use a separate config for the mid block. The other arguments for the mid block that re-use the config for the last encoder block make sense because they are dimensionality based and they have to match. But this constraint doesn't hold necessarily for the only_cross_attention flag
There was a problem hiding this comment.
Actually with this commit, 7329ead , we can do a better default to the value of only_cross_attention when it's given as a single boolean
williamberman
requested review from
patrickvonplaten,
pcuenca,
yiyixuxu and
sayakpaul
|
The documentation is not available anymore as the PR was closed or merged. |
williamberman
force-pushed
the
only_cross_attention_simple_attention_blocks
branch
from
9a56e8f to
ad99a33
Compare
patrickvonplaten
approved these changes
Apr 9, 2023
williamberman
force-pushed
the
only_cross_attention_simple_attention_blocks
branch
from
ad99a33 to
ea330a3
Compare
|
I rebased on top of #3046 because was needed for tests |
williamberman
force-pushed
the
only_cross_attention_simple_attention_blocks
branch
from
ea330a3 to
6a43702
Compare
|
@patrickvonplaten I'm not exactly sure what's best to test here. I added a sanity check test that the outputs using only_cross_attention vs not are different here 320d288 Could you take and let me know if that's sufficient before I merge? |
sayakpaul
reviewed
Apr 11, 2023
|
|
||
| only_cross_attn_out = attn(**forward_args) | ||
|
|
||
| self.assertTrue((only_cross_attn_out != self_and_cross_attn_out).all()) |
sayakpaul
approved these changes
Apr 11, 2023
allow mid_block_only_cross_attention to default to `only_cross_attention` when `only_cross_attention` is given as a single boolean
williamberman
force-pushed
the
only_cross_attention_simple_attention_blocks
branch
from
7329ead to
b9ffce2
Compare
patrickvonplaten
approved these changes
Apr 11, 2023
w4ffl35
pushed a commit
to w4ffl35/diffusers
that referenced
this pull request
Apr 14, 2023
* add only cross attention to simple attention blocks * add test for only_cross_attention re: @patrickvonplaten * mid_block_only_cross_attention better default allow mid_block_only_cross_attention to default to `only_cross_attention` when `only_cross_attention` is given as a single boolean
dg845
pushed a commit
to dg845/diffusers
that referenced
this pull request
May 6, 2023
* add only cross attention to simple attention blocks * add test for only_cross_attention re: @patrickvonplaten * mid_block_only_cross_attention better default allow mid_block_only_cross_attention to default to `only_cross_attention` when `only_cross_attention` is given as a single boolean
yoonseokjin
pushed a commit
to yoonseokjin/diffusers
that referenced
this pull request
Dec 25, 2023
* add only cross attention to simple attention blocks * add test for only_cross_attention re: @patrickvonplaten * mid_block_only_cross_attention better default allow mid_block_only_cross_attention to default to `only_cross_attention` when `only_cross_attention` is given as a single boolean
AmericanPresidentJimmyCarter
pushed a commit
to AmericanPresidentJimmyCarter/diffusers
that referenced
this pull request
Apr 26, 2024
* add only cross attention to simple attention blocks * add test for only_cross_attention re: @patrickvonplaten * mid_block_only_cross_attention better default allow mid_block_only_cross_attention to default to `only_cross_attention` when `only_cross_attention` is given as a single boolean
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR allows the added kv attention processor blocks to skip their self attention and only do cross attention
We also pass the only_cross_attention argument through the "simple" unet blocks