Change weight to channel-packing in Conv1d #7057
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7057
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 6b5a494 with merge base a35cb73 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D66417572 |
yipjustin
added a commit
to yipjustin/executorch
that referenced
this pull request
Nov 25, 2024
Summary: In a model we evaluate, we have a weight tensor (256, 1, 7) for conv 1d, 256 is the out-channel count and 7 is the weight. It leads to a non-optimal use of memory since this tensor is mapped to extents of `(7 / 4, 1, 256)` under weight-packing, using 1MB per tensor. Reason is that each (x, y) plane uses 4096 bytes in the test device (for both 'OPTIMAL' and 'LINEAR' tiling), despite we are using only 2 texels in each plane. A temporarily work-around is to use channel-packing instead. Then new tensor will be `(7, 1, 64)`, 75% less deep hence consume far less memory. Knowing that we will fetch 4 times more. But lab test shows that our model has no perf regression. ## Future work: A more optimal solution is mapping the weight tensor `(out-channel, in-channel, kernel)` into extents `(x=out-channel, y=kernel, z=in-channel)`. In our case, it leads to close to optimal layout. Reviewed By: nathanaelsee Differential Revision: D66417572
yipjustin
force-pushed
the
export-D66417572
branch
from
5353456 to
fa845e2
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66417572 |
yipjustin
added a commit
to yipjustin/executorch
that referenced
this pull request
Nov 25, 2024
Summary: In a model we evaluate, we have a weight tensor (256, 1, 7) for conv 1d, 256 is the out-channel count and 7 is the weight. It leads to a non-optimal use of memory since this tensor is mapped to extents of `(7 / 4, 1, 256)` under weight-packing, using 1MB per tensor. Reason is that each (x, y) plane uses 4096 bytes in the test device (for both 'OPTIMAL' and 'LINEAR' tiling), despite we are using only 2 texels in each plane. A temporarily work-around is to use channel-packing instead. Then new tensor will be `(7, 1, 64)`, 75% less deep hence consume far less memory. Knowing that we will fetch 4 times more. But lab test shows that our model has no perf regression. ## Future work: A more optimal solution is mapping the weight tensor `(out-channel, in-channel, kernel)` into extents `(x=out-channel, y=kernel, z=in-channel)`. In our case, it leads to close to optimal layout. Reviewed By: nathanaelsee Differential Revision: D66417572
yipjustin
force-pushed
the
export-D66417572
branch
from
fa845e2 to
44097ee
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66417572 |
yipjustin
added
the
release notes: backends [DO NOT USE]
jorgep31415
approved these changes
Nov 25, 2024
yipjustin
added a commit
to yipjustin/executorch
that referenced
this pull request
Nov 26, 2024
Summary: In a model we evaluate, we have a weight tensor (256, 1, 7) for conv 1d, 256 is the out-channel count and 7 is the weight. It leads to a non-optimal use of memory since this tensor is mapped to extents of `(7 / 4, 1, 256)` under weight-packing, using 1MB per tensor. Reason is that each (x, y) plane uses 4096 bytes in the test device (for both 'OPTIMAL' and 'LINEAR' tiling), despite we are using only 2 texels in each plane. A temporarily work-around is to use channel-packing instead. Then new tensor will be `(7, 1, 64)`, 75% less deep hence consume far less memory. Knowing that we will fetch 4 times more. But lab test shows that our model has no perf regression. ## Future work: A more optimal solution is mapping the weight tensor `(out-channel, in-channel, kernel)` into extents `(x=out-channel, y=kernel, z=in-channel)`. In our case, it leads to close to optimal layout. Reviewed By: nathanaelsee, jorgep31415 Differential Revision: D66417572
yipjustin
force-pushed
the
export-D66417572
branch
from
44097ee to
c19759d
Compare
Summary: In a model we evaluate, we have a weight tensor (256, 1, 7) for conv 1d, 256 is the out-channel count and 7 is the weight. It leads to a non-optimal use of memory since this tensor is mapped to extents of `(7 / 4, 1, 256)` under weight-packing, using 1MB per tensor. Reason is that each (x, y) plane uses 4096 bytes in the test device (for both 'OPTIMAL' and 'LINEAR' tiling), despite we are using only 2 texels in each plane. A temporarily work-around is to use channel-packing instead. Then new tensor will be `(7, 1, 64)`, 75% less deep hence consume far less memory. Knowing that we will fetch 4 times more. But lab test shows that our model has no perf regression. ## Future work: A more optimal solution is mapping the weight tensor `(out-channel, in-channel, kernel)` into extents `(x=out-channel, y=kernel, z=in-channel)`. In our case, it leads to close to optimal layout. Reviewed By: nathanaelsee, jorgep31415 Differential Revision: D66417572
yipjustin
force-pushed
the
export-D66417572
branch
from
c19759d to
6b5a494
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66417572 |
1 similar comment
|
This pull request was exported from Phabricator. Differential Revision: D66417572 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
In a model we evaluate, we have a weight tensor (256, 1, 7) for conv 1d, 256 is the out-channel count and 7 is the weight.
It leads to a non-optimal use of memory since this tensor is mapped to extents of
(7 / 4, 1, 256)under weight-packing, using 1MB per tensor. Reason is that each (x, y) plane uses 4096 bytes in the test device (for both 'OPTIMAL' and 'LINEAR' tiling), despite we are using only 2 texels in each plane.A temporarily work-around is to use channel-packing instead. Then new tensor will be
(7, 1, 64), 75% less deep hence consume far less memory. Knowing that we will fetch 4 times more. But lab test shows that our model has no perf regression.Future work:
A more optimal solution is mapping the weight tensor
(out-channel, in-channel, kernel)into extents(x=out-channel, y=kernel, z=in-channel). In our case, it leads to close to optimal layout.Reviewed By: nathanaelsee
Differential Revision: D66417572