Benchmarking updates for semi-structured sparse training

[jcaip]

Summary:

This PR does the following:

• adds e2e ViT benchmarks for semi-structured sparse training
• adds nn.Linear microbenchmarks
• removes extra xformers benchmarking utils I copied over
• removes MLP block benchmarks
• updated README.md with new benchmarks + accuracy benchmarks

Given we have nn.Linear microbenchmarks and e2e benchmarks, I felt that
the MLP block benchmarks were unnecessary

As a sanity check, I ran the MLP benchmarks with the new benchmarking
suite and the old one, and got the same results:

NEW:

OLD:

	dense	w24
f16 (44160,1024,4096,1024)	11534.3	9204.7

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/398

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 881ae2c with merge base 6b0ca2d ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Run Regression Tests / test (CUDA Nightly, linux.g5.12xlarge.nvidia.gpu, --pre torch --index-url https://download.pytorc... / linux-job (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[msaroufim]

Cool thank you! This is significantly clearer. I do want us to think a bit harder about the top line metric since 6% might not be super compelling to people not familiar with limitations of sparsity

[jcaip]

@msaroufim We could compare to masking based approaches (which are slower than dense training) for a larger number, but I think it'd be a bit confusing since I'm assuming most users are coming with a dense model and not an existing sparse training script they want to accelerate.