Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add CUTLASS-based row-wise scaled sparse FP8 kernel #1671

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add CUTLASS-based row-wise scaled sparse FP8 kernel #1671

wants to merge 1 commit into from

Conversation

alexsamardzic
Copy link
Collaborator

No description provided.

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Feb 5, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1671

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 5, 2025
@alexsamardzic
Copy link
Collaborator Author

alexsamardzic commented Feb 5, 2025 •
edited
Loading

The kernel is ready and passes smoke test.

Remaining tasks:

  • Write a converter to SM90 sparse semi-structured format
  • Validate the kernel on proper test inputs
  • Write the benchmark
  • Write Python-side code: sparsify/quantize method, Llama generator extension, etc.
  • Provide that kernel is built with SM90a flags when torchao detects H100 card as SM90
  • Further unify CUDA code with rowwise_scaled_linear_cutlass code
  • Implement a meaningful config selection.

@cpuhrsch @drisspg

@cpuhrsch cpuhrsch requested a review from jcaip February 5, 2025 21:00
@alexsamardzic alexsamardzic added float8 sparsity topic: new feature Use this tag if this PR adds a new feature labels Feb 6, 2025
@alexsamardzic alexsamardzic force-pushed the rowwise-scaled-sparse-fp8-cutlass branch 2 times, most recently from 5bbcc49 to 6d34b7e Compare February 6, 2025 23:41
@alexsamardzic alexsamardzic force-pushed the rowwise-scaled-sparse-fp8-cutlass branch 7 times, most recently from 7a4b4e2 to bd7288a Compare February 13, 2025 13:49
@alexsamardzic alexsamardzic force-pushed the rowwise-scaled-sparse-fp8-cutlass branch from bd7288a to f11fae4 Compare February 13, 2025 22:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jcaip jcaip jcaip left review comments

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. float8 sparsity topic: new feature Use this tag if this PR adds a new feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@alexsamardzic @jcaip @facebook-github-bot