Skip to content

Conversation

@ani300
Copy link
Contributor

@ani300 ani300 commented Jul 16, 2025

Description of the change

This PR adds an assertion that the FP8 scales have the same batch dimension as the inputs for paged attention. This is important for torch compile to not create extra dynamic dimensions, which is needed for AIU compilation and also accelerates tracing.

Related issues or PRs

Internal AIU issue tracker.

How to verify the PR

Was the PR tested

  • I have added >=1 unit test(s) for every new method I have added (if that coverage is difficult, please briefly explain the reason)
  • I have ensured all unit tests pass

Checklist for passing CI/CD:

  • All commits are signed showing "Signed-off-by: Name <email@domain.com>" with git commit -signoff or equivalent
  • PR title and commit messages adhere to Conventional Commits
  • Contribution is formatted with tox -e fix
  • Contribution passes linting with tox -e lint
  • Contribution passes spellcheck with tox -e spellcheck
  • Contribution passes all unit tests with tox -e unit

Note: CI/CD performs unit tests on multiple versions of Python from a fresh install. There may be differences with your local environment and the test environment.

ani300 added 2 commits July 16, 2025 15:23
Signed-off-by: Antoni Viros i Martin <aviros@ibm.com>
Signed-off-by: Antoni Viros i Martin <aviros@ibm.com>
@ani300 ani300 changed the title Mark FP8 scale to have the same batch size as input [bug] Mark FP8 scale to have the same batch size as input Jul 16, 2025
@ani300 ani300 changed the title [bug] Mark FP8 scale to have the same batch size as input [fix] Mark FP8 scale to have the same batch size as input Jul 16, 2025
@ani300 ani300 changed the title [fix] Mark FP8 scale to have the same batch size as input [fix]: Mark FP8 scale to have the same batch size as input Jul 16, 2025
@ani300 ani300 changed the title [fix]: Mark FP8 scale to have the same batch size as input fix: Mark FP8 scale to have the same batch size as input Jul 16, 2025
@github-actions github-actions bot added the fix label Jul 16, 2025
@ani300 ani300 merged commit fae7b28 into foundation-model-stack:main Jul 16, 2025
11 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants