Skip to content

fix(cross_entropy): preserve ignored logits without gradients - #1604

Merged
Tcc0403 merged 1 commit into
linkedin:mainfrom
LiRunGuo:fix/cross-entropy-no-grad-preserve-logits
Nov 8, 2026
Merged

Tcc0403 merged 1 commit into
linkedin:mainfrom
LiRunGuo:fix/cross-entropy-no-grad-preserve-logits

Conversation

@LiRunGuo

@LiRunGuo LiRunGuo commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #1594.

liger_cross_entropy_kernel reuses the input tensor as gradient storage. For rows whose target is ignore_index, it was zeroing the input unconditionally, including when the input did not require grad. This mutated caller-owned logits during evaluation or metrics collection even though no backward pass would consume the result.

Gate that store on HAS_GRADIENTS, matching the existing gradient path below it. The actual loss and accuracy outputs are unchanged.

Testing Done

  • Hardware Type: NVIDIA GeForce RTX 3090
  • run make test to ensure correctness
  • run make checkstyle to ensure code style
  • run make test-convergence to ensure convergence

GPU validation on Yokota:

$ python -m pytest -c pyproject.toml test/transformers/test_cross_entropy.py -q
187 passed, 1 warning in 60.68s

The new regression covers both float32 and bfloat16, verifies that ignored rows remain byte-identical under torch.no_grad(), and compares the non-ignored losses with PyTorch.

$ ruff check --output-format=concise .
All checks passed!

$ ruff format --check --diff .
500 files already formatted

AI assistance was used to investigate, implement, and validate this focused change.

@LiRunGuo

LiRunGuo commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Full CrossEntropy test file completed on an NVIDIA GeForce RTX 3090:

$ PYTHONPATH=src:. python -m pytest -c pyproject.toml test/transformers/test_cross_entropy.py -q
187 passed, 1 warning in 60.68s (0:01:00)

The two new parameterizations both pass and verify that ignored rows remain byte-identical under torch.no_grad(). The warning is the environment missing pytest-asyncio and is unrelated to this change.

@Tcc0403 Tcc0403 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch! Thanks for the contribution!

@Tcc0403
Tcc0403 enabled auto-merge November 8, 2026 15:44
@Tcc0403
Tcc0403 added this pull request to the merge queue Nov 8, 2026
Merged via the queue into linkedin:main with commit e44de70 Nov 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cross entropy zeroes the ignore_index rows of its input when no gradient is needed

2 participants