Skip to content

fix: avoid underflow close to mu in stats/base/dists/levy/logcdf - #15779

Open
Abhist17 wants to merge 2 commits into
stdlib-js:developfrom
Abhist17:fix/levy-logcdf-underflow
Open

Abhist17 wants to merge 2 commits into
stdlib-js:developfrom
Abhist17:fix/levy-logcdf-underflow

Conversation

@Abhist17

@Abhist17 Abhist17 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Description

What is the purpose of this pull request?

This pull request:

  • avoids the erfc underflow in stats/base/dists/levy/logcdf (JS, factory and C).

The function returned ln( erfc( z ) ) with z = sqrt( c / (2*(x-mu)) ). erfc(z) underflows to 0 for z > ~26.5, so for x - mu < c/1400 the result is -Infinity, although the log-CDF there is finite (about -c / (2*(x-mu))):

var logcdf = require( '@stdlib/stats/base/dists/levy/logcdf' );

logcdf( 1.0e-4, 0.0, 1.0 );
// develop: -Infinity
// this PR: -5004.831061513645

logcdf( 1.0e-300, 0.0, 1.0 );
// develop: -Infinity
// this PR: -4.9999999999999995e+299

For z >= 1 it now returns ln( erfcx( z ) ) - w, where w = c / (2*(x-mu)) is z^2 taken before the square root, so no rounding is added. erfcx is the scaled complementary error function, e^{z^2} erfc(z), which doesn't underflow. Below z = 1 the old ln( erfc( z ) ) stays.

Fixtures and tests:

  • runner.jl now evaluates the textbook formula ln(erfc(sqrt(c/(2(x-mu))))) in 2048-bit BigFloat (with SpecialFunctions.erfc) instead of Distributions.logcdf. REQUIRE is updated, and the three fixtures are regenerated with the same input ranges.
  • There's a new near_mu.json with x - mu log-spaced over [1e-15, 1]. Smaller gaps push exp(-w) beyond MPFR's exponent range.
  • Max error in JS and native is 3 ULP on the existing fixtures (the same as develop against the exact values) and 2 ULP on near_mu. The tolerances go from 0 to 3; the old 0 relied on the fixture using the same double-precision expression. On develop, 802 of the 4,016 assertions in test.logcdf.js fail, most of them returning -Infinity. With this change they all pass.

Related Issues

Does this pull request have any related issues?

None.

Questions

Any questions for reviewers of this pull request?

No.

Other

Any other information relevant to this pull request? This may include screenshots, references, and/or implementation notes.

No.

Checklist

Please ensure the following tasks are completed before submitting this pull request.

AI Assistance

When authoring the changes proposed in this PR, did you use any kind of AI assistance?

  • Yes
  • No

If you answered "yes" above, how did you use AI assistance?

  • Code generation (e.g., when writing an implementation or fixing a bug)
  • Test/benchmark generation
  • Documentation (including examples)
  • Research and understanding

Disclosure

If you answered "yes" to using AI assistance, please provide a short disclosure indicating how you used AI assistance. This helps reviewers determine how much scrutiny to apply when reviewing your contribution. Example disclosures: "This PR was written primarily by Claude Code." or "I consulted ChatGPT to understand the codebase, but the proposed changes were fully authored manually by myself.".

This PR was written primarily by Claude Code. I found the bug by evaluating every stats/base/dists logcdf just above the lower end of its support.


@stdlib-js/reviewers

`erfc(z)` underflows for `z > ~26.5`, so `ln( erfc(z) )` returned
`-Infinity` for `x - mu < c/1400` although the log-CDF is finite there.
Use `ln(erfc(z)) = ln(erfcx(z)) - z^2` for `z >= 1`. The fixtures are now
generated from the textbook formula in extended precision.
@Abhist17
Abhist17 requested a review from a team October 2, 2026 00:07
@stdlib-bot stdlib-bot added Statistics Issue or pull request related to statistical functionality. Needs Review A pull request which needs code review. labels Oct 2, 2026
@stdlib-bot

stdlib-bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Report

Package Statements Branches Functions Lines
stats/base/dists/levy/logcdf $\\color{green}325/325$
$\\color{green}+0.00\\%$
$\\color{green}27/27$
$\\color{green}+0.00\\%$
$\\color{green}4/4$
$\\color{green}+0.00\\%$
$\\color{green}325/325$
$\\color{green}+0.00\\%$

The above coverage report was generated for the changes in this PR.

@0PrashantYadav0 0PrashantYadav0 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separate from this fix, the large-x tail still loses precision: logcdf( 1e40, 0.0, 1.0 ) returns 0 (exact ≈ -7.98e-21), and log1p( -erf( z ) ) gets it within ~1 ULP. Are you planning a follow-up for that?

t.strictEqual( isAlmostSameValue( y, expected[ i ], 2 ), true, 'returns expected value' );
}
t.end();
});

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
});
});
tape( 'the function evaluates the logcdf for `x` very close to `mu`', function test( t ) {
var y;
y = logcdf( 1.0e-20, 0.0, 1.0 );
t.strictEqual( isAlmostSameValue( y, -5.0e19, 0 ), true, 'returns expected value' );
y = logcdf( 1.0e-200, 0.0, 1.0 );
t.strictEqual( isAlmostSameValue( y, -5.0e199, 0 ), true, 'returns expected value' );
t.end();
});
tape( 'if provided `x` equal to `mu`, the function returns `-infinity`', function test( t ) {
var y = logcdf( 2.0, 2.0, 1.0 );
t.strictEqual( y, NINF, 'returns expected value' );
t.end();
});

near_mu.json stops at x-mu = 1e-15, so the gaps cited in the PR description and x == mu (which now goes through erfcx) aren't tested. Same in test.factory.js:238 and test.native.js:189 (with opts).

Co-authored-by: 0PrashantYadav0 <0PrashantYadav0@users.noreply.github.com>
@Abhist17

Abhist17 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

@0PrashantYadav0 thanks. I added both tests in 55b4d8a, in test.logcdf.js, test.factory.js and test.native.js, and they pass in JS and native.

On the large-x tail: yes. ln( erfc( z ) ) for small z is the mirror case of this one, and log1p( -erf( z ) ) fixes it, as you say. It touches the same lines, so I'll send it as a follow-up once this one is merged, with a fixture for large x - mu.

@0PrashantYadav0

Copy link
Copy Markdown
Member

LGTM but we still have to wait for others review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Needs Review A pull request which needs code review. Statistics Issue or pull request related to statistical functionality.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants