Repository navigation
Statistical validation with known baselines #148
Description
This issue is in relation to JOSS review thread openjournals/joss-reviews#11047
It would be good to add more tests covering statistical correctness of the methods directly. This could be against simple baselines with hand-computed outcomes or slightly more complex simulations with known treatment effects in the DGP. I see such simulations in the underlying papers but having tests like that here could catch implementation bugs. I did find one such bug in a simple case for QTE where treatment effects are 5 everywhere but requesting the median returns 1, see below.
from dte_adj import SimpleDistributionEstimator
np.random.seed(0)
D = np.repeat([0, 1], 20)
X = np.zeros((40, 1))
Y = 5.0 * D
est = SimpleDistributionEstimator().fit(X, D, Y)
qte, _, _ = est.predict_qte(
1, 0, quantiles=np.array([0.5]), n_bootstrap=5, display_progress=False
)
np.testing.assert_allclose(qte, [5.0], rtol=0, atol=1e-10)
# AssertionError: actual [1.], desired [5.]```
Activity
Address JOSS review feedback (#147, #148, #149) (#150)
TomeHirata commentedon Oct 6, 2026
Fixed in #150. The QTE quantile search returned the largest location with
F(y) <= q; it now uses the generalized inverse, i.e. the smallestywithF(y) >= q. Your example now returns 5.The PR also adds
tests/test_statistical_baselines.py:Generated by Claude Code