Calibrate scores. Control discoveries. Monitor change.

Documentation · Batch workflow · Sequential workflow · API reference · Paper

nonconform calibrates anomaly scores for discovery, threshold certification, and sequential change monitoring. Wrap a supported scikit-learn estimator, a PyOD model, or a custom detector:

• Discovery: Use calibrated p-values directly or call select(...) to apply false discovery rate (FDR) control.
• Score thresholds: Build a simultaneous false-alarm/FPR certificate and choose a score threshold for future inliers.
• Stream: Use conformal martingales to accumulate evidence against exchangeability and trigger configured alarms.

Why nonconform?

• Calibrate anomaly scores into conformal p-values using reference data.
• Certify raw-score thresholds with simultaneous FPR certificates and target-driven threshold inversion.
• Control discoveries by accounting for multiple tests within a fixed family.
• Monitor streams for change with conformal martingales, anytime evidence against exchangeability, and configurable alarms.
• Keep your detector through support for PyOD, recognized scikit-learn estimators, and protocol-compliant custom models.

Works with              

Installation

nonconform requires Python 3.12 or newer. Both discovery control and sequential monitoring are included in the core installation.

pip install nonconform

For the PyOD detector collection and benchmark datasets:

pip install "nonconform[pyod,data]"

Optional extras

Extra	Adds
pyod	PyOD anomaly detectors
data	oddball benchmark datasets and PyArrow support
fdr	Streaming FDR procedures from online-fdr
probabilistic	KDE-based probabilistic estimation and tuning
all	Every optional feature

Quick start

Batch discovery control

This core-only example demonstrates the batch lane. The detector is trained on normal data, part of which is reserved automatically for conformal calibration.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split

rng = np.random.default_rng(42)
x_train = rng.normal(size=(3_000, 2))
x_test = np.vstack([
    rng.normal(size=(200, 2)),
    rng.normal(loc=5.0, size=(20, 2)),
])

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=1_000),
    seed=42,
).fit(x_train)

discoveries = detector.select(x_test, alpha=0.05)
p_values = detector.last_result.p_values

print(f"Selected {discoveries.sum()} of {len(x_test)} observations")

Note

discoveries is a Boolean mask. Here, alpha=0.05 is the target FDR level, not a per-observation score threshold. The underlying conformal p-values remain available through last_result for inspection or downstream analysis.

For threshold exploration in one fixed batch, use certificate = detector.fdp_bounds(x_test, confidence=0.95) and query its realized-FDP upper curve. certificate.select(0.01) applies the p-value cutoff 0.01; it does not request FDP at most 1%. The ordinary call prepares one default envelope using fresh Monte Carlo sampling, and subsequent queries reuse it. Coverage is joint over data and that sampling. Keep the scoring rule and family fixed, and do not cherry-pick repeated constructions. The FDP guide shows a complete example and the separate expert interface for method settings.

Raw-score false-alarm control

If the operational requirement is “flag no more than 5% of future inliers,” use the raw-score certificate instead of treating an FDR target as a score threshold:

certificate = detector.fpr_bounds(confidence=0.95)
scores = detector.score_samples(x_test)
selected = certificate.select(scores, target_fpr=0.05)
print(f"Certified threshold: {certificate.threshold_for(0.05)}")

With probability at least 95% over the clean calibration draw, the certificate bounds the population inlier false-alarm rate at every threshold, conditional on the independently fitted score map. On that event, the chosen threshold has a future-inlier false-alarm probability at most 5%. Calibration and future inlier scores must be i.i.d. from the same distribution; exchangeability alone is insufficient.

The certificate uses the deterministic finite-sample one-sided KS distribution. With 1,000 calibration scores at 95% confidence, its finite-threshold band floor is about 3.85%, allowing the 5% target here. Smaller targets can produce an infinite threshold and an empty selection. Native certificates support unweighted Split calibration independently of the p-value estimator. This does not guarantee a realized false-alarm proportion in a finite batch, FDR, FDP, recall, or sequential Ville control.

Sequential change monitoring

A fitted Split detector can initialize the stream lane without refitting its scoring model. The example is self-contained so it can be copied independently of the batch example.

Show sequential monitoring example

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split
from nonconform.martingales import AlarmConfig, SimpleJumperMartingale
from nonconform.monitoring import ExchangeabilityMonitor

rng = np.random.default_rng(42)
x_train = rng.normal(size=(1_000, 2))

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=0.3),
    seed=42,
).fit(x_train)

alpha = 0.05
monitor = ExchangeabilityMonitor.from_split_detector(
    detector,
    martingale=SimpleJumperMartingale(
        alarm_config=AlarmConfig(restarted_ville_threshold=1 / alpha)
    ),
    seed=42,
)

# Stable observations followed by a distribution shift
x_stream = np.vstack([
    rng.normal(size=(50, 2)),
    rng.normal(loc=3.0, size=(50, 2)),
])

for x_t in x_stream:
    state = monitor.update(x_t)
    if "restarted_ville" in state.triggered_alarms:
        print(f"Change alarm at step {state.evidence_step}")
        break

Under the sequential validity assumptions, the restarted Ville alarm at 1 / alpha controls the probability of ever crossing on one stream. It does not control FDR across multiple streams. See the sequential monitoring guide for the full guarantee scope and other alarm statistics.

Choose a workflow

Goal	Start with
Calibrate and select anomalies in a batch	Split and select(...)
Set a per-inlier raw-score false-alarm target	fpr_bounds(...) and FPR certificates
Aggregate evidence across random splits	DerandomizedSplits and e-BH
Monitor a stream for change	Exchangeability martingales
Reuse more data for fitting and calibration	CrossValidation or JackknifeBootstrap
Account for covariate shift	Weighted conformal inference
Certify realized FDP across p-value thresholds	fdp_bounds(...) and FDP certificates
Bring a custom or third-party detector	Detector compatibility

Statistical scope

Important

Guarantees are assumption-dependent. Standard conformal workflows require calibration data and null test cases to be exchangeable. FDR claims additionally require valid p-values or the applicable aggregate null-evidence condition, together with the assumptions of the selected multiple-testing procedure. FPR certificates require clean, i.i.d. calibration and future inlier scores from the same distribution, conditional on an independently fitted score map. Weighted workflows require a plausible covariate-shift model, support overlap, and reliable weights. Sequential martingales require valid sequential conformal p-values; Ville thresholds provide false-alarm control for one valid stream, while CUSUM and Shiryaev-Roberts thresholds are change-evidence triggers that require separate calibration.

nonconform calibrates detector scores; it cannot make an unsuitable detector or mismatched calibration set valid. Spatial or temporal dependence must be handled explicitly before applying standard exchangeability-based claims. See the guides to FDR control and sequential monitoring before relying on error-control statements in a new application.

Citation

If you use nonconform in academic work, please cite the accompanying paper:

@inproceedings{Hennhofer_Conformal_Anomaly_Detection_2026,
author = {Hennhöfer, Oliver and Kirsch, Maximilian and Preisach, Christine},
booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications},
pages = {613--632},
publisher = {PMLR},
title = {{Conformal Anomaly Detection in Python: Moving Beyond Heuristic Thresholds with nonconform}},
url = {https://proceedings.mlr.press/v329/hennhofer26a.html},
year = {2026}
}

Project

Read the documentation, browse the changelog, or report a problem in the issue tracker. Contributions are welcome; start with the contributing guide. nonconform is distributed under the BSD 3-Clause License.

---