Skip to content

histogram: a nullable Float64 column holding a NaN gets no histogram (regression from #1069) #1077

Open

Description

@paddymul

Problem

A nullable Float64 column can hold a NaN that isn't pd.NA: arithmetic such as 0.0 / 0.0 produces one, and isna() is False for it. For such a column base_summary_stats (pd_stats_v2.py:131) returns min and max as pd.NA. Since #1069, histogram_series drops the NaN along with the infinities and returns numeric histogram_args, so histogram calls numeric_histogram with the pd.NA min, and the math.isfinite check #1069 added to fmt_num (histogram.py:21) raises TypeError: must be real number, not NAType. The column gets no histogram. Before #1069 the NaN reached np.quantile, histogram_args came back empty, and the column got a categorical histogram listing nan as a value.

import pandas as pd
from buckaroo.customizations.pd_stats_v2 import PD_ANALYSIS_V2
from buckaroo.pluggable_analysis_framework.stat_pipeline import StatPipeline

num = pd.Series([0.0] + [float(i) for i in range(1, 100)], dtype="Float64")
den = pd.Series([0.0] + [1.0] * 99, dtype="Float64")
ser = num / den  # row 0 is NaN, not pd.NA
sd, errs = StatPipeline(PD_ANALYSIS_V2, unit_test=False).process_df(pd.DataFrame({"a": ser}))
print(sd["a"]["min"], sd["a"]["histogram"], [(e.stat_key, str(e.error)) for e in errs if e.stat_key == "histogram"])
# <NA> None [('histogram', 'must be real number, not NAType')]

mean, median and std fail on the same column (#1074).

Suggested fix

Treat a nullable column's NaN as missing when base_summary_stats takes min and max, so they span the numbers, and have fmt_num return a placeholder for a bound that isn't a real number instead of raising.

Context

Regression from #1069, found while probing histogram edge cases. Reproduces on main at e88ace9.

Activity

  1. paddymul commented on Oct 7, 2026

    @paddymul
    CollaboratorAuthor

    Probe script for this issue. Save it as probe_1077.py in a buckaroo checkout and run uv run python probe_1077.py. The output below is from main at e88ace9.

    probe_1077.py
    """Since #1069, histogram_series drops a nullable Float64 column's NaN (not pd.NA) with the infinities and returns
    numeric histogram_args, but min/max are pd.NA and fmt_num's math.isfinite raises on them."""
    import logging
    logging.disable(logging.CRITICAL)
    import warnings
    warnings.filterwarnings('ignore')
    import numpy as np
    import pandas as pd
    from buckaroo.customizations.pd_stats_v2 import PD_ANALYSIS_V2
    from buckaroo.pluggable_analysis_framework.stat_pipeline import StatPipeline
    
    num = pd.Series([0.0] + [float(i) for i in range(1, 100)], dtype='Float64')
    den = pd.Series([0.0] + [1.0] * 99, dtype='Float64')
    ser = num / den
    print('values', ser.head(3).tolist(), 'isna', ser.isna().head(3).tolist())
    print('dropna().min() =', ser.dropna().min(), '| np.isfinite keeps', int(np.isfinite(ser.dropna()).sum()), 'of', len(ser))
    sd, errs = StatPipeline(PD_ANALYSIS_V2, unit_test=False).process_df(pd.DataFrame({'a': ser}))
    for e in errs:
        print('error', e.stat_key, type(e.error).__name__, str(e.error)[:120])
    print('histogram_args empty:', sd['a']['histogram_args'] == {}, '| histogram:', sd['a']['histogram'])
    Output at e88ace9
    values [nan, 1.0, 2.0] isna [False, False, False]
    dropna().min() = <NA> | np.isfinite keeps 99 of 100
    error mean TypeError float() argument must be a string or a real number, not 'NAType'
    error std TypeError float() argument must be a string or a real number, not 'NAType'
    error median TypeError float() argument must be a string or a real number, not 'NAType'
    error histogram TypeError must be real number, not NAType
    histogram_args empty: False | histogram: None
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions