pandas stats: a nullable Float64/Int64 column loses mean, median and std when its std is pd.NA #1074

Activity

1. paddymul

   added

   Histograms

   on Oct 6, 2026
2. paddymul commented on Oct 7, 2026

   

   paddymul

   on Oct 7, 2026

   CollaboratorAuthor

   More actions

   Probe script for this issue. Save it as probe_1074.py in a buckaroo checkout and run uv run python probe_1074.py. The output below is from main at e88ace9.

   probe_1074.py

   """numeric_stats returns float(ser.mean()), float(ser.std()), float(ser.median()) from one function; on a
   nullable column pandas returns pd.NA where numpy dtypes return NaN, and float(pd.NA) raises."""
   import logging
   logging.disable(logging.CRITICAL)
   import warnings
   warnings.filterwarnings('ignore')
   import numpy as np
   import pandas as pd
   from buckaroo.customizations.pd_stats_v2 import PD_ANALYSIS_V2
   from buckaroo.pluggable_analysis_framework.stat_pipeline import StatPipeline
   
   for label, s in [('Float64 with inf', pd.Series([1.0, 2.0, 3.0, np.inf, None], dtype='Float64')),
                    ('Float64 with -inf', pd.Series([1.0, 2.0, -np.inf], dtype='Float64')),
                    ('Float64 one value', pd.Series([1.5, None], dtype='Float64')),
                    ('Int64 one value', pd.Series([1, None], dtype='Int64')),
                    ('Float64 all NA', pd.Series([None, None], dtype='Float64')),
                    ('float64 one value (numpy, for comparison)', pd.Series([1.5, None])),
                    ('float64 with inf (numpy, for comparison)', pd.Series([1.0, 2.0, 3.0, np.inf, None])),
                    ('Float64 spanning +-1e308 (std overflows to inf, no error)', pd.Series([1e308, -1e308, 5.0], dtype='Float64'))]:
       sd, errs = StatPipeline(PD_ANALYSIS_V2, unit_test=False).process_df(pd.DataFrame({'a': s}))
       print(f"{label:58} raw {s.mean()!s:>8} {s.median()!s:>8} {s.std()!s:>8} | sd {sd['a'].get('mean')!s:>8} "
             f"{sd['a'].get('median')!s:>8} {sd['a'].get('std')!s:>8} | errors {sorted({e.stat_key for e in errs})}")

   Output at e88ace9

   Float64 with inf                                           raw      inf      2.5     <NA> | sd     None     None     None | errors ['mean', 'median', 'std']
   Float64 with -inf                                          raw     -inf      1.0     <NA> | sd     None     None     None | errors ['mean', 'median', 'std']
   Float64 one value                                          raw      1.5      1.5     <NA> | sd     None     None     None | errors ['mean', 'median', 'std']
   Int64 one value                                            raw      1.0      1.0     <NA> | sd     None     None     None | errors ['mean', 'median', 'std']
   Float64 all NA                                             raw     <NA>     <NA>     <NA> | sd     None     None     None | errors ['mean', 'median', 'std']
   float64 one value (numpy, for comparison)                  raw      1.5      1.5      nan | sd      1.5      1.5      nan | errors []
   float64 with inf (numpy, for comparison)                   raw      inf      2.5      nan | sd      inf      2.5      nan | errors []
   Float64 spanning +-1e308 (std overflows to inf, no error)  raw 1.6666666666666667      5.0      inf | sd 1.6666666666666667      5.0      inf | errors []