step_climate()` aggregates the season containing each training row's target week — in-sample contamination in backtests

[docxology]

Channel: public issue · Severity: medium

---

step_climate() builds its multi-year seasonal average over all training rows, with no exclusion of the season(s) overlapping a row's own target week. When fitting on historical data (the package's standard backtesting usage), every training row's climate predictor contains its own (diluted) outcome value.

What happens

R/step_climate.R:280-296 (HEAD 43b352f / main 7d8539b): prep.step_climate() computes the climate_table over all of training via mutate(.idx = time_value %m-% ahead_period, ...) %>% reframe(roll_modular_multivec(...)) — there is no year/season exclusion. At bake (:320-326), each row is mapped to the multi-year average at week(t) + ahead. So for any training row t whose t + ahead lies within the data, y(t+ahead) is one of the averaged points. The docs only say the step "examines all available seasons"; the current-season contribution is not documented.

Repro (code-reading analysis; I don't have R available, so please confirm)

library(epipredict)
library(dplyr)

edf <- covid_case_death_rates |>
  filter(geo_value %in% c("ak", "ca"), time_value >= as.Date("2020-01-01"),
         time_value <= as.Date("2021-12-31"))   # two seasons

r <- epi_recipe(edf) |>
  step_epi_ahead(death_rate_7d_av, ahead = 14) |>
  step_climate(death_rate_7d_av, time_type = "epiweek")

p <- prep(r, edf)
p$steps[[2]]$climate_table
# for any training row t with t+14 inside the data, the aggregated week's
# average includes y(t+14) itself

Impact

In backtests, the climate→outcome association is inflated relative to real-time deployment (the predictor is partially the outcome), producing optimistic validation results that do not transfer to live forecasting. This is the step-level sibling of the as-of truncation question for climatological_forecaster().

Suggested fix

Either add an option (or documented guidance) to exclude, per row, the season(s) overlapping that row's target week from the average, or at minimum document the in-sample contamination explicitly in @details so backtesting users can compensate (e.g. by holding out the final season).