step_lag_difference()`/`step_growth_rate()` use `left_join` and cannot populate the synthetic forecast rows created by lag/ahead steps — empty predictions when all lags > 0

[docxology]

Channel: public issue · Severity: medium · Related: #359

---

Lag/ahead steps fabricate trailing forecast rows via full_join, but the derived-feature steps join onto existing rows only. In recipes where every lag is > 0 (which get_test_data() deliberately trims), the derived features are NA exactly on the rows that matter, and the forecast row disappears.

What happens

• add_shifted_columns() (used by bake.step_epi_lag/bake.step_epi_ahead) full_joins shifted frames onto new_data, re-adding rows beyond max(time_value) with raw values NA (R/epi_shift.R:59-65 on HEAD 43b352f / main 7d8539b).
• bake.step_lag_difference() and bake.step_growth_rate() use left_join(new_data, ..., by = ok) instead (R/step_lag_difference.R:146, R/step_growth_rate.R:194) — derived features are only computed for rows that already exist in the input.
• get_test_data() drops the top min_lags rows from the test set (R/get_test_data.R:72-73); the lag steps then re-add synthetic rows via their full_join, but the derived-feature steps never populate them.

Repro (code-reading analysis; I don't have R available, so please confirm)

library(epipredict)
library(dplyr)

edf <- covid_case_death_rates |>
  filter(geo_value %in% c("ak", "ca"), time_value >= as.Date("2021-10-01"))

r <- epi_recipe(edf) |>
  step_lag_difference(death_rate_7d_av, horizon = 7) |>
  step_epi_lag(death_rate_7d_av, lag = c(7, 14)) |>
  step_epi_ahead(death_rate_7d_av, ahead = 7) |>
  step_epi_naomit()

wf <- epi_workflow(r, parsnip::linear_reg()) |> fit(edf)
forecast(wf, get_test_data(r, edf))
# the forecast rows re-added by the lag full_join (above the min_lags cutoff)
# have NA lag_diff_/gr_ predictors, so step_epi_naomit removes them and the
# prediction at the reference date is empty (or the fit errors)

Canned recipes include a lag-0 term, so they are unaffected — this bites custom recipes with all lags > 0 plus a derived-feature step.

Impact

Any recipe combining lag-only features with step_lag_difference/step_growth_rate predictors yields no usable forecast row — silently empty (or confusingly errored) predictions.

Suggested fix

Use the same full-join-and-extend pattern as the lag/ahead steps in bake.step_lag_difference()/bake.step_growth_rate(), or document that these steps require a lag-0 term in the recipe. (#359 tracks related composition gaps in get_test_data() for derived features.)

Related issues

#359 (get_test_data doesn't consider lagged differences of lagged differences).