Modular pipeline for postfire raster processing (DEM, soils, burn severity, landcover) and ML training for dem_diff prediction.
Conda (recommended):
conda env create -f environment.yml
conda activate fire-debrisflow-mlUse existing ml_debris environment (already present on this machine):
conda activate ml_debrisKnown-good snapshot in ml_debris:
python 3.10.12numpy 2.2.6rasterio 1.4.4fiona 1.10.1geopandas 1.1.1shapely 2.1.2pyproj 3.7.1landlab 2.10.0bmi-topography 0.9.0scikit-learn 1.7.2xgboost 3.1.3torch 2.9.1+cpu
If you want to run repository tests/smoke checks from ml_debris, install:
conda install -n ml_debris -c conda-forge pytestQuick dependency check:
CONDA_NO_PLUGINS=true conda run -n ml_debris python -c "import importlib.util as u;mods=['numpy','yaml','requests','rasterio','fiona','geopandas','shapely','pyproj','landlab','bmi_topography','sklearn','joblib','xgboost','torch'];print({m: bool(u.find_spec(m)) for m in mods})"Pip (if your system geospatial stack is already available):
pip install -r requirements.txtUse an environment with required geospatial packages (geopandas, rasterio, fiona, landlab, bmi_topography, requests, pyyaml).
Update paths in config/base.yaml, then run:
python src/run_pipeline.py --config config/base.yaml --export-final-tifsThis builds aligned final .asc and .tif layers in paths.output_dir.
For dedicated USDA SOLUS soil data workflow (fetch/harmonize/run CLIs), see usda_solus.md.
config/base.yaml supports:
source: localsource: remotesource: remote_then_local(try fire-name/id remote first, then local)source: auto(same behavior asremote_then_local)
Generate target raster (dem_diff.tif) aligned to the pipeline grid:
python src/dem_difference.py \
--pre /path/to/pre_dem.tif \
--post /path/to/post_dem.tif \
--out-dir /path/to/output \
--config config/base.yaml \
--template /path/to/output/topographic__elevation.tifRandom Forest:
python scripts/train_model.py --config config/ml_rf.yamlXGBoost:
python scripts/train_model.py --config config/ml_xgb.yamlU-Net (CPU starter setup):
python scripts/train_unet.py --config config/ml_unet.yamlSimple CNN (CPU starter setup):
python scripts/train_cnn.py --config config/ml_cnn.yamlXGBoost (latest run):
python scripts/predict_model.py \
--model-path "$(ls -1dt models/xgb/* | head -n 1)/model.joblib" \
--feature-order "$(ls -1dt models/xgb/* | head -n 1)/feature_order.json" \
--data-dir /path/to/output \
--out-path /path/to/output/dem_diff_pred_xgb.tifU-Net (latest run):
python scripts/predict_unet.py \
--model-path "$(ls -1dt models/unet/* | head -n 1)/model.pt" \
--feature-order "$(ls -1dt models/unet/* | head -n 1)/feature_order.json" \
--data-dir /path/to/output \
--out-path /path/to/output/dem_diff_pred_unet.tifSimple CNN (latest run):
python scripts/predict_cnn.py \
--model-path "$(ls -1dt models/cnn/* | head -n 1)/model.pt" \
--feature-order "$(ls -1dt models/cnn/* | head -n 1)/feature_order.json" \
--data-dir /path/to/output \
--out-path /path/to/output/dem_diff_pred_cnn.tif- Training requires feature rasters and
dem_diff.tifon the same grid (CRS, transform, shape). - Pipeline needs internet access for remote DEM/feature downloads unless all sources are local/cached.
- Keep secrets (API keys, personal paths) out of commits.