BC I/O: refuse per-rank boundary files written for another decomposition

[sbryngelson]

Description

Restarting (or just running simulation) on a different rank count from pre_process silently corrupts Dirichlet (-17) inflow and boundary-patch data.

Root cause. When bc_io is on (any -17 boundary or num_bc_patches > 0), pre_process writes the boundary types and buffers per rank, to restart_data/boundary_conditions/bc_<rank>.dat (s_write_parallel_boundary_condition_files). simulation and post_process open bc_<proc_rank>.dat by their own rank index and read it unchecked (s_read_parallel_boundary_condition_files). The restart file itself is in a global layout and reads on any rank count, so the only thing tying a run to the pre_process decomposition is this directory, and nothing checks it. On another rank count each rank gets another rank's boundary slab, with the wrong position and possibly the wrong size. The per-rank MPI_File_open was not checked either, so a missing file went unnoticed too.

Fix (minimal).

• pre_process rank 0 also writes restart_data/boundary_conditions/decomposition.dat, containing num_procs num_procs_x num_procs_y num_procs_z.
• simulation aborts on a mismatch, with a message that names both decompositions.
• post_process only uses the data for boundary ghost cells, so it warns instead of aborting (new optional strict argument). Workflows that post-process on a different rank count keep working, but are told that boundary ghost values are wrong.
• Files written before this change have no decomposition.dat and are read as before, unchecked.
• Both per-rank MPI_File_open calls now go through s_check_mpi_file_open.

Not done here: the full fix. The full fix would write and read these arrays in a global parallel-I/O layout, like the restart file, so that any rank count works. I did not do it in this PR because it is not contained:

• the direction-2 and direction-3 arrays carry -buff_size:m+buff_size halos, which overlap between ranks and include corner cells, and buff_size can differ between targets;
• each face would need its own global subarray type, and only the ranks on that physical boundary hold data for it;
• it changes the file format for all three targets, so old files would need a fallback.

Verification

Production (OLCF Frontier, CCE 19, OpenACC). pre_process ran on 192 ranks and the restart ran on 64 ranks at step 15714. The inflow cell j = 0 reached ICFL 1.015 at step 15723. In a second run (360 ranks, then 240), p = -4e47 appeared at the inflow face at step 2. Runs that kept the rank count were unaffected.

This branch. I ran these on a Frontier CPU compute node (GNU 12.3 + Cray MPICH, Release). The case is a 2D uniform channel, 50x40, with bc_x%beg = -17.

pre_process	simulation	result
2 ranks, this branch	2 ranks, this branch	runs; decomposition.dat = 2 2 1 1
2 ranks, this branch	1 rank, this branch	aborts: ./restart_data/boundary_conditions was written for 2 ranks (2x1x1) but this run uses 1 ranks (1x1x1). These per-rank boundary files only work on the decomposition that wrote them: run on that rank count, or rerun pre_process on this one.
2 ranks, master	1 rank, master	runs to the end with no warning, reading the 2-rank files
2 ranks, master (no decomposition.dat)	2 ranks, this branch	runs (backward compatible)

Builds: pre_process, simulation and post_process compile with CCE 19 on CPU, and pre_process and simulation with GNU 12.3. The post_process warning path was compiled but not run. Precheck passes, apart from two test_thermochem cases that fail on the Frontier login node because they compile with the system /usr/bin/gfortran (addressed by #1943).

No regression test is added. The test harness compares goldens and has no way to expect an abort, and a test that restarts on a different rank count would need harness support.

Contribution Policy

We do not accept pull requests generated primarily by AI without genuine understanding or real-world usage context.

All contributions are expected to demonstrate:

• A clear understanding of the codebase
• Alignment with product direction
• Thoughtful reasoning behind changes
• Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

---

Acknowledgement

• I confirm this PR meets the above expectations and reflects my own understanding and real-world context.

---

This PR was prepared with the assistance of an AI tool (Claude Code). The bug was found in production runs on Frontier; the fix was exercised on the small cases above.

PR template credit: junegunn

[github-actions]

Lines of Code

File	Lines	Diff
src/common/m_boundary_io.fpp	309	+35

Directory	Lines	Diff
common	10461	+35
total	47043	+35

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open findings

• The code uses a hard-coded I/O unit (1) and does not check iostat on open/read/write.… · New
• The code uses a hard-coded I/O unit (1) and does not check iostat on open/read/write.… · New
• The warning print executes unconditionally after the strict abort call, relying on s_mpi_abort… · New

What changed in this PR

Adds a lightweight decomposition guard for per-rank boundary-condition restart files so runs don’t silently read the wrong slabs when restarted with a different MPI decomposition.

Changes:

• Write restart_data/boundary_conditions/decomposition.dat from rank 0 when generating per-rank BC files.
• Check decomposition on read (abort by default; optionally warn-only via strict=.false.).
• Route per-rank MPI_File_open through s_check_mpi_file_open and adjust post_process to use warn-only mode.

File	Description
src/​post_process/​m_data_input.f90	Post-process now calls BC read with strict=.false. (warn instead of abort on mismatch).
src/​common/​m_boundary_io.fpp	Writes decomposition metadata; validates it on read; adds MPI file-open error checking.

🧠 Review effort: Lite

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[codecov]

Codecov Report

❌ Patch coverage is 65.21739% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.66%. Comparing base (3dc5b2f) to head (aa76e9f).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
src/common/m_boundary_io.fpp	63.63%	6 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1947      +/-   ##
==========================================
+ Coverage   62.64%   62.66%   +0.01%     
==========================================
  Files          86       86              
  Lines       22425    22456      +31     
  Branches     3325     3329       +4     
==========================================
+ Hits        14048    14071      +23     
- Misses       6119     6125       +6     
- Partials     2258     2260       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.