An interrupted build leaves MFC permanently skipping a dependency or a configure step

[sbryngelson]

Summary

MFCTarget.is_configured() decides whether a target is ready by checking only for CMakeCache.txt:

https://github.com/MFlowCode/MFC/blob/master/toolchain/mfc/build.py#L369-L372

def is_configured(self, case: Case) -> bool:
    # We assume that if the CMakeCache.txt file exists, then the target is
    # configured. (this isn't perfect, but it's good enough for now)
    return os.path.isfile(os.sep.join([self.get_staging_dirpath(case), "CMakeCache.txt"]))

CMake writes CMakeCache.txt before it generates the build system and long before anything is installed, so the file surviving does not mean either finished. Two callers draw stronger conclusions than the check supports, and both fail in ways that point the blame somewhere unrelated.

Failure 1 — configure is skipped, build fails with a confusing make error

__build_target skips configure() whenever is_configured() is true:

if not target.is_configured(case):
    target.configure(case)
target.build(case)

If a configure is interrupted after the cache is written but before generation, the staging dir has a cache and no Makefile. Every subsequent build skips configure and dies with:

✗ Build failed for simulation
gmake: Makefile: No such file or directory
gmake: *** No rule to make target 'Makefile'.  Stop.

Nothing in that message suggests the fix is to delete a stale cache, and ./mfc.sh clean is a heavier hammer than needed.

Failure 2 — a dependency is skipped forever, surfacing as a find_package error

# Dependencies are pinned to fixed versions. If already configured
# (built & installed by a prior --deps-only step), skip entirely
# to avoid re-entering the superbuild (which may access the network).
if target.isDependency and target.is_configured(case):
    return

The comment states the assumption plainly — configured ⇒ built & installed — and that is false for any dependency build that failed. The superbuild's git clone step creates the staging dir and cache before it downloads, so a clone that fails (very easy to hit: compute nodes have no outbound network, so any dependency build submitted to a batch node fails here) leaves the cache behind with nothing installed. From then on MFC skips that dependency permanently, CMAKE_PREFIX_PATH points at an empty install tree, and the user sees an error from a completely different place:

CMake Error at .../FindPackageHandleStandardArgs.cmake:233 (message):
  Could NOT find SILO (missing: SILO_LIBRARY SILO_INCLUDE_DIR)
Call Stack (most recent call first):
  toolchain/cmake/regular/FindSILO.cmake:21
  cmake/MFCTargets.cmake:120 (find_package)

or, for hipfort, a find_package fallback onto ROCm's system copy, which on ROCm 7.2.0 is itself incomplete (missing hipfort-hipfft-targets.cmake) and produces a baffling error deep inside /opt/rocm-7.2.0.

Reproduce

# Failure 2, on any machine with no outbound network from the batch node
rm -rf build/staging/silo
srun ./mfc.sh build -t post_process     # clone fails, cache is left behind
./mfc.sh build -t post_process          # from a login node, with network:
                                        # silo is skipped, find_package(SILO) fails

Observed impact

Debugging a Cray CCE 21 port on OLCF Frontier, this cost six batch jobs and roughly an hour across three different worktrees before the common cause was spotted. In each case the visible error implicated the compiler or a missing system package rather than a skipped step.

Suggested fix

Ask the question each caller actually means:

• Ready to build — require the generator's build file as well as the cache (build.ninja or Makefile).
• Already installed — require install_manifest.txt, which CMake writes only after a successful install. Verified present for all five dependencies (fftw, hdf5, silo, lapack, hipfort) after a normal build, and absent exactly when the install did not happen.

A discriminating check on a real broken tree: silo had a cache and no manifest and was correctly rebuilt, while the four genuinely-installed dependencies were correctly skipped.

One behaviour change worth calling out: a dependency provisioned by hand, outside MFC's own install path, will no longer be recognised and will be rebuilt. That is arguably correct — MFC manages its own dependencies — but it turns one silent-wrong-answer case into a loud-failure case.

I have this fixed locally and will include it in a PR unless you would rather it went in separately.

[sbryngelson]

Folding in an additional symptom of the same defect, reported independently in #1692 (now closed as a duplicate of this).

A dependency build can report success having installed nothing. ./mfc.sh build --deps-only returned rc=0 with no hipfort under build/install/, and the next build then failed at M_FFTW with HIPFFTPLANMANY has no explicit type. Same root cause as the second failure mode above — the skip is decided from the cache rather than from the install — but the visible error lands in an unrelated module, which makes it even harder to trace back.

That strengthens the case for gating on the install artifact rather than the return code: a superbuild step can exit 0 having done nothing, so rc=0 is not evidence that the dependency is present.

Independently confirmed on the same platform (OLCF Frontier, cpe/26.03, cce/21.0.2, rocm/7.2.0) from a separate worktree, so this is reproducible rather than a one-off local state.

For what it's worth on severity: across one porting effort this predicate destroyed six batch jobs, produced a false negative in a compiler-flag experiment (a link reported as failing when it had never been attempted), broke a CCE 19 control run, and silently voided two full benchmark campaigns. In every instance the visible error implicated the compiler, a missing system package, or a slow machine — never the skip logic.

I have a fix implemented locally: require the generator's build file (Makefile / build.ninja) alongside the cache for "configured", and install_manifest.txt — which CMake writes only after a successful install — for "installed". Verified that all five dependencies (fftw, hdf5, silo, lapack, hipfort) produce that manifest on a normal build, and that it correctly discriminated on a real broken tree: rebuilt the one dependency that had a cache but no install, skipped the four that were genuinely installed.

One behaviour change worth flagging for review: a dependency provisioned by hand, outside MFC's own install path, will no longer be recognised and will be rebuilt. That is arguably correct, but it converts a silent-wrong-answer case into a loud-failure one.