Developing a DUNE package with MPD on AL9¶

One path, start to finish, for building a DUNE/LArSoft package with Spack and MPD on an AL9 gpvm against a published DUNE release environment. It was worked out between 8 and 19 September 2026 by rebuilding dunereco against dunesw-10_22_00d01-justin-01_06_04-prototype, then checked with a clean rerun from scratch.

It came out of #9 and FNALssi/spack-mpd#62. Thanks to Patrick Gartung for the recipe-commit fix and to Kyle Knoepfel for the MPD changes that followed.

The four things that matter¶

Each makes every later step look broken until it is right.

1. Create the subspack from v1.1.1/setup-env.sh. The packages behind the -prototype environments live under v1.1.1/opt/spack. A subspack from the top-level (v1.2.2) script reuses nothing, tries to build ten packages and dies fetching gcc-runtime@12.5.0 from the build cache.
2. Check the fnal_art recipe repo out at the commit the release was built with. For this release that is f80d956792. Newer recipes describe a dependency graph the installed tree does not contain, which produces Missing dependency not in database warnings and a view with no CLHEP in it.
3. Do not pin versions by hand. require: ["@=..."] entries taken from spack find name the newest installed version, not the one the environment uses (larsim@10.20.06 where the environment has 10.20.05), so they force rebuilds and shift every downstream hash.
4. Force the step you changed. refresh replays local/spack.lock and build skips configure when a build directory exists. Use refresh -f after any config change and build --clean after anything that affects configure.

Runbook¶

Set the variables once. Use a new INST for a clean run: instances are cheap, and stale state is what costs hours.

ssh dunegpvmNN.fnal.gov
htgettoken -a htvaultprod.fnal.gov -i dune

INST=/exp/dune/data/users/$USER/mpd1
PROJ=my-project
ENVNAME=dunesw-10_22_00d01-justin-01_06_04-prototype
TAG=v10_22_00d01
PKG=dunereco

1. Subspack from the right parent¶

source /cvmfs/dune.opensciencegrid.org/spack/v1.1.1/setup-env.sh
echo "ROOT=$SPACK_ROOT"   # MUST contain /v1.1.1/
echo "ENV=[$SPACK_ENV]"   # MUST be empty, or subspack writes onto read-only CVMFS

mkdir -p "$INST" && cd "$INST"
spack subspack --with-padding "$PWD/spack"
source spack/setup-env.sh
echo "ROOT=$SPACK_ROOT"   # MUST now be $INST/spack

2. Make the release environments resolvable, then init MPD¶

Environment names are instance-local. spack env list on the current CVMFS tree shows newer dunesw environments whose packages are not installed under v1.1.1, and they are unusable here.

ln -s /cvmfs/dune.opensciencegrid.org/spack/v1.1.1/var/spack/environments/* \
      "$SPACK_ROOT/var/spack/environments/"   # not needed if you sourced v1.1.1
spack env list | grep -F "$ENVNAME"           # MUST print
spack env deactivate 2>/dev/null              # git-clone and refresh need no env active
spack mpd init

3. Pin the recipe repo¶

R=$(spack repo list | awk '/fnal_art/{print $NF}')
echo "$R"
git -C "$R" log -1 --format='%H %cd'   # record it, so you can go back
git -C "$R" fetch --all
git -C "$R" checkout f80d956792

spack config get packages | grep -c require   # expect no hand pins

spack spec larsim > spec.log 2>&1; head -3 spec.log   # expect [^] larsim@10.20.05

If the repo path is under /cvmfs, it is read-only: clone it yourself, check out the commit, and spack repo add it ahead of the CVMFS copy.

4. Project and source¶

spack mpd n -C gcc@12.5.0 -T "$INST/$PROJ" -E "$ENVNAME" -d cetmodules@3
spack mpd git-clone "$PKG"

cd "$INST/$PROJ/srcs/$PKG"
git fetch --tags
git checkout -b my-feature "$TAG"   # git-clone lands on develop
git describe --tags                 # MUST print exactly $TAG
cd "$INST"

5. Concretize, and read the log before answering¶

spack mpd refresh -f generator=ninja 2>&1 | tee refresh.log
grep -c "not in database" refresh.log   # expect 0

A correct run, checked on 18 September 2026:

• Reused from /cvmfs/.../v1.1.1/opt/spack/: larsim@10.20.05, larreco@10.06.07, larana@10.02.07, larpandora@10.04.00, larrecodnn@10.04.00, artg4tk@13.00.00, art@3.14.04, geant4@11.2.2, cetmodules@3.27.04, clhep@2.4.7.1.
• Built from source: dunecore, dunecalib and duneprototypes at 10.23.00d01, about 8 minutes at 4 cores. Their recipes come from a different repo than fnal_art, so that layer is still unpinned.
• Answer the core-count prompt with the number you want. MPD keeps it in the project config, and -j on build does not override it.

6. Build¶

spack mpd build --clean -j12 2>&1 | tee build1.log

--clean is required on a first build, and again after adding any source file, because art_make globs .cxx at configure time. Deprecation warnings from cetmodules are expected. -Werror is on, so anything that stops the build is a real error.

ninja -C "$INST/$PROJ/build"   # "ninja: no work to do." means every target is current

Then check the plugins are visible:

spack mpd select -p "$PROJ"
lar --print-available module | grep -i <YourModule>
echo "$CET_PLUGIN_PATH" | tr ':' '\n' | head   # project build/install first

7. Resuming after a break¶

The instance, the project and the repo checkout all persist. A fresh shell needs only:

source "$INST/spack/setup-env.sh"   # NOT the CVMFS script
spack mpd list                      # project listed?
spack mpd select -p "$PROJ"         # marker moves to it
git -C "$(spack repo list | awk '/fnal_art/{print $NF}')" log -1 --format='%H'

refresh and build both refuse to run with no project selected, and selection does not carry across instances.

8. The cetmodules experiment¶

A few minutes, and nothing compiles.

spack mpd n -C gcc@12.5.0 -T "$INST/probe-nocet" -E "$ENVNAME" -S "$INST/$PROJ/srcs"
spack mpd refresh -f generator=ninja 2>&1 | tee nocet.log   # answer n at the prompt
grep -c "not in database" nocet.log
grep -E "^\[.\]" nocet.log | grep -E "cetmodules|clhep"
spack mpd select -p "$PROJ"                                  # switch back

A clean result means cetmodules@3 is not needed. Warnings coming back, or cetmodules@4 being chosen, means -d cetmodules@3 is load-bearing and belongs in every DUNE MPD recipe. Please post either answer on #9.

When something looks broken¶

Still open¶

• The dune* recipe layer is unpinned. dunecore, dunecalib and duneprototypes build from source every time.
• Recipe commits are not published. Nothing in the environment directory records which fnal_art commit built it; it had to be looked up by hand.
• Development-cycle cost. Concretize plus build of one package took over 30 minutes, the number to compare against the 1 October "Spack and MPD no worse than MRB" milestone.
• Whether cetmodules@3 is required (step 8).

Fallback: SL7 Apptainer and mrb¶

The SL7 container with mrb still works (checked 7 September 2026). Use it if the Spack path stalls.