Multi Repository Build (mrb) system

Overview

Teaching: 10 min
Exercises: 25 min

Questions

• How are different software versions handled?

Objectives

• Understand the roles of the tool mrb

• Create an mrb development area and read what it sets up

mrb

What is mrb and why do we need it?
Early on, the LArSoft team chose git and cmake as the software version manager and the build language, respectively, to keep up with industry standards and to take advantage of their new features. When we clone a git repository to a local copy and check out the code, we end up building it all. We would like LArSoft and DUNE code to be more modular, or at least the builds should reflect some of the inherent modularity of the code.

Ideally, we would like to only have to recompile a fraction of the software stack when we make a change. The granularity of the build in LArSoft and other art-based projects is the repository. So LArSoft and DUNE have divided code up into multiple repositories (DUNE ought to divide more than it has, but there are a few repositories already with different purposes). Sometimes one needs to modify code in multiple repositories at the same time for a particular project. This is where mrb comes in.

mrb stands for “multi-repository build”. mrb has features for cloning git repositories, setting up build and local products environments, building code, and checking for consistency (i.e. there are not two modules with the same name or two fcl files with the same name). mrb builds UPS products – when it installs the built code into the localProducts directory, it also makes the necessasry UPS table files and .version directories. mrb also has a tool for making a tarball of a build product for distribution to the grid. The software build example later in this tutorial exercises some of the features of mrb.

Link to the mrb reference guide

Exercise 1: explore an mrb development area

Building code with mrb takes more than one session slot, but setting up a development area and reading what it produces does not. This exercise is checkout and inspection only, no compile.

Run it in the SL7 container (see the setup episode for the dunesl7 alias). Start the container from a shell where the Spack dune-prototype environment is not active, otherwise its LD_LIBRARY_PATH leaks in and breaks the container’s tools.

source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
export DUNELAR_VERSION=v10_22_00d01
export DUNELAR_QUALIFIER=e26:prof

mkdir -p /exp/dune/app/users/$USER/mrbexplore
cd /exp/dune/app/users/$USER/mrbexplore
mrb newDev -q $DUNELAR_QUALIFIER
source localProducts*/setup
cd srcs
mrb g -t $DUNELAR_VERSION protoduneana
cd $MRB_BUILDDIR
mrbsetenv

Two warnings are expected here and are not errors: cannot find larsoft//releaseDB/base_dependency_database during mrb newDev, and could not identify a product/project version during mrbsetenv. Both are harmless for this setup.

Now answer these from the area you just made, without building anything:

1. Run git -C $MRB_SOURCE/protoduneana log --oneline -1 and git -C $MRB_SOURCE/protoduneana status. Which tag is checked out, and why does git say you are not on a branch?
2. Run grep -n protoduneana $MRB_SOURCE/CMakeLists.txt. What did mrb g add to that file, and what happens to it if you check out a second repository?
3. Run echo $MRB_SOURCE $MRB_BUILDDIR $MRB_INSTALL. These are three separate directories. Why does mrb keep source, build, and install areas apart?
4. Run echo $PRODUCTS. Your localProducts area is first in the list. After you build a package locally and then setup dunesw, which copy of that package does UPS use?
5. Run mrb s (short for mrb status). What does it show once more than one repository is checked out, and how does that relate to the consistency checks described above?

Solution

1. protoduneana is at tag v10_22_00d01 in “detached HEAD” state. mrb g -t checks out the tag as a commit, not a branch; you would run `git switch -c ` to start a branch for real edits.
2. mrb g added a line registering protoduneana (an ADD_SUBDIRECTORY) to $MRB_SOURCE/CMakeLists.txt. Each further mrb g adds another such line, which is how a single mrb b builds every checked-out repository together.
3. Out-of-source build: keeping build separate from srcs lets mrb z wipe build products without touching your edited code, and lets the same source build for more than one qualifier. localProducts (install) is separate again because it is the UPS-formatted output that other setups point at.
4. UPS reads $PRODUCTS left to right, so the locally built package in localProducts... shadows the CVMFS one. That is the point of a test release: your change is picked up with nothing reinstalled centrally.
5. mrb s reports the git branch and status of every repository under $MRB_SOURCE at once. With several repositories checked out it is how you keep them on a consistent set of tags; mrb also refuses to configure if two packages define a module or fcl file with the same name.

Exercise 2 (optional): build a package and watch an incremental rebuild

This one compiles code, so it needs a build node or a fast interactive node and about 15 minutes, most of it the first build. It uses larexamples, the LArSoft teaching package. Skip it if you are short on time; the point it makes also shows up in Episode 06.

Use a fresh development area, separate from Exercise 1. A development area holds one coherent set of packages from one release train. Mixing, say, protoduneana (the dunesw vXX_YY_ZZdNN scheme) and larexamples (the larsoft vXX_YY_ZZ scheme) in one area makes the configure step fail with a missing-dependency error such as Could not find a package configuration file provided by "dunereco".

Set up dunesw first, so the whole stack is on the build’s search path, then check out and build just larexamples. Setting up dunesw also exports LAREXAMPLES_VERSION into your environment (v10_02_02 for this release – the bare larsoft vXX_YY_ZZ scheme, not the dunesw dNN scheme), and that is the tag mrb g picks up below:

source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
setup dunesw v10_22_00d01 -q e26:prof

mkdir -p /exp/dune/app/users/$USER/mrbbuild
cd /exp/dune/app/users/$USER/mrbbuild
mrb newDev -q e26:prof
source localProducts*/setup
cd srcs
echo $LAREXAMPLES_VERSION            # set by 'setup dunesw'; v10_02_02 here
mrb g -t $LAREXAMPLES_VERSION larexamples
cd $MRB_BUILDDIR
mrbsetenv
time mrb i -j4

Note how long that first build took. Now change one file and rebuild:

1. Edit $MRB_SOURCE/larexamples/larexamples/AnalysisExample/AnalysisExample_module.cc and delete a semicolon somewhere in the analyze method. Run time mrb b -j4. It should fail in around ten seconds with a message like AnalysisExample_module.cc:437:32: error: expected ';' before 'fRun'. Restore the file afterwards with git -C $MRB_SOURCE/larexamples restore larexamples/AnalysisExample/AnalysisExample_module.cc.
2. Put the semicolon back. Add a message-logger line in analyze, for example mf::LogInfo("Exercise") << "my local build";. Run time mrb b -j4 again. It should take about 15 to 20 seconds: one file recompiles and the module library relinks. How does that compare to the first mrb i?
3. Run ls -l --time-style=+%H:%M:%S $MRB_INSTALL/*/lib/*AnalysisExample*.so. The timestamp is from your rebuild, not the original install.

Solution

1. mrb b (buildtool) recompiles only the translation units that changed, so it reaches the broken file in around ten seconds (12 s when this was checked) and stops with the compiler error, file, and line. Warnings are errors here too.
2. Only AnalysisExample_module.cc recompiles, then the library relinks: under twenty seconds (18 s when this was checked), against the minutes the first mrb i took. This is the point of mrb. You rebuild the one package you touched, not the stack it depends on.
3. The .so under localProducts is what source localProducts*/setup and mrbslp put first on the search path, so any job that loads that library now gets your build.

Key Points

• The multi-repository build (mrb) tool allows code modification in multiple repositories, which is relevant for a large project like LArSoft with different cases (end user and developers) demanding consistency between the builds.

• mrb keeps source, build, and localProducts areas separate, and puts localProducts first on the UPS path so a locally built package shadows the central one.

• Building code with mrb needs the SL7 container; it does not yet work under Spack on AL9.