Short submission with your own code – Batch Computing Basics for DUNE

this collects the sequence of steps for a batch submission with local code which produces both an artroot and root file.

It splits out different functions so you need to examine/adapt all of the support scripts which can be found here.

This sequence assumes you are in your top level mrb directory.

in your top level mrb directory

For example /exp/dune/app/users/$USER/myworkarea

need to have a name for it as you will be making a tarball

export DIRECTORY=myworkarea

copy these scripts into that top level directory

You can access a tarball with them all here.

Download that tarball into the top level directory for your build and

tar xBf usefulcode.tar

to get the code.

here are the scripts..

utilities you need

setup-grid (should not need to modify) This replaces setup in your local_build directory.
maketar.sh $DIRECTORY (should not need to modify) This takes the contents of $DIRECTORY and makes a tarball in /exp/data/users/$USER/
makerdcs.sh $DIRECTORY (should not need to modify) This takes the tarball and copies it to /cvmfs/ where grid jobs can find it. It places the location in the file $DIRECTORY/cvmfs.location so you can find it.

setup scripts

setup_before_submit.sh (customize versions for your code) You need to modify this to reflect the code version you are setting up. Normally only need to run/modify this once/session.
job_config.sh (you modify this to reflect your workflow. Sets things like $FCL_FILE). This sets up essential job parameters. You need to understand and modify these appropriately for your purpose.

Script to test and submit jobs

test_workflow.sh (script to do interactive tests of your jobscript)
submit_workflow.sh (writes output to scratch. Modify running time and memory)
submit_workflow_rucio.sh (writes output to rucio. Modify running time and memory)

scripts that run on the remote machine

extractor_new.py (this makes metadata for your files)

submit_local_code.jobscript.sh (may need to modify if expert)

How to run these scripts

modify two-three scripts (should not need to change the others)

edit

setup_before_submit.sh if you change code versions and
job_config.sh if you change more temporary things like fcl files.
choose your code version (code version has to match your build)
make certain the fcl file is either in the fcl path or in $DIRECTORY
add a string APP_TAG that will go in your filename
add a description in DESCRIPTION

Then run those scripts to set things up

source setup_before_submit.sh # sets up larsoft

from $DIRECTORY make a tarball and put in rcds

If you have changed any scripts or code, you must redo this.

./maketar.sh $DIRECTORY
./makercds.sh $DIRECTORY

will take a while, produce a tarball on /exp/dune/data/users/$USER/ and put the cvmfs location in cvmfs.location in $DIRECTORY

Configure your job with job_config.sh

Then edit job_config.sh to reflect the # of events you want and other run-time parameters.

Details of job_config.sh

Here is what is in job_config.sh

export FCL_FILE="run_analyseEvents.fcl" # fcl file
export OUTPUT_DATA_TIER1="full-reconstructed" # tier for artroot output
export OUTPUT_DATA_TIER2="root-tuple" # tier for root output
export MQL="files where dune.workflow['workflow_id']=3923 and core.data_tier=full-reconstructed limit 2 ordered " # metacat query for files
export APP_TAG="ana" # application name
export DESCRIPTION="$APP_TAG using $FCL_FILE" # appears as jobname in justin
export USERF=${USER} # make certain the grid knows who your are
export NUM_EVENTS=-1 # process them all
export FNALURL='https://fndcadoor.fnal.gov:2880/dune/scratch/users' # sends output to scratch
export NAMESPACE="usertests" # don't change this unless doing production

FCL_FILE= the top level fcl file - assumes it is in DIRECTORY or in the PHICL_FILE_PATH
OUTPUT_DATA_TIER1 data_tier for artroot output (full-reconstructed, …)
OUTPUT_DATA_TIER2 data_tier for root output - normally plain root-tuple
MQL Metacat query that you wish to run over
APP_TAG this is a tag that goes into the output filename, like reco2, ana …
DESCRIPTION the jobname that shows up in justIN
USERF make certain the grid knows who your are without overwriting whatever internal USER it has
NUM_EVENTS the -n argument of larsoft
FNALURL sends output to subdirectories of your area on scratch
NAMESPACE rucio/metacat namespace for your output, normally usertests or possibly your username or physics group unless you are doing production.

Test your jobscript interactively

test_workflow.sh

results will show up in the ‘tmp’ area on your local machine.

Submit the job

submit_workflow.sh

This one writes to /pnfs/dune/scratch

./submit_workflow.sh

submit_workflow_rucio.sh

This one writes to a rucio location specified by $NAMESPACE

./submit_workflow_rucio.sh

after submission

You should get a workflow number back

go to justin

to track your job.

Batch Computing Basics for DUNE - 2025 transition edition: Short submission with your own code