The purpose of this newsletter is to inform the collaboration of recent and upcoming changes in computing. We’re in the middle of an ‘exciting’ time for computing as we’re constantly adapting to new operating systems and authentication methods. This means there are frequent changes that affect all DUNE users. We offer tutorials, the wiki, and a FAQ page, but those move more slowly so we’re trying this newsletter as a way to bridge the gap between permanent documentation and real-time announcements.
You can find newsletters at https://wiki.dunescience.org/wiki/Computing_Newsletters.
If you want computing news and have questions, remember to join the DUNE Slack and the #computing-announcements and #computing-questions channels. (And if you know an answer, please help out.)
Slack: #computing-questions, #computing-announcements, #computing_training_group
Tired of reading pages of documentation to do something? We now have cheat-sheets for fast access to DUNE procedures.
We currently have:
| Git | MetaCat | FHiCL | Spack |
We’re working on
If you have such a sheet or comments on the ones we have, contact dune-computing-training@fnal.gov or put in an issue at https://github.com/DUNE/FAQ/issues .
There is now a PHLEX adoption working group which meets alternate Mondays.
Indico: https://indico.fnal.gov/category/1781/
DUNE Slack: phlex
Mailing list: dune-phlex
bi-weekly meeting: Mondays at 10:30 AM CT
(alternate with AI/ML)
Co-chairs: Gavin S. Davies (gavin.s.davies@gmail.com) and Michel Sorel (sorel@ific.uv.es)
This alternates with the AI/ML meeting
Phlex v0.3.0, the next version in the Phlex prototype series. Release notes can be found at https://github.com/Framework-R-D/phlex/releases/tag/v0.3.0. Notable features include:
phlex::source base class, used to generate data-cell indices, and to create providers for data products to be read from an external data source.This work represents the coordinated efforts of developers from Fermilab, LBNL, and ANL, and it includes contributions from members of the SHiP experiment at CERN. As in the previous prototype releases, Phlex v0.3.0 is known to be incomplete with respect to the Phlex design. It does, however, represent a significant step forward in how the IO subsystem (via FORM) integrates with core Phlex behavior.
We are currently vetting our installation procedures and making sure extant documentation is up-to-date. Any issues should be reported at https://github.com/Framework-R-D/phlex/issues/new.
Thank you for your patience and for your continued strong support.
We have created a set of physics group-specific quotas in the DUNE global pool.
All physics groups together can take up to 10% of the total cycles in the DUNE global pool. Work that will be used by several people in the physics group should be coordinated so we don’t have several independent private copies of the same private data sets. There are corresponding Rucio scopes for each physics group which can be found on the “Scopes” tab in the justin UI, each is allocated between 25-90TB. To use the corresponding group quota, specify that scope in the justin simple-workflow command. If there is a medium-large physics group workflow you can apply to the CRAB to have the physics group batch quota temporarily increased.
Note: physics group storage scopes require that files have valid metadata (unlike the usertests scope). If you have not done metadata creation before, download extractor_new.py.
Last month, four dozen next generation particle physicists participated in an immersive data analysis school focused specifically on DUNE software, computing, and analysis tools. This inaugural week of learning included physics lectures and hands-on exercises on topics that included TPC and light simulations, reconstruction using Pandora, SPINE, CAF selections, and MaCh3. Prior to the event, the training group offered Computing Basics support. Kudos to the organizing and program committees for their work to make this event a success, and to Vishvas Pandey and Mike Kirby for their pedagogical leadership and perennial vision for DUNE DAS.
Many people like to use VNC’s to connect to the gpvms. This allows you to, for example, open root windows more quickly. Unfortunately, the default VNC method (often gnome) launch a full desktop environment which spawns hundreds of unused processes.
Ken Herner has written up how to use **IceWM **instead of gnome. We ask that everyone move to this method as it does not bog down our systems.
https://wiki.dunescience.org/wiki/DUNE_Computing/Using_VNC_Connections_on_the_dunegpvms
Describes how to set up the IceWM on your gpvm
And also provides a script from Dom Brailsford that you can use to set up your VNC system.
source /nashome/d/dbrailsf/setupVNC.sh
There has been a lot of work on spack builds for DUNE software for use on AL9. Please join the #spack channel in Slack for frequent updates and to report issues. We are now up to v1.1 for spack itself and 0_2_0 for Phlex. For convenience, there are generic setups but they point to versioned instances.
larsoftsource /cvmfs/dune.opensciencegrid.org/spack/setup-env.sh
echo "Activate dune-prototype"
spack env activate dune-prototype
phlex (our new framework for testing)source /cvmfs/dune.opensciencegrid.org/spack/setup-env.sh
echo "Activate dune-phlex-prototype"
# currently phlex-0_2_0-prototype
spack env activate dune-phlex-prototype
Most host certificates in the United States institutions are currently issued by InCommon.
InCommon has begun the process of migrating Fermilab to its new certificate backend provider called CertiNext. These new host certificates will no longer be usable as an SSL client.
Key DUNE services such as the Global Pool frontend and the Rucio server still have certs that can be used as SSL clients, which expire in March of 2027. By that time DUNE must be in a situation where we use no more SSL client certificates.
There are only 3 DUNE compute sites which do not accept tokens. One of these three is in process of changing at the moment. We will also complete the transition of the internals of the DUNE Rucio server, several machines in the DUNE data ingest pipeline, and the GlideinWMS frontend to no longer rely on the InCommon certificates as clients, with the goal of finishing this work by Sep. of 2026.
Likely announcement on
DOE has announced a new funding opportunity for the Genesis Mission:
NOFO: DE-FOA-0003612
https://science.osti.gov/-/media/grants/pdf/foas/2026/DE-FOA-0003612.pdf
The deadline has passed for Phase 1 but people need to be thinking about Phase 2
Genesis Mission Q&A from DOE:
We will continue discussion and coordination of Genesis Mission proposals through the DUNE AI/ML Forum:
https://indico.fnal.gov/category/1676/
DUNE Slack: ai-ml
Mailing list: dune-ai-ml
DUNE AI/ML forum bi-weekly meeting: Mondays at 10:30 AM CT
Co-chairs: Jianming Bian (bianjm@uci.edu) and Leigh Whitehead (leigh.howard.whitehead@cern.ch)
The Production Group makes official datasets. The Rice group has created a website for finding them. There are also command line tools for exploring further. See the tutorial for details or go straight to the website to find your samples.
A reminder that the Fermilab compute nodes and disk servers undergo monthly maintenance on the 3rd Wednesday of each month. Systems are partially unavailable for 3-6 hours, which leads to things not working.
Just a reminder that running larsoft still requires Scientific Linux 7 in a container. You can find sample scripts for setting up your container here.
Tokens are the new way to authenticate to access dCache disk, our data catalogs (rucio) and submit batch jobs (justin and jobsub). On SL7, follow these instructions . On AL9, it still depends on which Spack environment you are using. The recent dune-prototype spack environment (above) can make tokens using justIN get-token. We suggest you upgrade to that one.
If you just want to access files via root or xrdcp and don’t need rucio or justin you can use this method to get an interactive token.
htgettoken -i dune --vaultserver htvaultprod.fnal.gov -r interactive
export BEARER_TOKEN_FILE=/run/user/`id -u`/bt_u`id -u`
export X509_CERT_DIR=/cvmfs/oasis.opensciencegrid.org/mis/certificates
`LD_PRELOAD=$XROOTD_LIB/libXrdPosixPreload.so lar -c standard_reco_protodunehd_keepup.fcl $DATA -n 1`