The purpose of this newsletter is to inform the collaboration of recent and upcoming changes in computing. We’re in the middle of an ‘exciting’ time for computing as we’re constantly adapting to new operating systems and authentication methods. This means there are frequent changes that affect all DUNE users. We offer tutorials, the wiki, and a FAQ page, but those move more slowly so we’re trying this newsletter as a way to bridge the gap between permanent documentation and real-time announcements.
You can find newsletters at https://wiki.dunescience.org/wiki/Computing_Newsletters.
If you want computing news and have questions, remember to join the DUNE Slack and the #computing-announcements and #computing-questions channels. (And if you know an answer, please help out.)
Slack: #computing-questions, #computing-announcements, #computing_training_group
The NPC DUNE Data Analysis School (NPC-DDAS) will be held Fermilab. The school will provide focused, practical training in DUNE analysis tools and workflows. 30 people have registered and will be attending in person. More info: https://npc.fnal.gov/npc-data-analysis-school/ and https://indico.fnal.gov/event/73408/.
There will be a pre-workshop online setup session for people new to using the Fermilab computers. It will be held on Wednesday May 6, 2-4 PM Fermilab time. This is a session to help people use the Fermilab computers before the NPC Analysis workshop. It is open to all DUNE collaborators. People not registered for the NPC workshop need to register here to receive the zoom link. We will cover the material in https://dune.github.io/computing-basics/ and parts of https://dune.github.io/computing-basics-larsoft/
We plan to hold in person computing help sessions. They are not scheduled yet but are likely to be during lunch breaks on Tuesday and Wednesday.
Many people like to use VNC’s to connect to the gpvms. This allows you to, for example, open root windows more quickly. Unfortunately, the default VNC method (often gnome) launch a full desktop environment which spawns hundreds of unused processes.
Ken Herner has written up how to use **IceWM **instead of gnome. We ask that everyone move to this method as it does not bog down our systems.
https://wiki.dunescience.org/wiki/DUNE_Computing/Using_VNC_Connections_on_the_dunegpvms
Describes how to set up the IceWM on your gpvm
And also provides a script from Dom Brailsford that you can use to set up your VNC system.
source /nashome/d/dbrailsf/setupVNC.sh
There has been a lot of work on spack builds for DUNE software for use on AL9. Please join the #spack channel in Slack for frequent updates and to report issues. We are now up to v1.1 for spack itself and 0_2_0 for Phlex. For convenience, there are generic setups but they point to versioned instances.
larsoftsource /cvmfs/dune.opensciencegrid.org/spack/setup-env.sh
echo "Activate dune-prototype"
spack env activate dune-prototype
phlex (our new framework for testing)source /cvmfs/dune.opensciencegrid.org/spack/setup-env.sh
echo "Activate dune-phlex-prototype"
# currently phlex-0_2_0-prototype
spack env activate dune-phlex-prototype
Most host certificates in the United States institutions are currently issued by InCommon.
InCommon has recently announced they will be using a different back-end certificate provider called CertiNext, no later than July. The schedule for Fermilab to migrate to this backend has not yet been finalized. In the interim time period it is still possible to get a certificate that can be used as a TLS Web Client.
We had reached out to 13 compute sites and several storage elements which still only take certificates. All but 3 of them have converted to tokens now. We will also complete the transition of the internals of the DUNE Rucio server, several machines in the DUNE data ingest pipeline, and the GlideinWMS frontend to no longer rely on the InCommon certificates as clients, with the goal of finishing this work by July 2026.
Users can currently run jobs with a fair-share priority. But we often find several people performing similar tasks at large scale and then storing the results, uncataloged, in personal areas. To preserve resources we plan to move to a tiered priority system where people acting on behalf of physics groups or the production group will have higher priority than individuals. Individuals can still submit their own jobs but will find they run faster if they collaborate within their physics groups. See Andrew McNab’s talk for a brief explanation. Implementation will occur over the next few months.
DOE has announced a new funding opportunity for the Genesis Mission:
NOFO: DE-FOA-0003612
https://science.osti.gov/-/media/grants/pdf/foas/2026/DE-FOA-0003612.pdf
The deadline has passed for Phase 1 but people need to be thinking about Phase 2
Genesis Mission Q&A from DOE:
We will continue discussion and coordination of Genesis Mission proposals through the DUNE AI/ML Forum:
https://indico.fnal.gov/category/1676/
DUNE Slack: ai-ml
Mailing list: dune-ai-ml
DUNE AI/ML forum bi-weekly meeting: **Mondays at 10:00 AM CT**
Co-chairs: Jianming Bian (bianjm@uci.edu) and Leigh Whitehead (leigh.howard.whitehead@cern.ch)
Held March 10–12 at Rice’s BioScience Research Collaborative, the three-day workshop (https://indico.fnal.gov/event/72868/) explored how the CS&SC consortium can better support the growing role of AI and ML in DUNE. On Day 1, participants reviewed the upcoming physics goals of the experiment and learned how the various working groups are incorporating new AI/ML trends into DUNE’s toolkits. Day 2 focused on available computing resources (GPUs), future plans to expand them, support for AI/ML workflows, and how large language models are being explored within DUNE. On the final day, participants examined emerging topics, from foundation models to differential programming and concluded by summarizing the workshop’s key action items.
The Production Group makes official datasets. The Rice group has created a website for finding them. There are also command line tools for exploring further. See the tutorial for details or go straight to the website to find your samples.
A reminder that the Fermilab compute nodes and disk servers undergo monthly maintenance on the 3rd Wednesday of each month. Systems are partially unavailable for 3-6 hours, which leads to things not working.
Just a reminder that running larsoft still requires Scientific Linux 7 in a container. You can find sample scripts for setting up your container here.
Tokens are the new way to authenticate to access dCache disk, our data catalogs (rucio) and submit batch jobs (justin and jobsub). On SL7, follow these instructions . On AL9, it still depends on which Spack environment you are using. The recent dune-workflow spack environment (above) can make tokens using justIN get-token. We suggest you upgrade to that one.
If you just want to access files via root or xrdcp and don’t need rucio or justin you can use this method to get an interactive token.
htgettoken -i dune --vaultserver htvaultprod.fnal.gov -r interactive
export BEARER_TOKEN_FILE=/run/user/`id -u`/bt_u`id -u`
export X509_CERT_DIR=/cvmfs/oasis.opensciencegrid.org/mis/certificates
LD_PRELOAD=$XROOTD_LIB/libXrdPosixPreload.so lar -c standard_reco_protodunehd_keepup.fcl $DATA -n 1