DUNE merge-utils Documentation

Indices and tables

• Index

• Module Index

• Search Page

Contents

setup

• Installation and Startup
  • Installation
  • Testing
  • Dependencies

Configuration

• Configuration
  • defaults.yaml
  • metadata.yaml
  • logging.json

Production scripts

• Production Utilities (prod_utils)
  • Setup

python API

• do_merge
• config
  • check_environment()
  • load()
  • update()
  • update_dict()
  • update_list()
  • uuid()
• local
• io_utils
  • expand_path()
  • find_cfg()
  • find_file()
  • find_runner()
  • get_inputs()
  • log_dict()
  • log_list()
  • log_nonzero()
  • log_print()
  • pkg_dir()
  • read_config_file()
  • set_log_level()
  • setup_log()
  • src_dir()
• metacat_utils
• meta
  • MergeMetaAll
    • MergeMetaAll.add()
    • MergeMetaAll.valid
    • MergeMetaAll.value
  • MergeMetaMax
    • MergeMetaMax.add()
    • MergeMetaMax.valid
  • MergeMetaMin
    • MergeMetaMin.add()
    • MergeMetaMin.valid
  • MergeMetaOverride
    • MergeMetaOverride.add()
    • MergeMetaOverride.valid
  • MergeMetaSubset
    • MergeMetaSubset.add()
    • MergeMetaSubset.valid
    • MergeMetaSubset.warn
  • MergeMetaSum
    • MergeMetaSum.add()
    • MergeMetaSum.valid
  • MergeMetaUnion
    • MergeMetaUnion.add()
    • MergeMetaUnion.valid
    • MergeMetaUnion.value
  • MergeMetaUnique
    • MergeMetaUnique.add()
    • MergeMetaUnique.valid
  • MetaNameDict
    • MetaNameDict.MetaKey
    • MetaNameDict.MetaVal
      • MetaNameDict.MetaVal.key
      • MetaNameDict.MetaVal.read_dict()
      • MetaNameDict.MetaVal.read_list()
      • MetaNameDict.MetaVal.value
    • MetaNameDict.eval()
    • MetaNameDict.format()
    • MetaNameDict.get()
  • auto_output()
  • check_method()
  • check_required()
  • fix()
  • log_method()
  • make_names()
  • match_method()
  • merge_cfg_keys()
  • merged_keys()
  • parents()
  • set_method()
  • set_method_auto()
  • set_method_custom()
  • validate()
• rucio_utils
• justin_utils
  • get_distances()
• merge_rse
  • MergeRSE
    • MergeRSE.distance()
    • MergeRSE.get_ping()
    • MergeRSE.nearest_site()
    • MergeRSE.pfns
  • MergeRSEs
    • MergeRSEs.add_pfn()
    • MergeRSEs.add_replicas()
    • MergeRSEs.cleanup()
    • MergeRSEs.connect()
    • MergeRSEs.get_pfns()
    • MergeRSEs.optimize_pfns()
    • MergeRSEs.site_pfns()
  • check_path()
• merge_set
• retriever
• scheduler
• CheckConfiguration
• MetaValidator
  • MetaValidator()

command line API

• merge

references

• Design docs
  • Design considerations
    • Motivation
    • Inputs
      • At the list generation stage, we need methods that
    • Move from Jobsub to Justin for batch running
      • Justin questions 2025-03-24
      • how do we get Justin to run over chunks of files in a list.
      • Methods needed:
    • Delivery methods
    • Merging methods
      • Note on Event counting and tracking
    • Consistency checks
    • Metadata merge:
      • Metadata fields added:
      • enhancements to TypeChecker.py
      • Sam4users
    • Where this runs
    • Suggestions for future versions
    • Design ideas:
• Notes on sphinx
  • rst format
  • conda install
  • Github actions
  • Google indexing Notes
• Related documentation
  • metacat
  • Data Dispatcher (dd)
  • JustIn

• Acknowledgements