DUNE merge-utils Documentation

Indices and tables

• Index

• Module Index

• Search Page

Contents

setup

• Installation and Startup
  • Installation
  • Testing
  • Dependencies

Configuration

• Configuration
  • defaults.yaml
  • metadata.yaml
  • logging.json
  • valid_values.json

python API

• do_merge
• config
  • check_environment()
  • load()
  • update()
  • update_dict()
  • update_list()
  • uuid()
• local
  • LocalMetaRetriever
    • LocalMetaRetriever.input_batches()
  • LocalPathFinder
    • LocalPathFinder.process()
  • get_local_files()
  • search()
• io_utils
  • expand_path()
  • find_cfg()
  • find_file()
  • find_runner()
  • get_inputs()
  • log_dict()
  • log_list()
  • log_nonzero()
  • log_print()
  • pkg_dir()
  • read_config_file()
  • set_log_level()
  • setup_log()
  • src_dir()
• metacat_utils
  • MetaCatRetriever
    • MetaCatRetriever.connect()
    • MetaCatRetriever.input_batches()
  • list_extensions()
  • list_field_values()
• meta
  • MergeMetaAll
    • MergeMetaAll.add()
    • MergeMetaAll.valid
    • MergeMetaAll.value
  • MergeMetaMax
    • MergeMetaMax.add()
    • MergeMetaMax.valid
  • MergeMetaMin
    • MergeMetaMin.add()
    • MergeMetaMin.valid
  • MergeMetaOverride
    • MergeMetaOverride.add()
    • MergeMetaOverride.valid
  • MergeMetaSubset
    • MergeMetaSubset.add()
    • MergeMetaSubset.valid
    • MergeMetaSubset.warn
  • MergeMetaSum
    • MergeMetaSum.add()
    • MergeMetaSum.valid
  • MergeMetaUnion
    • MergeMetaUnion.add()
    • MergeMetaUnion.valid
    • MergeMetaUnion.value
  • MergeMetaUnique
    • MergeMetaUnique.add()
    • MergeMetaUnique.valid
  • MetaNameDict
    • MetaNameDict.MetaKey
    • MetaNameDict.MetaVal
      • MetaNameDict.MetaVal.key
      • MetaNameDict.MetaVal.read_dict()
      • MetaNameDict.MetaVal.read_list()
      • MetaNameDict.MetaVal.value
    • MetaNameDict.eval()
    • MetaNameDict.format()
    • MetaNameDict.get()
  • auto_output()
  • check_method()
  • check_required()
  • fix()
  • log_method()
  • make_names()
  • match_method()
  • merge_cfg_keys()
  • merged_keys()
  • parents()
  • set_method()
  • set_method_auto()
  • set_method_custom()
  • validate()
• rucio_utils
  • RucioFinder
    • RucioFinder.checksum()
    • RucioFinder.connect()
    • RucioFinder.output_chunks()
    • RucioFinder.process()
    • RucioFinder.run()
• justin_utils
  • get_distances()
• merge_rse
  • MergeRSE
    • MergeRSE.distance()
    • MergeRSE.get_ping()
    • MergeRSE.nearest_site()
    • MergeRSE.pfns
  • MergeRSEs
    • MergeRSEs.add_pfn()
    • MergeRSEs.add_replicas()
    • MergeRSEs.cleanup()
    • MergeRSEs.connect()
    • MergeRSEs.get_pfns()
    • MergeRSEs.optimize_pfns()
    • MergeRSEs.site_pfns()
  • check_path()
• merge_set
  • MergeChunk
    • MergeChunk.add()
    • MergeChunk.chunk()
    • MergeChunk.get_output()
    • MergeChunk.inputs
    • MergeChunk.json
    • MergeChunk.make_name()
    • MergeChunk.metadata
    • MergeChunk.namespace
    • MergeChunk.outputs
    • MergeChunk.parents
    • MergeChunk.settings
    • MergeChunk.tier
    • MergeChunk.tier2_chunks()
  • MergeFile
    • MergeFile.did
    • MergeFile.format
    • MergeFile.get_fields()
    • MergeFile.name
    • MergeFile.namespace
  • MergeSet
    • MergeSet.add_file()
    • MergeSet.add_files()
    • MergeSet.check_consistency()
    • MergeSet.check_errors()
    • MergeSet.check_missing()
    • MergeSet.check_reachability()
    • MergeSet.check_uniqueness()
    • MergeSet.check_validity()
    • MergeSet.count_errors()
    • MergeSet.dupes
    • MergeSet.files
    • MergeSet.group_count()
    • MergeSet.group_size()
    • MergeSet.groups()
    • MergeSet.hash
    • MergeSet.set_unreachable()
    • MergeSet.size
  • check_remote_path()
• retriever
  • MetaRetriever
    • MetaRetriever.add()
    • MetaRetriever.connect()
    • MetaRetriever.dupes
    • MetaRetriever.files
    • MetaRetriever.input_batches()
    • MetaRetriever.missing
    • MetaRetriever.output_chunks()
    • MetaRetriever.run()
  • PathFinder
    • PathFinder.connect()
    • PathFinder.files
    • PathFinder.input_batches()
    • PathFinder.process()
• scheduler
  • JobScheduler
    • JobScheduler.run()
    • JobScheduler.write_json()
    • JobScheduler.write_script()
  • JustinScheduler
    • JustinScheduler.justin_cmd()
    • JustinScheduler.upload_cfg()
    • JustinScheduler.write_script()
  • LocalScheduler
    • LocalScheduler.write_json()
    • LocalScheduler.write_script()
• CheckConfiguration
• MetaValidator
  • MetaValidator()

command line API

• merge
  • main()

references

• Design docs
  • Design considerations
    • Motivation
    • Inputs
      • At the list generation stage, we need methods that
    • Move from Jobsub to Justin for batch running
      • Justin questions 2025-03-24
      • how do we get Justin to run over chunks of files in a list.
      • Methods needed:
    • Delivery methods
    • Merging methods
      • Note on Event counting and tracking
    • Consistency checks
    • Metadata merge:
      • Metadata fields added:
      • enhancements to TypeChecker.py
      • Sam4users
    • Where this runs
    • Suggestions for future versions
    • Design ideas:
• Notes on sphinx
  • rst format
  • conda install
  • Github actions
  • Google indexing Notes
• Related documentation
  • metacat
  • Data Dispatcher (dd)
  • samweb
  • JustIn

• Acknowledgements