DUNE merge-utils Documentation

Indices and tables

• Index

• Module Index

• Search Page

Contents

setup

• Installation and Startup
  • Installation
  • Testing
  • Dependencies

Configuration

• Configuration
• Types
  • defaults.yaml
  • metadata.yaml
  • logging.json
• Configuration
• Types
  • defaults.yaml
  • metadata.yaml
  • logging.json

command line API

• merge
  • get_inputs()
  • get_parser()
  • main()
  • print_metadata()
  • print_replicas()
  • resume_job()
  • start_job()

campaigns interface

• Campaigns
  • production scripts
    • utilities
  • sequence of actions

references

• Design docs
  • Design considerations
    • Motivation
    • Inputs
      • At the list generation stage, we need methods that
    • Move from Jobsub to Justin for batch running
      • Justin questions 2025-03-24
      • how do we get Justin to run over chunks of files in a list.
      • Methods needed:
    • Delivery methods
    • Merging methods
      • Note on Event counting and tracking
    • Consistency checks
    • Metadata merge:
      • Metadata fields added:
      • enhancements to TypeChecker.py
      • Sam4users
    • Where this runs
    • Suggestions for future versions
    • Design ideas:
• Notes on sphinx
  • rst format
  • conda install
  • Github actions
  • Google indexing Notes
• Related documentation
  • metacat
  • Data Dispatcher (dd)
  • samweb
  • JustIn

python API

• config_keys
  • ConfigBool
  • ConfigCollection
  • ConfigCondition
  • ConfigDict
    • ConfigDict.get()
    • ConfigDict.items()
  • ConfigFloat
  • ConfigInt
  • ConfigKey
  • ConfigList
    • ConfigList.append()
    • ConfigList.extend()
  • ConfigMap
    • ConfigMap.get()
    • ConfigMap.items()
    • ConfigMap.keys()
    • ConfigMap.update()
    • ConfigMap.values()
  • ConfigNum
  • ConfigOption
  • ConfigPath
  • ConfigSet
    • ConfigSet.extend()
  • ConfigSizeSpec
    • ConfigSizeSpec.a
    • ConfigSizeSpec.b
    • ConfigSizeSpec.n
    • ConfigSizeSpec.parse_spec()
    • ConfigSizeSpec.parse_term()
    • ConfigSizeSpec.s
  • ConfigString
    • ConfigString.format()
  • ConfigTuple
  • ConfigValue
    • ConfigValue.set()
    • ConfigValue.val()
    • ConfigValue.value
  • check_condition()
  • make_cfg_key()
  • parse_type()
• config
  • check_environment()
  • custom_serializer()
  • dump()
  • get_key()
  • load()
  • override()
  • resume()
  • set_cmd_opts()
  • set_error_handling()
  • set_host()
  • update()
  • uuid()
• io_utils
  • expand_path()
  • find_cfg()
  • find_file()
  • find_runner()
  • log_dict()
  • log_list()
  • log_nonzero()
  • log_print()
  • pkg_dir()
  • read_config_file()
  • read_json()
  • set_log_level()
  • setup_job_dir()
  • setup_log()
  • src_dir()
• justin_utils
  • get_site_rse_distances()
• merge_set
  • MergeChunk
    • MergeChunk.chunk_id
    • MergeChunk.inputs()
    • MergeChunk.make_child()
    • MergeChunk.make_name()
    • MergeChunk.metadata
    • MergeChunk.namespace
    • MergeChunk.outputs()
    • MergeChunk.parents
    • MergeChunk.settings()
    • MergeChunk.spec()
    • MergeChunk.specs
    • MergeChunk.tier
  • MergeFile
    • MergeFile.did
    • MergeFile.file_format
    • MergeFile.get_fields()
    • MergeFile.good
    • MergeFile.name
    • MergeFile.namespace
    • MergeFile.set_parents()
    • MergeFile.validate()
  • MergeFileError
    • MergeFileError.critical()
    • MergeFileError.first
    • MergeFileError.group
    • MergeFileError.handling
  • MergeSet
    • MergeSet.add()
    • MergeSet.all_files
    • MergeSet.at()
    • MergeSet.check_consistency()
    • MergeSet.check_errors()
    • MergeSet.end_idx
    • MergeSet.enum
    • MergeSet.enum_good
    • MergeSet.get_by_did()
    • MergeSet.get_by_idx()
    • MergeSet.get_slice()
    • MergeSet.good_files
    • MergeSet.group_by_count()
    • MergeSet.group_by_size()
    • MergeSet.groups()
    • MergeSet.insert()
    • MergeSet.set_error()
• meta
  • MergeMetaAll
    • MergeMetaAll.add()
    • MergeMetaAll.valid
    • MergeMetaAll.value
  • MergeMetaMax
    • MergeMetaMax.add()
    • MergeMetaMax.valid
  • MergeMetaMin
    • MergeMetaMin.add()
    • MergeMetaMin.valid
  • MergeMetaOverride
    • MergeMetaOverride.add()
    • MergeMetaOverride.valid
  • MergeMetaSubset
    • MergeMetaSubset.add()
    • MergeMetaSubset.valid
    • MergeMetaSubset.warn
  • MergeMetaSum
    • MergeMetaSum.add()
    • MergeMetaSum.valid
  • MergeMetaUnion
    • MergeMetaUnion.add()
    • MergeMetaUnion.valid
    • MergeMetaUnion.value
  • MergeMetaUnique
    • MergeMetaUnique.add()
    • MergeMetaUnique.valid
  • add_origin()
  • auto_output()
  • check_method()
  • check_required()
  • fix()
  • log_method()
  • make_names()
  • match_method()
  • merge_cfg_keys()
  • merged_keys()
  • parents()
  • set_method()
  • set_method_auto()
  • set_method_custom()
  • validate()
• metacat_utils
  • MetaCatWrapper
    • MetaCatWrapper.connect()
    • MetaCatWrapper.disconnect()
    • MetaCatWrapper.files()
    • MetaCatWrapper.query()
• config_keys
  • ConfigBool
  • ConfigCollection
  • ConfigCondition
  • ConfigDict
    • ConfigDict.get()
    • ConfigDict.items()
  • ConfigFloat
  • ConfigInt
  • ConfigKey
  • ConfigList
    • ConfigList.append()
    • ConfigList.extend()
  • ConfigMap
    • ConfigMap.get()
    • ConfigMap.items()
    • ConfigMap.keys()
    • ConfigMap.update()
    • ConfigMap.values()
  • ConfigNum
  • ConfigOption
  • ConfigPath
  • ConfigSet
    • ConfigSet.extend()
  • ConfigSizeSpec
    • ConfigSizeSpec.a
    • ConfigSizeSpec.b
    • ConfigSizeSpec.n
    • ConfigSizeSpec.parse_spec()
    • ConfigSizeSpec.parse_term()
    • ConfigSizeSpec.s
  • ConfigString
    • ConfigString.format()
  • ConfigTuple
  • ConfigValue
    • ConfigValue.set()
    • ConfigValue.val()
    • ConfigValue.value
  • check_condition()
  • make_cfg_key()
  • parse_type()
• replicas
  • BaseRSE
    • BaseRSE.cache_local()
    • BaseRSE.cache_xrootd()
    • BaseRSE.check()
    • BaseRSE.check_cache()
    • BaseRSE.check_local()
    • BaseRSE.check_xrootd()
    • BaseRSE.checksum_local()
    • BaseRSE.checksum_xrootd()
    • BaseRSE.ping()
    • BaseRSE.xrdfs()
  • GenericRSE
  • PathFinder
    • PathFinder.add_replica()
    • PathFinder.check_replica()
    • PathFinder.connect()
    • PathFinder.disconnect()
    • PathFinder.files
    • PathFinder.get_metadata()
    • PathFinder.get_paths()
    • PathFinder.input_batches()
    • PathFinder.replica_checker()
    • PathFinder.set_paths()
  • PathListFinder
    • PathListFinder.add_replica()
    • PathListFinder.add_rse()
    • PathListFinder.connect()
    • PathListFinder.get_paths()
    • PathListFinder.set_paths()
  • Replica
    • Replica.protocol
  • RucioFinder
    • RucioFinder.add_replica()
    • RucioFinder.checksum()
    • RucioFinder.connect()
    • RucioFinder.get_paths()
    • RucioFinder.set_paths()
  • RucioRSE
    • RucioRSE.set_distance()
    • RucioRSE.set_urls()
  • Status
    • Status.bad
    • Status.good
  • StatusMeta
  • get()
  • get_host()
  • get_path()
  • get_port()
  • get_protocol()
  • path_to_xrootd()
  • xrootd_to_path()
• retriever
  • DidRetriever
    • DidRetriever.check_duplicates()
    • DidRetriever.check_namespaces()
    • DidRetriever.get_metadata()
  • InputBatch
  • LocalMetaRetriever
    • LocalMetaRetriever.get_metadata()
  • MetaRetriever
    • MetaRetriever.check_existence()
    • MetaRetriever.check_parents()
    • MetaRetriever.connect()
    • MetaRetriever.disconnect()
    • MetaRetriever.files
    • MetaRetriever.get_batch()
    • MetaRetriever.get_done()
    • MetaRetriever.get_files()
    • MetaRetriever.get_metadata()
    • MetaRetriever.get_siblings()
    • MetaRetriever.input_batches()
    • MetaRetriever.namespace
    • MetaRetriever.run()
  • QueryRetriever
    • QueryRetriever.get_metadata()
  • file_serializer()
  • get()
• rucio_utils
  • RucioWrapper
    • RucioWrapper.connect()
    • RucioWrapper.disconnect()
    • RucioWrapper.get_replicas()
    • RucioWrapper.get_rse()
    • RucioWrapper.get_rses()
• scheduler
  • JobScheduler
    • JobScheduler.assign_site()
    • JobScheduler.chunk_distances()
    • JobScheduler.connect()
    • JobScheduler.disconnect()
    • JobScheduler.file_distances()
    • JobScheduler.files
    • JobScheduler.input_batches()
    • JobScheduler.replica_distances()
    • JobScheduler.run()
    • JobScheduler.run_loop()
    • JobScheduler.schedule()
    • JobScheduler.split_files()
    • JobScheduler.write_script()
    • JobScheduler.write_specs()
  • JustinScheduler
    • JustinScheduler.connect()
    • JustinScheduler.justin_cmd()
    • JustinScheduler.schedule()
    • JustinScheduler.upload_cfg()
    • JustinScheduler.write_script()
  • LocalScheduler
    • LocalScheduler.connect()
    • LocalScheduler.replica_distances()
    • LocalScheduler.schedule()
    • LocalScheduler.write_script()

• runners
  • do_merge
  • merge_hdf5
  • merge_tar
    • merge_tar()
  • pass2_fix
    • get_cfgs()
    • get_pfns()
    • main()
• jobscript

• Acknowledgements