local

Class variants for managing local file retrieval and merging operations.

class merge_utils.local.LocalMetaRetriever(files: dict)

MetaRetriever for local files

Initialize the LocalMetaRetriever with a list of json files.

Parameters:

files – dictionary of metadata file names and paths

async input_batches() → AsyncGenerator[dict, None]

Retrieve metadata for local files in batches

class merge_utils.local.LocalPathFinder(source: MetaRetriever, files: dict | None = None, dirs: list | None = None)

PathFinder for local files

Initialize the LocalMetaRetriever with a list of json files.

Parameters:

• source – MetaRetriever object to use as the source of file metadata

• files – dictionary of metadata file names and paths

• dirs – list of directories to search for data files

async process(files: dict) → None

Process a batch of files to find their physical locations.

Parameters:

files – dictionary of files to process

merge_utils.local.get_local_files(files: list, dirs: list) → PathFinder

Search local filesystem for pairs of data and metadata files. If both data and metadata files are found, return a LocalPathFinder using a LocalMetaRetriever. If only metadata files are found, return a RucioPathFinder using a LocalMetaRetriever. If only data files are found, return a LocalPathFinder using a MetaCatRetriever.

Parameters:

• files – List of input data or metadata file paths

• dirs – List of directories to search for corresponding data or metadata files

Returns:

Appropriate PathFinder object

merge_utils.local.search(file: str, dirs: list) → str

Search for a file in a list of directories.

Parameters:

• file – Name of the file to search for

• dirs – List of directories to search in

Returns:

Full path to the file if found, otherwise None