rucio_utils

Utility functions for interacting with the Rucio web API.

class merge_utils.rucio_utils.RucioRetriever(source: FileRetriever)

Class for managing asynchronous queries to the Rucio web API.

Initialize the RucioRetriever with a source of file metadata.

Parameters:

source – FileRetriever object to use as the source of file metadata

async checksum(file: MergeFile, rucio: dict) → bool

Ensure file sizes and checksums from Rucio agree with the input metadata.

Parameters:

• file – MergeFile object to check

• rucio – Rucio replicas dictionary

Returns:

True if files match, False otherwise

async connect() → None

Connect to the Rucio web API

property dupes: dict

Return the set of duplicate files from the source

property files: MergeSet

Return the set of files from the source

async input_batches() → AsyncGenerator[dict, None]

Asynchronously retrieve metadata for the next batch of files.

Returns:

dict of MergeFile objects that were added

property missing: dict

Return the set of missing files from the source

output_chunks() → Generator[MergeChunk, None, None]

Yield chunks of files for merging.

Returns:

yeilds a series of MergeChunk objects

async process(files: dict) → None

Process a batch of files to find their physical locations in Rucio.

Parameters:

files – dictionary of files to process

run() → None

Retrieve metadata for all files.