Compare actual data by chunk in exdf-compare
exdf-compare
compares the actual data of keys by default, but did so by loading the entire key into memory using KeyData.ndarray()
for both operands. This MR introduces a configurable memory cap (10G by default), which is split evenly and used to load as many trains as can fit into it.
Two other minor improvements made along the way:
- Add progress bars for sources/keys in
exdf-compare
when not verbose or quiet - Use
tqdm
directly as iterable inexdf-du
for better resource management
Edited by Philipp Schmidt