Skip to content

Compare actual data by chunk in exdf-compare

Philipp Schmidt requested to merge feat/compare-chunked into master

exdf-compare compares the actual data of keys by default, but did so by loading the entire key into memory using KeyData.ndarray() for both operands. This MR introduces a configurable memory cap (10G by default), which is split evenly and used to load as many trains as can fit into it.

Two other minor improvements made along the way:

  • Add progress bars for sources/keys in exdf-compare when not verbose or quiet
  • Use tqdm directly as iterable in exdf-du for better resource management
Edited by Philipp Schmidt

Merge request reports

Loading