Skip to content

Add reduction operation optimizer

Philipp Schmidt requested to merge feat/optimize-operations into master

The recent application of LFF (before introducing !38 (closed)) emitted a reduction operation per train and module, which caused performance issues both during initialization and excessive metadata in each file. So it seems prudent to apply data reduction to data reduction.

This MR adds the basic infrastructure to optimize the stream of reduction operations, starting with combining select-entries and select-xtdf operations when they apply the same pattern across multiple trains. With some help from Gemini, it became fairly clever in extracting any arithmetic patterns from the set of trains.

@esobolev

Merge request reports

Loading