Work queue file loader – Add possibility to aggregate compressed files

tvavrda · July 20, 2022, 9:28pm

Current work queue file loader supports the following scenarios:

reading each file in the folder as a single work queue entry
aggregating all the currently present files in the folder as a single work queue entry
splitting files by rows

We need a fourth option: Aggregating each compressed (ZIP) file as a single work queue entry, which would contain the files inside the archive.

tvavrda · July 20, 2022, 9:28pm

github.com/origam/origam

CHG: Introduced a possibility to aggregate content of single ZIP file…

origam:master ← origam:file-loader

opened 09:24PM - 20 Jul 22 UTC

tvavrda

+164 -118

…s instead of all the files in the directory. Connection string of work queue… file loader has been refactored and instead of "aggregate" boolean value it now uses a single "readType" property that can have the following values: - SingleFiles: Default, loads each file separately - AggregateCompressedFiles: Loads each compressed file separately and aggregates all files inside it - AggregateAllFiles: Loads all the files available as a single queue entry, including all the files inside compressed files - SplitByRows: Splits the files by each nth row specified using the "splitFileByRows" and "keepHeader" parameters