The Indexer builds a large array of all actual files on a storage
(identifiedFileUids). If many files exists, this array can get
very large.
This array was then passed to a QueryBuilder to fetch all
records NOT IN that array. Since a NOT IN query is passed as a
string to the database, it can exceed the string size allowed
in a query, making the whole task fail.
Since a NOT IN query cannot be chunked easily, the whole
logic has been adapted in a different way.
Instead of fetching a restricted list of database records,
all records are fetched and iterated. Even with a million
of sys_file_records of a single (!) storage this will
perform alright, and be within practical usage scenarios.
Each database record is then checked for a match in the
large array of known records, and then execute the
same logic as before.
To benchmark the implications, the following test was run:
Baseline:
* sys_file with 50.736 entries
* 16.912 marked as missing
* 33.824 marked as existing
* Filesystem with 8.771 actual files
Tested setup via a script which:
* Resets to baseline sys_file storage
* Executes scheduler task "File Abstraction Layer: Update storage index
(scheduler)"
* Flags 41.965 files as missing, 8.771 as found.
Script execution was performed 50 times, and a mean avery was calculated,
once once with the patch in place, once without.
Old variant (using NOT IN query): 11.787 seconds
New variant (fetching all records): 12.0544 seconds
On top of being within the same performance level, using the new method,
no database exception will be provoked (see ticket).