Issue Bug Closed

#109309 FileIndexRepository::findInStorageWithIndexOutstanding() includes missing=1 records, blocking metadata extraction of valid files

Sebastian Michaelsen 03/21/26, 06:53 updated 03/31/26, 08:09

Description

FileIndexRepository::findInStorageWithIndexOutstanding() is used by the file indexer’s second step (metadata extraction via FileStorageExtractionTask). The query fetches all sys_file records where tstamp > last_indexed for a given storage, but it does not filter out records marked missing=1.

Orphaned/deleted file records that have been flagged as missing still consume slots in the configurable batch limit. In installations with a significant number of orphan records, valid but unindexed files are pushed beyond the batch window and their metadata is never extracted until the orphan records are manually cleaned up.

Current query (FileIndexRepository.php):

  ->where(
      $queryBuilder->expr()->gt('tstamp', $queryBuilder->quoteIdentifier('last_indexed')),
      $queryBuilder->expr()->eq('storage', $queryBuilder->createNamedParameter($storage->getUid(), Connection::PARAM_INT))
  )

No missing = 0 condition is present.

Steps to reproduce:

1. Have a file storage with several hundred sys_file records marked missing=1 (e.g. after bulk-deleting files from disk without going through the FAL API).
2. Add new files to the same storage that have not yet been indexed (i.e. tstamp > last_indexed).
3. Run the Scheduler task FileStorageExtractionTask with a batch limit smaller than the number of missing records.
4. Observe: the new files’ metadata is never extracted because the batch is filled entirely by missing=1 records, which are skipped individually (exception caught) but still count against the limit.

Expected behaviour:

findInStorageWithIndexOutstanding() should only return records where missing = 0. Records already marked missing are handled by the first indexer step (detectMissingFiles) and have no useful metadata to extract.

Proposed fix:

Add $queryBuilder->expr()->eq('missing', $queryBuilder->createNamedParameter(0, Connection::PARAM_INT)) to the where() clause.

Reviews 2