Issue Bug Closed

#63615Huge memory consumption in DataHandler->processClearCacheQueue while bulk inserting

Stephan Großberndt 12/05/14, 19:28 updated 10/02/18, 11:48

Description

I found a memory consumption problem in
TYPO3\CMS\Core\DataHandling\DataHandler::processClearCacheQueue()
due to this dubious construct, which is called for each and every record added:
foreach (static::$recordsToClearCacheFor as $table => $uids) {
  foreach (array_unique($uids) as $uid) {
    ...

    foreach ($pageIdsThatNeedCacheFlush as $pageId) {
      if ($pageId >= 0) {
        $tagsToClear[] = 'pageId_' . $pageId;
      }
    }
    // Queue delete cache for current table and record
    $tagsToClear[] = $table;
    $tagsToClear[] = $table . '_' . $uid;
    $tagsToClear = array_unique($tagsToClear);
  }
}

I am adding 6500 records, so FOR EACH AND EVERY record ‘pageId_’ . $pageId, $table and $table . ‘‘ . $uid are added to $tagsToClear and afterwards ‘pageId’ . $pageId and $table are removed again for 6499 times. There is no reason to do it like this. array_unique() is meant here to reduce memory consumption in $tagsToClear but for big arrays with array_unique() this is terrible – xhprof showed 2,020,470,496 bytes (~2 gigabyte !!!) of used memory for array_unique() in processClearCacheQueue()

It is possible to set
clearCache_disable=1
to disable clearing of the cache but there is also a simple code solution to solve this since entries in this array are meant to be unique anyway:
foreach (static::$recordsToClearCacheFor as $table => $uids) {
  foreach (array_unique($uids) as $uid) {
    ...

    foreach ($pageIdsThatNeedCacheFlush as $pageId) {
      if ($pageId >= 0) {
        $tagsToClear['pageId_' . $pageId] = 'pageId_' . $pageId;
      }
    }
    // Queue delete cache for current table and record
    $tagsToClear[$table] = $table;
    $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
  }
}

array_unique memory consumption drops from

IMemUse%: 13064.1%
Incl. MemUse (bytes): 2,020,470,496 bytes (~2 gigabyte !!!)

to

IMemUse%: 4.1%
Incl. MemUse (bytes): 629,464 bytes

There may be optimizations possible like

foreach (static::$recordsToClearCacheFor as $table => $uids) {
  foreach (array_unique($uids) as $uid) {
    ...

    foreach ($pageIdsThatNeedCacheFlush as $pageId) {
      $pageKey = 'pageId_' . $pageId;
      if ($pageId >= 0 && !isset($tagsToClear[$pageKey])) {
        $tagsToClear[$pageKey] = $pageKey;
      }
    }
    // Queue delete cache for current table and record
    if (!isset($tagsToClear[$table])) {
      $tagsToClear[$table] = $table;
    }
    $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
  }
}

but even without those its SOOOO much better…

Regards,
Stephan

Reviews 3