#63615Huge memory consumption in DataHandler->processClearCacheQueue while bulk inserting
Description
I found a memory consumption problem in
TYPO3\CMS\Core\DataHandling\DataHandler::processClearCacheQueue()due to this dubious construct, which is called for each and every record added:
foreach (static::$recordsToClearCacheFor as $table => $uids) {
foreach (array_unique($uids) as $uid) {
...
foreach ($pageIdsThatNeedCacheFlush as $pageId) {
if ($pageId >= 0) {
$tagsToClear[] = 'pageId_' . $pageId;
}
}
// Queue delete cache for current table and record
$tagsToClear[] = $table;
$tagsToClear[] = $table . '_' . $uid;
$tagsToClear = array_unique($tagsToClear);
}
}
I am adding 6500 records, so FOR EACH AND EVERY record ‘pageId_’ . $pageId, $table and $table . ‘‘ . $uid are added to $tagsToClear and afterwards ‘pageId’ . $pageId and $table are removed again for 6499 times. There is no reason to do it like this. array_unique() is meant here to reduce memory consumption in $tagsToClear but for big arrays with array_unique() this is terrible – xhprof showed 2,020,470,496 bytes (~2 gigabyte !!!) of used memory for array_unique() in processClearCacheQueue()
It is possible to setclearCache_disable=1to disable clearing of the cache but there is also a simple code solution to solve this since entries in this array are meant to be unique anyway:
foreach (static::$recordsToClearCacheFor as $table => $uids) {
foreach (array_unique($uids) as $uid) {
...
foreach ($pageIdsThatNeedCacheFlush as $pageId) {
if ($pageId >= 0) {
$tagsToClear['pageId_' . $pageId] = 'pageId_' . $pageId;
}
}
// Queue delete cache for current table and record
$tagsToClear[$table] = $table;
$tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
}
}
array_unique memory consumption drops from
IMemUse%: 13064.1% Incl. MemUse (bytes): 2,020,470,496 bytes (~2 gigabyte !!!)
to
IMemUse%: 4.1% Incl. MemUse (bytes): 629,464 bytes
There may be optimizations possible like
foreach (static::$recordsToClearCacheFor as $table => $uids) {
foreach (array_unique($uids) as $uid) {
...
foreach ($pageIdsThatNeedCacheFlush as $pageId) {
$pageKey = 'pageId_' . $pageId;
if ($pageId >= 0 && !isset($tagsToClear[$pageKey])) {
$tagsToClear[$pageKey] = $pageKey;
}
}
// Queue delete cache for current table and record
if (!isset($tagsToClear[$table])) {
$tagsToClear[$table] = $table;
}
$tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
}
}
but even without those its SOOOO much better…
Regards,
Stephan