Issue Bug Closed

#86018FolderTreeView::getBrowseableTreeForStorage() will timeout on network file systems

Dmitry Dulepov 08/29/18, 12:34 updated 03/25/25, 13:41

Description

FolderTreeView::getBrowseableTreeForStorage() contains this code:

            // If the mount is expanded, go down:
            if ($isOpen && $storageObject->isBrowsable()) {
                // Set depth:
                $this->getFolderTree($rootLevelFolder, 999);
            }

While it could be ok to browse 999 levels on the local file system, it fails on network file systems with AWS or WebDav due to much longer time needed to fetch all those levels. If I lift php timeout restrictions, it took about 6 hours over high speed connection to fetch folder tree from the NextCloud with a single first level subfolder expanded. This is absolutely not acceptable. Folder tree fetches EVERY possible subfolder of each expanded first level folder in the tree. It fetches deep inside, even those subfolders that the user never opened. This takes huge time over network.

In general, FAL is not very usable with network storages.

Reviews 0

No change resolves or names this issue in its commit message, and no comment links one.

History 11

  1. Benni Mack commented <span data-processor="relativetime" data-value="2018-08-29T12:39:59+00:00" > 08/29/18, 12:39 </span>

    Hey Dmitry,

    nice to read you here.

    I agree with you. what would be the best solution to “mitigate”? seems like this is a deeper issue which will be hard to fix in existing LTS versions, and come up with a better folder tree separately?

    All the best,
    Benni.

  2. Dmitry Dulepov commented <span data-processor="relativetime" data-value="2018-08-29T12:55:55+00:00" > 08/29/18, 12:55 </span>

    I could not find a quick solution for this :(

    Ideally, folder tree should only load folders that the user expanded. Now it loads everything. Even more: when I expand another folder, that function is called again and it will fetch again all those expanded folders starting from the root. This means that the more folders you expand, the more it fetches over network and the slower the tree becomes. Important thing is that it always fetches all expanded folders from the root down to their last node.

    In the end user just hits the php timeout. Also he cannot insert images to content elements because image browser also uses the same tree component. The only cure for this is to clear all user settings to reset file tree’s expanded node list. Even more, users continue clicking on the file module to see the tree. They spawn more and more php processes and the server quickly becomes overloaded and network becomes saturated with requests. This is an unexpected DoS made by editors in attempt to see their files. The problem is very serious with network FAL drivers.

  3. Benni Mack commented <span data-processor="relativetime" data-value="2021-02-02T12:58:28+00:00" > 02/02/21, 12:58 </span>

    Hey Dmitry,

    I am trying to reproduce the issue. Can you tell me:

    • What remote driver you’re using (TER extension / custom) e.g. S3
    • How many files / data is in the remote storage?
  4. Dmitry Dulepov commented <span data-processor="relativetime" data-value="2021-02-02T13:15:52+00:00" > 02/02/21, 13:15 </span>

    It is webdav. We took the one from TER. Connection is to OwnCloud. There are about 5-10 subdirectories on each level, up to 10 levels deep, about 10-100 files on each level, mostly pdfs. Due to the way FAL works, it fetches the same data multiple times, so request to fetch the content of the same (!) directory comes more than once (I think 2 or 3 times in a row). This means a new http request each time. Same for getting each file’s properties (size, etc). So it gets the directory and then gets information for each file. So 2*(1 dir + 100 files) = 202 http requests just for one directory. And so on for each subdirectory. This happens for every directory, every file, and every level even when the tree is not expanded (and probably will not be expanded at all).

    If you do not have a remote OwnCloud, possibly you can reproduce it if you mount a directory with many files (like a TYPO3 core) using sshfs somewhere inside fileadmin/. But make sure that the server is remote on the Internet.

  5. Benni Mack commented and changed status Accepted Needs Feedback <span data-processor="relativetime" data-value="2022-01-07T13:24:15+00:00" > 01/07/22, 13:24 </span>

    Hi Dmitry,

    thanks for your report.

    We have now built the file storage tree based on AJAX (like the pagetree) with multiple levels. Does this problem still occur in v11?

  6. Dmitry Dulepov commented <span data-processor="relativetime" data-value="2022-01-07T13:29:48+00:00" > 01/07/22, 13:29 </span>

    Hi Benni!

    I will not be able to test it. We decided not to use webdav directly but instead make a local copy of necessary files. It has its disadvantages but it works for us.

    Sorry :(

  7. Benni Mack commented <span data-processor="relativetime" data-value="2022-01-10T12:06:58+00:00" > 01/10/22, 12:06 </span>

    Dmitry Dulepov wrote in #note-8:
    > Hi Benni!
    >
    > I will not be able to test it. We decided not to use webdav directly but instead make a local copy of necessary files. It has its disadvantages but it works for us.
    >
    > Sorry :(

    Hi Dmitry,

    Thanks for the quick reply. Guess we need somebody else to validate this issue then

    All the best,
    Benni.

  8. Benni Mack commented and changed status New Closed <span data-processor="relativetime" data-value="2025-03-25T13:41:59+00:00" > 03/25/25, 13:41 </span>

    Hi Dmitry,

    I will now close this issue. I hope this is OK with you, Dmitry, otherwise feel free to reach out to me.

    The mentioned class was removed in v12/v13. The conceptual issues regarding remote drivers in FAL is a separate topic, if you have some ideas and examples, let me know, (I also saw a POC for Flysystem implementation for FAL), but we should have a different / new issue for this.

    All the best,
    Benni.