86091[FEATURE] Introduce dashboard widget for duplicate files
Commit message
Test this change
Run a command in your clone of the TYPO3 Core. Then try what the change does and vote on Verified in Gerrit. How to review a patch
- Check out patch set 29
-
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/91/86091/29 && git checkout FETCH_HEAD - Cherry-pick onto main
-
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/91/86091/29 && git cherry-pick FETCH_HEAD
Review
-
Code review CR 0 Needs a vote
-
Verified V -1 Blocked
- core-ci -1 CI
Merge conflicts 5
- main 91900[TASK] Display 3-line dashboard widget without scrollbar
- main 91358[FEATURE] Introduce "Activity Stream" dasboard widget
- main 92326[FEATURE] Introduce dashboard widget for largest unused files
- main 92677[FEATURE] Make dashboard widgets resizable
- main 92232[TASK] Refine border radius mechanics for UI
Issues 1
Files 11
- Build/Sources/Sass/dashboard/_widget_table.scss1+6 −0
- typo3/sysext/core/Documentation/Changelog/14.2/Feature-104966-IntroduceDashboardWidgetForDuplicateFiles.rstnew+74 −0
- typo3/sysext/dashboard/Classes/Widgets/DuplicateFilesWidget.php33new+180 −0
- typo3/sysext/dashboard/Configuration/Services.yaml2+16 −1
- typo3/sysext/dashboard/Resources/Private/Language/locallang.xlf4+18 −0
- typo3/sysext/dashboard/Resources/Private/Templates/Widget/DuplicateFilesWidget.html3new+95 −0
- typo3/sysext/dashboard/Resources/Public/Css/dashboard.css+1 −0
- typo3/sysext/dashboard/Tests/Functional/Fixtures/be_users.csv+7 −6
- typo3/sysext/dashboard/Tests/Functional/Fixtures/sys_file.csvnew+15 −0
- typo3/sysext/dashboard/Tests/Functional/Fixtures/sys_filemounts.csvnew+3 −0
- typo3/sysext/dashboard/Tests/Functional/DuplicateFilesWidgetTest.php4new+89 −0
Review log 6595
Show 4970 earlier messages
-
Ulrich Mathes uploaded patch set 1 09/13/24, 22:04
-
core-ci voted on patch set 1 V +1 09/13/24, 22:11 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65713
-
Ulrich Mathes commented on patch set 1 09/13/24, 22:19
DuplicateFilesWidget.php:103This check is necessary to prevent errors while generating thumbnails for missing files. Maybe we could handle this in another way?DuplicateFilesWidget.php:111AFAIK this information is not available in fluid.locallang.xlf:246This could be misleading as we also show files which have been uploaded multiple times. But i can't think of another short term for this. -
Ulrich Mathes uploaded patch set 2
V +109/13/24, 22:29 -
Ulrich Mathes commented on patch set 2 09/13/24, 22:30
DuplicateFilesWidget.php:111ReplyIt is, `{duplicate.file.image}`. Applied in patch set 2 -
core-ci voted on patch set 2 V +1 09/13/24, 22:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65714
-
Stefan Bürk commented on patch set 2 CR -1 09/15/24, 00:26
DuplicateFilesWidget.php:74This whole query build up needs to be refactored/rethinked. On top of that, it **must** be moved to a dedicated service and at least the service coverd with functional test to verify it works at least to some grade for all TYPO3 supported databases as fare the test setup allowes for it.DuplicateFilesWidget.php:75GROUP_CONCAT has evil dependencies and size limits - that can be configured, but if you do don't have access to the server you cannot do it on global manner. THat means it needs to be increased here. And even done so, the list **may** exceed that limit, which leads to a "bad" behaviour: If the group_concat size limit is hit, it silenctly cuts at that point - even within a uid which leads to another UID for the last element which gets cutted, for example: ``` '123, 456, ......., 1002302932' '123, 456, ......., 10023' ``` Beside that, literals are used here. Missing identifier quoting. should be at least done using sprintf: ``` ->selectLiteral( sprintf('GROUP_CONCAT(%s) AS uidlist', $queryBuilder->quoteIdentifier('uid')), sprintf('COUNT(*) as %s', $queryBuilder->quoteIdentifier('counting'), ) ``` The group by field should be contained as "first" field in the select field list.DuplicateFilesWidget.php:78No literlas here - use QueryBuilder/ExpressionBuilder API along with createNamedParameters or quoting methods. End for the like the correspinding escpate method needs to be used. Beside that. using the `where()` method and just provde the lines makes this more readable than the chain of andWheres for each statement.DuplicateFilesWidget.php:83Using groupBy with a field not contained in the `select field` is not supported for all database systems, and on some depends on the exact server configuration.DuplicateFilesWidget.php:87Why only the first column ? would it not make sense to retrieve the count also to display how many occurances happend ?DuplicateFilesWidget.php:104throwable variable unused, can be left out.DuplicateFilesWidget.php:1133rd argument should be set to true to remove empty values.DuplicateFilesWidget.php:122Bad that the request does not contain the backend user authentication object. for simply isAdmin checks it may be usable) but not for the extended access checks. Not an issue of this change. -
Frank Nägler commented on patch set 2 CR -1 09/15/24, 09:42
on itDuplicateFilesWidget.php:75ReplyAcknowledgedDuplicateFilesWidget.php:78ReplyAcknowledgedDuplicateFilesWidget.php:83ReplyAcknowledgedDuplicateFilesWidget.php:87Reply> Why only the first column ? would it not make sense to retrieve the count also to display how many occurances happend ? IMHO not relevant for the purpose of this widget.DuplicateFilesWidget.php:103ReplyI will push an update with a $file->isMissing() condition in the previous condition, this should be enough, please correct me if I am wrong.DuplicateFilesWidget.php:104ReplyAcknowledgedDuplicateFilesWidget.php:113ReplyAcknowledged -
Frank Nägler commented on patch set 2 09/15/24, 09:43
locallang.xlf:246ReplyDone -
Frank Nägler uploaded patch set 3
CR -1V +109/15/24, 10:04 -
Frank Nägler commented on patch set 3 09/15/24, 10:04
DuplicateFilesWidget.php:72I will remove this, default restriction should work here and we don't have the risk to expose files without access. -
Frank Nägler commented on patch set 3 09/15/24, 10:05
DuplicateFilesWidget.php:103ReplyDone -
core-ci voted on patch set 3 V -1 09/15/24, 10:11 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65804
-
Frank Nägler uploaded patch set 4
V -109/15/24, 10:54 -
core-ci voted on patch set 4 V -1 09/15/24, 10:59 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65807
-
Frank Nägler uploaded patch set 5
V -109/16/24, 09:03 -
core-ci voted on patch set 5 V +1 09/16/24, 09:10 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65862
-
Ulrich Mathes uploaded patch set 6
V +109/16/24, 11:35 -
core-ci voted on patch set 6 V +1 09/16/24, 11:41 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65882
-
Ulrich Mathes uploaded patch set 7
V +109/16/24, 12:33 -
core-ci voted on patch set 7 V -1 09/16/24, 12:43 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65883
-
Ulrich Mathes uploaded patch set 8
V -109/16/24, 13:41 -
Ulrich Mathes commented on patch set 8 09/16/24, 13:46
DuplicateFilesWidget.php:74ReplyI'm not sure to what extent the widgets should be self-contained. I fear that splitting them up could lead to a confusing proliferation of classes. I added a test for database access - which failed on Postgres. I had to remove `GROUP_CONCAT`. So with the new approach there are no more literal selects. The disadvantage is that I have to do a separate database query for each sha1 that is found more than 1 time.DuplicateFilesWidget.php:87ReplyThe number of occurrences may be reduced later by user permissions.DuplicateFilesWidget.php:103ReplyIt is possible that the file does not yet know that it is missing. I reinserted the `$file->getParentFolder()` function call. When I took a closer look, I realised that this function checks the backend user access for this folder. So we don't need to check the access ourselves. -
Ulrich Mathes commented on patch set 8 09/16/24, 13:47
DuplicateFilesWidget.php:87ReplyDone -
core-ci voted on patch set 8 V -1 09/16/24, 13:51 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65892
-
Ulrich Mathes uploaded patch set 9 09/16/24, 17:13
Copied Votes: * Verified-1 (copy condition: "**changekind:NO_CHANGE**") -
core-ci voted on patch set 9 V -1 09/16/24, 17:29 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65915
-
core-ci voted on patch set 9 V +1 09/16/24, 18:58 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65915
-
Ulrich Mathes commented on patch set 9 09/16/24, 19:15
DuplicateFilesWidget.php:74ReplyDone -
Philipp Kuhlmay commented on patch set 9 10/04/24, 10:13
locallang.xlf:249I would change this to "Displays files that exists multiple times". A file does not need to be uploaded before, when it exists in the fileadmin, it could also be "copied" from another folder. So this duplicate file is not "uploaded", but "copied".DuplicateFilesWidget.html:21Currently, all images are shown twice. When a file is shown in this widget, I already assume that these images exist multiple times inside my storage, and therefore I only want to see the image once, not twice or more. Maybe you could only show the image itself one time and only show the separate information for the other ones like path and count of references. Example: When I have an image 10 times in my storage, I would see the exact image 10 times in the list. But I only want to see it once. -
Ulrich Mathes commented on patch set 9 10/08/24, 06:53
locallang.xlf:249ReplyAcknowledgedDuplicateFilesWidget.html:21ReplyDisplaying the images multiple times was an active decision. The user **immediately** realises that an image is available multiple times. It also makes it clear that it really is the same image. -
Philipp Kuhlmay commented on patch set 9 10/08/24, 06:55
DuplicateFilesWidget.html:21ReplyOkay when it is an active decision, I am okay with it. I just wanted to mention it. -
Ulrich Mathes uploaded patch set 10
V +110/08/24, 07:02 -
core-ci voted on patch set 10 V +1 10/08/24, 07:09 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67320
-
Fabian Cloïn commented on patch set 10 05/20/25, 11:12
DuplicateFilesWidget.php:54Just out of curiosity. My first idea would be to put this logic in a repository. I see that there is already a repository that does something with sys_file https://github.com/TYPO3/typo3/blob/main/typo3/sysext/core/Classes/Resource/FileRepository.php. It also would be more aligned with the other widget classes in this directory. (Although this shouldn't be the solely reason to refactor.) Maybe in this scenario it would be okay to leave it in this class i don't know. Im curious of what you all think about it. -
Oli Bartsch uploaded patch set 11
V +105/23/25, 09:02 -
core-ci voted on patch set 11 V +1 05/23/25, 09:08 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/79415
-
Ulrich Mathes uploaded patch set 12
V +105/27/25, 09:44 -
Ulrich Mathes commented on patch set 12 05/27/25, 09:49
DuplicateFilesWidget.php:54ReplyThis is a question in the direction of core development. I suspect we will make as few adjustments to the core as possible.DuplicateFilesWidget.php:86This can be done within one database query. Thanks @schneider.maik@me.com -
core-ci voted on patch set 12 V +1 05/27/25, 09:51 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/79548
-
Ulrich Mathes uploaded patch set 13
V +110/22/25, 12:48Show icon and file size -
Ulrich Mathes commented on patch set 13 10/22/25, 12:50
DuplicateFilesWidget.php:54ReplyDone -
core-ci voted on patch set 13 V +1 10/22/25, 12:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/87259
-
Ulrich Mathes uploaded patch set 14
V +101/07/26, 10:26solve merge conflict -
core-ci voted on patch set 14 V +1 01/07/26, 10:34 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/93047
-
Josef Glatz commented on patch set 14 01/07/26, 15:29
I really like the feature. Would it makes sense to move the logic for finding duplicates in service? - e.g. adding the duplicate files functionality as API feature to TYPO3 (not in ext:dashboard but in ext:core) - e.g. to make dependency of ext:dashboard not mandatory if someone wants to use the feature in an own extension I mean, why not adding the core functionality as feature for other places too? -
Ulrich Mathes commented on patch set 14 01/08/26, 06:45
Reply
I might think that the core doesn't want to design services for reusable logic without a concrete idea for which they might be used. Do you have any ideas about where such a service could be beneficial? By encapsulating all the logic within the widgets themselves, overhead is not introduced in other places. However, I am not in a position to make this decision. -
Josef Glatz commented on patch set 14 01/08/26, 08:46
Reply
My advice follows the note from Stefan Bürk in one of his first notes on this patch -
Ulrich Mathes commented on patch set 14 01/08/26, 10:22
Reply
His note was related to the previous patchset which had used `selectLiteral` to fetch the duplicates. Since the widget now relies completely on normal QueryBuilder selects, he resolved this issue. -
Simon Schaufelberger uploaded patch set 15
V +101/13/26, 01:19 -
Simon Schaufelberger voted on patch set 15 CR +1 01/13/26, 01:20
-
Simon Schaufelberger voted on patch set 15 V +1 01/13/26, 01:21
-
core-ci voted on patch set 15 V +1 01/13/26, 01:27 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/93298
-
Ulrich Mathes uploaded patch set 16
CR +1V +101/21/26, 15:52 -
core-ci voted on patch set 16 V +1 01/21/26, 16:01 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/93962
-
Ulrich Mathes uploaded patch set 17
V +101/21/26, 16:04 -
core-ci voted on patch set 17 V +1 01/21/26, 16:16 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/93966
-
Simon Schaufelberger commented on patch set 17 01/21/26, 18:52
Services.yaml:236i think the indentation is too deep here -
Stefan Bürk commented on patch set 17 01/21/26, 23:28
DuplicateFilesWidget.php:72I'd say / guess that we can combine `getDuplicatedSha1()` and `getFileUuidsFromSha1()` into a single datbase query / method, reducing database hoops. Either with a good table join and splitting the result into the nested array struc like `getFileUuidsFromSha1()` returns or with a `common table expression query (CTE)` Would reduce the dispatched query roundtrups from `201` to `1` using default `$this->options['dupliateLimit'] === 200`. `getDuplicates()` is heavy enough with additional queries, but using FAL API to retrieve the files is okay I guess. However, not sure if we really need it. Should we really re-evaluate FAL/File status here ? -
Ulrich Mathes commented on patch set 17 01/22/26, 10:49
DuplicateFilesWidget.php:72ReplyAs I understand it, we currently only dispatch two queries, as the `getFileUidsFromSha1()` function uses IN with an array of SHA-1 strings. So the overhead is not that significant.However, I think we should move to a CTE, as `sys_file` can be very large.`getDuplicates()` is resource-intensive, but perhaps we could eliminate some checks? We need to retrieve the FileObject for Fluid.We can remove the call to `isMissing()` as we already select only files that are not missing, and `isMissing()` does not actually check if the file is present (as `exists()` already has done).The user permission check (via `InsufficientFolderAccessPermissionsException`) could be carried out much earlier using `FolderMountsRestriction`. This might reduce the amount of SHA1 drastically. This late permission check may return a smaller resultset than the `duplicateLimit`, even if there are more duplicates present.Thanks for your in-depth analysis; I will address these points. -
Ulrich Mathes uploaded patch set 18
V +101/22/26, 11:25Use CTE and FolderMountRestriction for BEuser -
Ulrich Mathes commented on patch set 18 01/22/26, 11:26
DuplicateFilesWidget.php:72ReplyAcknowledgedServices.yaml:236ReplyDone -
core-ci voted on patch set 18 V +1 01/22/26, 11:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/94030
-
Stefan Bürk commented on patch set 18 01/22/26, 16:36
DuplicateFilesWidget.php:72Reply> As I understand it, we currently only dispatch two queries Correct. Was to late looking on it. But still, we have one query retrieving a result set which needs to be added as "payload" to a second query. Having it combined would remove that roundtrip ;) -
Guido Schmechel commented on patch set 18 01/25/26, 09:46
I tested the whole thing. It worked well for my simple example with two duplicates. What I would expect: - The limit number should be controllable in the widget. As with ‘TYPO3 security advisories’. What I would give as feedback: - I would have expected the service to be in the core so that it could be used in several places. Example: Option to report duplicates by email as an overview. - I would have expected the function to be a submodule in Media rather than in a dashboard. I realise that this is a bigger issue. But perhaps it could be discussed as a glimpse into the future. However, it should not block the feature for v14. Just bringing in a different perspective :-) -
Ulrich Mathes uploaded patch set 19
V +101/26/26, 08:02Add limit configuration -
core-ci voted on patch set 19 V -1 01/26/26, 08:10 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/94342
-
Ulrich Mathes commented on patch set 19 01/26/26, 08:15
Reply
I gave the widget a limit configuration. I kept the hard limit to be controlled by administrators, since the number of duplicates could be surprisingly high. The bigger picture in terms of using this data is, as you mention, a bigger issue. I could think of a feature where the duplicate check is performed directly while uploading a new image. There are many possibilities and places to show this data. From my perspective, we could start by enabling the editors to view these duplicates in the hope that they will want to keep their data clean. -
Ulrich Mathes uploaded patch set 20
V -101/26/26, 08:19 -
core-ci voted on patch set 20 V +1 01/26/26, 08:28 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/94345
-
Guido Schmechel commented on patch set 20 01/26/26, 10:43
Thanks for the update. I was able to successfully test the limit. I have two more ideas in mind: 1.) Does it make sense for us to establish a ‘Media’ group instead of registering it in the content? 2.) Even though it's a list, should we perhaps keep the icon synchronised as in the other patch? Related patch: https://review.typo3.org/c/Packages/TYPO3.CMS/+/92326 -
Ulrich Mathes rebased onto patch set 21
V +101/30/26, 14:24Fetch thumbnail size from tsconfig -
Ulrich Mathes uploaded patch set 22 01/30/26, 14:26
Fetch thumbnail size from tsconfig -
Ulrich Mathes uploaded patch set 23 01/30/26, 14:29
-
core-ci voted on patch set 23 V +1 01/30/26, 14:46 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/94675
-
core-ci voted on patch set 21 V +1 01/30/26, 14:59 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/94673
-
core-ci voted on patch set 22 V +1 01/30/26, 15:02 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/94674
-
Ulrich Mathes commented on patch set 23 02/02/26, 06:18
Reply
Done -
Garvin Hicking commented on patch set 23 02/02/26, 07:21
DuplicateFilesWidgetTest.php:48Maybe add one more scenario for a non-admin who sees one file in a mount? Also, please also include a _recycler_ record in there as a dupe thats filtered to cover all query parts. So ones with missing=1 and atorage=0 would be helpful. (More a not forcmyself: What I have to check out, how do the "file exists" checks succeed, where are the files added to the test setup?) -
Ulrich Mathes uploaded patch set 24
V +103/18/26, 08:09Resolve conflicts -
core-ci voted on patch set 24 V +1 03/18/26, 08:19 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/98142
-
Ulrich Mathes uploaded patch set 25
V +103/18/26, 13:10Use domain based translation syntax -
Ulrich Mathes uploaded patch set 26 03/18/26, 13:20
Clarify widget options and settings -
core-ci voted on patch set 25 V +1 03/18/26, 13:20 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/98188
-
Ulrich Mathes uploaded patch set 27 03/18/26, 13:22
-
Ulrich Mathes commented on patch set 27 03/18/26, 13:28
DuplicateFilesWidgetTest.php:48ReplyAcknowledged -
Ulrich Mathes commented on patch set 27 03/18/26, 13:29
DuplicateFilesWidgetTest.php:48Replysorry, not resolved yet. -
core-ci voted on patch set 27 V -1 03/18/26, 13:37 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/98191
-
core-ci voted on patch set 26 V +1 03/18/26, 13:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/98190
-
Ulrich Mathes rebased onto patch set 28
V -103/30/26, 10:20test every predicate and add missing FolderMountsRestriction -
Ulrich Mathes commented on patch set 28 03/30/26, 10:23
Adding more fixtures to the tests to test against every predicate has showed that the FolderMountsRestriction was missing in the CTE. Thanks for pushing me to write more tests.DuplicateFilesWidgetTest.php:48ReplyDone -
core-ci voted on patch set 28 V -1 03/30/26, 10:30 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/99035
-
Ulrich Mathes uploaded patch set 29
V -103/30/26, 10:34missing changes -
core-ci voted on patch set 29 V -1 03/30/26, 10:44 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/99038
-
Josef Glatz wrote 06/08/26, 03:43
Removed cc Josef Glatz.
_widget_table.scss:32