Review main Ready for review Stale

86091[FEATURE] Introduce dashboard widget for duplicate files

Ulrich Mathes CR 0 V -1 Hard +504 −7 updated 06/08/26, 03:43

Commit message

This patch adds a new widget for EXT:dashboard. The widget
fetches a list of files that are identical in terms of
their SHA1 hash.

Test this change

Run a command in your clone of the TYPO3 Core. Then try what the change does and vote on Verified in Gerrit. How to review a patch

Check out patch set 29
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/91/86091/29 && git checkout FETCH_HEAD
Cherry-pick onto main
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/91/86091/29 && git cherry-pick FETCH_HEAD

Review

  • Code review CR 0 Needs a vote
  • Verified V -1 Blocked
    • core-ci -1 CI

Merge conflicts 5

Issues 1

Files 11

Review log 6595

Show 4970 earlier messages
  1. Ulrich Mathes uploaded patch set 1 09/13/24, 22:04

  2. core-ci voted on patch set 1 V +1 09/13/24, 22:11 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65713

  3. Ulrich Mathes commented on patch set 1 09/13/24, 22:19

    _widget_table.scss:32

    I wasn't happy about adding some styles but i wanted to separate the different groups of duplicate files. Another solution would be a table within the widget-table.

    DuplicateFilesWidget.php:103

    This check is necessary to prevent errors while generating thumbnails for missing files. Maybe we could handle this in another way?

    DuplicateFilesWidget.php:111

    AFAIK this information is not available in fluid.

    locallang.xlf:246

    This could be misleading as we also show files which have been uploaded multiple times. But i can't think of another short term for this.
  4. Ulrich Mathes uploaded patch set 2 V +1 09/13/24, 22:29

  5. Ulrich Mathes commented on patch set 2 09/13/24, 22:30

    DuplicateFilesWidget.php:111 Reply

    It is, `{duplicate.file.image}`. Applied in patch set 2
  6. core-ci voted on patch set 2 V +1 09/13/24, 22:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65714

  7. Stefan Bürk commented on patch set 2 CR -1 09/15/24, 00:26

    DuplicateFilesWidget.php:74

    This whole query build up needs to be refactored/rethinked. On top of that, it **must** be moved to a dedicated service and at least the service coverd with functional test to verify it works at least to some grade for all TYPO3 supported databases as fare the test setup allowes for it.

    DuplicateFilesWidget.php:75

    GROUP_CONCAT has evil dependencies and size limits - that can be configured, but if you do don't have access to the server you cannot do it on global manner. THat means it needs to be increased here. And even done so, the list **may** exceed that limit, which leads to a "bad" behaviour: If the group_concat size limit is hit, it silenctly cuts at that point - even within a uid which leads to another UID for the last element which gets cutted, for example: ``` '123, 456, ......., 1002302932' '123, 456, ......., 10023' ``` Beside that, literals are used here. Missing identifier quoting. should be at least done using sprintf: ``` ->selectLiteral( sprintf('GROUP_CONCAT(%s) AS uidlist', $queryBuilder->quoteIdentifier('uid')), sprintf('COUNT(*) as %s', $queryBuilder->quoteIdentifier('counting'), ) ``` The group by field should be contained as "first" field in the select field list.

    DuplicateFilesWidget.php:78

    No literlas here - use QueryBuilder/ExpressionBuilder API along with createNamedParameters or quoting methods. End for the like the correspinding escpate method needs to be used. Beside that. using the `where()` method and just provde the lines makes this more readable than the chain of andWheres for each statement.

    DuplicateFilesWidget.php:83

    Using groupBy with a field not contained in the `select field` is not supported for all database systems, and on some depends on the exact server configuration.

    DuplicateFilesWidget.php:87

    Why only the first column ? would it not make sense to retrieve the count also to display how many occurances happend ?

    DuplicateFilesWidget.php:104

    throwable variable unused, can be left out.

    DuplicateFilesWidget.php:113

    3rd argument should be set to true to remove empty values.

    DuplicateFilesWidget.php:122

    Bad that the request does not contain the backend user authentication object. for simply isAdmin checks it may be usable) but not for the extended access checks. Not an issue of this change.
  8. Frank Nägler commented on patch set 2 CR -1 09/15/24, 09:42

    on it

    DuplicateFilesWidget.php:75 Reply

    Acknowledged

    DuplicateFilesWidget.php:78 Reply

    Acknowledged

    DuplicateFilesWidget.php:83 Reply

    Acknowledged

    DuplicateFilesWidget.php:87 Reply

    > Why only the first column ? would it not make sense to retrieve the count also to display how many occurances happend ? IMHO not relevant for the purpose of this widget.

    DuplicateFilesWidget.php:103 Reply

    I will push an update with a $file->isMissing() condition in the previous condition, this should be enough, please correct me if I am wrong.

    DuplicateFilesWidget.php:104 Reply

    Acknowledged

    DuplicateFilesWidget.php:113 Reply

    Acknowledged
  9. Frank Nägler commented on patch set 2 09/15/24, 09:43

    locallang.xlf:246 Reply

    Done
  10. Frank Nägler uploaded patch set 3 CR -1 V +1 09/15/24, 10:04

  11. Frank Nägler commented on patch set 3 09/15/24, 10:04

    DuplicateFilesWidget.php:72

    I will remove this, default restriction should work here and we don't have the risk to expose files without access.
  12. Frank Nägler commented on patch set 3 09/15/24, 10:05

    DuplicateFilesWidget.php:103 Reply

    Done
  13. core-ci voted on patch set 3 V -1 09/15/24, 10:11 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65804

  14. Frank Nägler uploaded patch set 4 V -1 09/15/24, 10:54

  15. core-ci voted on patch set 4 V -1 09/15/24, 10:59 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65807

  16. Frank Nägler uploaded patch set 5 V -1 09/16/24, 09:03

  17. core-ci voted on patch set 5 V +1 09/16/24, 09:10 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65862

  18. Ulrich Mathes uploaded patch set 6 V +1 09/16/24, 11:35

  19. core-ci voted on patch set 6 V +1 09/16/24, 11:41 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65882

  20. Ulrich Mathes uploaded patch set 7 V +1 09/16/24, 12:33

  21. core-ci voted on patch set 7 V -1 09/16/24, 12:43 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65883

  22. Ulrich Mathes uploaded patch set 8 V -1 09/16/24, 13:41

  23. Ulrich Mathes commented on patch set 8 09/16/24, 13:46

    DuplicateFilesWidget.php:74 Reply

    I'm not sure to what extent the widgets should be self-contained. I fear that splitting them up could lead to a confusing proliferation of classes. I added a test for database access - which failed on Postgres. I had to remove `GROUP_CONCAT`. So with the new approach there are no more literal selects. The disadvantage is that I have to do a separate database query for each sha1 that is found more than 1 time.

    DuplicateFilesWidget.php:87 Reply

    The number of occurrences may be reduced later by user permissions.

    DuplicateFilesWidget.php:103 Reply

    It is possible that the file does not yet know that it is missing. I reinserted the `$file->getParentFolder()` function call. When I took a closer look, I realised that this function checks the backend user access for this folder. So we don't need to check the access ourselves.
  24. Ulrich Mathes commented on patch set 8 09/16/24, 13:47

    DuplicateFilesWidget.php:87 Reply

    Done
  25. core-ci voted on patch set 8 V -1 09/16/24, 13:51 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65892

  26. Ulrich Mathes uploaded patch set 9 09/16/24, 17:13

    Copied Votes: * Verified-1 (copy condition: "**changekind:NO_CHANGE**")
  27. core-ci voted on patch set 9 V -1 09/16/24, 17:29 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65915

  28. core-ci voted on patch set 9 V +1 09/16/24, 18:58 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/65915

  29. Ulrich Mathes commented on patch set 9 09/16/24, 19:15

    DuplicateFilesWidget.php:74 Reply

    Done
  30. Philipp Kuhlmay commented on patch set 9 10/04/24, 10:13

    locallang.xlf:249

    I would change this to "Displays files that exists multiple times". A file does not need to be uploaded before, when it exists in the fileadmin, it could also be "copied" from another folder. So this duplicate file is not "uploaded", but "copied".

    DuplicateFilesWidget.html:21

    Currently, all images are shown twice. When a file is shown in this widget, I already assume that these images exist multiple times inside my storage, and therefore I only want to see the image once, not twice or more. Maybe you could only show the image itself one time and only show the separate information for the other ones like path and count of references. Example: When I have an image 10 times in my storage, I would see the exact image 10 times in the list. But I only want to see it once.
  31. Ulrich Mathes commented on patch set 9 10/08/24, 06:53

    locallang.xlf:249 Reply

    Acknowledged

    DuplicateFilesWidget.html:21 Reply

    Displaying the images multiple times was an active decision. The user **immediately** realises that an image is available multiple times. It also makes it clear that it really is the same image.
  32. Philipp Kuhlmay commented on patch set 9 10/08/24, 06:55

    DuplicateFilesWidget.html:21 Reply

    Okay when it is an active decision, I am okay with it. I just wanted to mention it.
  33. Ulrich Mathes uploaded patch set 10 V +1 10/08/24, 07:02

  34. core-ci voted on patch set 10 V +1 10/08/24, 07:09 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67320

  35. Fabian Cloïn commented on patch set 10 05/20/25, 11:12

    DuplicateFilesWidget.php:54

    Just out of curiosity. My first idea would be to put this logic in a repository. I see that there is already a repository that does something with sys_file https://github.com/TYPO3/typo3/blob/main/typo3/sysext/core/Classes/Resource/FileRepository.php. It also would be more aligned with the other widget classes in this directory. (Although this shouldn't be the solely reason to refactor.) Maybe in this scenario it would be okay to leave it in this class i don't know. Im curious of what you all think about it.
  36. Oli Bartsch uploaded patch set 11 V +1 05/23/25, 09:02

  37. core-ci voted on patch set 11 V +1 05/23/25, 09:08 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/79415

  38. Ulrich Mathes uploaded patch set 12 V +1 05/27/25, 09:44

  39. Ulrich Mathes commented on patch set 12 05/27/25, 09:49

    DuplicateFilesWidget.php:54 Reply

    This is a question in the direction of core development. I suspect we will make as few adjustments to the core as possible.

    DuplicateFilesWidget.php:86

    This can be done within one database query. Thanks @schneider.maik@me.com
  40. core-ci voted on patch set 12 V +1 05/27/25, 09:51 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/79548

  41. Ulrich Mathes uploaded patch set 13 V +1 10/22/25, 12:48

    Show icon and file size
  42. Ulrich Mathes commented on patch set 13 10/22/25, 12:50

    DuplicateFilesWidget.php:54 Reply

    Done
  43. core-ci voted on patch set 13 V +1 10/22/25, 12:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/87259

  44. Ulrich Mathes uploaded patch set 14 V +1 01/07/26, 10:26

    solve merge conflict
  45. core-ci voted on patch set 14 V +1 01/07/26, 10:34 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/93047

  46. Josef Glatz commented on patch set 14 01/07/26, 15:29

    I really like the feature. Would it makes sense to move the logic for finding duplicates in service? - e.g. adding the duplicate files functionality as API feature to TYPO3 (not in ext:dashboard but in ext:core) - e.g. to make dependency of ext:dashboard not mandatory if someone wants to use the feature in an own extension I mean, why not adding the core functionality as feature for other places too?
  47. Ulrich Mathes commented on patch set 14 01/08/26, 06:45

    Reply

    I might think that the core doesn't want to design services for reusable logic without a concrete idea for which they might be used. Do you have any ideas about where such a service could be beneficial? By encapsulating all the logic within the widgets themselves, overhead is not introduced in other places. However, I am not in a position to make this decision.
  48. Josef Glatz commented on patch set 14 01/08/26, 08:46

    Reply

    My advice follows the note from Stefan Bürk in one of his first notes on this patch
  49. Ulrich Mathes commented on patch set 14 01/08/26, 10:22

    Reply

    His note was related to the previous patchset which had used `selectLiteral` to fetch the duplicates. Since the widget now relies completely on normal QueryBuilder selects, he resolved this issue.
  50. Simon Schaufelberger uploaded patch set 15 V +1 01/13/26, 01:19

  51. Simon Schaufelberger voted on patch set 15 CR +1 01/13/26, 01:20

  52. Simon Schaufelberger voted on patch set 15 V +1 01/13/26, 01:21

  53. core-ci voted on patch set 15 V +1 01/13/26, 01:27 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/93298

  54. Ulrich Mathes uploaded patch set 16 CR +1 V +1 01/21/26, 15:52

  55. core-ci voted on patch set 16 V +1 01/21/26, 16:01 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/93962

  56. Ulrich Mathes uploaded patch set 17 V +1 01/21/26, 16:04

  57. core-ci voted on patch set 17 V +1 01/21/26, 16:16 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/93966

  58. Simon Schaufelberger commented on patch set 17 01/21/26, 18:52

    Services.yaml:236

    i think the indentation is too deep here
  59. Stefan Bürk commented on patch set 17 01/21/26, 23:28

    DuplicateFilesWidget.php:72

    I'd say / guess that we can combine `getDuplicatedSha1()` and `getFileUuidsFromSha1()` into a single datbase query / method, reducing database hoops. Either with a good table join and splitting the result into the nested array struc like `getFileUuidsFromSha1()` returns or with a `common table expression query (CTE)` Would reduce the dispatched query roundtrups from `201` to `1` using default `$this->options['dupliateLimit'] === 200`. `getDuplicates()` is heavy enough with additional queries, but using FAL API to retrieve the files is okay I guess. However, not sure if we really need it. Should we really re-evaluate FAL/File status here ?
  60. Ulrich Mathes commented on patch set 17 01/22/26, 10:49

    DuplicateFilesWidget.php:72 Reply

    As I understand it, we currently only dispatch two queries, as the `getFileUidsFromSha1()` function uses IN with an array of SHA-1 strings. So the overhead is not that significant.However, I think we should move to a CTE, as `sys_file` can be very large.`getDuplicates()` is resource-intensive, but perhaps we could eliminate some checks? We need to retrieve the FileObject for Fluid.We can remove the call to `isMissing()` as we already select only files that are not missing, and `isMissing()` does not actually check if the file is present (as `exists()` already has done).The user permission check (via `InsufficientFolderAccessPermissionsException`) could be carried out much earlier using `FolderMountsRestriction`. This might reduce the amount of SHA1 drastically. This late permission check may return a smaller resultset than the `duplicateLimit`, even if there are more duplicates present.Thanks for your in-depth analysis; I will address these points.
  61. Ulrich Mathes uploaded patch set 18 V +1 01/22/26, 11:25

    Use CTE and FolderMountRestriction for BEuser
  62. Ulrich Mathes commented on patch set 18 01/22/26, 11:26

    DuplicateFilesWidget.php:72 Reply

    Acknowledged

    Services.yaml:236 Reply

    Done
  63. core-ci voted on patch set 18 V +1 01/22/26, 11:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/94030

  64. Stefan Bürk commented on patch set 18 01/22/26, 16:36

    DuplicateFilesWidget.php:72 Reply

    > As I understand it, we currently only dispatch two queries Correct. Was to late looking on it. But still, we have one query retrieving a result set which needs to be added as "payload" to a second query. Having it combined would remove that roundtrip ;)
  65. Guido Schmechel commented on patch set 18 01/25/26, 09:46

    I tested the whole thing. It worked well for my simple example with two duplicates. What I would expect: - The limit number should be controllable in the widget. As with ‘TYPO3 security advisories’. What I would give as feedback: - I would have expected the service to be in the core so that it could be used in several places. Example: Option to report duplicates by email as an overview. - I would have expected the function to be a submodule in Media rather than in a dashboard. I realise that this is a bigger issue. But perhaps it could be discussed as a glimpse into the future. However, it should not block the feature for v14. Just bringing in a different perspective :-)
  66. Ulrich Mathes uploaded patch set 19 V +1 01/26/26, 08:02

    Add limit configuration
  67. core-ci voted on patch set 19 V -1 01/26/26, 08:10 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/94342

  68. Ulrich Mathes commented on patch set 19 01/26/26, 08:15

    Reply

    I gave the widget a limit configuration. I kept the hard limit to be controlled by administrators, since the number of duplicates could be surprisingly high. The bigger picture in terms of using this data is, as you mention, a bigger issue. I could think of a feature where the duplicate check is performed directly while uploading a new image. There are many possibilities and places to show this data. From my perspective, we could start by enabling the editors to view these duplicates in the hope that they will want to keep their data clean.
  69. Ulrich Mathes uploaded patch set 20 V -1 01/26/26, 08:19

  70. core-ci voted on patch set 20 V +1 01/26/26, 08:28 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/94345

  1. Guido Schmechel commented on patch set 20 01/26/26, 10:43

    Thanks for the update. I was able to successfully test the limit. I have two more ideas in mind: 1.) Does it make sense for us to establish a ‘Media’ group instead of registering it in the content? 2.) Even though it's a list, should we perhaps keep the icon synchronised as in the other patch? Related patch: https://review.typo3.org/c/Packages/TYPO3.CMS/+/92326
  2. Ulrich Mathes rebased onto patch set 21 V +1 01/30/26, 14:24

    Fetch thumbnail size from tsconfig
  3. Ulrich Mathes uploaded patch set 22 01/30/26, 14:26

    Fetch thumbnail size from tsconfig
  4. Ulrich Mathes uploaded patch set 23 01/30/26, 14:29

  5. core-ci voted on patch set 23 V +1 01/30/26, 14:46 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/94675

  6. core-ci voted on patch set 21 V +1 01/30/26, 14:59 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/94673

  7. core-ci voted on patch set 22 V +1 01/30/26, 15:02 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/94674

  8. Ulrich Mathes commented on patch set 23 02/02/26, 06:18

    Reply

    Done
  9. Garvin Hicking commented on patch set 23 02/02/26, 07:21

    DuplicateFilesWidgetTest.php:48

    Maybe add one more scenario for a non-admin who sees one file in a mount? Also, please also include a _recycler_ record in there as a dupe thats filtered to cover all query parts. So ones with missing=1 and atorage=0 would be helpful. (More a not forcmyself: What I have to check out, how do the "file exists" checks succeed, where are the files added to the test setup?)
  10. Ulrich Mathes uploaded patch set 24 V +1 03/18/26, 08:09

    Resolve conflicts
  11. core-ci voted on patch set 24 V +1 03/18/26, 08:19 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/98142

  12. Ulrich Mathes uploaded patch set 25 V +1 03/18/26, 13:10

    Use domain based translation syntax
  13. Ulrich Mathes uploaded patch set 26 03/18/26, 13:20

    Clarify widget options and settings
  14. core-ci voted on patch set 25 V +1 03/18/26, 13:20 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/98188

  15. Ulrich Mathes uploaded patch set 27 03/18/26, 13:22

  16. Ulrich Mathes commented on patch set 27 03/18/26, 13:28

    DuplicateFilesWidgetTest.php:48 Reply

    Acknowledged
  17. Ulrich Mathes commented on patch set 27 03/18/26, 13:29

    DuplicateFilesWidgetTest.php:48 Reply

    sorry, not resolved yet.
  18. core-ci voted on patch set 27 V -1 03/18/26, 13:37 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/98191

  19. core-ci voted on patch set 26 V +1 03/18/26, 13:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/98190

  20. Ulrich Mathes rebased onto patch set 28 V -1 03/30/26, 10:20

    test every predicate and add missing FolderMountsRestriction
  21. Ulrich Mathes commented on patch set 28 03/30/26, 10:23

    Adding more fixtures to the tests to test against every predicate has showed that the FolderMountsRestriction was missing in the CTE. Thanks for pushing me to write more tests.

    DuplicateFilesWidgetTest.php:48 Reply

    Done
  22. core-ci voted on patch set 28 V -1 03/30/26, 10:30 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/99035

  23. Ulrich Mathes uploaded patch set 29 V -1 03/30/26, 10:34

    missing changes
  24. core-ci voted on patch set 29 V -1 03/30/26, 10:44 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/99038

  25. Josef Glatz wrote 06/08/26, 03:43

    Removed cc Josef Glatz.