Review main Merged

75867[BUGFIX] Generate correct asset symlinks on Windows

Robert Kärner CR +2 V +2 Easy +6 −1 updated 08/30/23, 06:18

Commit message

This patch fixes two issues when generating symlinks to the public
extension resources (Public/Resources) into the _assets folder during
Composer installation.

First, directory symlinks can't be created the same way on Windows as
on Unix platforms. The creation requires elevated rights, and the
required command is named differently as well. Junctions should be used
on Windows instead, because they do not need elevated privileges to be
created.

When installing a TYPO3 extension via Composer, its Resources/Public
folder is getting symlinked to the publicly accessible _assets folder.
Until now, this process only used the Unix specific functions provided
by Composer. It did not create any symlinks on Windows based
installations, without generating any error message.

This patch adds the same checks and platform-specific functions used by
Composer when symlinking packages from a local path repository.

The second issue that has been addressed is the naming of the symlinks
themselves. When a Resources/Public directory is symlinked into the
_assets folder, the symlink will be named using an md5 hash based on
parts of the extension's installation path on the filesystem. During
Composer installation, this relative path is derived from the
absolute path provided by Composer. As Windows uses a different default
path separator (backslash) compared to Unix (forward slash), the
relative path will be different between both platforms, resulting in a
different md5 hash being generated.

Absolute path on Unix:
/path/to/typo3/vendor/mycompany/myext

Absolute path on Windows:
C:\path\to\typo3\vendor/mycompany/myext

Relative path to be used for the hash on Unix:
/vendor/mycompany/myext

Relative path to be used for the hash on Windows:
\vendor/mycompany/myext

If an uri into the _assets folder is to be generated in either frontend
or backend, the md5 hash will be always be generated by using forward
slashes, resulting in incorrect links on Windows installations.

This difference has been solved by replacing all backslashes in the
absolute extension installation path into forward slashes on Windows
installations, so the symlink will be named correctly from the
beginning.

Review

  • Code review CR +2 Done
    • Stefan Bürk +2
    • Christian Kuhn +1
    • Nicole Hummel +1
  • Verified V +2 Done
    • Stefan Bürk +2
    • Christian Kuhn +1
    • Nicole Hummel +1
    • core-ci +1 CI

Issues 2

Files 1

Review log 3342

Show 1517 earlier messages
  1. Robert Kärner uploaded patch set 1 09/26/22, 09:28

  2. core-ci voted on patch set 1 V +1 09/26/22, 09:36 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/31530

  3. Simon Gilli commented on patch set 1 CR -1 09/26/22, 11:44

    Thanks for the patch! This is a workaround only without fixing the source of the problem. Since Windows 10 there is the user privilege `SeCreateSymbolicLinkPrivilege` which should be properly set for trusted users to enable the creation of symlinks. Admins have this privilege set by default. On developer machines this can be achieved by enabling the developer mode. On production system one should follow the instructions here https://learn.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/create-symbolic-links But we for sure should discuss, if the core needs a fix here. In my opinion the suggested change should only be used as fallback, the definition of Microsoft is pretty clear on how to work with symlinks. On Windows platforms I would prefer a warning in case of a failure and maybe asking the user for the confirmation if he likes to use a junction instead or abort by default and let the user fix his configuration which should be the prefered solution. Maybe a warning only with a hint on how to fix the problem is also fine.
  4. Robert Kärner commented on patch set 1 09/26/22, 12:06

    Reply

    Valid point. Maybe this is not the right place and this discussion should move to the coredev Slack channel, but can you elaborate a bit further why forcing an OS configuration change is the preferred way contrary to just using junctions like Composer does? Are there any benefits of using symlinks instead of junctions? Also, I believe a fix of some nature is needed here indeed, because even if the symlink rights are granted, the command to create a symlink differs between Windows (mklink) and Unix (symlink). The currently used Composer function $fileSystem->relativeSymlink() only checks for "symlink" being available and exits if not. So it does nothing on a Windows installation.
  5. Simon Gilli commented on patch set 1 09/26/22, 12:56

    Reply

    It's fine discussing it here. Here are some discussions about junction vs symbolic link topic https://superuser.com/questions/343074/directory-junction-vs-directory-symbolic-link And some additional content of Microsoft regarding hard links and junctions (soft links) https://learn.microsoft.com/en-us/windows/win32/fileio/hard-links-and-junctions Not directly related to the decision, just wondering, is your case related to local development or used for production hosting on Windows Server?
  6. Robert Kärner commented on patch set 1 09/26/22, 13:29

    Reply

    Neither at the moment. I was fiddling and experimenting around, and noticed that typo3/cms-composer-installers v4 doesn't work on Windows as intended, so I decided to supply a patch. We have a client running a TYPO3 instance on a Windows Server, but that's Intranet stuff and I'm not sure if it's gonna receive an upgrade eventually. Your links show the differences between symlinks and junctions on Windows, and I know about them. But they don't show a statement like: "Junctions are deprecated, avoid them in the future, Symlinks are the way to go." So the motivation to avoid junctions and give Windows installations extra configuration pain is still unclear for me. Of course, DDEV is the recommended way to setup local development environments and skip the Windows hassle (and gaining headaches elsewhere, but that's not the topic here). Maybe TYPO3 should officially stop supporting Windows environments completely and refer to DDEV only. This would also solve the getenv() issue that still causes random install tool logouts on Windows machines. But as long as Windows is an officially supported platform, I think all features should be supported. And since T3 v12 comes with typo3/cms-composer-installers v4 as a requirement (when installed in Composer mode), a solution should be found and applied. Unfortunately I'm lacking the knowledge about the Composer installation process and how exactly to implement your suggested installation warning ("Do you want to continue using junctions?"). Maybe Helmut has some ideas how to tackle the whole problem best.
  7. Simon Gilli commented on patch set 1 09/26/22, 14:24

    Reply

    I have no final opinion which one is the correct way, that's why I'm trying to collect some information about the various possible scenarios. My feeling tells me, we should support symbolic links in the first place and maybe not support junctions at all. But so far I did not find a source that proves this feeling. And it's not about changing your patch immediately. Implementing a user confirmation is no big deal. But first, there must be a decision on how to fix the issue. Of course while preparing cms-composer-installers v4 we also talked about Windows support but decided to postpone the topic until first issues show up.
  8. Simon Gilli commented on patch set 1 09/27/22, 09:10

    Reply

    I think we should be fine by using junctions on Windows. Will cause less problems for Windows users. Composer does the same with the symlink strategie see https://github.com/composer/composer/blob/5770fb027c3af5b29f1542bde0407c37961ee950/src/Composer/Downloader/PathDownloader.php#L107-L112.

    PackageArtifactBuilder.php:280

    The path may already exist as symlink, should be checked too to avoid errors. Composer solves this case by always remove the path in advance which also maybe a good idea here see https://github.com/composer/composer/blob/5770fb027c3af5b29f1542bde0407c37961ee950/src/Composer/Downloader/PathDownloader.php#L97-L98
  9. Robert Kärner commented on patch set 1 09/27/22, 09:43

    Reply

    I am also fine with postponing the issue, since we are not affected yet. No pressure.

    PackageArtifactBuilder.php:280 Reply

    ACK, good catch. Will do some testing on multiple possible scenarios.
  10. Robert Kärner uploaded patch set 2 09/27/22, 09:58

  11. Robert Kärner commented on patch set 2 09/27/22, 10:01

    PackageArtifactBuilder.php:280 Reply

    I have verified that Composer's $fileSystem->removeDirectory() function handles all cases (directory, symlink, junction) just fine, so no further checks are needed on this. This has been solved in patchset 2.
  12. core-ci voted on patch set 2 V +1 09/27/22, 10:06 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/31579

  13. Simon Gilli voted on patch set 2 CR +1 09/27/22, 10:10

  14. Simon Gilli commented on patch set 2 V +1 09/27/22, 15:30

    Works fine. Tested with PHP 8.1 and Composer 2.4
  15. Kevin Appelt voted on patch set 2 CR +1 10/02/22, 22:26

  16. Robert Kärner uploaded patch set 3 CR +1 V +1 01/17/23, 06:42

  17. Robert Kärner commented on patch set 3 01/17/23, 06:44

    I have merged the changes from https://review.typo3.org/c/Packages/TYPO3.CMS/+/75880 into here, so we have a single patch to review now.
  1. core-ci voted on patch set 3 V +1 01/17/23, 06:50 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/35907

  2. Benjamin Franzke commented on patch set 3 CR -1 01/17/23, 17:45

    PackageArtifactBuilder.php:277

    The is using `rm -rf` underneath, let's never call that, please. In case we mess any path calculations up, things can go really bad. Please do only remove stuff if it is an empty directory or a symlink.
  3. Robert Kärner commented on patch set 3 01/19/23, 13:50

    PackageArtifactBuilder.php:277 Reply

    Ack, on it
  4. Robert Kärner uploaded patch set 4 CR -1 V +1 01/19/23, 14:29

  5. core-ci voted on patch set 4 V +1 01/19/23, 14:39 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/36254

  6. Robert Kärner commented on patch set 4 01/19/23, 14:52

    PackageArtifactBuilder.php:277 Reply

    Only empty directories and symlinks/junctions are removed now. Do you think this is fine, or should we just remove the whole check altogether?
  7. Helmut Hummel commented on patch set 4 03/15/23, 09:52

    PackageArtifactBuilder.php:277 Reply

    I would for now not remove anything. Instead put both check for existence and creation of the reference into each branch of the platform check if (isWindows) { if (!junction) createJuction() else { if (!symlink) createSumlink() }
  8. Benni Mack updated the commit message in patch set 5 V +1 05/03/23, 09:35

  9. Benni Mack rebased onto patch set 6 07/07/23, 13:08

  10. core-ci voted on patch set 6 V +1 07/07/23, 13:26 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/43908

  11. Benni Mack rebased onto patch set 7 V +1 08/07/23, 09:33

  12. core-ci voted on patch set 7 V +1 08/07/23, 09:40 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/45177

  13. Helmut Hummel uploaded patch set 8 V +1 08/22/23, 09:16

  14. Helmut Hummel voted on patch set 8 V +1 CR +1 08/22/23, 09:16

  15. Helmut Hummel commented on patch set 8 08/22/23, 09:16

    PackageArtifactBuilder.php:277 Reply

    Done
  16. core-ci voted on patch set 8 V +1 08/22/23, 09:22 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/45666

  17. Christian Kuhn updated the commit message in patch set 9 CR +1 V +1 08/23/23, 16:14

  18. Christian Kuhn rebased onto patch set 10 08/23/23, 16:14

  19. Christian Kuhn voted on patch set 10 V +1 CR +1 08/23/23, 16:14

  20. core-ci voted on patch set 9 V +1 08/23/23, 16:23 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/45758

  21. core-ci voted on patch set 10 V +1 08/23/23, 16:23 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/45759

  22. Nicole Hummel voted on patch set 10 V +1 08/28/23, 09:07

  23. Nicole Hummel voted on patch set 10 CR +1 08/28/23, 09:08

  24. Stefan Bürk voted on patch set 10 V +2 CR +2 08/30/23, 06:02

  25. Stefan Bürk merged patch set 11 08/30/23, 06:18