Review main Merged

72642[TASK] Migrate from yarn to npm

Benjamin Franzke CR +2 V +2 Impossible +18767 −8553 updated 08/23/22, 19:53

Commit message

Yarn has been added to TYPO3 back in 2017 because
npm shrinkwrap files were not stable and package-lock.json
and `npm ci` not yet established.

By now the yarn version in TYPO3 (v1) got outdated
and an update to yarn v3 would be required sooner
or later. The update to yarn v3 brings quite some
problems for the TYPO3 codebase:

 - Yarn v3 is executed/proxied through yarn v1
   (installed on the developer machine) – therefore the
   yarn v3 executable needs to be tracked within git.
   See https://yarnpkg.com/cli/set/version#details

 - Yarn v3 pulls in @types/node (through the "got" package) for
   unknown reasons, causing clashes with TypeScript declarations
   for requirejs (window.require vs node's require())

As package-lock.json is solid thesedays, there is no real
need to depend on yarn anymore. We basically switch
to npm because that switch is way more straightforward than
migrating to yarn v3 and it removes one dependency.

Note that `npm install` properly honors `package-lock.json`
these days (no implicit version upgrades due to the lock file),
but is faster than `npm ci`, therefore it is used in `grunt build`
and runTests.sh for all developer oriented build-targets.
All CI checks use `npm ci` to enfore a clean install.

We set the required npm verson to v8 in order to use the
newest package-lock.json v3 format and .nvmrc is updated to
suggest the recent nodejs v16 LTS release, as npm v8 is
shipped with nodejs v16 by default.

Note that node v14 may still be used to compile packages, but
npm v8 is needed for dependency installation (installable
via `npm install -g npm` in node v14 installations). For that
reason the CI docker images are updated to use nodejs v16
instead of nodejs v14, see following ticket for details:
https://git.typo3.org/typo3/CI/testing-infrastructure/-/issues/3

Executed Commands:

  cd Build/
  # be sure the node_modules folder dir and is updated
  yarn install
  sed -i 's/"yarn": "^1.22.0"/"npm": ">=8.0.0 <9.0.0"/' package.json
  rm -f package-lock.json
  # npm install >= v7 can read yarn.lock and will create
  # package-lock.json as a result
  npm install
  git add package-lock.json
  git rm -f yarn.lock
  npm remove "jquery-ui"
  # The use of `git@` in the HTTPS URLs is a workaround for NPM's
  # auto-rewrite that results in an SSH URL being used instead,
  # which fails to install in CI.
  # See npm/cli#2610 for more information.
  npm install "git+https://git@github.com/jquery/jquery-ui#1.11.4"
  # Fix fsevents not being installed
  npm install rollup@~2.32.0
  # Explicitly pull in popperjs which would be included by npm's
  # resolver in npm >8.5
  npm install @popperjs/core@^2.11.6
  # Let npm build a new, fully clean tree, without old yarn cruft
  rm -rf node_modules/
  npm install
  grunt default

Review

  • Code review CR +2 Done
    • Benjamin Franzke +2
    • Stefan Bürk +1
    • Benni Mack +1
  • Verified V +2 Done
    • Benjamin Franzke +2
    • Stefan Bürk +1
    • Benni Mack +1
    • core-ci +1 CI

Issues 2

Files 12

Review log 4262

Show 2537 earlier messages
  1. Benjamin Franzke uploaded patch set 1 12/13/21, 12:18

  2. core-ci voted on patch set 1 V -1 12/13/21, 12:35 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21027

  3. Benjamin Franzke uploaded patch set 2 12/13/21, 13:12

  4. core-ci voted on patch set 2 V -1 12/13/21, 13:25 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21029

  5. Benjamin Franzke uploaded patch set 3 12/13/21, 18:40

  6. core-ci voted on patch set 3 V -1 12/13/21, 18:58 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21041

  7. Benjamin Franzke uploaded patch set 4 12/13/21, 19:25

  8. core-ci voted on patch set 4 V -1 12/13/21, 19:46 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21042

  9. Benjamin Franzke uploaded patch set 5 12/14/21, 20:43

  10. core-ci voted on patch set 5 V -1 12/14/21, 20:47 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21104

  11. Benjamin Franzke uploaded patch set 6 12/14/21, 20:56

  12. core-ci voted on patch set 6 V -1 12/14/21, 21:10 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21105

  13. Benjamin Franzke uploaded patch set 7 12/14/21, 21:33

  14. Benjamin Franzke uploaded patch set 8 12/14/21, 21:40

  15. core-ci voted on patch set 7 V -1 12/14/21, 21:41 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21108

  16. Benjamin Franzke uploaded patch set 9 12/14/21, 21:48

  17. core-ci voted on patch set 8 V -1 12/14/21, 21:51 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21109

  18. core-ci voted on patch set 9 V +1 12/14/21, 22:03 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21110

  19. Benjamin Franzke commented on patch set 9 12/14/21, 22:09

    Note: npm ci is slower than yarn install. rm -rf node_modules && time npm ci real 0m22,517s user 0m27,196s sys 0m7,538s rm -rf node_modules && time yarn install real 0m11,201s user 0m12,597s sys 0m5,261s Feel free to CR-1 if anyone sees that as a blocker ;)
  20. Oliver Hader commented on patch set 9 12/15/21, 07:45

    Just want to understand the reasons behind switching from yarn to npm... for instance, what's the "binary blob" thing in that regard & why is it better to use npm instead?
  21. Benjamin Franzke commented on patch set 9 CR -1 12/15/21, 08:04

    Reply

    This "blob" topic (i call it blob, although of course, it's only minified code, not binary) came up with the following patch that upgrades to yarn v3 and ships yarn-berry.cjs (Note: that's actually v2, so don't exactly know why that is shipped there): https://review.typo3.org/c/Packages/TYPO3.CMS/+/72622/10/Build/.yarn/releases/yarn-berry.cjs I asked Jochen why there is no yarn v3 release on npm, and if shipping it in-tree is the supposed way to use yarn-v3. He wondered as well, but didn't know another way. (It seems you have to install yarn v3 using yarn v1/v2). Other than that there are no technical reasons for switch, besides that I like to use "vanilla" software (npm) and avoid unnecessary addons (yarn), if those addons are no longer required. For sure, we can see yarn's speed improvements as requirements, so I'd fine to abandon this. Anyway, the commit message clearly needs clarification regarding the blob topic.
    needs more explanation and ideally a source-link
  22. Jochen Roth commented on patch set 9 12/17/21, 15:43

    Here some details about performance on install (without cache) npm install (v7) 24.16s user 14.04s system 136% cpu 28.048 total yarn install (v1) 21.43s user 28.56s system 192% cpu 25.904 total yarn install (v3) 20.91s user 11.02s system 147% cpu 21.670 total yarn install (v3 without `nodeLinker: node-modules`) 8.79s user 2.89s system 162% cpu 7.174 total
  23. Benni Mack commented on patch set 9 12/21/21, 11:40

    I think yarn3 seems like the best approach (for the time being), however we'd need to sort out this binary/blob thingy.
  24. Jochen Roth commented on patch set 9 12/21/21, 12:05

    Reply

    Well, if we really don't want this in the repo, we need to use `yarn set version berry` (berry is yarn >= 2) prior install which creates the blob/binary iirc.
  25. Jochen Roth commented on patch set 9 12/21/21, 12:08

    Reply

    > This command will download a specific release of Yarn directly from the Yarn GitHub repository, will store it inside your project, and will change the yarnPath settings from your project .yarnrc.yml file to point to the new file. https://yarnpkg.com/cli/set/version#details
  26. Benjamin Franzke uploaded patch set 10 05/04/22, 14:38

  27. Benjamin Franzke commented on patch set 10 05/04/22, 14:38

    Reply

    Done

    Reply

    Done
  28. core-ci voted on patch set 10 V -1 05/04/22, 14:48 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26827

  29. Benjamin Franzke commented on patch set 10 05/04/22, 14:58

  30. Benjamin Franzke uploaded patch set 11 05/04/22, 15:59

  31. Benjamin Franzke commented on patch set 11 05/04/22, 15:59

    Reply

    Fixed be removing the useless package.json file in codemirror/ folder.
  32. core-ci voted on patch set 11 V -1 05/04/22, 16:09 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26832

  33. Benjamin Franzke uploaded patch set 12 05/04/22, 16:27

  34. core-ci voted on patch set 12 V +1 05/04/22, 16:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26833

  35. Benni Mack commented on patch set 12 05/04/22, 17:29

    Dude, 37K lines for a lock file, this stuff (in general) is just super nuts....
  36. Benjamin Franzke commented on patch set 12 05/04/22, 18:38

    Reply

    Very large indeed, I also wondered about it, and checked if a clean `npm install` procuded the same amout of lines – it did :\ It turns out the lock file contains all information both in v2 and v1 format (one "packages" array and one v1-compatible redundant "dependencies" array), which makes this file soo large. The size will grow to 50% size will lockFileVersion v3: https://docs.npmjs.com/cli/v8/configuring-npm/package-lock-json#lockfileversion We could directly switch to lock-file v3 with lockfile-version=3 in Build/.npmrc but for that to work we need node16 in the docker images, see https://git.typo3.org/typo3/CI/testing-infrastructure/-/issues/3
  37. Benjamin Franzke uploaded patch set 13 05/04/22, 21:05

  1. core-ci voted on patch set 13 V +1 05/04/22, 21:15 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26842

  2. Benjamin Franzke uploaded patch set 14 05/04/22, 22:56

  3. core-ci voted on patch set 14 V -1 05/04/22, 23:13 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26859

  4. Stefan Bürk commented on patch set 14 05/05/22, 00:30

    .nvmrc:1

    If i'm right, this is the "to be used node version" through the multi node thing "nvm". Right ? Which only takes effect directly on developer machines. the core ci js image "typo3/core-testing-js" inherits from node-v14 images, so this is there irrelevant. Also seen some other (wip) patches - should we discuss if we need a newer image / dedicated image with seperate settings ? Should we update or have dedicated js images for core version / nodeversions etc ? Maybe we can have a call the next days if there are need for images/image updates which I can prepare/provide (if wanted/needed).
  5. Josef Glatz commented on patch set 14 05/05/22, 05:27

    .nvmrc:1 Reply

    yes. (cd Build && nvm use)
  6. Benjamin Franzke uploaded patch set 15 05/05/22, 06:07

  7. Benjamin Franzke commented on patch set 15 05/05/22, 06:07

    .nvmrc:1 Reply

    Yup, nvm isn't use by core. So this is basically just for (some) developers, that use nvm. > Maybe we can have a call the next days if there are need for images/image updates which I can prepare/provide (if wanted/needed). Yup, as outlined in https://git.typo3.org/typo3/CI/testing-infrastructure/-/issues/3 I think the best solution would be to use version tags run docker-composer instead of :latest. Other than that we actually don't have a strict need for node 16 right now, it's rather npm v8 that is needed (which is shipped with node v16). Another "quick" fix could be to update to `npm` LTS in the core-testing-js images: https://git.typo3.org/typo3/CI/testing-infrastructure/-/blob/main/docker-images/core-testing-js/Dockerfile#L20 npm install -g --force npm:~8.5 yarn optipng-bin instead of npm install -g --force yarn optipng-bin That would even be safe for v10 and v11 as we could stay with node v14 for the time being. Also, please see for a demo why we need npm v8: https://review.typo3.org/c/Packages/TYPO3.CMS/+/74522/4
  8. core-ci voted on patch set 15 V +1 05/05/22, 06:15 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26867

  9. Benjamin Franzke uploaded patch set 16 05/05/22, 06:21

  10. core-ci voted on patch set 16 V +1 05/05/22, 06:32 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26868

  11. Benjamin Franzke commented on patch set 16 CR -1 05/05/22, 07:34

  12. Benni Mack commented on patch set 16 05/05/22, 09:11

    .nvmrc:1 Reply

    Let's first migrate to npm v8 and then continue later-on.
  13. Benjamin Franzke uploaded patch set 17 08/12/22, 05:01

  14. core-ci voted on patch set 17 V +1 08/12/22, 05:09 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/30061

  15. Benjamin Franzke uploaded patch set 18 08/23/22, 17:44

  16. Benjamin Franzke commented on patch set 18 08/23/22, 17:45

    .nvmrc:1 Reply

    Yea, let's do so. That allows to use package-lock.json format v3 right away, without the v2 compat layer 👍 Done.
  17. Benjamin Franzke commented on patch set 18 08/23/22, 17:48

    Reply

    Marking as Resolved, as the yarn patch has already been abandoned, see https://review.typo3.org/c/Packages/TYPO3.CMS/+/72622/38#message-e8272c4c749bf48fa7947b093feaddf976b790bb
  18. core-ci voted on patch set 18 V +1 08/23/22, 17:51 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/30383

  19. Stefan Bürk commented on patch set 18 08/23/22, 19:01

  20. Benni Mack voted on patch set 18 V +1 CR +1 08/23/22, 19:29

  21. core-ci voted on patch set 18 V -1 08/23/22, 19:31 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/30386

  22. core-ci voted on patch set 18 V +1 08/23/22, 19:42 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/30386

  23. Stefan Bürk voted on patch set 18 V +1 CR +1 08/23/22, 19:42

  24. Benjamin Franzke voted on patch set 18 V +2 CR +2 08/23/22, 19:52

  25. Benjamin Franzke merged patch set 19 08/23/22, 19:53