Review main Merged

72637[TASK] Migrate backend TypeScript to ECMAScript modules (ESM)

Benjamin Franzke CR +2 V +2 Nightmare +2865 −2864 updated 01/21/22, 09:22

Commit message

Use the JavaScript module and importmap Feature introduced
in #96510 to import JavaScript as native browser modules
(commonly referred to as ES6 modules, or short ESM).
To be specific: we actually use ES2020 (ES11) modules,
as we import modules dynamically via import() in
combination with an importmap (#96510).

Backwards compatibility for existing RequireJS imports
is provided by a shim as introduced in #96510.

How to (re)view this change
---------------------------

Exclude all automatic TypeScript/JavaScript/lockfile changes with:

git show --oneline -- . \
    ':(exclude)typo3/sysext/*.js' \
    ':(exclude)Build/Sources/TypeScript/*.ts' \
    ':(exclude)Build/yarn.lock'

git show --oneline -- $(find Build/ \
    -name DragUploader.ts -o \
    -name FormEngine.ts -o \
    -name Helper.ts -o \
    -name CKEditorLoader.ts -o \
    -name CodeMirrorElement.ts)

Gruntfile & tsconfig.json
-------------------------

Gruntfile is adapted to transform all our dependencies into ES6
modules.

Also tsconfig is adapted to emit es2020 modules (es2020 is required
for dynamic module imports with import()).
Also allowSyntheticDefaultImports is enabled while esModuleInterop
is now longer needed as we target ES modules and therefore removed.
allowSyntheticDefaultImports had been enabled implicitly by
esModuleInterop, we still need this enabled as the type declarations
of our dependencies still imply AMD/CJS exports
(which we fix in our build steps).

JSUnit
------

JSUnit configuration is moved from testing-framework into core for more
flexibility when adaptions are required. The testing is revamped
to use karma-rollup-preprocessor instead of karma-requirejs in
order to support the newly compiled ES6 modules.

TypeScript Exports and Imports
------------------------------

All export and import definitions have been auto-converted
from AMD pseudo import/exports to native ES6 import/exports.

Executed commands:

  find Build/Sources/TypeScript/ -name '*.ts' -exec sed -i \
    -e 's/export =/export default/' \
    -e 's/ = require(\(.*\))/ from \1/g' \
    -e "s/^import 'tablesort';/import Tablesort from 'tablesort';/" \
    '{}' +

Note that the switch from `export =` to `export default` is
not considered breaking because the require-adapter in
requirejs-loader wil transparently resolve the .default in moduls.
This is possible because AMD modules could not use `export =`
and `export {}` (for named exports) at the same time.

Explicitly use RequireJS for CodeMirror and ext:form for now
------------------------------------------------------------

CodeMirror can't be easily transformed to ES6 right now,
therefore this patch explicitly configures to use the
global RequireJS instance.

The ext:form JavaScript is currently not written in TypeScript
and will therefore not be automatically transformed from AND
to ES6 when updating tsconfig.json to emit ES6 modules.
Therefore requirejs is uses.

This partially reverts #96326 which changed some require()
calls to import() syntax too early.

New npm dependencies:
---------------------

 # For module-postprocessing in Gruntfile
 # This is the same parser as used for the
 # frontend polyfill es-module-shims
yarn add --dev es-module-lexer

 # A rollup bundle for unit testing in karma is created
 # usin a karma preprocessor
yarn remove karma-requirejs
yarn add --dev karma-rollup-preprocessor rollup-plugin-glob-import

 # Update grunt-terser to support es2020 syntax
 # (uses terser v5 which we already depended on, but grunt v1
 # used the older v4 version)
yarn add --dev grunt-terser@^2.0.0

Review

  • Code review CR +2 Done
    • Benjamin Franzke +2
    • Andreas Kienast +1
    • Benni Mack +1
  • Verified V +2 Done
    • Benjamin Franzke +2
    • Andreas Kienast +1
    • Benni Mack +1
    • core-ci +1 CI

Issues 3

Files 797

Show 777 more files

Review log 3867

Show 2342 earlier messages
  1. Benjamin Franzke uploaded patch set 1 12/12/21, 19:01

  2. Benjamin Franzke wrote 12/12/21, 19:05

    Topic set to esm
  3. core-ci voted on patch set 1 V -1 12/12/21, 19:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21003

  4. Benjamin Franzke uploaded patch set 2 12/12/21, 19:48

  5. core-ci voted on patch set 2 V -1 12/12/21, 20:02 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21004

  6. Benjamin Franzke uploaded patch set 3 12/12/21, 22:01

  7. core-ci voted on patch set 3 V -1 12/12/21, 22:13 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21006

  8. Benjamin Franzke updated the commit message in patch set 4 12/13/21, 20:37

  9. core-ci voted on patch set 4 V -1 12/13/21, 20:52 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21045

  10. Benjamin Franzke uploaded patch set 5 12/13/21, 20:58

  11. core-ci voted on patch set 5 V -1 12/13/21, 21:13 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21046

  12. Benjamin Franzke uploaded patch set 6 12/13/21, 21:55

  13. core-ci voted on patch set 6 V -1 12/13/21, 22:09 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21048

  14. Benjamin Franzke uploaded patch set 7 12/13/21, 22:41

  15. core-ci voted on patch set 7 V -1 12/13/21, 23:00 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21049

  16. Benjamin Franzke uploaded patch set 8 12/13/21, 23:39

  17. core-ci voted on patch set 8 V -1 12/13/21, 23:53 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21051

  18. Benjamin Franzke uploaded patch set 9 12/14/21, 08:17

  19. core-ci voted on patch set 9 V -1 12/14/21, 08:34 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21062

  20. Benjamin Franzke uploaded patch set 10 12/14/21, 09:09

  21. core-ci voted on patch set 10 V -1 12/14/21, 09:25 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21070

  22. Oliver Hader commented on patch set 10 12/14/21, 10:12

    DeferredAction.ts:35

    General question: Should we use `export default class ...` instead? (I see, the current approach used here was simpler grep-ing lots of files ;-)

    devbridge-autocomplete+1.3.0.patch:1

  23. Benjamin Franzke uploaded patch set 11 12/14/21, 15:08

  24. Benjamin Franzke commented on patch set 11 12/14/21, 15:08

    DeferredAction.ts:35 Reply

    would be fine with that once we have proper es6 module syntax, yea. For now this eases automatic migration (as I didn't change this by hand, but rather using sed-expressions). And I want to keep doing so, to be able to rebase.

    devbridge-autocomplete+1.3.0.patch:1 Reply

    Thanks, rebased.
  25. core-ci voted on patch set 11 V -1 12/14/21, 15:23 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21090

  26. Benjamin Franzke uploaded patch set 12 12/15/21, 14:42

  27. Benjamin Franzke uploaded patch set 13 12/15/21, 14:53

  28. core-ci voted on patch set 13 V +1 12/15/21, 15:09 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21135

  29. Benjamin Franzke uploaded patch set 14 12/20/21, 18:54

  30. core-ci voted on patch set 14 V -1 12/20/21, 19:08 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21314

  31. Benjamin Franzke uploaded patch set 15 12/20/21, 20:14

  32. core-ci voted on patch set 15 V +1 12/20/21, 20:28 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21317

  33. Benjamin Franzke uploaded patch set 16 12/20/21, 22:37

  34. core-ci voted on patch set 16 V -1 12/20/21, 23:45 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21323

  35. Benjamin Franzke uploaded patch set 17 01/05/22, 11:25

  36. core-ci voted on patch set 17 V +1 01/05/22, 11:39 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21637

  37. Benjamin Franzke uploaded patch set 18 01/06/22, 19:26

  38. core-ci voted on patch set 18 V -1 01/06/22, 19:44 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21730

  39. Benjamin Franzke uploaded patch set 19 01/06/22, 19:59

  40. core-ci voted on patch set 19 V +1 01/06/22, 20:14 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21732

  41. Benjamin Franzke uploaded patch set 20 01/13/22, 09:15

  42. core-ci voted on patch set 20 V -1 01/13/22, 09:37 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22008

  1. Benjamin Franzke uploaded patch set 21 01/13/22, 09:39

  2. core-ci voted on patch set 21 V -1 01/13/22, 09:42 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22009

  3. Benjamin Franzke uploaded patch set 22 01/13/22, 09:49

  4. core-ci voted on patch set 22 V -1 01/13/22, 10:11 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22010

  5. Benjamin Franzke uploaded patch set 23 01/13/22, 12:12

  6. core-ci voted on patch set 23 V +1 01/13/22, 12:27 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22017

  7. Benjamin Franzke uploaded patch set 24 01/14/22, 19:01

  8. core-ci voted on patch set 24 V -1 01/14/22, 19:22 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22085

  9. Benjamin Franzke uploaded patch set 25 01/14/22, 19:25

  10. Benjamin Franzke commented on patch set 25 01/14/22, 19:33

    DeferredAction.ts:35 Reply

    I guess we can postpone that decision until later.
  11. core-ci voted on patch set 25 V +1 01/14/22, 19:40 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22090

  12. Benjamin Franzke uploaded patch set 26 01/15/22, 16:54

  13. core-ci voted on patch set 26 V -1 01/15/22, 17:06 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22123

  14. Benjamin Franzke uploaded patch set 27 01/15/22, 18:04

  15. core-ci voted on patch set 27 V +1 01/15/22, 18:20 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22124

  16. Benjamin Franzke uploaded patch set 28 01/17/22, 20:24

  17. core-ci voted on patch set 28 V +1 01/17/22, 20:38 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22197

  18. Benjamin Franzke uploaded patch set 29 01/18/22, 10:21

  19. core-ci voted on patch set 29 V +1 01/18/22, 10:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22218

  20. Benjamin Franzke uploaded patch set 30 01/19/22, 14:57

    Resolve Merge Conflict
  21. core-ci voted on patch set 30 V +1 01/19/22, 15:11 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22272

  22. Andreas Kienast voted on patch set 30 V +1 CR +1 01/19/22, 16:55

  23. Benni Mack voted on patch set 30 V +1 CR +1 01/21/22, 07:51

  24. Benjamin Franzke voted on patch set 30 V +2 CR +2 01/21/22, 09:22

  25. Benjamin Franzke merged patch set 31 01/21/22, 09:22