#92526Concept for improved EXT:impexp
Description
Scope
The TYPO3 system extension EXT:impexp seems to be a bit abandoned (“stepmotherly supervised”):
The UI is not intuitive, an official documentation is missing and the support of CLI commands could be extended. The community seems to be eager to use it, but at the moment there has evolved a group of unofficially certified “TYPO3 Impexp Integrators” who know how to do it right.
As this extension is a very well encapsulated, small piece of code which should not rely too much on other extensions, this is maybe a good project for someone who wants to dive into TYPO3 Core development and Clean Code step by step.
The following diagrams and lists are the result of some brainstorming on a revised EXT:impexp. It is open for discussion and in the end this page should contain a final wish list and rough concept which is ready for implementation.
(and I would like to do / participate in the implementation for the above reasons:)
Concept
Entity “Export Preset”
- Fully-Qualified TCA
- display records in Web > List module
- make all fields editable
- fields:
- [ctrl] hideTable = false;
- [ctrl] rootLevel = -1;
- uid
- pid
- title
- identifier (new)
- description (new)
- configuration (former: preset_data)
- cruser_id (former: user_uid)
- tstamp
- crdate
- hidden (new)
- deleted (new)
- remove fields:
- public (replace by native TYPO3 access management)
- item_uid (replace by pid)
- CRUD
- Creation:
- create export preset if
- export preset by this backend user + pid + title does not exist
- export module accessible by backend user
- record creation allowed to backend user on this pid
- pid = root page of exported page subtree
- title + description = form fields in export module tab “Meta Data”
- identifier = auto-generated from backend user + pid + title (unique, used for CLI commands)
- cruser_id = backend user using export module
- configuration = serialized export configuration
- tstamp + crdate = creation timestamp
- hidden = false
- deleted = false
- create export preset if
- Read:
- show export preset if
- pid = current root page of export module
- export module accessible by backend user
- page / record reading allowed to backend user on this pid
- show export preset if
- Update:
- update export preset if
- export preset by this backend user + pid + title does exist
- export module accessible by backend user
- record editing allowed to backend user on this pid
- description = form field in export module tab “Meta Data”
- configuration = serialized export configuration
- tstamp = update timestamp
- update export preset if
- Deletion:
- delete export preset if
- export module accessible by backend user
- record deletion allowed to backend user on this pid
- deleted = true
- delete export preset if
- Creation:
- General: Sync handling of TCEforms and import/export module:
- soft / hard deletion
- record history
- access management
Entity “Import Preset”
- basic setting as entity “Export Preset” but used for import configurations
Documentation
- create full extension docs at /Documentation
- add good explanations for all 3 export configuration fields:
- “Include tables”
- “Include relations to tables”
- “Use static relations for tables” => what happens if selected / is missing / which scenario is it good for?
UI
- Export module
- tab “Preset”
- list all existing presets of page with
- normal TCA list item
- action: load (jump to tab “Page Tree” with preset loaded + success message)
- list all existing presets of subtree with
- grouped by page ID (page title)
- normal TCA list item
- action: load (jump to page in page tree + tab “Page Tree” with preset loaded + success message + info about page tree jump)
- grouped by page ID (page title)
- Button “Next”
- list all existing presets of page with
- tab “Page Tree”
- Page ID
- Page Tree (former: Tree)
- Depth (former: Levels)
- Include tables
- Include relations to tables (find better wording)
- Use static relations for tables (find better wording)
- Show static relations (find better wording)
- Exclude elements: Add same row as for “Preview export”: Include [check] [symbol] [title/table+id if no title]
- Exclude disabled elements (is it really required or does it add complexity?)
- Button “Apply” (former: “Update”)
- Button “Prev” + Button “Next”
- Preview export (single records removable)
- tab “Files”
- Exclude media which is linked in HTML / CSS files
- Button “Apply”
- Button “Prev” + Button “Next”
- Preview export (single records removable?)
- tab “Export”
- Section “Save configuration as preset”
- title
- description
- button “Save preset” => on save: “Configuration saved to path/file successfully.”
- Section “Save export”
- File format
- Subsection “Save export on server”
- Readonly server path + input field for filename (if empty, suggest something)
- Button “Save export on server” => on save: “Export saved to path/file successfully.”
- Subsection “Download export”
- Button “Download export”
- Button “Prev”
- Preview export
- Section “Save configuration as preset”
- tab “Preset”
- Import module
- tab “Preset”
- list all existing presets of page with
- normal TCA list item
- action: load (jump to tab “Load” with preset loaded + success message)
- list all existing presets of subtree with
- grouped by page ID (page title)
- normal TCA list item
- action: load (jump to page in page tree + tab “Page Tree” with preset loaded + success message)
- grouped by page ID (page title)
- Button “Next”
- list all existing presets of page with
- tab “Load”
- Select file to import
- Upload file from local computer
- Overwrite existing files
- Button “Load” (former: “Update”)
- Button “Next”
- tab “Page Tree”
- Page ID
- Page Tree (former: Tree)
- Depth (former: Levels)
- Include tables (as in view “Export”)
- Method: Insert records / Update records / Update records but ignore the pid (former: Update records)
- Do not show differences in records => Why, for performance reasons?
- Force uids of import file (former: Force ALL UIDs values)
- Log all database actions (former: Write individual DB actions during import to the log)
- Exclude elements: Add same row as for “Preview import”: Include [check] [symbol] [title/table+id if no title]
- Button “Apply”
- Button “Prev” + Button “Next”
- Preview import (single records removable)
- tab “Import”
- Section “Save configuration as preset”
- title
- description
- button “Save preset” => on save: “Configuration saved to path/file successfully.”
- Section “Import”
- Button “Import”
- Button “Prev”
- Preview import
- Section “Save configuration as preset”
- tab “Preset”
- General: perform database actions by DataHandler
- General: Sort lists alphabetically, if no other sorting is more appropriate.
- General: Add info overlays with detailed explanations for all fields that are not super obvious.
- General: Always return to the current tab when submitting a form.
- General: Are there UI elements available for progress / proceeding with a multistep form?
- General: Add more apply buttons to UI, maybe after each configuration section
- General: Apply button maybe with loading overlay, maybe with enabled and disabled state (depending on change in formular?)
- General: Prev and next buttons maybe contain titles of prev + next tabs
CLI
- Export command (https://forge.typo3.org/issues/84718)
- params
- filepath (required) (The path and filename to export to)
- filetype (required) (t3d/t3dz/xml)
- preset (optional) (The preset identifier to use for export)
- pid (optional) (The pid indicates root page of export)
- depth (optional) (Traversal depth, same as levels in export module)
- includeTable (optional, multiple) (if added, only records of these tables will be exported, same as in export module)
- includeRelated (optional, multiple) (if added, records of these tables will be exported only if linked by another record, same as in export module)
- includeStatic (optional, multiple) (if added, foreign keys will be kept in records of these tables, same as in export module)
- excludeRecord (optional, multiple) (if added, this particular record (table:uid) will be exluded from export, same as in export module)
- includeRecord (optional, multiple) (if added, this particular record (table:uid) will be included into export, same as in export module)
- dry-run (optional) (if added, the command returns all queries instead of executing them, same as “Export Preview” in export module)
- preset and other params can be used together: they get merged in a sensible and well documented way
- params
- Import command
- params
- file -> filepath (filetype gets auto-detected)
- preset (optional) (The preset identifier to use for import)
- pid (optional)
- depth (optional) (Traversal depth, same as levels in import module)
- includeTable (optional, multiple) (if added, only records of these tables will be imported, same as in import module)
- excludeRecord (optional, multiple) (if added, this particular record (table:uid) will be exluded from import, same as in import module)
- method [insert(default)/update/updateButIgnorePid] (optional) (former: updateRecords, ignorePid)
- forceUid (optional) (if added, uids of imported records will be forced)
- enableLog (optional) (if added, database queries get logged)
- dry-run (optional) (if added, the command returns all queries instead of executing them, same as “Import Preview” in import module)
- preset and other params can be used together: they get merged in a sensible and well documented way
- params
- General: Add shortcuts for each param
- General: Add summary log with configuration and statistics of import/export
- General: Make all module configuration fields available as CLI command params (such that an integrator can fully switch from TYPO3 backend to CLI for large TYPO3 instances which exceed PHP maximum execution time)
Wordings
- levels -> depth (see e.g. DeletedRecordsCommand)
- pageId -> pid (see e.g. DeletedRecordsCommand)
- No tree exported – only tables on the page. -> No pages exported – only records on this page.
Misc
- Check and maybe adjust indentation of entries in section “Preview import” / “Preview export”
- General: Use as many core functionality as possible
- General: Refactor to smaller yet better testable portions of code (e.g. small controllers)
- Check and maybe add missing TCA configurations for all existing TYPO3 entities to make them exportable, e.g. scheduler tasks
Questions
- Merge tabs “Page Tree” and “Files” into “Configuration” to reduce tabbing?
- File formats of preset and dump files still state of the art?
- How to best visualize TCA field “configuration”? If visualization for XML or JSON or YAML or FlexForm available, maybe switching from serialized object to that format is preferred?
- Is there any advantage of T3D over XML? If not, could we skip support?
- Is there any advantage of XML over JSON? Maybe having a scheme? If not, should we switch to the slim JSON format?
- How to best support a user-friendly copy & paste / instant preset, e.g. User A -> User B: “Please export your TYPO3 instance with this preset snippet and paste in here the export ..”
- How to best replace access field “public” by official TYPO3 Core access management
- Are there already TYPO3 Core UI elements for showing progress and handling a multistep form in a linear direction?
Reviews 0
No change resolves or names this issue in its commit message, and no comment links one.
Related issues 13
- Bug #83580GeneralUtility::xml2array() can't parse bigger files (> 10MB)
- Feature #84718Add CLI export command to impexp
- Story #85430Improve usability of impexp and add documentation
- Bug #92974Re-import of website with form partially fails
- Bug #93470XML export data does not include page translations.
- Task #94772Cleanup impexp extension
- Feature #94939Keep UIDs when installing distributions
- Feature #94940Manually excluded UIDs in EXT:impexp rely naivly on volatile UIDs
- Feature #94941Exports via export tools sometimes contain previous exports
- Bug #94972PHP 8 warnings: Undefined array key in EXT:impexp
- Bug #95007Checking the file checksum on export or import may fail due to sanitizers
- Task #95135Create an extension manual for EXT:impexp
- Task #95471Add more details to the manual of EXT:impexp
Files 2
History 27
Show 2 earlier messages
-
Alexander Nitsche changed Related to #85430 <span data-processor="relativetime" data-value="2020-10-09T09:15:51+00:00" > 10/09/20, 09:15 </span>
-
Alexander Nitsche changed Related to #84718 <span data-processor="relativetime" data-value="2020-10-09T09:23:56+00:00" > 10/09/20, 09:23 </span>
-
Alexander Nitsche changed description <span data-processor="relativetime" data-value="2020-10-09T09:28:40+00:00" > 10/09/20, 09:28 </span>
-
Alexander Nitsche changed attached impexp_export_module.svg attached impexp_import_module.svg description <span data-processor="relativetime" data-value="2020-10-09T09:42:16+00:00" > 10/09/20, 09:42 </span>
-
Alexander Nitsche changed description <span data-processor="relativetime" data-value="2020-10-09T09:43:26+00:00" > 10/09/20, 09:43 </span>
-
Alexander Nitsche changed attached impexp_export_module.png attached impexp_import_module.png description <span data-processor="relativetime" data-value="2020-10-09T09:49:59+00:00" > 10/09/20, 09:49 </span>
-
Alexander Nitsche changed
impexp_export_module.svg<span data-processor="relativetime" data-value="2020-10-09T09:51:14+00:00" > 10/09/20, 09:51 </span> -
Alexander Nitsche changed
impexp_import_module.svg<span data-processor="relativetime" data-value="2020-10-09T09:51:24+00:00" > 10/09/20, 09:51 </span> -
Alexander Nitsche changed Related to #72584 <span data-processor="relativetime" data-value="2020-10-09T09:52:25+00:00" > 10/09/20, 09:52 </span>
-
Alexander Nitsche changed
Related to #72584<span data-processor="relativetime" data-value="2020-10-09T09:52:58+00:00" > 10/09/20, 09:52 </span> -
Georg Ringer changed status
NewAccepted <span data-processor="relativetime" data-value="2020-10-09T10:06:20+00:00" > 10/09/20, 10:06 </span> -
Alexander Nitsche changed description <span data-processor="relativetime" data-value="2020-11-18T07:18:38+00:00" > 11/18/20, 07:18 </span>
-
Simon Gilli changed Related to #92974 <span data-processor="relativetime" data-value="2020-12-02T13:12:25+00:00" > 12/02/20, 13:12 </span>
-
Matthias Druske changed tracker
StoryBug TYPO3 Version1110 PHP Version 7.3 Tags multilingual translation page <span data-processor="relativetime" data-value="2021-02-09T08:44:31+00:00" > 02/09/21, 08:44 </span> -
Matthias Druske changed Related to #93470 <span data-processor="relativetime" data-value="2021-02-09T08:56:09+00:00" > 02/09/21, 08:56 </span>
-
Alexander Nitsche changed tracker
BugStory TYPO3 Version1011 PHP Version7.3Tagsmultilingual translation pageimpexp <span data-processor="relativetime" data-value="2021-02-09T09:21:38+00:00" > 02/09/21, 09:21 </span> -
Alexander Nitsche changed Related to #94939 <span data-processor="relativetime" data-value="2021-08-20T08:23:03+00:00" > 08/20/21, 08:23 </span>
-
Alexander Nitsche changed Related to #94940 <span data-processor="relativetime" data-value="2021-08-20T08:28:14+00:00" > 08/20/21, 08:28 </span>
-
Alexander Nitsche changed Related to #94941 <span data-processor="relativetime" data-value="2021-08-20T08:31:29+00:00" > 08/20/21, 08:31 </span>
-
Alexander Nitsche changed Related to #94972 <span data-processor="relativetime" data-value="2021-08-24T09:08:56+00:00" > 08/24/21, 09:08 </span>
-
Alexander Nitsche changed Related to #83580 <span data-processor="relativetime" data-value="2021-08-29T16:30:50+00:00" > 08/29/21, 16:30 </span>
-
Alexander Nitsche changed Related to #95135 <span data-processor="relativetime" data-value="2021-09-07T12:03:56+00:00" > 09/07/21, 12:03 </span>
-
Alexander Nitsche changed Related to #95007 <span data-processor="relativetime" data-value="2021-09-17T13:43:37+00:00" > 09/17/21, 13:43 </span>
-
Alexander Nitsche changed Related to #95471 <span data-processor="relativetime" data-value="2021-10-07T10:40:51+00:00" > 10/07/21, 10:40 </span>
-
Alexander Nitsche changed Related to #94772 <span data-processor="relativetime" data-value="2022-02-03T14:46:46+00:00" > 02/03/22, 14:46 </span>
-
Georg Ringer commented and changed status
AcceptedClosed <span data-processor="relativetime" data-value="2026-11-13T18:43:23+00:00" > 11/13/26, 18:43 </span>Closing as part of a backlog cleanup
This story collected ideas for a revised EXT:impexp in 2020. The parts that were taken up have been delivered through their own issues: the CLI commands (#84718), the extension manual (#95135, #95471) and the general cleanup (#94772). The remaining main idea, turning the presets into a fully qualified TCA entity with identifier, description, hidden/deleted and native access handling, was never picked up.
tx_impexp_presetsstill uses the original fields (public,item_uid,user_uid,preset_data) and the concept has not been touched since 2021.As the concept is six years old and the module and its surrounding APIs (TCA schema API, backend modules) have changed considerably since, it is no longer a useful work list. Remaining ideas should be filed as separate, focused issues, as it is already done for #94940, #94941 and #95007.
of course feel free to pick up the concept again if still interested or got time.
I take the freedom to dump some notes here I once wrote down in my personal notes: