Merge conflict Gerrit cannot merge this change. It needs a rebase onto main.

Review main Work in progress Quiet

96543[WIP][POC][TASK] Write and read records by the same rules and rights

Benjamin Kott CR 0 V -1 Impossible +206489 −22679 updated 10/23/26, 13:41

Commit message

Records are written and read by the same rules, and one access
concept decides who may do what. This change is about data being
written and read correctly:

* Writing: the DataHandler keeps its API and its hooks but no longer
  writes records itself. It translates the data map and the command
  map into typed commands for the new RecordWriter, which places
  records by position, processes values with a processor per field
  type and keeps translations, workspace versions, history,
  reference index and page caches in sync, in one transaction.
* Reading: the RecordReader reads records and their relations the
  way the RecordWriter writes them; the readers of the core
  (FormEngine, Record API, Extbase, reference index, workspace
  module) read relations through it.
* Access: every write has an actor (a backend user, the system or an
  application) that answers rights, and record voters decide each
  operation, refusing a record or a single field of it. Every read
  names its actor too; read voters of the core decide what it sees,
  and writing requires reading.

One goal of the rewrite is a behaviour of relations, 1:n and m:n,
that is defined in one place, documented in the code and pinned by
tests, instead of being spread over the DataHandler, the
RelationHandler, the DataMapProcessor and the workspace hook.

Documentation in this change:
* Feature "Write and read records by the same rules": the
  architecture -
  entry points, the context of a write, the pipeline from the plan
  to the events after the commit, relations, workspaces, storage
  and its cache, the read layer, results, access rules and rights,
  field processors, extension points, public API and internals,
  and a table of where each part of the former engine went.
* Important "Relations follow defined rules on writing": the
  specification of
  every relation kind with a matrix, the behaviour kept from the
  former engine, and the defects of the former engine this change
  fixes.
* Breaking changelogs for each change of behaviour or API.

Status of this proof of concept, kept up to date with each patch
set:

Done
* The DataHandler is a facade on the RecordWriter; its public and
  protected API and all hooks are kept, see the breaking changelog
  for what is removed.
* The behaviour is the one of the former engine, pinned by the
  existing functional tests, except for the fixes below.
* Queries per scenario are fewer than before, measured by
  QueryBudgetTest against a budget of the former engine plus 10%.
* When a relation defines foreign_sortby, that field is the stack of
  the children in every operation, publishing included.
* Raw columns of a new record never set uid, page, deletion,
  permission or t3ver_* (engine state is the internal Origin).
* The test_mm fixture names its MM_opposite_field correctly.
* Publishing only a child's deletion leaves a sibling's own change
  in the workspace.
* A single placeholder in an MM, inline or file field counts as a
  list of one; a placeholder no command creates is a violation.
  WriteResult::orThrow() throws WriteFailed with all violations; a
  site with pages, content, categories and inline children is set
  up in one change set.
* FormEngine accepts the value of a field a connected translation
  takes from its original.
* The relation graph is split: StoredRelationGraph reads, MmRows,
  RelationPersister, RelationCleanup and ParentIndex write.
* QueryBudgetTest also covers moving content, deleting, publishing
  and discarding inline children, restoring with children (rows
  read capped as well) and moving into a full set.
* DataScenarios write their site through one TestSiteTrait;
  reference index differences are declared and checked in
  tearDown().
* WriteResult tells copies and translations (copyOf(),
  translationOf()); uidOf() never hands out a version uid.
* Position::firstOn() with a page version or move pointer places
  on the live page.
* Live editing and stage rights are checked in the workspace of
  the write.
* A copy carries only the TCA fields of its original, and
  "required" applies to the fields the record type shows.
* The read cache forgets per uid what a write may change, sets are
  dropped per table; MM rows, child pointers and list cleanup write
  through the store; a rolled-back nested write leaves no cached
  rows behind.
* What goes with a version is defined once (VersionLifecycle,
  VersionCarriers);
  publish and discard map the same companions; publishing runs in
  steps (values, relations, stacks, counts), discarding has its own
  handler.
* Carried followers are move pointers copied from their live row,
  read with one query.
* StackReader reads and StackNumbering numbers the children of a
  parent, ParentIndex finds their parents; ParentField and Stack
  replace the parent array shape; finders ask the relation map.
* children() reads counted children through the RecordReader.
* The relation graph hands out ParentField, parents come from
  ParentIndex; StackNumbering::track() marks the places a workspace
  write gives.
* The DataHandler gets the relation graph injected, with the mirror
  refresh of connected translations.
* Commands the engine issues carry an internal Origin; records
  deleted later come from WriteOptions only; RecordReference
  follows by a mode.
* The changelogs describe the current building blocks and rules.
* Violations of records the engine writes along name the caller's
  command and keep the record they were found at; a translation
  asked for in the other mode than one the change set makes is
  reported as localize.duplicate.
* Engine methods are kept short (about 40 lines, depth 3), code
  lines below 140 characters except messages; class comments state
  rules; the behaviour kept from the former engine is a named list
  in the changelog.
* Labels added by this change use translation domains; test
  fixture comments carry no internal decision ids.
* The store notices writes around it through a DBAL driver
  middleware (WriteObserver); a statement on one table forgets that
  table, one it cannot place forgets all.
* Writing in a workspace the user has no access to is refused with
  access.workspace; an update whose version is refused reports the
  refusal.
* A record may be copied several times in one change set; each copy
  gets its own copies of what goes along, named copies are read
  with uidOf(); a copy or translation the change set makes already
  is resolved silently.
* Children copied into a translation are its own content, not
  hidden or labelled, and each translation gets its own.
* The engine reads and writes relations itself: stacks by their
  pointers (PointerChildren), MM rows through the store
  (MmRelations, MM_table_where and rows without tablenames
  included), child pointers (PointerArms) and relation lists
  (RelationItems); a symmetric record changing its pointer in a
  workspace versions both parents.
* The RelationHandler is unchanged from main; the engine does not
  use it. The reference index, FormEngine, the Record API,
  BackendUtility, the FileRepository, Extbase in a
  workspace and the workspace module read relations as the engine
  stores them through StoredRelations.
* Publishing takes carried child versions along itself; discarding
  a child's change recounts a parent with own change and drops the
  sibling places in it.
* The site listener has a feature switch; alike copies in one
  change set are one copy known by each name; updates of the delete
  field or version state reindex records pointing to the record.
* A translation's l10n_state sent with a value in one request
  decides, for FlexForm fields as well.
* Restore finds children deleted on their own by their pointer,
  the parent's list and history, and reads only their history.
* Restoring and discarding ask checkRecordEditAccess() as the
  DataHandler did. Updates and moves in a workspace are decided on
  the live record before its version is made or written, marked by
  AccessRequest::$throughVersion.
* preProcessFieldArray hooks place new records by "pid" again.
* File references whose file the field does not allow are
  reported: a write refuses those it would attach and warns when it
  deletes one the record already held.
* The record history is written inside the transaction, through
  BeforeChangeSetCommittedEvent; the reference index and caches
  follow after the commit.
* Atomic and SkipFailedFieldsAtomically refuse a change set whose
  records go to more than one database connection
  (storage.other_connection); the other policies take further
  connections into the transaction.
* A listener failing after the commit leaves the other changes
  announced and follow-ups written; its exception comes last.
* Listeners write through RecordWriter::followUp(): after all
  listeners, as the same actor in the same Context, with the same
  correlation id, and their changes join the result. System rights
  only on request, logged in sys_log; loops and chains deeper than
  ten are refused. Every write has a correlation id.
* The site listener writes the slug of a new site root as the actor
  who created the page, as the DataHandler did.
* Values that BeforeRecordPersistedEvent listeners, interceptors and
  postProcessFieldArray hooks change are processed and checked like
  submitted values.
* In a workspace, records of versioned tables become versions
  whoever writes, the SystemActor included; a backend user writes
  live only with live access of their groups.
* When write() throws, the DataHandler still logs the violations of
  the write and the changes it committed; errors that hooks log
  during a write are written to sys_log after it, so a rollback
  keeps them.
* write() without an actor writes as the backend user of a backend
  request; elsewhere an anonymous actor refuses every write.
* A Delegation next to the actor names for whom a write acts and
  why; voters and RecordChangedEvent listeners see it, the history
  takes the original user from it and every escalation is logged
  in one place.
* Extension guards are record voters on every write path: workspace
  publish rights and stages, redirect hosts, form definition
  grants, short URLs, content restrictions per column, scheduler
  tasks.
* The delete placeholder of a child moved to another parent in a
  workspace is indexed at the parent of its live row.
* Records deleted with their page are decided like live in a
  workspace; translations deleted along are carried by that
  deletion.
* A uid without a table in a group field of several tables warns
  with field.ambiguous_table on the API and Extbase paths.
* BeforeRecordPersistedEvent names the actor, the delegation and
  whether the record is imported; defaults of core extensions are
  its listeners: credentials of new backend users, owners of
  dashboards and notes, defaults of redirects, QR codes and short
  URLs.
* A publish is the same on every path: its history is written in
  its transaction, AfterRecordPublishedEvent, the log and one
  notification per write follow the commit as listeners; deleting a
  workspace or a stage is cleaned up by listeners.
* ChangeSetWrittenEvent announces a write once after its changes;
  PublishVersion carries the comment and recipients of a publish; a
  follow-up write may name its own Context.
* Side effects after the commit are RecordChangedEvent listeners on
  every write path: sessions end on a password change, the file
  index, slug redirects (URLs taken before the write), indexed_search
  (restored and moved records included), recent documents, template
  caches, page modification webhooks. indexed_search follows a
  publish; deletions in a workspace leave the live index until they
  are published.
* The facade keeps no static state for its error log and failed
  writes: errors are held back per outermost write run and written
  after it, the result of a failed write comes back with its
  exception. AbstractWriteInterceptor gives neutral defaults; a
  correlation id without subject casts to its scope prefix.
* Translations to follow are read once per table when they settle; a
  record's translation is found by its pointers and the record it
  goes after as one row; the facade reads the records a command map
  translates at once (QueryBudgetTest: reads of an update 206 ->
  59, rows read to localize 4126 -> 199).
* Each relation reader of the core has a pin of its read restrictions
  (deleted, workspaces, placeholders, language); the engine and the
  readers read the items of a relation field by one implementation.
* Every read carries an actor, the writing one for the reads of a
  write; the reader hides hidden and scheduled records when a
  ReadContext asks for it.
* Readers pass their ReadContext to StoredRelations; the engine reads
  the structure, complete, for the writing actor; read voters decide
  reads only where the caller asks, and a record they hide is unseen.
* The anonymous actor is refused with actor.anonymous; escalated
  follow-ups of a user switch keep the administrator; the slug
  service writes subpage slugs and redirects as follow-ups of the
  change, so writes from the CLI create redirects too.
* Slug changes are followed in a workspace too: subpages get versions,
  publishing creates the redirects from the former live URLs;
  WriteOptions::$followSlugChanges switches following off for a write;
  subpage versions made only for the slug change are carried and go
  with the page on publish and discard; imports do not follow slugs
  unless asked.
* A listener can mark a follow-up as derived from the change it
  follows (followUp(..., derived: true)); the engine records it in
  t3ver_carries and publishes or discards the derived versions with
  the last change that carries them, once its publish or discard is
  written.
* Changing the type of a record requires the fields of the new type
  on every path, field.required_by_type; a FormEngine reload holds a
  changed type back and shows it pending until the record is saved,
  for inline children as well.
* Extbase persists for the logged-in frontend user as a Delegation;
  the history names that user, escalated follow-ups included.
* Removing an Extbase object loaded in a translation deletes only
  that translation; the history names the translation.
* "Forgot password" writes through the RecordWriter as a task with
  escalation, backend and felogin; sessions end through the listener.
* The content types a backend layout allows bind every actor; values
  changed late during a move are decided as an update; WriteAccess
  asks the voters without writing.
* An ApplicationActor writes only the tables it declares (WriteTable)
  and is no admin; Extbase declares the tables of its objects unless a
  plugin declares, and writes as the backend user in the backend.
* A hard delete drops MM rows that fields of other tables hold to the
  record; changing the field that picks a FlexForm data structure
  reindexes; DetachTranslation makes a connected translation free.
* Core read voters decide what a backend user sees (page show and
  web mounts, tables, languages) and fe_group in the frontend; writing
  requires reading: a record the actor does not see is not found.
* FormEngine, record labels, the record history and the workspace
  comparison show related records the user may not read as "Not
  readable"; forms submit them unchanged, so saving keeps them.
* Writes are decided by the read voters on the record as the
  workspace of the write sees it; writing for a frontend user applies
  its frontend groups. Frontend groups no longer hide related records
  from a backend user in the Record API.
* In a workspace, reads and writes are decided on the page as the
  workspace sees it: its permissions, edit lock and type; subpages
  copied along likewise.
* Passwords and reset tokens stay out of the record history (TCA
  option 'history' => false); a wizard removes them from v14 entries;
  a reset by link ends all sessions of the user.
* A record new in the workspace is changed directly, without a second
  version; a new record placed after a translated one gets its own
  sorting value, its set is renumbered where no value is free;
  publishing a version of an edit-locked record is refused.
* A version keeps the live uid in every relation, translation
  pointers included; a translation moved along with its original is
  marked t3ver_change 4, also in a stack where it holds its place,
  and goes with it on discard; an upgrade wizard repoints the
  translation pointers of v14 versions.
* A child naming another parent in a FlexForm field by its pointer,
  or a new parent by its placeholder, is followed as in TCA fields;
  the reference index is read only for tables a data structure
  relates to.
* A parent new in the change set that lists a child places it
  itself; the child's pointer adds nothing.
* Every read names its actor, there is no default: a request reads
  as its backend user, else as its logged-in frontend user
  (FrontendUserActor, which may not write), else as an anonymous
  actor with the groups of the context; the engine reads as the
  actor that writes. The reference index reads for an actor too.
* Inline and file children take the language their parent is stored
  with, for every writer: a LanguageEffect in the settle phase, after
  the stacks and before translations follow, for parents created with
  children or joined by a new child, whose language changed or that
  got new children submitted; nested children follow, each once along
  a chain of parents.
* The language is bookkeeping like the pointer and the place of a
  child: no voters or interceptor; in a workspace a live child gets a
  version carried by the parent that holds only its language.
* Every defect of the former engine that this change records has a
  regression test in Regression/ of core DataHandling and Extbase
  Persistence, as pairs ...ByCore/...ByExtbase where Extbase is
  involved. Each one ran unchanged on 14.3 and main and failed there
  on the defect; the tests of defects this change does not fix are
  skipped.

Fixes of the former engine
Each one turns an expectation that the existing tests marked as
wrong, or that is otherwise documented as a flaw, into the correct
result:
* Values unique in their page are checked against the page a record
  is moved to, live and in workspaces, slug fields with
  "uniqueInPid" included. Before, the page the record came from was
  used, and slugs unique in their page were not checked on move.
* Publishing a version takes its values of fields with "unique" or
  "uniqueInPid" live. Before, the live values of input and email
  fields were kept and the change was dropped silently (#52070).
* A record and its versions count as one record when a value is
  made unique, so a record no longer gets a number appended to a
  value that one of its own versions holds.
* A workspace never writes the sorting of live inline children:
  versioning, moving or deleting a record in a workspace numbers its
  children anew only in the workspace, publishing takes the order
  live. Before, the live children were numbered anew right away.
* Inline children are ordered by their parent's stack, not by the
  page: a child translated, copied or synchronized along with its
  parent gets the sorting of its original and never shifts other
  records. Before, all following records on the page were shifted,
  children of other parents and, from a workspace, live records
  included, and copies in a workspace got other values than live.
* Inline children in FlexForm fields follow the rules of children of
  TCA fields; publishing takes their workspace order live, file
  references included (was: publishing numbered only children of
  TCA fields).
* Discarding a version drops its MM rows, inline MM included.
* Versions keep live uids in their relations, translation pointers
  excepted; publishing a symmetric inline change no longer relinks
  other records' relations to the removed version.
* Restoring a record restores the inline children of its FlexForm
  fields and their reference index.
* Localizing a record with inline MM relations and synchronization
  works (was: TypeError); copying a page with inline MM relations
  failed with a database error in TYPO3 v14 before #79712.
* A file field with l10n_mode exclude gets translations of the
  default references without label prefix, like inline fields.
* A child synchronized to a translation in a workspace is counted
  on the version of the translation instead of its live record.
* Published inline children keep the order they had in the
  workspace, also when they go live before their parent.
* A child synchronized to a translation of a translation in one go
  gets that translation as its translation source instead of the
  child of the default language.
* Children moving along with their parent keep their stack. Before,
  they got page-level sorting, and only the publish gave 1..n.
* Moving a child after its sibling numbers the stack 1..n. Before,
  it got a page sorting and the reference index went stale.
* Moving a record on its page in a workspace versions none of its
  children; publishing leaves unchanged children as live. Before,
  the whole child tree got versions and was renumbered.
* Deleting a child saves its parents without it: count or list
  and 1..n follow, in a workspace through versions. Before, the
  parent kept the deleted child.
* Versions copy their MM rows 1:1, inline MM included, and publish
  moves them as they are. Before, publishing reordered the other
  side, versions got no rows of inline MM fields, and publishing
  did not take inline MM rows live.
* Publishing keeps MM rows to records of other workspaces. Before,
  the versions of another workspace lost these relations.
* A resort from the opposite side of an m:n relation updates
  the reference index of the local side. Before, it kept the
  old order.
* A cleared category list is stored as '' like other lists.
  Before, it was stored as "0".
* enableCascadingDelete=false is honoured in workspaces. Before, the
  children got delete placeholders and publishing deleted them.
* A hard delete removes the MM rows of FlexForm fields. Before, they
  were left behind.
* Publishing a new record removes its workspace rows from the
  reference index. Before, children published along re-added them.
* New inline children get their place in their parent's stack only.
  Before, they were placed on the page and shifted other records,
  from a workspace also live ones.
* Publishing reads foreign_sortby of the record type. Before, an
  override in columnsOverrides lost the workspace order on publish.
* Deleting an inline child in a workspace versions its parent with
  the new count or list only, and a sibling only for a new place.
  Discarding the deletion keeps other changes of the siblings.
* Restoring the opposite side of an inline MM relation re-indexes
  the records on its local side. Before, they stayed indexed
  without it.
* Children copied along in a workspace take over the columns TCA
  does not define, like live copies. Before, they got defaults.
* A version records in t3ver_change whether it is an own change or
  carried by another change (children versioned along, cascades,
  stack places, parents that only lost a child). The workspace
  module lists own changes only; carried versions go live and are
  discarded with their cause. An editor's edit makes them own.
* Discarding a parent takes own changes of children only when they
  are deleted with it (enableCascadingDelete). Before, children of
  a field with enableCascadingDelete = false kept all their
  versions.
* The record history lists the related records of every relation
  kind before and after a change, reordering included, and names
  the parent of a new child; undo restores the list. Before, inline
  and file fields were recorded as a count, and undo took it for
  a uid.
* Records on a page are placed only among the records shown with
  them: same page, language (all languages count everywhere) and
  the grouping columns of the new TCA ctrl option
  sortingGroupFields (colPos for content). A workspace never moves
  live records. Before, any record of the page counted and a
  workspace could shift live records.
* New inline children with foreign_sortby are not placed on the
  page. Before, they could shift the records of the whole page.
* Without room in a workspace, the following live records of the
  set move up by carried move pointers that hold only their place
  (hidden in the module, published and discarded with their cause);
  never below 0 in unsigned sorting columns. Before, a workspace
  could shift live records.
* m:n relations keep both sides in order and counted. Before, a new
  relation sorted first on the other side and its count stayed.
* MM relations written from the local side get the first field of
  MM_oppositeUsage that the record type shows, with a warning if it
  shows several. Before, the fieldname stayed empty.
* Saving the local side keeps the rows an item has in several fields.
  Before, an item related in two fields lost both rows.
* A hard delete removes the record from the lists and counts that
  point to it and drops its MM rows on both sides of its own
  fields. Before, it left dangling uids and wrong counts.
* A child left out of its parent's value is deleted, its connected
  translations with it; one moved or listed elsewhere stays. Before,
  it stayed attached and was no longer counted.
* Moving a list child after a sibling sorts the parent's list; in a
  workspace the parent gets a version with the new list only.
* Restoring a child puts it back where it hung, kept with the
  history of its deletion, list and FlexForm children included.
* Publishing counts a parent's children from its live stack and in
  the parent's versions of other workspaces.
* Publishing or discarding a record takes the versions of list and
  FlexForm children it removed or deleted along; a child moved to
  another parent in a workspace goes live with it.
* A later publish of another workspace takes only its own changes
  of a stack (places an editor gave, new children); lists and MM
  rows of other workspaces follow a publish. Before, the last
  publish restored the order its versions were made with.
* Versions in other workspaces of a record deleted live stay as a
  conflict: the workspace module shows them, publishing them is
  refused, undelete restores the record as new in the workspace.
  Before, they were discarded.
* Records created with a parent through inline MM or intermediate
  tables go live and are discarded with it.
* In a workspace, moving, deleting or re-sorting a parent whose
  children have no workspace support is refused.
* The reference index reads the configuration of the record type.
* RecordStore caches the rows a write run reads; a write forgets
  the rows it may change, writes around the store are noticed by
  the WriteObserver middleware.
* Reference index updates of soft deletes, of records pointing to
  deleted ones and of local MM sides are collected and batched;
  the engine skips MM sortings and child pointers that hold their
  value already.
* Extbase writes no relation for empty properties of new objects;
  an Extbase QueryBudgetTest pins the cost of persisting a blog.
* The writer reads records and stacks of children through an
  internal RecordReader (DataHandling\Read), in one query per
  stack. Hidden records on the other side of m:n relations are
  counted too.
* ReadContext carries the read side of the Context: workspace,
  language with fallback chain and overlay type, visibility, time,
  user groups. TranslatedReferences decides whose relations a
  translated reference field has, as the important changelog
  defines it; the readers below use it.
* The writer writes where its WriteContext says (WorkspaceAspect
  of the Context); the Actor keeps who writes and what they may do.
* Connected translations hold no relations of their own: read-only
  mirror MM rows, no reference index rows, a read-only copy of the
  value of their original, kept up to date; submitted values are
  refused (translation.connected_reference), opposite-side writes
  go to the original; copies to a language keep their MM rows.
  Before, a translation on the local side of an MM relation started
  without relations and could hold its own.
* tt_content.pages and records allow language synchronization.
* Readers use the relations a connected translation takes from its
  original: Record API (RelationResolver), FormEngine (read-only,
  hint), BackendUtility::getProcessedValue; targets are shown in the
  language read, excluded fields as the original relates them.
* References inside FlexForm fields of connected translations are a
  read-only copy of the original's value, with read-only mirror MM
  rows like TCA fields.
* Extbase reads, queries and drops the references of connected
  translations by the rule: relations of the original, joins and
  contains() through l18n_parent, changed references not written.
* FlexForm reference copies of translations follow the original;
  the wizard looks into FlexForm fields and workspace versions.
* FormEngine links read-only references to the original.
* Connected translations keep read-only MM rows mirroring their
  original's (SQL and TypoScript joins keep working); mirrors are
  not counted, not stacked on the other side and not indexed.
* l10n_mode exclude shows the original's related records in the
  language read, as before.
* tt_content.pi_flexform allows language synchronization.
* References a translation takes are derived: publishing never
  takes them from a version, they are recomputed from the live
  original. The wizard never widens access, keeps equal MM rows as
  mirror and switches differing FlexForm fields to own values.
* A translation of a translation gets the parent of its source.
* RecordWriter::write() takes an optional Context: it overrides
  only the aspects it sets, the current Context (read at write
  time) decides the rest; the DataHandler passes its user's
  workspace and never mutates the global Context. Context can be
  built with aspects and tells which are set (isAspectSet()).
  ReadContext and WriteContext hold the core aspects, no copies.
* A connected page translation resolves the relation fields it
  takes from its original in the rootline.
* Readers list only real relations on the other side of an MM
  relation, not the mirror rows of connected translations; mirrors
  follow relations written from the other side and the original's
  rows, not its count.
* The wizard keeps a synchronizable FlexForm value as own value
  whenever it differs from the original.
* A child left out of its parent's value keeps its place in the
  history of its deletion; restoring puts it back there, also when
  its parent is restored; restoring without that entry warns.
* A child copied after a sibling joins that stack; copied elsewhere
  it is appended to the stack of its original's parent.
* A relation record moved after a sibling reorders its own stack
  (intermediate: the end with foreign_selector).
* A parent losing a FlexForm child gets a shallow carried version;
  discarding follows enableCascadingDelete in FlexForm fields;
  FlexForm stacks with their own foreign_sortby publish places.
* Copied relation records take their own place at the end of the
  other parent's stack; a copy keeps every MM row of its original
  in all MM_oppositeUsage fields.
* Hiding a record reindexes the records pointing to it; inline
  children moved to another parent in a workspace are read and
  indexed there; publish history records uid lists;
  cleanup:versions reports versions of records deleted for good.
* A version of another workspace keeps its own value of a single
  valued relation when a publish changes it; a record deleted for
  good leaves the FlexForm fields of other records.
* Publishing only a child's deletion takes only its effect on the
  parent live (recount, list, FlexForm list); the parent's own
  change stays in the workspace and its stages.
* Each child deleted because it was left out of its parent's value
  is logged; the rollback preview names the children a rollback
  deletes.
* The workspace module names the children moved in a list.
* Versions of a record deleted live for good stay as a conflict;
  undelete in the workspace restores it as a new record.
* Extbase persists through the RecordWriter: one change set per
  persistAll() as the ApplicationActor, in the workspace of the
  Context; uids, events and clean states follow the write. Related
  records Extbase never loaded (hidden, scheduled, restricted) are
  kept. Before, Extbase wrote live rows in a workspace preview.
* FailurePolicy::SkipFailedFieldsAtomically: refused values are
  left out, a failing record rolls back the change set.
* Actor::placesAnywhere(): the application ignores rootLevel and
  page types; WriteOptions::recordsHistory for own histories.
* JSON text in JSON columns is stored as JSON; the writer keeps
  null in json fields, the DataHandler still stores [].
* Upgrade wizard connectedTranslationRelationsMigration and
  command cleanup:translationrelations for existing translations.
* A value equal to the stored child count of an inline or file
  field leaves the children as they are, so extensions writing a
  whole FlexForm or record back delete nothing; the import writes
  only changed FlexForm values.
* Engine classes are @internal; the API classes carry an example
  each. PlannedCommandHandler and RecordReader are internal for now;
  FieldProcessor, RecordVoter and WriteInterceptor extend.
* Relation fields of a record are enumerated in one place,
  FlexForm sheets included; which fields relate to a table is
  answered by one map built per TCA, used by the handlers.
* Extbase loads relations stored as a list of uids in the order of
  the list (was: in the order the database returned them, so
  PostgreSQL shuffled the list when it was written back).
* A child that names its parent by its own pointer joins the
  parent's stack: the parent counts it, the stack is numbered, live
  and through carried versions in a workspace; a child changing its
  pointer leaves its former parent counted (was: stale counts, the
  child took a place on the page).
* Publishing a child's pointer change counts the live parents it
  left or joined again and takes the sibling places along when the
  parent keeps its own change; a parent saved with a child of
  another parent takes it, the other parent counts and numbers
  anew, live and in workspaces (was: stale counts, gaps).
* DataHandler::checkValue() keeps the children of a stored count
  of an inline or file field (was: read as a uid list, which took
  a child of another parent).
* A new root page gets its site configuration from a listener of
  the RecordChangedEvent, for DataHandler, RecordWriter and Extbase
  (was: a DataHandler hook, now removed).
* "required" holds for the fields the record type shows (CType,
  doktype, FlexForm data structure of the type); unknown types are
  checked as type 0/1 like FormEngine (was: all submitted fields).
* ext:redirects rebuilds its cache from a RecordChangedEvent
  listener for every write path, once per host (was: a DataHandler
  hook that rebuilt all hosts for a new redirect).
* Typolink soft references keep t3://page links and their
  parameters through export and import (was: bare page uid).
* FieldProcessor is a public extension point with a priority;
  extension TCA types are in the schema (was: dropped). RecordVoter
  gets AccessRequest values and an AccessContext; every actor holds
  rights it is asked for (Actor::holds(Right), DeclaredRights).
* Core guards are record voters asking the actor for rights: page
  TSconfig, the file index, file mounts, system maintainers and sudo
  mode hold for every write path (was: DataHandler hooks); Extbase
  references files in declared folders, its uploads by default; the
  access hooks of the DataHandler are deprecated.
* Synchronized FlexForm fields keep the inline children of the
  translation when they follow a change of the original (was: the
  original's children were taken over); versions a translation gets
  only for following its original go live and are discarded with
  it (was: discarding kept them, with the values synchronized from
  the discarded change).
* Pre-existing long methods the change touches are split into
  steps (e.g. BackendUtility::getProcessedValue, WorkspaceService,
  GridDataService, Import), pinned by tests.
* Extbase keeps inline list children it never loaded (hidden,
  restricted) when it writes the list (was: dropped them from the
  list).
* The reference index is written per table and workspace in one
  batch, inline children of a batch are looked up at once, rows
  are dropped per table (was: one record at a time); an update
  that changes no indexed field is not reindexed.
* A parent saved only because a child joins or leaves its stack is
  decided with the child: metadata of a file is created, translated
  and deleted without a right on sys_file.
* The record history names the administrator who switched to the
  backend user who wrote (originaluserid), as the DataHandler did;
  RecordChangedEvent tells it in $onBehalfOf->impersonatedBy.
* field.no_processor stays out of sys_log like field.unknown.
* Equal new value objects in one persistAll() are one record, as
  on main: the second shares the placeholder of the first.
* RelatedRecord names a record with its table, so a group field of
  several tables takes placeholders and uids of any of them; an
  unknown placeholder in it is a field error.
* The frontend FileRepository reads references through
  StoredRelations, so the workspace preview shows a moved reference
  at its new parent, as the backend does (was on main: every reader
  showed the old parent until publishing).
* $datamap holds translations kept in sync once they are written,
  with the values of the synchronized fields.
* In a workspace, the hooks of a translation kept in sync are called
  as for a record of the data map: the live uid up to its version,
  made by the "version" command, the uid of the version after it.
* A WriteInterceptor sees an update in a workspace for the live
  record, before its version is made, and once only.
* A parent version carried by the changes of several children stays
  until the last of them is published or discarded; discarding one
  counts or lists its child in the kept version again, inside
  FlexForm fields too.
* The rollback preview leaves out children the workspace deletes.
* A write runs flat and linear: each command of the caller is a
  segment, planned in rounds and then written; a frozen round takes
  no new work, and no command is visited twice. What depends on
  earlier writes is collected as effects and settled once at the
  end: stacks numbered once per parent with one history and log
  entry, counts, orders, translations that follow their original.
* A relation field is written in two steps: RelationDiff computes
  the change from the submitted and stored values and plans its
  outcome, RelationWrite only applies it.
* A command map is one change set in one transaction; the paste
  values are a segment after their command, publishing a deletion
  plans the live deletion, and no DataHandler runs nested in another.
* WriteInterceptor has fixed points in the order of the former
  hooks (admit, decideAccess, versioned, beforePersist,
  afterPersist, beforeCommand, afterCommand, afterLinking,
  beforeFlexFormMerge, beforeTranslate) with an
  InterceptedRecord and a public WriteReason; the version of an
  update is made in the same change set.
* The life of a version is in one place: VersionMarks writes
  t3ver_change, VersionCarriers derives who carries a version, and
  publish and discard follow one rule: the last carrier takes it
  along, otherwise only its effect is applied or reverted.
* A move, a new record or a copy into a full set in a workspace is
  refused before any version is made when a follower cannot be
  carried.
* Publishing some of the changes a workspace made in a stack takes
  only their effect live; place versions another change still
  carries stay in the workspace.
* Tests follow the structure: unit tests for the plan, its effects,
  stack settlement, version marks and carriers, the access gate and
  the interception points; functional pins for stack bookkeeping,
  the hook order, a command map as one change set and the carrier
  rule on publish.
* One workspace view decides which rows a workspace sees, owns,
  may write and counts (WorkspaceView, RowView, Reach); relation and
  versioning code compares no workspace ids itself (guarded by a
  test). RelationFollowers names who follows a change.
* Every command has an origin (RecordCommand::why()); what an origin
  decides is one row of OriginKind::policy(), and every access
  decision goes through the AccessGate.
* Parents and stacks are read through ParentIndex and StackReader
  and written through StackNumbering only; pointer arms are read
  from one PointerArms value.
* A deleted relation record is no longer counted on the other side
  in a workspace; a save carried by another change is decided with
  it in a workspace too; MM mirror rows of translation versions
  follow live writes; a move whose record was deleted live takes
  what it carried along when discarded.
* Mirror rows of connected translations follow relations written
  from the other side in a workspace, as rows of that version.
* A record deleted for good leaves the lists of workspace versions
  as well.
* Restoring a record as new in a workspace takes the versions of
  its translations along.
* The reference index has no rows to a soft-deleted record on any
  path of the TCA, through a list as through an MM table; restoring
  the record or deleting it for good finds the lists that hold it.
* In a workspace, a live record that gains a relation record of an
  intermediate table or a symmetric relation gets a version with
  its new count, carried by the relation record, as one that loses
  it does; this was wrong in FlexForm fields and in the TCA alike.
* All rows of one write are stamped with the time of the date aspect
  of the Context, as the DataHandler took it from the request; a
  Context passed with a date aspect sets it.
* Defaults of a new record are resolved by the DefaultValuesResolver
  only, and the access check of a new record sees them: an omitted
  CType, colPos or language cannot bypass content restrictions or
  the language check.
* A copy of a new child of a table without language support names
  that child in its origUid field, as TYPO3 v14 does since #111070.
* An MM row names a table in tablenames only if the field allows
  that table, as the RelationHandler reads it since #75145.
* The shadowed state of redirects is updated by a listener after the
  commit, for every write path; the frontend preview highlights
  changes of their own only.
* Publishing in segments numbers a stack only once a child of it is
  published, so a stack keeps its places on every database.
* An empty password is stored as empty string unless the column is
  nullable; the update log entry names its history entry; linking
  that changes a record's relations sets its change time; a parent
  deleted with a translated child keeps its count; fields of a
  connected translation without stored state follow the original.
* Extbase keeps the time zone of dates and reads the children of a
  translation in a workspace from the translation; the conformance
  suite of main runs both twins where the patch fixes Extbase.

Kept on purpose as before
* Inserting, sorting and deleting children renumber their stack.
* Defaults are resolved by the DefaultValuesResolver as on main:
  page TSconfig before user TSconfig, the value of the record type
  before that of the field, for all fields of a new record; a
  version checking a unique value ignores the translations of its
  live record, as on main.
* Inline children created together with their parent are copied
  into the translations of the parent from the submitted values.
* The rules are written down in the important changelog
  "Relations follow defined rules on writing".

Changed on purpose, see the breaking changelog
* checkValue() writes relations right away and leaves out
  references to records not written yet.
* addRemapAction() calls its callback right away.
* Values set by postProcessFieldArray hooks are processed and
  checked; columns without TCA need a passthrough field.
* Extbase refuses a persist run whose records go to more than one
  database connection.
* Extbase loads the m:n relations of a connected translation from
  its original; the conformance suite pins that.
* The publish commands of a command map are published together at
  the first; hooks of the later commands see their records live.

Open
* Decided, not built yet: the upgrade wizard and the translation
  relation finder read the FlexForm data structure of the record
  type (columnsOverrides), as the ParentIndex does.
* Known gaps:
  - the reader applies access, visibility and time only when asked
  - the relation classes build their reference index updater
    without an actor; these index reads run as the system actor
  - the language of inline children is not aligned for an existing
    child that joins a new parent by its pointer, nor for the
    children of a copy; main aligns both
  - Extbase: tables without TCA still write SQL directly; Extbase
    clears page caches through its CacheService as before, not by
    the rules of the DataHandler
  - old workspace data with version uids in relation fields is not
    repaired on publish
  - the history of Extbase is written after the commit
* Remaining DataHandler hooks of the core: the scheduler form
  converter and open documents
* API: logical RecordChangedEvent; RecordReader, ReadContext and
  StoredRelations public later
* Decided, not built yet, from the review against the vision paper
  and the access concept:
  - dry runs of whole change sets; asynchronous runs with progress;
    a durable outbox for side effects after the commit, with retry
  - serializable, versioned events; an audit header per change set
    with idempotency keys; an event before a run;
    waiting for projections
  - optimistic locking by the latest history entry; an upsert
    command; write-protected fields in the schema
  - preview links and password resets as token-bound actors
  - a load test of Extbase with 10.000 objects
* Reading (audited): every reader moved to StoredRelations applies
  the restrictions it applied on main; in the frontend workspace
  preview a child moved to another parent is still listed at the
  old one by TypoScript CONTENT/select and plain SQL readers
* Verification: full functional suite on all databases (CI)
* Forge issues, changelog numbers and the split into reviewable
  steps

Verified
* This patch set, on main 429bf552fc5 (sqlite): CGL, PHPStan,
  unit tests of core, Extbase, backend, workspaces and redirects
  (11572); functional tests of DataScenarios (2932, 1 skipped) and
  of DataHandling, Database, Extbase, workspaces, redirects, info
  and the tests main added since 8dfb324534e (4466, 4 failing on an
  open question about the order of copies of excluded children);
  the full suite is left to CI.
* Query budget, measured on main e4f361cbd44 (October 2026) and on
  this change with the same scenarios, 50 records, admin, sqlite,
  reads/writes:
  create 700/155 -> 56/156, update 852/150 -> 256/151, localize
  1999/209 -> 355/203, copy page 3499/450 -> 686/345, move page
  28/12 -> 20/13, delete page 2466/544 -> 547/333, update in a
  workspace 4606/500 -> 1007/351, publish 2450/550 -> 760/352,
  delete page in a workspace 97/15 -> 35/13. Counts are the same on
  MariaDB 10.4 and PostgreSQL 10 (without sqlite's lock statement)
  and linear at 200 records; wall time of these nine -50% (sqlite),
  -51% (MariaDB), -57% (PostgreSQL). Main updates the reference
  index per record, re-reads rows per check and resolves every
  child through RelationHandler; the writer batches the index, reads
  a row once per write run and reads relations itself.
* Inline children in workspaces cost more than on main: deleting a
  child versions the parent's count and the siblings' places
  (1360/710, was 600/200), publishing and discarding take them
  along, and moving into a full set carries 48 followers (138/152,
  was 51/8, where main shifted live records). Restoring 50 contents
  reads 1801 rows (main 1701) without reading unrelated history.
* PostgreSQL 10 and MariaDB 10.4: core, Extbase, backend and impexp
  suites green on the relation takeover; the full suite runs in CI.

Test this change

Run a command in your clone of the TYPO3 Core. Then try what the change does and vote on Verified in Gerrit. How to review a patch

Check out patch set 30
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/43/96543/30 && git checkout FETCH_HEAD
Cherry-pick onto main
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/43/96543/30 && git cherry-pick FETCH_HEAD

Review

  • Code review CR 0 Needs a vote
  • Verified V -1 Blocked
    • core-ci -1 CI

Files 3281

Show 3261 more files

Review log 3060

Show 1835 earlier messages
  1. Benjamin Kott uploaded patch set 1 10/08/26, 21:32

  2. core-ci voted on patch set 1 V -1 10/08/26, 21:40 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110530

  3. Benjamin Kott uploaded patch set 2 V -1 10/08/26, 22:33

  4. core-ci voted on patch set 2 V -1 10/08/26, 22:41 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110531

  5. Benjamin Kott uploaded patch set 3 V -1 10/08/26, 23:18

  6. core-ci voted on patch set 3 V -1 10/08/26, 23:26 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110532

  7. Benjamin Kott uploaded patch set 4 V -1 10/09/26, 07:07

  8. core-ci voted on patch set 4 V -1 10/09/26, 07:20 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110557

  9. Benjamin Kott uploaded patch set 5 V -1 10/09/26, 11:33

  10. core-ci voted on patch set 5 V -1 10/09/26, 11:45 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110593

  11. Benjamin Kott uploaded patch set 6 V -1 10/09/26, 18:28

  12. core-ci voted on patch set 6 V -1 10/09/26, 18:48 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110677

  13. Benjamin Kott uploaded patch set 7 V -1 10/09/26, 21:26

  14. core-ci voted on patch set 7 V -1 10/09/26, 21:50 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110728

  15. Benjamin Kott uploaded patch set 8 V -1 10/10/26, 04:25

  16. core-ci voted on patch set 8 V -1 10/10/26, 04:34 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110744

  17. Benjamin Kott uploaded patch set 9 V -1 10/10/26, 05:43

  18. core-ci voted on patch set 9 V -1 10/10/26, 05:59 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110754

  19. Benjamin Kott uploaded patch set 10 V -1 10/10/26, 06:55

  20. core-ci voted on patch set 10 V -1 10/10/26, 07:03 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110758

  21. Benjamin Kott uploaded patch set 11 V -1 10/10/26, 07:53

  22. Benjamin Kott updated the commit message in patch set 12 10/10/26, 07:54

  23. core-ci voted on patch set 11 V -1 10/10/26, 08:06 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110764

  24. core-ci voted on patch set 12 V -1 10/10/26, 08:12 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110765

  25. Benjamin Kott uploaded patch set 13 V -1 10/10/26, 09:03

  26. core-ci voted on patch set 13 V -1 10/10/26, 09:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110777

  27. Benjamin Kott uploaded patch set 14 V -1 10/10/26, 09:53

  28. core-ci voted on patch set 14 V -1 10/10/26, 10:03 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110786

  29. Benjamin Kott uploaded patch set 15 V -1 10/10/26, 12:25

  30. core-ci voted on patch set 15 V -1 10/10/26, 13:26 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110803

  31. Benjamin Kott uploaded patch set 16 V -1 10/10/26, 13:30

  32. core-ci voted on patch set 16 V -1 10/10/26, 14:32 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110804

  33. Benjamin Kott uploaded patch set 17 V -1 10/10/26, 16:47

  34. core-ci voted on patch set 17 V -1 10/10/26, 16:55 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110805

  35. Benjamin Kott uploaded patch set 18 V -1 10/10/26, 21:42

  1. core-ci voted on patch set 18 V -1 10/10/26, 21:54 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110820

  2. Benjamin Kott uploaded patch set 19 V -1 10/11/26, 01:49

  3. core-ci voted on patch set 19 V -1 10/11/26, 02:02 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110824

  4. Benjamin Kott uploaded patch set 20 V -1 10/11/26, 11:59

  5. core-ci voted on patch set 20 V -1 10/11/26, 12:13 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110850

  6. Benjamin Kott uploaded patch set 21 V -1 10/11/26, 13:17

  7. core-ci voted on patch set 21 V -1 10/11/26, 13:31 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110856

  8. Benjamin Kott uploaded patch set 22 V -1 10/15/26, 15:48

  9. core-ci voted on patch set 22 V -1 10/15/26, 15:59 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/111219

  10. Benjamin Kott uploaded patch set 23 V -1 10/15/26, 16:53

  11. core-ci voted on patch set 23 V -1 10/15/26, 17:09 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/111243

  12. Benjamin Kott uploaded patch set 24 V -1 10/16/26, 11:31

  13. core-ci voted on patch set 24 V -1 10/16/26, 11:43 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/111352

  14. Benjamin Kott uploaded patch set 25 V -1 10/16/26, 18:24

  15. core-ci voted on patch set 25 V -1 10/16/26, 18:41 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/111391

  16. Benjamin Kott uploaded patch set 26 V -1 10/16/26, 22:55

  17. core-ci voted on patch set 26 V -1 10/16/26, 23:13 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/111402

  18. Benjamin Kott uploaded patch set 27 V -1 10/18/26, 02:17

  19. core-ci voted on patch set 27 V -1 10/18/26, 02:33 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/111491

  20. Benjamin Kott uploaded patch set 28 V -1 10/18/26, 08:41

  21. core-ci voted on patch set 28 V -1 10/18/26, 09:00 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/111509

  22. Benjamin Kott uploaded patch set 29 V -1 10/18/26, 14:56

  23. core-ci voted on patch set 29 V -1 10/18/26, 15:12 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/111538

  24. Benjamin Kott uploaded patch set 30 V -1 10/23/26, 13:17

  25. core-ci voted on patch set 30 V -1 10/23/26, 13:41 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112305