Issue Feature Needs Feedback

#110808Extension Scanner: the mandatory restFiles reference makes rules impossible for undocumented API removals

Olivier Dobberkau 09/26/26, 20:51 updated 11/13/26, 09:23

Description

This is a request for a decision, not a patch proposal. I ran into a structural limit of the Extension Scanner while measuring, and I would like to know which direction the team considers acceptable before writing anything.

The constraint

A scanner rule must reference a changelog file, and this is enforced twice:

  • Load time — AbstractCoreMatcher::validateMatcherDefinitionKeys() throws InvalidArgumentException 1500496068, “Each configuration must have at least one referenced restFiles entry”, when restFiles is empty.
  • Display time — UpgradeController resolves every entry with Finder->name($fileName) against EXT:core/Documentation/Changelog and throws RuntimeException 1499803909 when the result is not exactly one file.

So a rule can only exist for an API change that has its own .rst file in the core. There is no placeholder, no external link, and no way to express “this was removed, and it is not documented anywhere”.

Why that is a practical limit, measured

The core’s own changelog metadata says that for v14.0, of 99 breaking changes 34 are FullyScanned, 5 PartiallyScanned and 58 NotScanned. This ticket is about a structural reason why that ratio is hard to improve.

I diffed the public API of typo3/cms-core v13.4.35 against v14.3.7 and generated candidate scanner rules from the diff: 636 rules, of which 590 are not covered by the shipped rule set (2424 rules in 13.4.35). Then I tried to give each one a restFiles reference by matching it against a corpus of TYPO3 changelog entries (3824 entries; 787 of them in the 13.0–14.3.x series).

Result for the 590 rules count share
symbol matched verbatim in an entry 30 5.1 %
only the class name matched 321 54.4 %
nothing matched at all 239 40.5 %
usable as a restFiles reference 45 7.6 %

The 321 class-only hits are mostly coincidence. In a sample of six, five did not mention the member anywhere in the matched document — the class simply appears in an unrelated entry. Entering those as restFiles would show users a wrong explanation, which is worse than none. Relaxing the rule to “class matched and the member name appears somewhere in the same document” raises the yield to 93, but also produces mismatches. The defensible statement is a range:

> 45 to 93 of 590 rules — 8 to 16 % — can be given a changelog reference mechanically.

For the 239 with no match I checked whether they might be documented in an older series (deprecated in v12, removed in v14). Across all 3824 entries exactly 4 more appear, and those are coincidental too.

> 235 of 590 rules — 39.8 % — describe API removals that appear in no changelog entry at all.

Limits of the measurement

  • The corpus is from 16 July 2026, v14.3.7 is from 8 September 2026. Entries added after that are missing, so the range is a lower bound.
  • The corpus was converted to Markdown, so the exact .rst file names are not available. The measurement answers “is there an entry”, not “which file”.
  • The assessment of the class-only hits is a sample: 6 of 321 were read.
  • Only typo3/cms-core was diffed for the rule generation.

Three ways forward

1. Contribute the 45–90 rules that can be given a hand-checked reference. Honest, small, immediately doable — and it covers roughly a sixth of the gap.
2. Change what restFiles requires. For instance: allow a rule without a reference and let the UI say “no changelog entry”; or allow one shared umbrella entry for undocumented removals; or make the display-time lookup non-fatal. This is the decision I am asking about.
3. Write the ~235 missing changelog entries. Arguably the correct answer, but a different order of magnitude, and not something one person does on the side.

I have no preference to push. Option 3 may well be what the team wants, and then the useful contribution of this measurement is simply the number: 235.

A side observation

The display-time check throws RuntimeException 1499803909 when a referenced file is not found. A renamed or removed .rst therefore does not degrade to “link missing” — it makes the whole scan result view fail. That may deserve its own ticket regardless of what is decided here.

What I can contribute

The diff tooling and the mapping script exist and the measurement is reproducible. I am happy to hand over the data, to hand-check the 45–90 references, or to write the patch once there is a direction.

Reviews 0

No change resolves or names this issue in its commit message, and no comment links one.

Fix this issue

Start the commit message of your patch with this. Add what the patch does, and the versions it goes to in "Releases:". How to create a patch

Commit message
[FEATURE] Extension Scanner: the mandatory restFiles reference makes rules impossible for undocumented API removals Resolves: #110808 Releases: main

Files 7

History 6

  1. Olivier Dobberkau commented and changed attached lab-api-diff.sh attached lab-rulegen.sh attached lab-restfiles-map.py <span data-processor="relativetime" data-value="2026-09-26T20:58:33+00:00" > 09/26/26, 20:58 </span>

    How the measurement was produced

    The three scripts are attached. Two caveats up front: their inline comments are in German, and they were written for one lab, not as a product — I am happy to translate or clean them up if anyone wants to build on them. The method matters more than the scripts, so here is what each one does.

    1. The API diff — lab-api-diff.sh

    lab-api-diff.sh <from> <to> [package …|--all], for example lab-api-diff.sh v13.4.35 v14.3.7 --all over the 34 system extensions.

    • Downloads both versions as dist archives from Packagist.
    • Reads autoload.psr-4 from each composer.json to find the namespace and the source directory, instead of assuming Classes/. That is not pedantry: typo3fluid/fluid uses src/, and the first version of the script skipped it silently.
    • Indexes both trees into a code graph and asks for the exported methods with signature, return type and file path.
    • Classifies into removed / signature changed / return type changed, and separates public from internal@ by looking at the docblock above the class declaration.
    • Writes one JSON per package plus a summary table.

    Two honest limits. It is method-level: removed classes only show up through their methods, and constants, properties and interfaces are not measured at all — so 1287 is a floor, not a total. And the graph step is not essential to the method: the query is just “give me the exported methods with their signatures”, which any static analyser over nikic/php-parser can answer. I used an off-the-shelf indexer (codebase-memory-mcp) because it was at hand.

    2. Rule generation — lab-rulegen.sh

    lab-rulegen.sh <api-diff.json|--all>. It must run against an installation of the target version, because it resolves every symbol by reflection through the real autoloader — class aliases, traits and inherited methods cannot be resolved reliably from the outside. As attached it shells out via ddev exec php; that is the one line a reviewer would adapt.

    finding result
    resolvable and signature compatible discarded as a false positive
    class gone ClassNameMatcher
    class there, method gone MethodCallMatcher / MethodCallStaticMatcher
    resolvable, signature narrowed MethodArgumentDroppedMatcher

    Both halves of that filter are needed. A pure existence check classified two real breaks as false positives: FlexFormService survives through a class alias but went from three parameters to one.

    Calibration against the hand-written rules: of the 35 hand-written v14.0 method rules in TYPO3\CMS\Core, the generator reproduced 34. The single outlier turned out to be a defect in the hand-written rule, not a gap in the generator — that is #110807. Measured precision over 157 symbols: 152 correct, 96.8 %.

    The generated file carries a header saying restFiles still has to be added before anything can go into the core. That is how I ran into the subject of this ticket.

    3. The changelog mapping — lab-restfiles-map.py

    lab-restfiles-map.py --corpus <changelog-root> --rules <rules.json> [--out report.json] [--all-series] — the script behind the numbers in the description. The corpus is one directory per series (13.0, 13.4, 14.0 …); --rules is a JSON list of {matcher, key}.

    Three hit levels per rule:

    • member — class and member appear verbatim in one entry
    • class — only the class name appears; sufficient for ClassNameMatcher, where the class is the symbol, usually a coincidence for everything else
    • none — no entry mentions the class

    By default it searches only the series that can plausibly document the measured jump (v13 announces, v14 removes). --all-series is the cross-check I ran over all 3824 entries: it added exactly 4 further hits, all coincidental, so the restriction costs nothing.

    What a reviewer would need to redo it

    A checkout of both versions, one installation of the target version for the reflection step, and a changelog corpus. The corpus I used was a Markdown conversion, which is why the exact .rst file names are missing from my numbers — for real restFiles entries one would run the same mapping against EXT:core/Documentation/Changelog of a v14 core, where the file names are what Finder->name() expects.

  2. Georg Ringer commented and changed priority Should have Could have <span data-processor="relativetime" data-value="2026-11-12T18:59:05+00:00" > 11/12/26, 18:59 </span>

    Thanks for the measurement and for laying out the options. I could not reproduce the headline number, so here is what I checked, with its limits.

    Constraint confirmed

    Both checks exist as described: AbstractCoreMatcher::validateMatcherDefinitionKeys() (1500496068) and the Finder lookup in UpgradeController (1499803909). The side observation is valid as well: a referenced .rst that is renamed or removed breaks the whole scan result view instead of degrading to a missing link.

    Spot check of the “undocumented” share

    I took all public methods of typo3/sysext/core/Classes that exist in v13.4.0 but not in v14.3.7. Only files present in both versions are considered, so removed or moved classes are not part of this count. Each method was then searched by name in the changelog entries of 13.* and 14.*.

    Result Methods
    removed public methods 115
    of those marked internal@ (method or class) 59
    named in a changelog entry 47
    named in no changelog entry 9

    So in this subset about 8 % of the removed public methods have no changelog entry, not about 40 %. Half of the removals were internal@ and need no scanner rule in the first place. Spot-checked examples that do have an entry but no scanner rule yet: CharsetConverter->conv() (Breaking-105695) and PageRenderer->getCompressJavascript() (Breaking-108055). Both name the method verbatim, so a restFiles reference is possible for them.

    Limits of this check

    • The name match is loose. A hit can be coincidental (e.g. instance), so 47 is an upper bound and 9 a lower bound.
    • I did not check whether the 9 moved to a parent class or trait.
    • Removed classes, signature changes and other packages are not covered. The difference to the 590 candidates might partly come from there.

    Question

    Could you tell whether the 235 “no changelog entry” candidates contain internal@ symbols, and whether the class-only/member matching handles the :php: / :php-short: notation of the changelog entries? In the cases I looked at, the entries did name the member.

    I would hold off on option 3 (writing about 235 entries) until that number is confirmed. Independent of the outcome, making the display-time lookup non-fatal looks worthwhile on its own.

  3. Olivier Dobberkau commented <span data-processor="relativetime" data-value="2026-11-13T09:11:52+00:00" > 11/13/26, 09:11 </span>

    Thanks Georg, your spot check was right, and checking your two questions against my scripts turned up three defects in my measurement. I withdraw the 235 / 40 % figure. Details below.

    1. @internal

    lab-api-diff.sh only filters internal@ on the class docblock (everything above the class keyword). Methods marked internal@ inside a public class are not filtered. So yes, the 590 and the 235 contain internal@ symbols. Together with the missing inheritance and alias resolution, this is the main reason why the corrected run below finds 113 candidates for cms-core instead of 636.

    2. :php: / :php-short: notation

    The mapping only counts a member hit if <FQCN>->member or <FQCN>::member appears verbatim. That fails in two common cases:

    • The Markdown conversion of my corpus turned :php-short:`\TYPO3\CMS\Core\Charset\CharsetConverter->conv()` into CharsetConverter->conv(). The FQCN is gone, so Breaking-105695 counts only as a “class” hit, although it names the method.
    • Many entries list bare method names under the class, e.g. Breaking-108055: “removed from the class \TYPO3\CMS\Core\Page\PageRenderer:” followed by getCompressJavascript() and eleven others. The script cannot match that pattern at all.

    So the “class only” bucket contains real hits, and entries that use only the short class name land in “nothing matched”. Both effects push the “undocumented” number up.

    3. Wrong baseline rule set

    The “not covered by the shipped rule set” filter used the rules of v13.4.35 (2425 restFiles blocks), not those of v14.3.7 (2624). The v14 rules added with the removals were therefore counted as missing. One correction to your examples as well: CharsetConverter->conv and PageRenderer->getCompressJavascript both already have a MethodCallMatcher rule in v14.3.7. They were added in the same commits as the removals (e.g. 0020f3463fa for the CharsetConverter).

    Corrected measurement for typo3/cms-core

    I rebuilt the candidates from git with three fixes: internal@ on class and method level, v14.3.7 as the rule baseline, and matching against the original .rst files. Method inheritance via parent classes, traits and interfaces is resolved, and the v14 class aliases are honoured (e.g. FlexFormService). Basis: public methods and classes of typo3/sysext/core/Classes, v13.4.35 → v14.3.7. Candidates are removed classes, removed methods, dropped arguments and newly required arguments.

    Result count
    public, non-@internal@ candidates 113
    already have a rule in v14.3.7 46
    (of those, already in the v13.4.35 rule set) (20)
    no rule: constructors, almost all of DI services 25
    no rule: other API 42

    The 42 without a rule:

    Changelog situation count
    entry names class and member (incl. bare method list below the class) 17
    entry names the member through its interface (Breaking-33747, Breaking-107310) 4
    no 13.x/14.x entry names the member 21

    14 of the 21 are cache backend setters and getters (RedisBackend->setHostname() and seven more, Typo3DatabaseBackend->getCacheTable() etc., NullBackend->setCompression(), AbstractBackend->flushByTags()). They disappeared in the cache typing rework (Breaking-107315), but that entry does not list them. The remaining 7: CharsetConverter->UnumberToChar(), DataHandler->deleteEl(), DataHandler->updateDB(), LocalImageProcessor->processTaskWithLocalFile(), the new required argument of JavaScriptRenderer->render(), and the exception classes UnknownCharsetException and InvalidSinglePointerFieldException.

    So for cms-core the honest numbers are: about 21 undocumented removals, not 235, and at most 21 rule candidates with a usable changelog reference. Several will drop out on a hand check: AbstractFile->copyTo(), moveTo() and rename() only moved down to File, so a rule matching by method name would flag every valid File->copyTo() call. PageRepository->enableFields() and BackendUserAuthentication->returnWebmounts() already have rules under other keys (see #110807). All 46 existing rules were found by the corrected matching (sanity check), so the matching itself looks sound. Limits: methods and classes only (no constants or properties), cms-core only, and the name match is loose, so the 21 per bucket are approximate. The complete list per symbol is attached as CSV.

    What this means

    • Option 3 (writing ~235 entries) is off the table. At most a handful of entries are missing; the cache setters might deserve a list in Breaking-107315.
    • Option 1 becomes concrete: I can prepare rules for the 21 documented candidates, each reference hand-checked. That would be one patch with its own ticket.
    • I have to take back the side observation as well, and we both agreed to it too quickly. The integrity check runTests.sh -s checkExtensionScannerRst (Build/Scripts/extensionScannerRstFileReferences.php, part of the pre-merge pipeline) requires every restFiles name to exist exactly once, which is the same condition as the runtime check. The rule set cannot be extended by extensions, and Documentation/Changelog is part of the dist package. Exception 1499803909 is therefore only reachable when a deployment strips the documentation or the core is patched. I see no need for a separate ticket.

    Disclosure: I checked the scripts and drafted this reply with AI assistance (Claude).

  4. Olivier Dobberkau commented and changed attached cms-core-v13.4.35-v14.3.7-scanner-candidates.csv attached nameonly.py attached analyze.py attached extract-api.php <span data-processor="relativetime" data-value="2026-11-13T09:23:19+00:00" > 11/13/26, 09:23 </span>

    Added artifacts.