61431[FEATURE] Rework external crawler support
Commit message
Review
-
Code review CR 0
-
Verified V 0
Issues 1
Files 7
- typo3/sysext/core/Documentation/Changelog/10.4.x/Feature-83603-ReworkExternalCrawlerSupport.rst2new+48 −0
- typo3/sysext/indexed_search/Classes/Hook/TypoScriptFrontendHook.php+2 −1
- typo3/sysext/indexed_search/Classes/Worker/Factory.php4new+86 −0
- typo3/sysext/indexed_search/Classes/Worker/IndexerWorker.php4new+33 −0
- typo3/sysext/indexed_search/Classes/Worker/WorkerInterface.php2new+30 −0
- typo3/sysext/indexed_search/Classes/Indexer.php+16 −35
- typo3/sysext/indexed_search/ext_localconf.php+1 −1
Review log 6185
Show 4160 earlier messages
-
Tizian Schmidlin uploaded patch set 1 08/03/19, 13:27
-
TYPO3com voted on patch set 1 V -1 08/03/19, 13:30 Completed build in 21s on Sat, 3 Aug, 01:30 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35551
-
Tomas Norre Mikkelsen commented on patch set 1 08/03/19, 15:10
IndexerWorker.php:27The array $conf is not used in method -
Tomas Norre Mikkelsen commented on patch set 1 08/04/19, 09:07
IndexerWorker.php:27ReplyI see.. From the Crawler PR why the $conf is needed. -
Tizian Schmidlin uploaded patch set 2 08/04/19, 12:09
-
TYPO3com voted on patch set 2 V +1 08/04/19, 12:18 Completed build in 09m 42s on Sun, 4 Aug, 12:18 PM Test Summary: 47172 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35571
-
Georg Ringer commented on patch set 2 08/04/19, 13:33
as this is a feature, it is 10 only and requires a rst file! -
Georg Ringer commented on patch set 2 08/04/19, 13:33
as this is a feature, it is 10 only and requires a rst file! -
Tizian Schmidlin commented on patch set 2 08/05/19, 09:10
> Patch Set 2: > > as this is a feature, it is 10 only and requires a rst file! Could this not be in 10 and 9? Because e need the feature in 9 in order to get crawler work and Benni Mack meant that this should be possible to integrate in TYPO3 9 too. -
Tizian Schmidlin uploaded patch set 3 08/05/19, 09:31
-
TYPO3com voted on patch set 3 V +1 08/05/19, 09:43 Completed build in 11m 35s on Mon, 5 Aug, 09:43 AM Test Summary: 47172 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35584
-
Tizian Schmidlin uploaded patch set 4 08/05/19, 13:07
-
Tizian Schmidlin updated the commit message in patch set 5 08/05/19, 13:08
-
TYPO3com commented on patch set 4 08/05/19, 13:22 Completed build in 15m 38s on Mon, 5 Aug, 01:22 PM Test Summary: 47172 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35587
-
TYPO3com voted on patch set 5 V +1 08/05/19, 13:28 Completed build in 14m 05s on Mon, 5 Aug, 01:28 PM Test Summary: 47172 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35588
-
Tizian Schmidlin uploaded patch set 6 08/05/19, 13:34
-
TYPO3com voted on patch set 6 V +1 08/05/19, 13:51 Completed build in 14m 22s on Mon, 5 Aug, 01:51 PM Test Summary: 47172 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35591
-
Tizian Schmidlin uploaded patch set 7 08/05/19, 14:23
-
Tizian Schmidlin updated the commit message in patch set 8 08/05/19, 14:25
-
TYPO3com voted on patch set 7 V -1 08/05/19, 14:28 Completed build in 10s on Mon, 5 Aug, 02:28 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35595
-
TYPO3com voted on patch set 8 V -1 08/05/19, 14:28 Completed build in 20s on Mon, 5 Aug, 02:28 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35596
-
Tizian Schmidlin uploaded patch set 9 08/05/19, 14:30
-
TYPO3com voted on patch set 9 V -1 08/05/19, 14:30 Completed build in 06s on Mon, 5 Aug, 02:30 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35597
-
Tizian Schmidlin wrote 08/05/19, 14:31
Description set to "This was added because of a faulty merge from a separate working branch, it does not belong here." -
Tizian Schmidlin wrote 08/05/19, 14:31
Description "This was added because of a faulty merge from a separate working branch, it does not belong here." removed -
Tizian Schmidlin wrote 08/05/19, 14:32
Description set to "This has some faulty files because of a local merge request which went wrong." -
Tizian Schmidlin wrote 08/05/19, 14:33
Description "This has some faulty files because of a local merge request which went wrong." removed -
Tizian Schmidlin updated the commit message in patch set 10 08/05/19, 14:49
-
TYPO3com voted on patch set 10 V -1 08/05/19, 14:49 Completed build in 10s on Mon, 5 Aug, 02:49 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35599
-
Tizian Schmidlin uploaded patch set 11 08/05/19, 15:53
-
TYPO3com voted on patch set 11 V -1 08/05/19, 16:03 Completed build in 10m 06s on Mon, 5 Aug, 04:03 PM Test Summary: 47170 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35602
-
Tizian Schmidlin uploaded patch set 12 08/06/19, 06:36
-
TYPO3com commented on patch set 12 08/06/19, 06:49 Completed build in 12m 18s on Tue, 6 Aug, 06:49 AM Test Summary: 47172 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35606
-
Tizian Schmidlin uploaded patch set 13 08/14/19, 16:26
-
TYPO3com voted on patch set 13 V -1 08/14/19, 16:26 Completed build in 08s on Wed, 14 Aug, 04:26 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35695
-
Tizian Schmidlin uploaded patch set 14 08/14/19, 16:41
-
TYPO3com voted on patch set 14 V +1 08/14/19, 16:53 Completed build in 11m 41s on Wed, 14 Aug, 04:53 PM Test Summary: 47176 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-35696
-
Tomas Norre Mikkelsen voted on patch set 14 CR +1 08/19/19, 11:00
-
Frank Nägler commented on patch set 14 CR -1 09/06/19, 12:05
first: I like this patch very much, decouple the dependency to ext:crawler is very great. but... I would like to see here PSR-14 events instead of the worker approach. What do you think? Is it possible to replace the worker with PSR-14 events? -
Tizian Schmidlin commented on patch set 14 09/09/19, 07:17
> Patch Set 14: Code-Review-1 > > first: I like this patch very much, decouple the dependency to ext:crawler is very great. > but... I would like to see here PSR-14 events instead of the worker approach. > What do you think? Is it possible to replace the worker with PSR-14 events? Since this is not going to happen before V11, I think the PSR-14 events approach can be taken, but I have to say that I wont have time to do any work on this until June 2020 (I'm off to military service for 6 months) at best, so this might not make it into V11 if I miss the deadline :-/ But I'll try my best. -
Tizian Schmidlin commented on patch set 14 09/09/19, 12:17
> Patch Set 14: > > > Patch Set 14: Code-Review-1 > > > > first: I like this patch very much, decouple the dependency to ext:crawler is very great. > > but... I would like to see here PSR-14 events instead of the worker approach. > > What do you think? Is it possible to replace the worker with PSR-14 events? > > Since this is not going to happen before V11, I think the PSR-14 events approach can be taken, but I have to say that I wont have time to do any work on this until June 2020 (I'm off to military service for 6 months) at best, so this might not make it into V11 if I miss the deadline :-/ > > But I'll try my best. @Frank Nägler: I've looked into this, the idea of using PSR-14 is certainly the right way to go, but this would mean to rewrite the way the whole indexing inside indexed_search works. Right now, indexed search relies on the fact that there is exactly *one* indexer, which would call for the implementation of an IndexerEvent which would be stoppable, but since the order of registration should not matter, we cannot rely on potentially arbitrarly loaded event listeners. I understand that introducing a new design pattern at this point might seem a bit much, but for this use case this is necessary to ensure the intended behaviour (having one defined indexer). I've put some thoughts into it before building it and the fact that there can only be one indexer at the time was the main reason for me to choose this approach over PSR-14. But keeping my mind open for suggestions, I think that in order to introduce PSR-14 in the part of the core, we'd need to rewrite the way the indexer works. The reason for this is, as you see in the PR, that there are some dependencies inside of the Indexer class, in order to know if frontend indexing has to be done or not and only further down the line the indexer is indeed instantiated, if we're in the indexing context. This worker is then used once for each entry in the list of pages to index. This would mean wrapping the whole indexing process with a *second* check to know if any event has been registered and then either loop over all registered events and stop at the first *or* execute the backup code if nothing was registered, which has some bad code smell to me. In order to implement this, we'd need to build an event listener that checks if any external event listener has been registered in order to provide this information. Also, this listener, which would be called by the Indexer (who becomes an event dispatcher at this point, too), would have to pass the arguments to the very specifically built event that takes two types of arguments-sets (see lines 772 and 780) and then check that the event has been stopped. This is quite some work, I think that throwing 16 to 24 hours at it would be enough to make the changes necessary to have it work and pass the tests. And then someone has to implement this in to crawler, which currently is still the only extension that actually would need to support this new event dispatching, so there go another 24 hours with implementing and testing. All in all, I think that throwing PSR-14 at this problem is not the right way to go, just because it's (excuse my language) "the hot new shit right now", not because I'm unwilling of implementing it, but because it does not fit the use case properly. The event-dispatch design pattern does not apply to the use case of the TYPO3 indexing the way it is now and therefore should not be used for the purpose of indexing content through a possible third party extension. Best Regards Tizian -
Daniel Siepmann uploaded patch set 15 09/27/19, 14:15
-
TYPO3com voted on patch set 15 V +1 09/27/19, 14:31 Completed build in 12m 04s on Fri, 27 Sep, 02:31 PM Test Summary: 47714 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36901
-
Jörg Bösche uploaded patch set 16 10/07/19, 11:39
-
TYPO3com voted on patch set 16 V +1 10/07/19, 11:53 Completed build in 14m 23s on Mon, 7 Oct, 11:53 AM Test Summary: 47719 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-37099
-
Susi Moog commented on patch set 16 10/26/19, 09:14
Is there a way to get this patch non-breaking so we can still get this into 10? -
Tizian Schmidlin uploaded patch set 17 10/28/19, 09:45
-
TYPO3com voted on patch set 17 V +1 10/28/19, 09:55 Completed build in 09m 50s on Mon, 28 Oct, 09:55 AM Test Summary: 48067 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-37820
-
Tizian Schmidlin commented on patch set 17 10/28/19, 09:59
> Patch Set 16: > > Is there a way to get this patch non-breaking so we can still get this into 10? Done, see patch set 17. -
Jonas Eberle wrote 12/03/19, 08:13
Removed reviewer Jonas Eberle. -
Daniel Siepmann wrote 02/20/20, 10:04
Removed reviewer Daniel Siepmann. -
Susi Moog uploaded patch set 18 02/21/20, 13:44
-
Susi Moog voted on patch set 18 CR +1 02/21/20, 13:46
-
TYPO3com voted on patch set 18 V +1 02/21/20, 14:29 Completed build in 28m 52s on Fri, 21 Feb, 02:29 PM Test Summary: 71464 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-400
-
Benni Mack uploaded patch set 19 05/22/20, 11:40
-
TYPO3com voted on patch set 19 V +1 05/22/20, 11:51 Completed build in 11m 17s on Fri, 22 May, 11:51 AM Test Summary: 73805 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3488
-
Tomas Norre Mikkelsen voted on patch set 19 CR +1 06/05/20, 08:09
-
Frank Nägler commented on patch set 19 CR -1 06/05/20, 12:55
Feature-83603-ReworkExternalCrawlerSupport.rst:1new feature RST should go into Changelog/master/ folderFactory.php:21The naming Factory is a bit too "simple", could we use a better Naming like IndexerWorkerFactory ?IndexerWorker.php:29Is it possible to be a bit more specific here? For example instead of `object $caller` I would prefer an Interface. The Interface should declare the indexRegularDocument() function. To be non-breaking, in EXT:indexed_search/Classes/Indexer.php we could throw a deprecation notice if the caller does not implements the interface. With a wrapper class as a deprecation layer, the old caller can be "migrated" to the new system. This way, we get better and more stable code and a smooth migration from the old system, to the new system.WorkerInterface.php:19the same here, I guess IndexerWorkerInterface would be a better naming. -
Frank Nägler commented on patch set 19 CR 0 06/05/20, 13:37
maybe I don't understand the code to 100%, tell me if I am wrong and my comments are stupid ;) -
Tizian Schmidlin uploaded patch set 20 06/05/20, 13:40
-
Tizian Schmidlin uploaded patch set 21 06/05/20, 13:41
-
Tizian Schmidlin uploaded patch set 22 06/05/20, 13:42
-
Tizian Schmidlin commented on patch set 22 V +1 CR +1 06/05/20, 13:44
> Patch Set 19: Code-Review-1 > > (4 comments) I added the ChangeLog to 9.5.x and master and adapted the merge request to be for all 3 branches.Feature-83603-ReworkExternalCrawlerSupport.rst:1ReplyI think it got moved here after 10.4 release. Should it be copied to master and 9.5 then?Factory.php:21Reply> The naming Factory is a bit too "simple", could we use a better Naming like IndexerWorkerFactory ? I'm not sure about the benefit of it. This factory gets used in one single place and could/should maybe be marked as @internal but apart from that I'm not sure that naming the Factory anything else than "Factory" has any purpose at all.IndexerWorker.php:29Reply> Is it possible to be a bit more specific here? > For example instead of `object $caller` I would prefer an Interface. > The Interface should declare the indexRegularDocument() function. > To be non-breaking, in EXT:indexed_search/Classes/Indexer.php we could throw a deprecation notice if the caller does not implements the interface. With a wrapper class as a deprecation layer, the old caller can be "migrated" to the new system. > This way, we get better and more stable code and a smooth migration from the old system, to the new system. I don't think this is necessary, as I explained, the idea is to replace the IndexerWorker completely if it is not compatible and therefore you keep freedom on how you want to handle this call. In this case, the IndexerWorker from the core exactly knows who is the caller. If you build a new IndexerWorker and override the default, you still know who the caller is and what methods to expect and we do not have to restrict the Interface early on.WorkerInterface.php:19ReplyI would agree with you in any broader context, but since this is the worker of the IndexedSearch Package and that this is the only imaginable worker for this Package (because if something else would be doing any work and therefore be a worker that is not about indexing, it would probably have to be located somewhere else). -
Andreas Kienast commented on patch set 22 CR -1 06/05/20, 13:51
Has there been an agreement to backport a feature back to v9? Anyway, the changelog is already in 9.5.x, you don't have to add it in 10.4.x as well. -
TYPO3com voted on patch set 20 V +1 06/05/20, 13:54 Completed build in 13m 29s on Fri, 5 Jun, 01:54 PM Test Summary: 72800 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3860
-
Tizian Schmidlin commented on patch set 22 06/05/20, 13:55
> Patch Set 19: -Code-Review > > maybe I don't understand the code to 100%, tell me if I am wrong and my comments are stupid ;) You point out good points that I guess would be valid in most cases but I think that it would be harmful to this patch to specify too much. The idea is to free up the core from restrictions regarding the indexed search crawling process. It used to be mandatory to use the crawler extension, since the core was actively checking if the crawler extension was loaded this came out after a long night at the dev days and from what I gather as one of the people involved in the development of the crawler extension and after some talks at the T3DD last year in Karlsruhe with Tomas and Benni, this solution should hold up in the future too, by letting others extend the indexing process as they need. Some examples to illustrate my point: - if we define an interface for the caller given to the IndexerWorker defining a method name with predefined parameters, we force a potential developer to create this function in his indexer, even though they might no work in the same way at all. In this way the developer is free to implement whatever they want. - if we name the Factory anything else, someone might want to add another factory thinking, this is the place where all the factories go. Seeing that there is already a factory, the developer are challenged to ask themselves "Why is that a plain factory? Ah yes, it belongs here, mine probably does not." and therefore saving them the frustration to try to publish something and getting a negativ remark that this does not belong there. In the end it's all about keeping things as simple as possible. -
Tizian Schmidlin commented on patch set 22 06/05/20, 13:56
> Patch Set 22: Code-Review-1 > > Has there been an agreement to backport a feature back to v9? Anyway, the changelog is already in 9.5.x, you don't have to add it in 10.4.x as well. No, indeed there wasn't, I was overeager there. I'll fix it. So if the changelog is in 10.4.x, does it have to go to master? -
Andreas Kienast commented on patch set 22 06/05/20, 13:56
Factory.php:21ReplyIn TypoScriptFrontendHook you use `if (Factory::isOverridden())`. It's not clear by reading TypoScriptFrontendHook only what this factory factors. -
Andreas Kienast commented on patch set 22 06/05/20, 13:58
> Patch Set 22: > > So if the changelog is in 10.4.x, does it have to go to master? It's fine when the changelog goes into 10.4.x only. -
TYPO3com voted on patch set 21 V +1 06/05/20, 13:59 Completed build in 15m 51s on Fri, 5 Jun, 01:59 PM Test Summary: 72799 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3861
-
TYPO3com voted on patch set 22 V +1 06/05/20, 14:05 Completed build in 21m 19s on Fri, 5 Jun, 02:05 PM Test Summary: 72798 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3862
-
Tizian Schmidlin uploaded patch set 23 06/05/20, 14:08
-
Tizian Schmidlin commented on patch set 23 06/05/20, 14:10
> Patch Set 22: > > (1 comment) So, as discussed with Benni Mack: 1. This patch will be put without an interface into TYPO3 10, since adding an interface would break and we are not allowed to do this 2. I split this patch up into another review just for TYPO3 11 which will break the way indexed search does it's thing and therefore we will be able to provide a proper interface I hope this suits all needs :-) -
Tizian Schmidlin uploaded patch set 24 06/05/20, 14:11
-
TYPO3com voted on patch set 23 V +1 06/05/20, 14:35 Completed build in 26m 14s on Fri, 5 Jun, 02:35 PM Test Summary: 72800 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3867
-
TYPO3com voted on patch set 24 V +1 06/05/20, 14:48 Completed build in 19m 23s on Fri, 5 Jun, 02:48 PM Test Summary: 72800 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3868
-
Stephan Großberndt commented on patch set 24 CR -1 06/06/20, 10:12
This apparently is the patch which should go to the 10.4 branch but it is currently target at master, not at the 10.4 branch. -
Tizian Schmidlin commented on patch set 24 V +1 CR +1 06/08/20, 06:41
> Patch Set 24: Code-Review-1 > > This apparently is the patch which should go to the 10.4 branch but it is currently target at master, not at the 10.4 branch. This was solved with the patch-set 24 -
Stephan Großberndt commented on patch set 24 06/08/20, 07:50
Sorry, but thats not true. If you look up in the patch description it still says "Branch master". I don't think its possible to change the branch a change is directed at. You will have to abandon this change for 10.4 and upload it as a new one targeted at 10.4 -
Tizian Schmidlin commented on patch set 24 06/08/20, 08:13
> Patch Set 24: > > Sorry, but thats not true. If you look up in the patch description it still says "Branch master". I don't think its possible to change the branch a change is directed at. You will have to abandon this change for 10.4 and upload it as a new one targeted at 10.4 Ok, that's weird, I changed it in the commit message, so I assumed it updated automatically, my bad. I'll correct this as soon as I find out how. -
Tizian Schmidlin wrote 06/08/20, 08:14
Change destination moved from master to 10.4 -
Tizian Schmidlin commented on patch set 24 06/08/20, 08:14
> Patch Set 24: > > > Patch Set 24: > > > > Sorry, but thats not true. If you look up in the patch description it still says "Branch master". I don't think its possible to change the branch a change is directed at. You will have to abandon this change for 10.4 and upload it as a new one targeted at 10.4 > > Ok, that's weird, I changed it in the commit message, so I assumed it updated automatically, my bad. I'll correct this as soon as I find out how. Done -
Stephan Großberndt commented on patch set 24 06/08/20, 08:43
thanks, that looks better! :) -
Christian Kuhn commented on patch set 24 11/09/21, 21:38
Hey. I hope it is ok to abandon this change for now: This patch stalled for quite a while without much happening. If work continues here in v12, it should probably change the indexed_search hooks to psr-14 nowadays and act on top of this. This change can of course be resurrected if work continues or a new one can be pushed. For now, I'll abandon the change and reject the forge issue. -
Christian Kuhn abandoned the change 11/09/21, 21:38
Factory.php:49Unresolved